Skip to content

Commit f329dd4

Browse files
fix(jdbi3): chunk IN-list batch queries to stay under DB parameter limit (#28752) (#28777)
* fix(jdbi3): chunk IN-list batch queries to stay under DB parameter limit Bulk subtree delete/restore fans an entire tree level's ids into a single IN (...) query. JDBI @BindList emits one bind parameter per element, so a level with ~100k entities (e.g. a service with 100k tables) exceeds PostgreSQL's 65,535 parameter ceiling ("Given query has 100,002 parameters") and the operation fails. The reported findToBatchAllTypes crash was only the first uncaught offender: getExtensionsBatch hits the same limit earlier but is swallowed by populateRelationFields' per-entity fallback, and ~13 other findToBatch*/findFromBatch* and tag batches were vulnerable in other paths. Add EntityDAO.queryInChunks / updateInChunks helpers (chunk at MAX_IN_LIST_CHUNK_SIZE, pass straight through for small lists so single-query behavior is preserved) and route every @BindList batch method through them: findToBatch*/findFromBatch*, entity_extension deleteAllBatch / getExtensionsBatch / getExtensionBatch, and tag_usage getTagsInternalBatch / getCertTagsInternalBatch. Also extract the duplicated deletedCondition(include) clause builder. Adds InListChunkingTest covering the chunk-size cap, exact input coverage (no dropped/duplicated ids), and per-chunk result aggregation. * fix(jdbi3): dedup ids before chunking IN-list batches queryInChunks / updateInChunks split a large id list into chunks, so a duplicate value landing in two different chunks would be queried (or deleted) twice — duplicating result rows / issuing a redundant statement, unlike a single IN (...) which ignores duplicate values. De-duplicate (encounter order preserved) before chunking, matching the existing findEntitiesByIds pattern. Adds InListChunkingTest cases covering duplicates split across a chunk boundary for both the query and update helpers. * fix(jdbi3): chunk three more unbounded IN-list paths Audit of the remaining @BindList paths surfaced three more methods that fan a caller-controlled, catalog-scaled list into one IN (...) and can hit the same 65,535 parameter ceiling. All routed through the queryInChunks/updateInChunks helpers: - EntityRelationshipDAO.bulkUpdateFromId — reparenting a glossary term updates every nested descendant's relationship row in one UPDATE; a subtree with >65k terms blew the cap. Now chunks toIds via updateInChunks. - TagUsageDAO.deleteTagsByTargets — clearing column tags on a bulk table update/import passes every column FQN across the batch (wide/nested tables x ~100 per batch). Now chunks via updateInChunks. - EntityTimeSeriesDAO.getLatestExtensionBatch — listing ingestion pipelines with fields=pipelineStatuses and a large limit (capped at 1,000,000) binds one FQN-hash per row. Now chunks via queryInChunks. Adds InListChunkingTest cases for all three. * test(jdbi3): cover IN-list chunking pass-through with duplicate ids The existing tests all feed an over-limit list, leaving queryInChunks / updateInChunks' else branch (size <= MAX_IN_LIST_CHUNK_SIZE) untested. Add two cases proving a within-limit list with duplicates issues exactly one query/statement and is passed through verbatim (no dedup) — the database collapses the duplicates via IN(...) set semantics. Complements the existing over-limit dedup tests, which assert against the distinct set. * fix(jdbi3): chunk five more catalog-scaled IN-list paths Follow-up to the IN-list chunking work — five more @BindList paths fan a catalog-scaled list into one IN (...) and can still hit the 65,535 parameter ceiling. All routed through the existing queryInChunks/updateInChunks helpers. - EntityRelationshipDAO.bulkRemoveTo / bulkRemoveFrom: the 'remove' siblings of bulkUpdateFromId (already chunked). Reached via the bulkRemove{To,From}Relationship wrappers (their only callers) from DataProductRepository.batchMigrateAssetDomains (all assets of a type under one data product), TeamRepository, DomainRepository. - UsageDAO.getLatestUsageBatch: GET /v1/tables?fields=usageSummary (and dashboards/pipelines/topics/mlmodels/...) resolves the field over the whole page via setFieldsInBulk; list limit is @max(1000000). - DataQualityDataTimeSeriesDAO.getLatestRecordBatch: GET /v1/dataQuality/testCases?fields=testCaseResult, same whole-page resolution. - TestCaseResultTimeSeriesDAO.listResultSummariesForTestSuites: GET /v1/dataQuality/testSuites?fields=summary, binds one id per executable test suite. Adds InListChunkingTest cases for all five (16 tests pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com> [1.13 backport] Dropped the findToBatchAllTypes(List<Integer> relations, Include) chunking hunk and its findToBatchAllTypesWithRelationsCondition delegate: that overload originates in temporal lineage (#28426), which is not on the 1.13 branch. The two queryInChunks dedup / pass-through tests that exercised it were retargeted to the existing int-relation findToBatchAllTypes overload (same queryInChunks path). All 16 InListChunkingTest cases pass. (cherry picked from commit 5af1ba5) Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
1 parent 9ce0855 commit f329dd4

4 files changed

Lines changed: 632 additions & 80 deletions

File tree

0 commit comments

Comments
 (0)