Commit f329dd4
* fix(jdbi3): chunk IN-list batch queries to stay under DB parameter limit
Bulk subtree delete/restore fans an entire tree level's ids into a single
IN (...) query. JDBI @BindList emits one bind parameter per element, so a
level with ~100k entities (e.g. a service with 100k tables) exceeds
PostgreSQL's 65,535 parameter ceiling ("Given query has 100,002 parameters")
and the operation fails. The reported findToBatchAllTypes crash was only the
first uncaught offender: getExtensionsBatch hits the same limit earlier but is
swallowed by populateRelationFields' per-entity fallback, and ~13 other
findToBatch*/findFromBatch* and tag batches were vulnerable in other paths.
Add EntityDAO.queryInChunks / updateInChunks helpers (chunk at
MAX_IN_LIST_CHUNK_SIZE, pass straight through for small lists so single-query
behavior is preserved) and route every @BindList batch method through them:
findToBatch*/findFromBatch*, entity_extension deleteAllBatch /
getExtensionsBatch / getExtensionBatch, and tag_usage getTagsInternalBatch /
getCertTagsInternalBatch. Also extract the duplicated deletedCondition(include)
clause builder.
Adds InListChunkingTest covering the chunk-size cap, exact input coverage
(no dropped/duplicated ids), and per-chunk result aggregation.
* fix(jdbi3): dedup ids before chunking IN-list batches
queryInChunks / updateInChunks split a large id list into chunks, so a
duplicate value landing in two different chunks would be queried (or deleted)
twice — duplicating result rows / issuing a redundant statement, unlike a
single IN (...) which ignores duplicate values. De-duplicate (encounter order
preserved) before chunking, matching the existing findEntitiesByIds pattern.
Adds InListChunkingTest cases covering duplicates split across a chunk
boundary for both the query and update helpers.
* fix(jdbi3): chunk three more unbounded IN-list paths
Audit of the remaining @BindList paths surfaced three more methods that fan a
caller-controlled, catalog-scaled list into one IN (...) and can hit the same
65,535 parameter ceiling. All routed through the queryInChunks/updateInChunks
helpers:
- EntityRelationshipDAO.bulkUpdateFromId — reparenting a glossary term updates
every nested descendant's relationship row in one UPDATE; a subtree with
>65k terms blew the cap. Now chunks toIds via updateInChunks.
- TagUsageDAO.deleteTagsByTargets — clearing column tags on a bulk table
update/import passes every column FQN across the batch (wide/nested tables x
~100 per batch). Now chunks via updateInChunks.
- EntityTimeSeriesDAO.getLatestExtensionBatch — listing ingestion pipelines
with fields=pipelineStatuses and a large limit (capped at 1,000,000) binds
one FQN-hash per row. Now chunks via queryInChunks.
Adds InListChunkingTest cases for all three.
* test(jdbi3): cover IN-list chunking pass-through with duplicate ids
The existing tests all feed an over-limit list, leaving queryInChunks /
updateInChunks' else branch (size <= MAX_IN_LIST_CHUNK_SIZE) untested. Add two
cases proving a within-limit list with duplicates issues exactly one
query/statement and is passed through verbatim (no dedup) — the database
collapses the duplicates via IN(...) set semantics. Complements the existing
over-limit dedup tests, which assert against the distinct set.
* fix(jdbi3): chunk five more catalog-scaled IN-list paths
Follow-up to the IN-list chunking work — five more @BindList paths fan a catalog-scaled list into one IN (...) and can still hit the 65,535 parameter ceiling. All routed through the existing queryInChunks/updateInChunks helpers.
- EntityRelationshipDAO.bulkRemoveTo / bulkRemoveFrom: the 'remove' siblings of bulkUpdateFromId (already chunked). Reached via the bulkRemove{To,From}Relationship wrappers (their only callers) from DataProductRepository.batchMigrateAssetDomains (all assets of a type under one data product), TeamRepository, DomainRepository.
- UsageDAO.getLatestUsageBatch: GET /v1/tables?fields=usageSummary (and dashboards/pipelines/topics/mlmodels/...) resolves the field over the whole page via setFieldsInBulk; list limit is @max(1000000).
- DataQualityDataTimeSeriesDAO.getLatestRecordBatch: GET /v1/dataQuality/testCases?fields=testCaseResult, same whole-page resolution.
- TestCaseResultTimeSeriesDAO.listResultSummariesForTestSuites: GET /v1/dataQuality/testSuites?fields=summary, binds one id per executable test suite.
Adds InListChunkingTest cases for all five (16 tests pass).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
[1.13 backport] Dropped the findToBatchAllTypes(List<Integer> relations, Include)
chunking hunk and its findToBatchAllTypesWithRelationsCondition delegate: that
overload originates in temporal lineage (#28426), which is not on the 1.13 branch.
The two queryInChunks dedup / pass-through tests that exercised it were retargeted
to the existing int-relation findToBatchAllTypes overload (same queryInChunks path).
All 16 InListChunkingTest cases pass.
(cherry picked from commit 5af1ba5)
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
1 parent 9ce0855 commit f329dd4
4 files changed
Lines changed: 632 additions & 80 deletions
File tree
- openmetadata-service/src
- main/java/org/openmetadata/service/jdbi3
- test/java/org/openmetadata/service/jdbi3
0 commit comments