feat(dataRetention): sweep orphan time-series rows for per-type tables#28384
Conversation
Adds `cleanOrphanedTimeSeriesRows()` to `DataRetention` alongside the existing orphan-relationship and orphan-tag sweeps. Each of the five affected DAOs gets a `deleteOrphanedRecords(int limit)` query (MySQL + PostgreSQL) that left-joins to its parent and deletes rows the parent no longer covers: - `testCaseResolutionStatus`: parent link via `entity_relationship` PARENT_OF - `agentExecution`: `agentId` → `ai_application_entity.id` - `mcpExecution`: `serverId` → `mcp_server_entity.id` - `profile_data`: `entityFQNHash` → `table_entity.fqnHash` - `query_cost_time_series`: `entityFQNHash` → `query_entity.fqnHash` The sweep runs after `cleanOrphanedRelationshipsAndHierarchies()` so the PARENT_OF check sees the post-cleanup `entity_relationship` state. Pairs with PR #28367, where the bulk hard-delete cascade now skips time-series children and relies on `DataRetention` to reclaim them out-of-band. Adds `OrphanedTimeSeriesCleanupIT` covering all five per-type queries: inserts a real-parent row and a bogus-parent row through the existing DAO `insert(...)` paths, runs `deleteOrphanedRecords(BATCH)`, asserts the orphan is gone and the valid row is preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
backend
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
…ion table arg - McpExecutionDAO.insertWithoutExtension uses `<table>` placeholder; the test was passing `null`, which made Jdbi fail to render the SQL template. Pass the literal table name `mcp_execution_entity`. - `ns.prefix(...)` embeds class + method names, so chaining it through database -> schema -> table -> auto-created test_suite pushed the test_suite `name` column past its VARCHAR(256) bound. Use `ns.uniqueShortId()` for the hierarchy components and shorten the test method names so the resulting FQN stays well under the column limit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rify PARENT_OF=9 Addresses two PR review findings: - `profiler_data_time_series.deleteOrphanedRecords` previously LIMITed distinct `entityFQNHash` values, then deleted every row for each hash — a batch could delete tens of millions of rows if many orphan hashes each had thousands of rows. Switch to row-level limiting: single-table `DELETE ... WHERE NOT EXISTS (...) LIMIT N` on MySQL, and `ctid IN (SELECT ... LIMIT N)` on PostgreSQL (the table has no `id` column, so we use Postgres ctid for the inner subquery). This matches the row-count cap used by the other four orphan-cleanup queries. - Annotate `er.relation = 9` in the testCaseResolutionStatus query with a `// 9 = Relationship.PARENT_OF` inline comment plus a leading block comment noting the ordinal is stable because the enum appends new values. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code Review ✅ Approved 2 resolved / 2 findingsImplements orphaned time-series data cleanup in the DataRetention job to reclaim storage after entity deletions, with row-level limits and clarified parent-relationship constants. ✅ 2 resolved✅ Performance: Profiler delete unbounded: LIMIT caps hashes, not rows deleted
✅ Quality: Magic number 9 for PARENT_OF relation in SQL annotation
OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|
🟡 Playwright Results — all passed (14 flaky)✅ 4241 passed · ❌ 0 failed · 🟡 14 flaky · ⏭️ 87 skipped
🟡 14 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Describe your changes:
Adds
cleanOrphanedTimeSeriesRows()toDataRetentionalongside the existing orphan-relationship and orphan-tag sweeps. Five DAOs each get adeleteOrphanedRecords(int limit)query (MySQL + PostgreSQL) that left-joins to the parent and deletes rows whose parent is gone:testCaseResolutionStatus(viaentity_relationshipPARENT_OF),agentExecution(agentId→ai_application_entity.id),mcpExecution(serverId→mcp_server_entity.id),profile_data(entityFQNHash→table_entity.fqnHash), andquery_cost_time_series(entityFQNHash→query_entity.fqnHash).Pairs with #28367 — that PR makes the bulk hard-delete cascade skip time-series children (they can hold millions of rows per parent and a synchronous DELETE would stall the request), and explicitly defers orphan reclamation to
DataRetention. The new sweep is wired to run aftercleanOrphanedRelationshipsAndHierarchies()so the testCaseResolutionStatus check sees the post-cleanupentity_relationshipstate.Type of change:
Tests:
Use cases covered
aiApplication/mcpServer/testCase/table/query, the DataRetention job reclaims the danglingagentExecution/mcpExecution/testCaseResolutionStatus/profile_data/query_costrows on its next run.Backend integration tests
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/OrphanedTimeSeriesCleanupIT.javawith one test per DAO. Each seeds a valid + an orphan row through the existing DAOinsert(...)paths, runsdeleteOrphanedRecords(BATCH), and asserts orphan-gone / valid-preserved via raw JDBI count queries.UI screen recording / screenshots:
Not applicable.
Checklist:
🤖 Generated with Claude Code