Fix - 27308: Mark Charts deleted#27309
Merged
harshsoni2024 merged 8 commits intomainfrom Apr 17, 2026
Merged
Conversation
github-actions
Bot
added
Ingestion
safe to test
Contributor
There was a problem hiding this comment.
Pull request overview
Adds first-class support in the dashboard ingestion topology to track ingested chart FQNs and soft-delete charts that no longer exist in the upstream source (fixing #27308).
Changes:
- Introduces a new ingestion config flag
markDeletedCharts(defaulttrue) for dashboard services. - Tracks scanned chart FQNs (
chart_source_state) and runs a new post-process step to mark missing charts as deleted. - Updates multiple dashboard connectors to register each emitted
CreateChartRequestinto the chart source-state.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json | Adds markDeletedCharts configuration option to dashboard ingestion pipeline schema. |
| ingestion/src/metadata/ingestion/source/dashboard/dashboard_service.py | Adds chart source-state tracking, chart record registration, and chart deletion post-process hook. |
| ingestion/src/metadata/ingestion/source/dashboard/tableau/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/superset/db_source.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/superset/api_source.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/ssrs/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/sigma/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/redash/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/quicksight/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/qliksense/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/qlikcloud/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/powerbi/metadata.py | Registers emitted charts into the shared chart source-state (but includes leftover no-op debug block). |
| ingestion/src/metadata/ingestion/source/dashboard/mode/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/microstrategy/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/metabase/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/looker/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/lightdash/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/grafana/metadata.py | Registers emitted charts into the shared chart source-state. |
| ingestion/src/metadata/ingestion/source/dashboard/domodashboard/metadata.py | Registers emitted charts into the shared chart source-state. |
Contributor
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
Contributor
🟡 Playwright Results — all passed (31 flaky)✅ 3597 passed · ❌ 0 failed · 🟡 31 flaky · ⏭️ 207 skipped
🟡 31 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Contributor
Jest test CoverageUI tests summary
|
|
|
ShaileshParmar11
approved these changes
Apr 14, 2026
Code Review ✅ ApprovedImplements logic to correctly mark charts as deleted in the database. No issues found. OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
pmbrull
reviewed
Apr 14, 2026
| ), | ||
| ) | ||
| yield Either(right=chart_request) | ||
| self.register_record_chart(chart_request=chart_request) |
Collaborator
There was a problem hiding this comment.
should we take the chance to improve the framework a bit. I think we have to keep adding this register_record_xyz when we miss it for this or that entity. Maybe we should rely on the yield Either right and the topology itself to say "entity registered" so we know we are not missing any connectors at all. thoughts?
Contributor
Author
There was a problem hiding this comment.
this improvement makes sense. since we have to manually keep that in each connector X for each entity(for e.g. 3 in case of dashboard).
We can couple this with topology manager to keep source state to maintain unchaged/changed/deleted entities & update them on ingestion based on markDeleted flag.
I am thinking of handling this in different pr something like this, wdyt ?
#27450
pmbrull
approved these changes
Apr 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes:
Fixes #27308
Type of change:
Checklist:
Fixes <issue-number>: <short explanation>