fix(dbt): remove Automated tags absent from incoming dbt schema (#26737) by aniruddhaadak80 · Pull Request #27249 · open-metadata/OpenMetadata

aniruddhaadak80 · 2026-04-10T14:38:36Z

Fixes #26737. Adds a reconciliation step in addDataModel to remove tags that were automatically applied by the dbt ingestion bot but are now absent from the incoming dataModel payload, resolving the indefinitely persisting tags bug.

…en-metadata#26737)

github-actions · 2026-04-10T14:39:02Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

Adds reconciliation logic in the backend TableRepository.addDataModel flow intended to remove dbt-ingestion-applied tags that are no longer present in the incoming DataModel payload (addressing the “automated tags persist forever” bug described in #26737).

Changes:

After merging tags with mergeTagsWithIncomingPrecedence, removes existing LabelType.AUTOMATED table tags that are absent from the incoming dataModel.tags.
Applies the same “remove absent automated tags” reconciliation to column tags from dataModel.columns[*].tags.

Copilot · 2026-04-10T14:46:04Z

+    if (table.getTags() != null) {
+      java.util.List<String> incomingTags = dataModel.getTags() != null 
+          ? dataModel.getTags().stream().map(org.openmetadata.schema.type.TagLabel::getTagFQN).collect(java.util.stream.Collectors.toList())
+          : java.util.Collections.emptyList();
+      mergedTableTags.removeIf(t -> t.getLabelType() == org.openmetadata.schema.type.TagLabel.LabelType.AUTOMATED && !incomingTags.contains(t.getTagFQN()));


This reconciliation removes all existing LabelType.AUTOMATED tags that are not present in the incoming dataModel payload. Since addDataModel is a general Table API (not dbt-only), this can unintentionally delete automated tags applied by other systems (e.g., auto-classification), causing data loss on any dataModel update. Please scope the removal to dbt-ingestion-owned tags (e.g., t.getAppliedBy() equals Entity.INGESTION_BOT_NAME or another dbt bot identity) and ideally gate it on dataModel.getModelType() == DBT so non-dbt updates don’t reconcile bot-applied tags.

Copilot · 2026-04-10T14:46:04Z

+      if (stored.getTags() != null) {
+        java.util.List<String> incomingColTags = modelColumn.getTags() != null 
+            ? modelColumn.getTags().stream().map(org.openmetadata.schema.type.TagLabel::getTagFQN).collect(java.util.stream.Collectors.toList())
+            : java.util.Collections.emptyList();
+        mergedColumnTags.removeIf(t -> t.getLabelType() == org.openmetadata.schema.type.TagLabel.LabelType.AUTOMATED && !incomingColTags.contains(t.getTagFQN()));


Same issue for column tags: this removes every existing LabelType.AUTOMATED column tag not present in the incoming model, which can wipe automated tags applied by other bots/recognizers whenever a data model is updated. Please restrict the reconciliation to tags applied by the ingestion/dbt bot (e.g., Entity.INGESTION_BOT_NAME via t.getAppliedBy()) and consider only applying this logic for DBT dataModel.getModelType().

aniruddhaadak80 · 2026-04-11T13:47:53Z

Hello! I am participating in the WeMakeDevs hackathon. Could a maintainer please assign the safe to test label so the GitHub Actions workflows can validate my fixes? Thank you!

aniruddhaadak80 · 2026-04-13T13:49:20Z

Hello team, could we get the safe to test label added to this PR? Let me know if any updates are needed before that.

github-actions · 2026-04-13T14:06:05Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

github-actions · 2026-04-13T14:09:31Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

Copilot · 2026-04-13T14:17:03Z

+    if (table.getTags() != null) {
+      java.util.List<String> incomingTags = dataModel.getTags() != null 
+          ? dataModel.getTags().stream().map(org.openmetadata.schema.type.TagLabel::getTagFQN).collect(java.util.stream.Collectors.toList())
+          : java.util.Collections.emptyList();
+      mergedTableTags.removeIf(t -> t.getLabelType() == org.openmetadata.schema.type.TagLabel.LabelType.AUTOMATED && !incomingTags.contains(t.getTagFQN()));
+    }


The reconciliation removes all existing AUTOMATED tags that are not present in the incoming dataModel.getTags(). Since AUTOMATED is also used for non-dbt automation (e.g., tier/certification/other workflows), this can unintentionally delete tags managed by other subsystems whenever /tables/{id}/dataModel is called. Consider restricting deletions to tags that are known to be dbt-managed (e.g., those present in the previous stored table.getDataModel().getTags() or another provenance signal), rather than all LabelType.AUTOMATED tags.

Copilot · 2026-04-13T14:17:04Z

+      if (stored.getTags() != null) {
+        java.util.List<String> incomingColTags = modelColumn.getTags() != null 
+            ? modelColumn.getTags().stream().map(org.openmetadata.schema.type.TagLabel::getTagFQN).collect(java.util.stream.Collectors.toList())
+            : java.util.Collections.emptyList();
+        mergedColumnTags.removeIf(t -> t.getLabelType() == org.openmetadata.schema.type.TagLabel.LabelType.AUTOMATED && !incomingColTags.contains(t.getTagFQN()));
+      }


Same concern as table-level tags: this removes every existing column tag with LabelType.AUTOMATED if it’s missing from the incoming dbt column payload. Column-level automated tags can also be produced by other automation (e.g., classification workflows), so this risks deleting non-dbt tags on every dbt run. Narrow the deletion scope to dbt-managed column tags (e.g., compare against the previous stored table.getDataModel().getColumns() tags for the same column) instead of all automated tags.

Copilot · 2026-04-13T14:17:04Z

      return listCount(
-          getTableName(), mySqlCondition, postgresCondition, team, Relationship.HAS.ordinal());
+          getTableName(), mySqlCondition, postgresCondition, team, Relationship.HAS.ordinal(), filter.getQueryParams());
    }


Passing filter.getQueryParams() as a @BindMap can bind a raw team value to :team, which conflicts with the explicit @BindFQN("team") binding (expected to hash the team). This can lead to incorrect/unstable binding for :team (and potentially wrong results) when the team filter is used. Pass a copy of the params map with team removed (or rename one of the bind parameters) to avoid double-binding :team.

Copilot · 2026-04-13T14:17:05Z

      return listBefore(
          getTableName(),
          mySqlCondition,
          postgresCondition,
          team,
          limit,
          beforeName,
          beforeId,
-          Relationship.HAS.ordinal());
+          Relationship.HAS.ordinal(),
+          filter.getQueryParams());


Same :team double-binding risk as in listCount: @BindFQN("team") and @BindMap(filter.getQueryParams()) can both bind :team when a team filter is present. Use a params map with team removed (or use a different bind name) before passing it into the DAO method.

Copilot · 2026-04-13T14:17:05Z

      return listAfter(
          getTableName(),
          mySqlCondition,
          postgresCondition,
          team,
          limit,
          afterName,
          afterId,
-          Relationship.HAS.ordinal());
+          Relationship.HAS.ordinal(),
+          filter.getQueryParams());


Same :team double-binding risk as in listCount/listBefore: @BindFQN("team") can be overridden/conflicted by @BindMap(filter.getQueryParams()) if the query params include team. Remove team from the bind map (or rename binds) before passing params to the SQL method.

github-actions · 2026-04-13T17:14:13Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

gitar-bot · 2026-04-13T17:15:23Z

Code Review ✅ Approved 2 resolved / 2 findings

Removes Automated tags absent from incoming dbt schema, addressing UTF-16LE encoding in DummyBackendTest.java and cleaning up unnecessary fully-qualified class names. No issues remain.

✅ 2 resolved

✅ Bug: DummyBackendTest.java is UTF-16LE encoded, will break builds

The new test file DummyBackendTest.java is encoded in UTF-16LE (with BOM FF FE), which is why Git treats it as a binary file. Java source files must be UTF-8 (or ASCII). Most Java compilers and build tools (Maven/Gradle) default to UTF-8 and will fail to parse this file correctly, causing compilation errors or the test being silently skipped.

This file needs to be re-saved as UTF-8 without BOM.

✅ Quality: Unnecessary fully-qualified class names; imports already exist

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java:1411-1414 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java:1434-1437 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java:1412 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java:1434
The new code uses fully-qualified names (java.util.List, java.util.stream.Collectors, java.util.Collections, org.openmetadata.schema.type.TagLabel.LabelType, org.openmetadata.schema.type.TagLabel) even though all of these types are already imported at the top of the file (lines 53, 58, 65, 107). This hurts readability and is inconsistent with the rest of the file.

Options

Display: compact → Showing less information.

Comment with these commands to change:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar}

PubChimps · 2026-04-16T00:52:29Z

Hi @aniruddhaadak80, thank you for your pr here, but we have another pr that is being developed here, please make sure that you are assigned to an issue before adding a fix to it, we will reopen this pr if necessary.

aniruddhaadak80 · 2026-04-16T04:54:24Z

Hi @aniruddhaadak80, thank you for your pr here, but we have another pr that is being developed here, please make sure that you are assigned to an issue before adding a fix to it, we will reopen this pr if necessary.

ok , thank you for transparency.

fix(dbt): remove stale automated tags absent from incoming schema (op…

8934b5f

…en-metadata#26737)

Copilot AI review requested due to automatic review settings April 10, 2026 14:38

Copilot started reviewing on behalf of aniruddhaadak80 April 10, 2026 14:39 View session

gitar-bot Bot reviewed Apr 10, 2026

View reviewed changes

Comment thread openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java Outdated

Copilot AI reviewed Apr 10, 2026

View reviewed changes

Fix UserDAO missing parameter domainEntityType open-metadata#27190

0dd985e

test: add backend test file

252ae1b

chore: remove placeholder java test breaking CI

86f6903

Copilot AI review requested due to automatic review settings April 13, 2026 14:09

Copilot started reviewing on behalf of aniruddhaadak80 April 13, 2026 14:10 View session

Copilot AI reviewed Apr 13, 2026

View reviewed changes

refactor: extract tag removal logic to helper method, use Set lookup

a2a9d42

PubChimps closed this Apr 16, 2026

Conversation

aniruddhaadak80 commented Apr 10, 2026

Uh oh!

github-actions Bot commented Apr 10, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

aniruddhaadak80 commented Apr 11, 2026

Uh oh!

aniruddhaadak80 commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

gitar-bot Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PubChimps commented Apr 16, 2026

Uh oh!

aniruddhaadak80 commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aniruddhaadak80 commented Apr 13, 2026 •

edited

Loading

gitar-bot Bot commented Apr 13, 2026 •

edited

Loading