Skip to content

Commit f6a8593

Browse files
canbekleysd-db
andauthored
feat: databricks tags merge behavior (#1340)
<!-- Please review our pull request review process in CONTRIBUTING.md before your proceed. --> Resolves #1184 Resolves #1188 ### Description defining `databricks_tags` on a lower configuration level currently completely replaces/clobbers any tag configuration defined at higher level (e.g. in `dbt_project.yml`). this is very unintuitive for dictionary type configurations and is usually solved with additive dictionary merging behaviour (e.g. `meta`, `tags`, `grants` etc.). dbt-common implements the `MergeBehavior` enum that can be used to set the meta_data of the dataclass fields in DatabricksConfig, which is handled by the BaseConfig. defining `MergeBehavior.Update` for `databricks_tags` basically makes it an additive dictionary field. Note: - this implementation could probably be extended to the `tblproperties` configuration as well, as those suffer the same limitations at the moment. I could add it in here if you like. ### Checklist - [x] I have run this code in development and it appears to resolve the stated issue - [x] This PR includes tests, or tests are not required/relevant for this PR - [x] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-databricks next" section. --------- Signed-off-by: Can Bekleyici <can.bekleyici@deepl.com> Co-authored-by: Shubham Dhal <shubham.dhal@databricks.com>
1 parent 8035fac commit f6a8593

4 files changed

Lines changed: 36 additions & 3 deletions

File tree

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@
77
- Add support for Python UDFs ([#1336](https://github.com/databricks/dbt-databricks/pull/1336))
88
- Add support for key-only `databricks_tags` for table and column tagging. This can now be configured by setting tag values to empty strings `""` or `None`. ([#1339](https://github.com/databricks/dbt-databricks/pull/1339))
99

10+
### Under the Hood
11+
12+
- **BREAKING:** `databricks_tags` defined at different hierarchy levels (e.g. project-level and model-level) now merge additively instead of the child config completely replacing the parent.
13+
1014
## dbt-databricks 1.11.7 (Apr 17, 2026)
1115

1216
### Features

dbt/adapters/databricks/impl.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
from collections.abc import Iterable, Iterator
66
from concurrent.futures import Future
77
from contextlib import contextmanager
8-
from dataclasses import dataclass
8+
from dataclasses import dataclass, field
99
from importlib import metadata
1010
from multiprocessing.context import SpawnContext
1111
from typing import TYPE_CHECKING, Any, ClassVar, Generic, NamedTuple, Optional, Union, cast
@@ -29,7 +29,7 @@
2929
SparkAdapter,
3030
)
3131
from dbt_common.behavior_flags import BehaviorFlag
32-
from dbt_common.contracts.config.base import BaseConfig
32+
from dbt_common.contracts.config.base import BaseConfig, MergeBehavior
3333
from dbt_common.exceptions import DbtConfigError, DbtInternalError
3434
from dbt_common.utils import executor
3535
from dbt_common.utils.dict import AttrDict
@@ -176,7 +176,9 @@ class DatabricksConfig(AdapterConfig):
176176
options: Optional[dict[str, str]] = None
177177
merge_update_columns: Optional[str] = None
178178
merge_exclude_columns: Optional[str] = None
179-
databricks_tags: Optional[dict[str, str]] = None
179+
databricks_tags: Optional[dict[str, str]] = field(
180+
default=None, metadata=MergeBehavior.Update.meta()
181+
)
180182
query_tags: Optional[str] = None
181183
tblproperties: Optional[dict[str, str]] = None
182184
zorder: Optional[Union[list[str], str]] = None

tests/functional/adapter/tags/fixtures.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,15 @@
1616
select cast(1 as bigint) as id, 'hello' as msg, 'blue' as color
1717
"""
1818

19+
tags_merged_sql = """
20+
{{ config(
21+
materialized = 'table',
22+
databricks_tags = {'c': 'd', 'k': ''},
23+
) }}
24+
25+
select cast(1 as bigint) as id, 'hello' as msg, 'blue' as color
26+
"""
27+
1928
streaming_table_tags_sql = """
2029
{{ config(
2130
materialized='streaming_table',

tests/functional/adapter/tags/test_databricks_tags.py

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,24 @@ def test_updated_tags(self, project):
6262
assert actual_tags == expected_tags
6363

6464

65+
@pytest.mark.skip_profile("databricks_cluster")
66+
class TestTableTagsMerged(BaseTestTags):
67+
@pytest.fixture(scope="class")
68+
def models(self):
69+
return {"tags.sql": fixtures.tags_merged_sql}
70+
71+
@pytest.fixture(scope="class")
72+
def project_config_update(self):
73+
return {
74+
"models": {
75+
"+databricks_tags": {
76+
"a": "b",
77+
"c": "TO_BE_REPLACED_AT_MODEL_LEVEL",
78+
}
79+
}
80+
}
81+
82+
6583
@pytest.mark.skip_profile("databricks_cluster")
6684
class TestTableTagsUpdateViaAlter(BaseTestTagsUpdateViaAlter):
6785
pass

0 commit comments

Comments
 (0)