Skip to content

Commit 0c6a10e

Browse files
committed
docs(databricks): merge databricks_tags additively across config hierarchy levels (databricks/dbt-databricks#1340)
`databricks_tags` now merges additively across dbt config hierarchy levels: a model-level `databricks_tags` value no longer clobbers tags defined at the project level — instead the dictionaries are merged (child keys override matching parent keys, other parent keys are retained). This is a breaking change for projects that relied on lower-level configs fully replacing higher-level ones.
1 parent fc1ffb6 commit 0c6a10e

1 file changed

Lines changed: 32 additions & 2 deletions

File tree

website/docs/reference/resource-configs/databricks-configs.md

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ dbt-databricks v1.9 adds support for the `table_format: iceberg` config. Try it
2929

3030
† When `table_format` is `iceberg`, `file_format` must be `delta`.
3131

32-
`databricks_tags` are applied via `ALTER` statements. Tags cannot be removed via dbt-databricks once applied. To remove tags, use Databricks directly or a post-hook.
32+
`databricks_tags` are applied via `ALTER` statements. Tags cannot be removed via dbt-databricks once applied. To remove tags, use Databricks directly or a post-hook. As of dbt-databricks v1.12.0, `databricks_tags` set at multiple config hierarchy levels [merge additively](#databricks_tags) rather than the lower level fully replacing the higher one.
3333

3434
<sup>^</sup> When `liquid_clustered_by` is enabled, dbt-databricks issues an `OPTIMIZE` (Liquid Clustering) operation after each run. To disable this behavior, set the variable `DATABRICKS_SKIP_OPTIMIZE=true`, which can be passed into the dbt run command (`dbt run --vars "{'databricks_skip_optimize': true}"`) or set as an environment variable. See [issue #802](https://github.com/databricks/dbt-databricks/issues/802).
3535

@@ -150,7 +150,7 @@ When materializing models of various types, you may include several optional col
150150
\* `using_columns` supports all parameter types listed in [Databricks column mask parameters](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-column-mask#parameters).
151151

152152

153-
† `databricks_tags` are applied via `ALTER` statements. Tags cannot be removed via dbt-databricks once applied. To remove tags, use Databricks directly or a post-hook.
153+
† `databricks_tags` are applied via `ALTER` statements. Tags cannot be removed via dbt-databricks once applied. To remove tags, use Databricks directly or a post-hook. As of dbt-databricks v1.12.0, `databricks_tags` set at multiple config hierarchy levels [merge additively](#databricks_tags) rather than the lower level fully replacing the higher one.
154154

155155
This example uses the column-level configurations in the previous table:
156156

@@ -1208,6 +1208,36 @@ _Available in versions 1.11 or higher_
12081208

12091209
Tags are applied via `ALTER` statements after the materialization is created. Once applied, tags cannot be removed through dbt-databricks configuration changes. To remove tags, you must use Databricks directly or a post-hook.
12101210

1211+
:::caution Breaking change in v1.12.0
1212+
As of dbt-databricks v1.12.0, `databricks_tags` set at different config hierarchy levels (for example, project-level and model-level) merge additively instead of the lower-level config completely replacing the higher-level one. Keys defined at the lower level override matching keys from the higher level, and any other keys from the higher level are retained. This applies wherever `databricks_tags` is set, including tables, columns, materialized views, and streaming tables.
1213+
:::
1214+
1215+
For example, with the following project-level and model-level configs:
1216+
1217+
<File name='dbt_project.yml'>
1218+
1219+
```yaml
1220+
models:
1221+
my_project:
1222+
+databricks_tags:
1223+
a: "b"
1224+
c: "project_value"
1225+
```
1226+
1227+
</File>
1228+
1229+
<File name='models/my_model.sql'>
1230+
1231+
```sql
1232+
{{ config(
1233+
databricks_tags={'c': 'model_value', 'k': 'v'}
1234+
) }}
1235+
```
1236+
1237+
</File>
1238+
1239+
The model is tagged with `a: b` (kept from the project level), `c: model_value` (the model-level value overrides the project-level `c`), and `k: v` (added at the model level).
1240+
12111241
#### description
12121242
As with views and tables, adding a `description` to your configuration will lead to a table-level comment getting added to your materialization.
12131243

0 commit comments

Comments
 (0)