feat: add skip_optimize model config to opt out of post-materialization OPTIMIZE

Adds a `skip_optimize` model config that lets users opt out of the
post-materialization OPTIMIZE call without removing
zorder/liquid_clustered_by/auto_liquid_cluster from the table
definition. Complements the existing run-wide DATABRICKS_SKIP_OPTIMIZE
var by allowing project-, folder-, or model-level opt-out via standard
dbt config inheritance.

Resolves #703

Co-authored-by: Isaac

Author: moomindani

CHANGELOG.md

@@ -3,6 +3,7 @@
 ### Features
 
 - Expose `job_id`, `job_run_id`, and `task_run_id` from the Databricks Jobs `dbt_task` runtime in `adapter_response`, enabling correlation between dbt runs and Databricks workflow executions via `run_results.json` ([#1451](https://github.com/databricks/dbt-databricks/pull/1451) closes [#722](https://github.com/databricks/dbt-databricks/issues/722))
+- Add `skip_optimize` model config to opt out of the post-materialization `OPTIMIZE` call without dropping `zorder` / `liquid_clustered_by` / `auto_liquid_cluster` from the table definition. Useful when `OPTIMIZE` is delegated to Predictive Optimization or scheduled out of band. Complements the existing run-wide `DATABRICKS_SKIP_OPTIMIZE` var by allowing project-, folder-, or model-level opt-out via standard dbt config inheritance ([#703](https://github.com/databricks/dbt-databricks/issues/703)).
 
 ### Fixes
 

dbt/adapters/databricks/impl.py

@@ -197,6 +197,7 @@ class DatabricksConfig(AdapterConfig):
     query_tags: Optional[str] = None
     tblproperties: Optional[dict[str, str]] = None
     zorder: Optional[Union[list[str], str]] = None
+    skip_optimize: Optional[bool] = None
     unique_tmp_table_suffix: bool = False
     skip_non_matched_step: Optional[bool] = None
     skip_matched_step: Optional[bool] = None

dbt/include/databricks/macros/relations/optimize.sql

@@ -3,7 +3,8 @@
 {% endmacro %}
 
 {%- macro databricks__optimize(relation) -%}
-  {%- if var('DATABRICKS_SKIP_OPTIMIZE', 'false')|lower != 'true' and
+  {%- if config.get('skip_optimize', false) -%}
+  {%- elif var('DATABRICKS_SKIP_OPTIMIZE', 'false')|lower != 'true' and
         var('databricks_skip_optimize', 'false')|lower != 'true' and
         adapter.resolve_file_format(config) == 'delta' -%}
     {%- if (config.get('zorder', False) or config.get('liquid_clustered_by', False)) or config.get('auto_liquid_cluster', False) -%}

tests/unit/macros/relations/test_optimize_macros.py

@@ -41,3 +41,20 @@ def test_macros_optimize_with_skip(self, key_val, var, template_bundle):
         r = self.render_bundle(template_bundle, "optimize")
 
         assert r == ""
+
+    @pytest.mark.parametrize(
+        "cluster_key,cluster_val",
+        [
+            ("zorder", "foo"),
+            ("liquid_clustered_by", ["foo"]),
+            ("auto_liquid_cluster", True),
+        ],
+    )
+    def test_macros_optimize_with_skip_optimize_config(
+        self, cluster_key, cluster_val, config, template_bundle
+    ):
+        config[cluster_key] = cluster_val
+        config["skip_optimize"] = True
+        r = self.render_bundle(template_bundle, "optimize")
+
+        assert r == ""