feat: add skip_optimize model config to opt out of post-materialization OPTIMIZE (#1485)

Resolves #703

### Description

Adds a `skip_optimize` model config that lets users opt out of the
post-materialization `OPTIMIZE` call without removing `zorder` /
`liquid_clustered_by` / `auto_liquid_cluster` from the table definition.

**Motivation**: the existing opt-out today is the run-wide
`DATABRICKS_SKIP_OPTIMIZE` var, which forces an all-or-nothing decision
for the entire invocation. Several users in #703 asked for a
config-level opt-out so they can:

- delegate `OPTIMIZE` to Predictive Optimization while keeping
`auto_liquid_cluster=true` on the table
- skip `OPTIMIZE` only for specific high-churn models and let it run for
the rest
- schedule `OPTIMIZE` out of band (workflow / job) instead of on the dbt
critical path

**Behavior**:

- New model config `skip_optimize` (bool, default `false`)
- When truthy, `databricks__optimize` short-circuits to a no-op even if
`zorder` / `liquid_clustered_by` / `auto_liquid_cluster` is set on the
model — the clustering declaration remains in the table DDL, only the
`OPTIMIZE` SQL emission is suppressed
- Inherits via standard dbt config resolution: project → folder → model
(more specific wins). Example:
  ```yaml
  # dbt_project.yml
  models:
    my_project:
      +skip_optimize: true
      high_read_models:
        +skip_optimize: false
  ```
- `DATABRICKS_SKIP_OPTIMIZE` var is unchanged (still skips run-wide)

### Docs follow-up

User-facing config reference lives in `dbt-labs/docs.getdbt.com`
(`databricks-configs.md`). A companion docs PR will be opened there to
document `skip_optimize` alongside the existing `zorder` /
`liquid_clustered_by` / `auto_liquid_cluster` entries.

### Checklist

- [x] I have run this code in development and it appears to resolve the
stated issue
- [x] This PR includes tests, or tests are not required/relevant for
this PR
- [x] I have updated the `CHANGELOG.md` and added information about my
change to the "dbt-databricks next" section.

---------

Co-authored-by: Shubham Dhal <shubham.dhal@databricks.com>

Author: moomindani

CHANGELOG.md

@@ -3,6 +3,7 @@
 ### Features
 
 - Add catalogs.yml v2 support (requires `use_catalogs_v2: true` in dbt-core) ([1440](https://github.com/databricks/dbt-databricks/pull/1440))
+- Add `skip_optimize` model config to opt out of the post-materialization `OPTIMIZE` call without dropping `zorder` / `liquid_clustered_by` / `auto_liquid_cluster` from the table definition. Useful when `OPTIMIZE` is delegated to Predictive Optimization or scheduled out of band. Complements the existing run-wide `DATABRICKS_SKIP_OPTIMIZE` var by allowing project-, folder-, or model-level opt-out via standard dbt config inheritance ([#703](https://github.com/databricks/dbt-databricks/issues/703)).
 
 ### Fixes
 - Apply column-level `databricks_tags` for incremental models on the V1 materialization path (`use_materialization_v2: false`, the default). They were silently dropped at create and on subsequent tag changes; the V1 incremental materialization now applies them, matching the `table` materialization and the V2 path. ([#1520](https://github.com/databricks/dbt-databricks/pull/1520) closes [#1307](https://github.com/databricks/dbt-databricks/issues/1307))

dbt/adapters/databricks/impl.py

@@ -205,6 +205,7 @@ class DatabricksConfig(AdapterConfig):
     query_tags: Optional[str] = None
     tblproperties: Optional[dict[str, str]] = None
     zorder: Optional[Union[list[str], str]] = None
+    skip_optimize: Optional[bool] = None
     unique_tmp_table_suffix: bool = False
     skip_non_matched_step: Optional[bool] = None
     skip_matched_step: Optional[bool] = None

dbt/include/databricks/macros/relations/optimize.sql

@@ -3,7 +3,8 @@
 {% endmacro %}
 
 {%- macro databricks__optimize(relation) -%}
-  {%- if var('DATABRICKS_SKIP_OPTIMIZE', 'false')|lower != 'true' and
+  {%- if config.get('skip_optimize', false) | as_bool -%}
+  {%- elif var('DATABRICKS_SKIP_OPTIMIZE', 'false')|lower != 'true' and
         var('databricks_skip_optimize', 'false')|lower != 'true' and
         adapter.resolve_file_format(config) == 'delta' -%}
     {%- if (config.get('zorder', False) or config.get('liquid_clustered_by', False)) or config.get('auto_liquid_cluster', False) -%}

tests/unit/macros/base.py

@@ -116,13 +116,25 @@ def databricks_env(self, macro_folders_to_load) -> Environment:
         """
         The environment used for rendering Databricks macros
         """
-        return Environment(
+        env = Environment(
             loader=FileSystemLoader(
                 [f"dbt/include/databricks/{folder}" for folder in macro_folders_to_load]
             ),
             extensions=["jinja2.ext.do"],
         )
 
+        def _as_bool(value):
+            if isinstance(value, bool):
+                return value
+            if str(value).lower() in ("true", "1", "yes"):
+                return True
+            if str(value).lower() in ("false", "0", "no"):
+                return False
+            raise ValueError(f"Cannot convert {value!r} to bool")
+
+        env.filters["as_bool"] = _as_bool
+        return env
+
     @pytest.fixture
     def databricks_template_names(self) -> list:
         """

tests/unit/macros/relations/test_optimize_macros.py

@@ -41,3 +41,20 @@ def test_macros_optimize_with_skip(self, key_val, var, template_bundle):
         r = self.render_bundle(template_bundle, "optimize")
 
         assert r == ""
+
+    @pytest.mark.parametrize(
+        "cluster_key,cluster_val",
+        [
+            ("zorder", "foo"),
+            ("liquid_clustered_by", ["foo"]),
+            ("auto_liquid_cluster", True),
+        ],
+    )
+    def test_macros_optimize_with_skip_optimize_config(
+        self, cluster_key, cluster_val, config, template_bundle
+    ):
+        config[cluster_key] = cluster_val
+        config["skip_optimize"] = True
+        r = self.render_bundle(template_bundle, "optimize")
+
+        assert r == ""