feat: add skip_optimize model config to opt out of post-materialization OPTIMIZE

[moomindani]

Resolves #703

Description

Adds a skip_optimize model config that lets users opt out of the post-materialization OPTIMIZE call without removing zorder / liquid_clustered_by / auto_liquid_cluster from the table definition.

Motivation: the existing opt-out today is the run-wide DATABRICKS_SKIP_OPTIMIZE var, which forces an all-or-nothing decision for the entire invocation. Several users in #703 asked for a config-level opt-out so they can:

• delegate OPTIMIZE to Predictive Optimization while keeping auto_liquid_cluster=true on the table
• skip OPTIMIZE only for specific high-churn models and let it run for the rest
• schedule OPTIMIZE out of band (workflow / job) instead of on the dbt critical path

Behavior:

• New model config skip_optimize (bool, default false)
• When truthy, databricks__optimize short-circuits to a no-op even if zorder / liquid_clustered_by / auto_liquid_cluster is set on the model — the clustering declaration remains in the table DDL, only the OPTIMIZE SQL emission is suppressed
• Inherits via standard dbt config resolution: project → folder → model (more specific wins). Example:

  # dbt_project.yml
  models:
    my_project:
      +skip_optimize: true
      high_read_models:
        +skip_optimize: false

• DATABRICKS_SKIP_OPTIMIZE var is unchanged (still skips run-wide)

Docs follow-up

User-facing config reference lives in dbt-labs/docs.getdbt.com (databricks-configs.md). A companion docs PR will be opened there to document skip_optimize alongside the existing zorder / liquid_clustered_by / auto_liquid_cluster entries.

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[sd-db]

Overall change looks good, added a minor comment on hardening.

We would also need to make a docs PR for the new config. Also worth to note in the same on how this config interacts with DATABRICKS_SKIP_OPTIMIZE (DATABRICKS_SKIP_OPTIMIZE cannot be overridden per-model with skip_optimize: false)

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks
impl.py
Project Total

This report was generated by python-coverage-comment-action

[sd-db]

/integration-test

[github-actions]

Integration tests dispatched for PR #1485 by @sd-db. Track progress in the Actions tab.

[github-actions]

Integration results for PR #1485 — UC cluster ✅ success · SQL warehouse ✅ success · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

[moomindani]

Thanks for the review! Opened the docs PR for the new config: dbt-labs/docs.getdbt.com#9496

It documents skip_optimize on the Databricks configurations page and notes the interaction with DATABRICKS_SKIP_OPTIMIZE — i.e. the run-wide variable takes precedence and OPTIMIZE cannot be re-enabled per-model with skip_optimize: false once it's set.