dbt-materialize: Support connection overrides

[SangJunBak]

Motivation

Unblocks OIDC auth because our dbt adapter doesn't allow inputting arbitrary options: https://materializeinc.slack.com/archives/C0A23PK183Z/p1776360806044789

Verification

1. Had a local materialize instance running via bin/environmentd
2. Saved following profiles.yml in the current directory:

dbt_test:
  target: dev
  outputs:
    dev:
      type: materialize
      host: localhost
      port: 6875
      user: materialize
      pass: materialize
      database: materialize
      schema: public
      options:
        oidc_auth_enabled: "true"
        welcome_message: "on"

3. Ran

cd misc/dbt-materialize
uv venv dbt-venv
uv pip install --python dbt-venv/bin/python pip
dbt-venv/bin/pip install .
dbt-venv/bin/dbt debug --profiles-dir .

4. Verified output had:

18:43:49    application_name: dbt-materialize v1.9.8
18:43:49    options: {'oidc_auth_enabled': 'true', 'welcome_message': 'on'}

Wasn't sure the best way to automate a unit test for this however.

[SangJunBak]

@bobbyiliev Unsure how we actually deploy an update to our dbt adapter!

Also I decided to go with the approach of allowing an options override instead of accepting a oidc_auth_enabled boolean as a connection option. This is because:

1. This oidc_auth_enabled option doesn't exist for cloud
2. Future flexibility if we need to specify a connection option

[bobbyiliev]

@bobbyiliev Unsure how we actually deploy an update to our dbt adapter!

@SangJunBak This gets auto released after bumping the version bump (version.py + setup.py). What we usually do is to split the version bump itself out into a follow-up release PR. We usually merge the feature + CHANGELOG entry first, then do a separate PR to cut the release to PyPI.

Would you mind reverting the verions bump and adding a changelog entry here? Then we can have a follow-up PR for the release itself. That way if there are other pending changes they all get released together.

[bobbyiliev]

3. Ran

cd misc/dbt-materialize
uv venv dbt-venv
uv pip install --python dbt-venv/bin/python pip
dbt-venv/bin/pip install .
dbt-venv/bin/dbt debug --profiles-dir .

Did you by any chance test this with a full dbt run / dbt build, not just dbt debug? The verification shows the options making it into the connection string, but I'd feel better knowing a real query round-trip works with an override that actually changes behavior (e.g. setting auto_route_catalog_queries: off). Happy to try this on my end if you hit any blockers setting this all up!

[bobbyiliev]

I totally hear you that this is tricky to unit-test, but do you think we could add an integration test in tests/adapter/ e.g. set a non-default option like welcome_message: on via a dbt_profile_target override and assert the server honored it? The options_dict.update(credentials.options) merge is the whole point of the feature. Happy to help sketch something out if useful!

[bobbyiliev]

This new open method copies a chunk of PostgresConnectionManager.open from dbt-postgres. Did you consider overriding a smaller surface instead (e.g. just the options-string construction and then super().open())? My worry is that when dbt-postgres adds new kwargs or changes retry logic, we'll silently fall behind. Totally fine if you already looked at this and it wasn't workable, just curious.

[SangJunBak]

My worry is that when dbt-postgres adds new kwargs or changes retry logic, we'll silently fall behind

I think this is a valid concern :( I tried to but because kwargs is defined internally and there's no way to override options through the credentials object, we can't use super().open(). The best way I could think of is extending off the previous approach of monkey patching psycopg, then doing some weird python thread blocking such that the monkey patch can accept input from our adapter's .open, but that seems worse than copying their implementation. FWIW prior art to this is something like Clickhouse https://github.com/ClickHouse/dbt-clickhouse/blob/main/dbt/adapters/clickhouse/connections.py

[SangJunBak]

Maybe the correct implementation here however is inheriting off SQLConnectionManager instead of PostgresqlConnectionManager?

[bobbyiliev]

Mentioned this in the other comment but could we split the version bump (version.py + setup.py) out of this PR? Normally I prefer to merge the feature + document it under "Unreleased" in the CHANGELOG first, then do a separate release PR that bumps the version and publishes to PyPI. Keeps the feature history and release history cleanly separate.

[SangJunBak]

For sure!

[SangJunBak]

Did you by any chance test this with a full dbt run / dbt build, not just dbt debug? The verification shows the options making it into the connection string, but I'd feel better knowing a real query round-trip works with an override that actually changes behavior (e.g. setting auto_route_catalog_queries: off). Happy to try this on my end if you hit any blockers setting this all up!

I think the verification also tries to connect, so the way I actually tested it was toggling welcome_message to on and off and seeing the output! I can write a unit test for this however!

[SangJunBak]

Created an integration test in the latest commit. Can test via bin/mzcompose --find dbt-materialize run default -k test_options_override

[SangJunBak]

Followup PR to bump the versions #36213