feat: add kernel integration test lane + use_kernel auth support#1576
Merged
Conversation
sd-db
force-pushed
the
sd-db/kernel-integration-test
branch
from
87bdea7 to
ddc574e
Compare
sd-db
changed the title
sd-db
force-pushed
the
sd-db/kernel-integration-test
branch
from
ddc574e to
ef3c482
Compare
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
sd-db
force-pushed
the
sd-db/kernel-integration-test
branch
from
b5a0794 to
23e2dcb
Compare
| "(azure_client_id/azure_client_secret). Use a personal access token " | ||
| "or Databricks OAuth (client_id/client_secret) instead, or drop " | ||
| "use_kernel to use the default backend." | ||
| ) |
There was a problem hiding this comment.
can we let this error be raised at the kernel level?
vikrantpuppala
approved these changes
Jul 6, 2026
Adds a weekly SQL-warehouse integration workflow that runs the functional suite through the databricks-sql-connector Rust kernel backend (use_kernel=True), and the adapter change needed for connections to authenticate on that path. The kernel routes over SEA and owns the OAuth token lifecycle itself, so its auth bridge rejects the connector credentials_provider dbt passes on the Thrift path. prepare_connection_arguments now forwards the raw credential the bridge understands (PAT, Databricks OAuth M2M, or OAuth U2M) when use_kernel is set, and raises a clear error for native Azure service-principal auth (which the kernel cannot serve) rather than misrouting it. SEA is warehouse-only, so the workflow runs only the databricks_uc_sql_endpoint profile and tests that pin a model to a cluster are marked @pytest.mark.skip_kernel.
sd-db
force-pushed
the
sd-db/kernel-integration-test
branch
from
23e2dcb to
ffa6ae3
Compare
The kernel's oauth-m2m flow owns the full token lifecycle including refresh (re-mints via workspace OIDC), so forwarding raw client_id/client_secret keeps long-running connections authenticated. Reverts to raw-cred forwarding as the primary path for PAT and Databricks OAuth (M2M/U2M). Azure service principals have no kernel-native flow, so they remain a fallback: dbt resolves the Azure exchange and forwards the resulting bearer token to the kernel's PAT path, with a warning that this token does not auto-refresh. Routing uses dbt's resolved config.auth_type, since an Azure SP may arrive either via azure_client_id/secret or (legacily) the client_id/client_secret fields. Fixes the CI invalid_client failure (peco is an Azure SP) without giving up token refresh for the common PAT/Databricks-OAuth cases.
Split _add_kernel_auth_arguments into three clear identification paths per review — PAT, U2M, and M2M — and reject/route the Azure SP up front: - Native Azure SP (azure_client_id/secret) routes to the bearer fallback before the M2M resolution, since it has no kernel-native flow. - U2M is identified simply as auth_type == "oauth" with no client secret. - M2M forwards raw creds when the resolved config.auth_type is oauth-m2m (kernel owns refresh); a legacy Azure SP passed via client_id/client_secret resolves to azure-client-secret and takes the bearer fallback. Extract the bearer-fallback into a helper. Terser docstring and CHANGELOG.
…t setup Per review, the production kernel path should support only what the kernel natively does: PAT, Databricks OAuth M2M (raw creds so the kernel owns refresh), and OAuth U2M. Any other auth — notably an Azure service principal, which the kernel has no flow for — is rejected with a DbtConfigError directing the user to the default backend. Removes the in-adapter Azure->bearer-token conversion that was only needed to run CI on a peco Azure SP. That conversion now lives in test setup (tests/profiles.py): when DBT_DATABRICKS_USE_KERNEL=1 and the credential is an SP, resolve it to a workspace token and hand the kernel a PAT. Marked temporary — the resolved token does not auto-refresh, and it goes away once the kernel gains native Azure support. Also restores test concurrency to -n 10 to match the other integration lanes.
…ken TTL The kernel evicts SEA sessions under sustained -n 10 load, so late in-test verification queries intermittently fail with "Cannot execute_command without an open session". Drop the kernel lane to -n 4 to keep the concurrent session count low. At -n 4 the 3-shard run took ~60min/shard — right at the ~1h lifetime of the Azure token the tests use — so split into 5 shards (~35min each) to keep every job well clear of that boundary. Both are kernel-lane workarounds documented in the workflow; the other lanes stay at -n 10 / 3 shards.
The temp trigger was added to validate the workflow on PR #1576; CI is green across all shards (run 28882012926), so revert to the intended workflow_dispatch + weekly schedule triggers.
jprakash-db
approved these changes
Jul 8, 2026
Forward kernel OAuth fields from the credential manager so default client IDs and scopes match the normal auth path while avoiding SDK auth initialization on short-circuit flows.
sd-db
added a commit
that referenced
this pull request
Jul 9, 2026
## Summary Release prep for **dbt-databricks 1.12.2**. Touches only the two release files: - `CHANGELOG.md` — set the `1.12.2` heading date to `Jul 9, 2026` and finalize the section (conformance cleanup + newly-added entries; see below). - `dbt/adapters/databricks/__version__.py` — bump `1.12.1` → `1.12.2`. ## CHANGELOG changes in this PR - **Add `#1508`** (Under the Hood): record/replay instrumentation for `add_query`, `get_relation_config`, `is_uniform`, `has_dbr_capability`, `is_cluster` (test-only, no runtime impact). - **Consolidate the coverage-run test work** into a single Under-the-Hood entry spanning the release's ~38 test/CI PRs, replacing the several partial test lines that were scattered across the section. The `#1576` kernel integration-test workflow line is kept separate (tied to the kernel feature). - **Conformance pass** per `AGENTS.md`: fix bare PR links (`[1440]`/`[1467]`/`[1539]` → `[#…]`), normalize `partially solves` → `partially resolves` and `closes` → `resolves` on the touched entries. ## Test plan - [x] `hatch run code-quality` — ruff / ruff format / mypy pass. (`uv-lock --check` is a known local-config false-negative; this PR does not touch `pyproject.toml` / `uv.lock`.) - [x] `hatch run unit` — 1258 passed, 6 skipped, 0 failures. - [x] `hatch run pre-commit run --files CHANGELOG.md dbt/adapters/databricks/__version__.py` — pass. - [x] CI (code quality, unit 3.10–3.13, build) to run on this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a weekly integration-test lane that exercises the dbt-databricks functional suite through the databricks-sql-connector Rust kernel backend (
use_kernel=True, connector 4.3.0), plus the adapter change required for connections to authenticate on that path.Why an adapter change is needed
The kernel routes over the SEA (Statement Execution API) HTTP transport and owns the OAuth token lifecycle itself, so its auth bridge rejects the connector
credentials_providerthat dbt passes on the default Thrift path. Without a change, every connection underuse_kernel=Truefails at open.SqlUtils.prepare_connection_argumentsnow, whenuse_kernelis set, forwards the raw credential the bridge understands instead:token)access_tokenclient_id/client_secret)oauth_client_id/oauth_client_secret+ the credential manager's resolvedoauth_scopesauth_type: oauth)auth_type=databricks-oauth, forwarding the credential manager's resolvedoauth_client_id+oauth_scopes(dbt's defaults, not the kernel's)azure_client_id/azure_client_secret)DbtConfigError— the kernel has no Azure-AD flow, so fail loud rather than misrouteThe default (Thrift) path is unchanged.
Workflow
.github/workflows/integration-kernel.ymlmirrors the SPOG scheduled smoke but runs a single job: it installs the optionaldatabricks-sql-connector[kernel]extra and runs thedatabricks_uc_sql_endpointprofile withDBT_DATABRICKS_USE_KERNEL=1. Scheduled Saturday 09:17 UTC, clear of the daily nightly and Sunday SPOG slots that share the peco warehouse.SEA is warehouse-only (the connector rejects an all-purpose-cluster http_path), so only the SQL-warehouse profile runs, and the handful of tests that pin a model to a cluster path/compute are marked
@pytest.mark.skip_kernel(a new marker enforced by an autouse fixture, gated onDBT_DATABRICKS_USE_KERNEL).Verification
tests/unit/test_handle.pypin each auth shape's forwarded kwargs (PAT, M2M with/withoutdoseprefix,oauth_scopes, U2M with/without explicitclient_id, native-Azure guard).use_kernel=Trueopens aKernelDatabricksClientand a basic table materialization passes (6/6); serverless Python models pass (their SQL connection stays on the warehouse).skip_kernel-marked tests skip under the flag and run normally without it.