Merge branch 'jb/aitools-interface' into jb/aitools-list-json

Author: simonfaltum

.agent/skills/pr-checklist/SKILL.md

@@ -16,8 +16,9 @@ Before submitting a PR, run these commands to match what CI checks. CI uses the
 # 3. Tests (CI runs with both deployment engines)
 ./task test
 
-# 4. If you changed bundle config structs or schema-related code:
+# 4. If you changed bundle config structs, schema, or direct-engine resource code:
 ./task generate-schema
+./task generate-direct
 
 # 5. If you changed files in python/:
 ./task pydabs-codegen pydabs-test pydabs-lint pydabs-docs

NEXT_CHANGELOG.md

@@ -4,12 +4,16 @@
 
 ### Notable Changes
 
-### CLI
+* Breaking change: OAuth tokens for interactive logins (`auth_type = databricks-cli`) are now stored in the OS-native secure store by default (Keychain on macOS, Credential Manager on Windows, Secret Service on Linux) instead of `~/.databricks/token-cache.json`. After upgrading, run `databricks auth login` once per profile to re-authenticate; cached tokens from older versions are not migrated. To keep the previous file-backed storage, set `DATABRICKS_AUTH_STORAGE=plaintext` or add `auth_storage = plaintext` under `[__settings__]` in `~/.databrickscfg` (the env var takes precedence over the config setting), then re-run `databricks auth login`. On systems where the OS keyring is not reachable (e.g. Linux containers without a D-Bus session bus), the CLI transparently falls back to the file cache when reading tokens so legacy `token-cache.json` entries remain accessible without manual configuration.
 
-* Added `databricks aitools` command group for installing Databricks skills into your coding agents (Claude Code, Cursor, Codex CLI, OpenCode, GitHub Copilot, Antigravity). Skills are fetched from [github.com/databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills) and either symlinked into each agent's skills directory or copied into the current project. Use `databricks aitools install` to set up, `update` to pull newer versions, `list` to see what's available, and `uninstall` to remove them. Pick where they go with `--scope=project|global` (`--scope=both` is accepted on `update` and `list`).
+### CLI
+* Added `databricks aitools` command group for installing Databricks skills into your coding agents (Claude Code, Cursor, Codex CLI, OpenCode, GitHub Copilot, Antigravity). Skills are fetched from [github.com/databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills) and either symlinked into each agent's skills directory or copied into the current project. Use `databricks aitools install` to set up, `update` to pull newer versions, `list` to see what's available, and `uninstall` to remove them.
 * `databricks aitools list` honors `--output json`, emitting a structured `{release, skills[...], summary{}}` document so coding agents and CI can consume the skill/version/installation matrix without scraping the tabular text output.
 
 ### Bundles
 * Make sure warnings asking for approval are understood by agents ([#5239](https://github.com/databricks/cli/pull/5239))
+* Support `replace_existing: true` on `postgres_branches` and `postgres_endpoints` so bundles can manage the implicitly-created production branch and primary read-write endpoint of a Lakebase project.
+* Add `postgres_catalogs` resource to bind a Unity Catalog catalog to a Postgres database on a Lakebase Autoscaling branch ([#5265](https://github.com/databricks/cli/pull/5265)).
+* engine/direct: Changes to state file now persisted to .wal file right away instead of being saved in the end ([#5149](https://github.com/databricks/cli/pull/5149))
 
 ### Dependency updates

acceptance/auth/bundle_default_profile/databricks.yml

@@ -0,0 +1,5 @@
+bundle:
+  name: test-default-profile
+
+# No workspace.host on purpose: this is the surface where
+# [__settings__].default_profile should be applied.

…ribe/u2m-plaintext-default/out.test.toml …uth/bundle_default_profile/out.test.tomlacceptance/cmd/auth/describe/u2m-plaintext-default/out.test.toml renamed to acceptance/auth/bundle_default_profile/out.test.toml acceptance/cmd/auth/describe/u2m-plaintext-default/out.test.toml renamed to acceptance/auth/bundle_default_profile/out.test.toml

@@ -0,0 +1,41 @@
+
+=== Bundle without workspace.host: default_profile is honored
+
+>>> [CLI] bundle validate -o json
+{
+  "host": null,
+  "profile": "default-target"
+}
+
+=== --profile overrides default_profile (negative case)
+
+>>> errcode [CLI] bundle validate -p other -o json
+Warn: [hostmetadata] failed to fetch host metadata for https://other.test, will skip for 1m0s
+Error: Get "https://other.test/api/2.0/preview/scim/v2/Me": (redacted)
+
+
+Exit code: 1
+{
+  "host": null,
+  "profile": "other"
+}
+
+=== Bundle with workspace.host: default_profile is NOT applied
+
+>>> errcode [CLI] bundle validate -o json
+{
+  "host": "[DATABRICKS_URL]",
+  "profile": null
+}
+
+=== Bundle with workspace.profile: pinned profile wins over default_profile
+
+>>> errcode [CLI] bundle validate -o json
+Error: Get "https://other.test/api/2.0/preview/scim/v2/Me": (redacted)
+
+
+Exit code: 1
+{
+  "host": null,
+  "profile": "other"
+}

acceptance/auth/bundle_default_profile/output.txt

@@ -0,0 +1,58 @@
+sethome "./home"
+
+# Save the test server host so we can pin a bundle-with-host variant below.
+host_value="$DATABRICKS_HOST"
+
+cat > "./home/.databrickscfg" <<EOF
+[default-target]
+host  = $DATABRICKS_HOST
+token = $DATABRICKS_TOKEN
+
+[other]
+host  = https://other.test
+token = other-token
+
+[__settings__]
+default_profile = default-target
+EOF
+
+unset DATABRICKS_HOST
+unset DATABRICKS_TOKEN
+unset DATABRICKS_CONFIG_PROFILE
+
+title "Bundle without workspace.host: default_profile is honored\n"
+trace $CLI bundle validate -o json | jq '{host: .workspace.host, profile: .workspace.profile}'
+
+title "--profile overrides default_profile (negative case)\n"
+trace errcode $CLI bundle validate -p other -o json | jq '{host: .workspace.host, profile: .workspace.profile}'
+
+# Switch to a bundle that pins workspace.host. The default_profile guard in
+# configureProfile must NOT apply default_profile here, because that would
+# silently route the user to a profile pointing at a different host than the
+# bundle declares.
+mkdir -p ./bundle-with-host
+cat > ./bundle-with-host/databricks.yml <<EOF
+bundle:
+  name: bundle-with-host
+
+workspace:
+  host: $host_value
+EOF
+
+title "Bundle with workspace.host: default_profile is NOT applied\n"
+(cd ./bundle-with-host && trace errcode $CLI bundle validate -o json | jq '{host: .workspace.host, profile: .workspace.profile}')
+
+# Switch to a bundle that pins workspace.profile but no host. The pinned
+# profile must win over default_profile — configureProfile's guard skips
+# default_profile when workspace.profile is already set.
+mkdir -p ./bundle-with-profile
+cat > ./bundle-with-profile/databricks.yml <<EOF
+bundle:
+  name: bundle-with-profile
+
+workspace:
+  profile: other
+EOF
+
+title "Bundle with workspace.profile: pinned profile wins over default_profile\n"
+(cd ./bundle-with-profile && trace errcode $CLI bundle validate -o json | jq '{host: .workspace.host, profile: .workspace.profile}')

acceptance/auth/bundle_default_profile/script

@@ -0,0 +1,12 @@
+Ignore = [
+    "home",
+    ".databricks",
+    "bundle-with-host",
+    "bundle-with-profile",
+]
+
+# Negative case: -p other tries to reach a non-existing host. Redact the
+# OS-/network-dependent suffix so the test is stable across runners.
+[[Repls]]
+Old = 'Get "https://other.test/api/2.0/preview/scim/v2/Me": .*'
+New = 'Get "https://other.test/api/2.0/preview/scim/v2/Me": (redacted)'

acceptance/auth/bundle_default_profile/test.toml

@@ -0,0 +1,12 @@
+#!/usr/bin/env python3
+import os, sys
+
+errors = 0
+
+for filename in sys.argv[1:]:
+    if not os.path.exists(filename):
+        sys.stderr.write(f"Unexpected: {filename} does not exist.\n")
+        errors += 1
+
+if errors:
+    sys.exit(1)

acceptance/bin/assert_exists.py

@@ -0,0 +1,12 @@
+#!/usr/bin/env python3
+import os, sys
+
+errors = 0
+
+for filename in sys.argv[1:]:
+    if os.path.exists(filename):
+        sys.stderr.write(f"Unexpected: {filename} exists.\n")
+        errors += 1
+
+if errors:
+    sys.exit(1)

acceptance/bin/assert_not_exists.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""Set up a kill rule on the testserver for the current test token.
+
+Usage: kill_after.py PATTERN OFFSET TIMES
+
+  PATTERN   HTTP method and path, e.g. "POST /api/2.2/jobs/create"
+  OFFSET    number of requests to let through before killing starts
+  TIMES     number of times to kill the caller
+
+The rule is scoped to the current DATABRICKS_TOKEN so it only affects
+the test that registers it, even when tests share a server.
+"""
+
+import json
+import os
+import sys
+import urllib.request
+
+host = os.environ.get("DATABRICKS_HOST", "")
+token = os.environ.get("DATABRICKS_TOKEN", "")
+
+if not host:
+    print("DATABRICKS_HOST not set", file=sys.stderr)
+    sys.exit(1)
+
+if len(sys.argv) != 4:
+    print(f"usage: {sys.argv[0]} PATTERN OFFSET TIMES", file=sys.stderr)
+    sys.exit(1)
+
+pattern, offset, times = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
+
+data = json.dumps({"pattern": pattern, "offset": offset, "times": times}).encode()
+req = urllib.request.Request(
+    f"{host}/__testserver/kill",
+    data=data,
+    headers={"Content-Type": "application/json", "Authorization": f"Bearer {token}"},
+    method="POST",
+)
+urllib.request.urlopen(req)