Skip to content

Commit e27c99f

Browse files
authored
Cache /.well-known/databricks-config lookups in the CLI (#5011)
## Why Every CLI command (`databricks auth profiles`, `bundle validate`, every workspace or account call) goes through `Config.EnsureResolved`, which triggers an unauthenticated GET to `{host}/.well-known/databricks-config` to populate host metadata. That round trip is ~700ms against production and gets paid on every invocation, doubling the latency of otherwise single-request commands. ## Changes **Before:** every CLI invocation hits the well-known endpoint once (or more when multiple configs get constructed). **Now:** the first invocation populates a local disk cache under `~/.cache/databricks/<version>/host-metadata/`; subsequent invocations read from it. Failures are negatively cached for 60s (except for `context.Canceled` / `context.DeadlineExceeded`, which are transient and never cached). The integration hooks into SDK `v0.128.0`'s `config.DefaultHostMetadataResolverFactory` (added in databricks/databricks-sdk-go#1636) via two pieces: - `libs/hostmetadata/resolver.go`: `init()` registers a factory that wraps `cfg.DefaultHostMetadataResolver()` in the caching resolver. `NewResolver(fetch)` remains the unit-testable primary API. - `main.go`: a blank import of `libs/hostmetadata` triggers that `init()` at startup, so every `*config.Config` the CLI constructs now and in the future picks up the cached lookup automatically. No per-site wiring, no guardrail test. Positive cache wraps the miss path, so the hit path is a single disk read; negative cache is only consulted when positive misses. `internal/testutil/env.go` pins `DATABRICKS_CACHE_DIR` to a temp dir in test cleanup so tests don't leak cache files into `HOME`. ### Collateral cleanups - `libs/cache/file_cache.go`: drop the `failed to stat cache file` debug log when the file is simply missing (`fs.ErrNotExist`). It was pure noise (the next line, `cache miss, computing`, conveys the same info) and its OS-specific error text diverged between Unix (`no such file or directory`) and Windows (`The system cannot find the file specified.`), breaking cross-platform acceptance goldens. Genuine stat failures (permission, corruption) still log. - `libs/testdiff/replacement.go`: `devVersionRegex` now accepts either `+SHA` or `-SHA` after `0.0.0-dev`. `build.GetSanitizedVersion()` swaps `+` to `-` for filesystem safety when the version is used in cache paths, and the old regex only covered the `+` form. ## Test plan - [x] `make checks` clean - [x] `make lint` clean (0 issues) - [x] `go test ./libs/hostmetadata/... -race` clean (factory-installed assertion + cache hit + fetch error + cancellation-not-cached + host isolation + end-to-end integration) - [x] `go test ./cmd/root/... ./bundle/config/... ./cmd/auth/... ./libs/auth/... -race` clean - [x] End-to-end acceptance test `acceptance/auth/host-metadata-cache/` asserts exactly ONE `/.well-known/databricks-config` GET across two `auth profiles` invocations sharing a `DATABRICKS_CACHE_DIR` - [x] Existing acceptance tests regenerated: fewer well-known GETs in `out.requests.txt` (caching works), new `[Local Cache]` debug lines in cache/telemetry tests, two `Warn: Failed to resolve host metadata` lines removed (intentional: the resolver returns `(nil, nil)` on fetch errors, which is how the SDK interprets "no metadata available"), stat-not-found lines removed (see Collateral cleanups) ### Live validation against dogfood (from previous push) Built locally and ran `databricks -p e2-dogfood current-user me` with and without a warm cache: | Scenario | Elapsed well-known time | Cache log output | |---|---|---| | Cold cache (fresh `DATABRICKS_CACHE_DIR`) | ~713ms fetch | `cache miss, computing` -> `GET /.well-known/databricks-config` -> `computed and stored result` | | Warm cache (second invocation) | ~1ms | single `[Local Cache] cache hit` line | Net per-command savings: ~700ms, matching the Why.
1 parent 488e86a commit e27c99f

37 files changed

Lines changed: 438 additions & 108 deletions

File tree

NEXT_CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
### CLI
66
* Added `--limit` flag to all paginated list commands for client-side result capping ([#4984](https://github.com/databricks/cli/pull/4984)). On `jobs list` and `jobs list-runs` the former API page-size flag was renamed to `--page-size` (hidden) to avoid collision.
77
* Accept `yes` in addition to `y` for confirmation prompts, and show `[y/N]` to indicate that no is the default.
8+
* Cache `/.well-known/databricks-config` lookups under `~/.cache/databricks/<version>/host-metadata/` so repeat CLI invocations against the same host skip the ~700ms discovery round trip.
89
* Deprecated `auth env`. The command is hidden from help listings and prints a deprecation warning to stderr; it will be removed in a future release.
910

1011
### Bundles

acceptance/auth/bundle_and_profile/output.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313

1414
=== Inside the bundle, profile flag not matching bundle host. Should use profile from the flag and not the bundle.
1515
>>> errcode [CLI] current-user me -p profile_name
16-
Warn: Failed to resolve host metadata: (redacted). Falling back to user config.
16+
Warn: [hostmetadata] failed to fetch host metadata for https://non.existing.subdomain.databricks.com, will skip for 1m0s
1717
Error: Get "https://non.existing.subdomain.databricks.com/api/2.0/preview/scim/v2/Me": (redacted)
1818

1919
Exit code: 1
@@ -73,7 +73,7 @@ Validation OK!
7373

7474
=== Bundle commands load bundle configuration with -t and -p flag, validation not OK (profile host don't match bundle host)
7575
>>> errcode [CLI] bundle validate -t prod -p DEFAULT
76-
Warn: Failed to resolve host metadata: (redacted). Falling back to user config.
76+
Warn: [hostmetadata] failed to fetch host metadata for https://bar.com, will skip for 1m0s
7777
Error: cannot resolve bundle auth configuration: the host in the profile ([DATABRICKS_TARGET]) doesn’t match the host configured in the bundle (https://bar.com)
7878

7979
Name: test-auth

acceptance/auth/bundle_and_profile/test.toml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,6 @@ New='DATABRICKS_TARGET'
99
Old='DATABRICKS_URL'
1010
New='DATABRICKS_TARGET'
1111

12-
[[Repls]]
13-
Old='Warn: Failed to resolve host metadata: .*\. Falling back to user config\.'
14-
New='Warn: Failed to resolve host metadata: (redacted). Falling back to user config.'
15-
1612
[[Repls]]
1713
Old='Get "https://non.existing.subdomain.databricks.com/api/2.0/preview/scim/v2/Me": .*'
1814
New='Get "https://non.existing.subdomain.databricks.com/api/2.0/preview/scim/v2/Me": (redacted)'

acceptance/auth/credentials/unified-host/out.requests.txt

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -22,15 +22,6 @@
2222
"method": "GET",
2323
"path": "/api/2.0/preview/scim/v2/Me"
2424
}
25-
{
26-
"headers": {
27-
"User-Agent": [
28-
"cli/[DEV_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS]"
29-
]
30-
},
31-
"method": "GET",
32-
"path": "/.well-known/databricks-config"
33-
}
3425
{
3526
"headers": {
3627
"Authorization": [

acceptance/auth/host-metadata-cache/out.test.toml

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
2+
=== First invocation populates the cache
3+
{
4+
"profiles": [
5+
{
6+
"name":"cached",
7+
"host":"[DATABRICKS_URL]",
8+
"cloud":"aws",
9+
"auth_type":"",
10+
"valid":false
11+
}
12+
]
13+
}
14+
15+
=== Second invocation should read from the cache
16+
{
17+
"profiles": [
18+
{
19+
"name":"cached",
20+
"host":"[DATABRICKS_URL]",
21+
"cloud":"aws",
22+
"auth_type":"",
23+
"valid":false
24+
}
25+
]
26+
}
27+
28+
=== Only one /.well-known/databricks-config request recorded
29+
{
30+
"method": "GET",
31+
"path": "/.well-known/databricks-config"
32+
}
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
sethome "./home"
2+
export DATABRICKS_CACHE_DIR="$TEST_TMP_DIR/cache"
3+
4+
# Point a profile at the mock server so auth profiles triggers a host metadata
5+
# fetch. Without a profile the command does nothing and the cache is never read.
6+
cat > "./home/.databrickscfg" <<EOF
7+
[cached]
8+
host = ${DATABRICKS_HOST}
9+
token = test-token
10+
EOF
11+
12+
title "First invocation populates the cache\n"
13+
$CLI auth profiles --skip-validate --output json
14+
15+
title "Second invocation should read from the cache\n"
16+
$CLI auth profiles --skip-validate --output json
17+
18+
title "Only one /.well-known/databricks-config request recorded\n"
19+
print_requests.py //.well-known/databricks-config --get --sort
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Ignore = [
2+
"home",
3+
"cache",
4+
]
5+
RecordRequests = true

acceptance/bundle/resources/volumes/change-schema-name/out.requests.txt

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,7 @@
1-
{
2-
"method": "GET",
3-
"path": "/.well-known/databricks-config"
4-
}
51
{
62
"method": "GET",
73
"path": "/api/2.1/unity-catalog/volumes/main.myschema.myvolume"
84
}
9-
{
10-
"method": "GET",
11-
"path": "/.well-known/databricks-config"
12-
}
135
{
146
"method": "GET",
157
"path": "/api/2.1/unity-catalog/volumes/main.myschema.mynewvolume"

acceptance/bundle/run/inline-script/databricks-cli/profile-is-passed/from_flag/out.requests.txt

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,6 @@
1616
"path": "/oidc/v1/token",
1717
"raw_body": "grant_type=client_credentials\u0026scope=all-apis"
1818
}
19-
{
20-
"method": "GET",
21-
"path": "/.well-known/databricks-config"
22-
}
2319
{
2420
"method": "GET",
2521
"path": "/oidc/.well-known/oauth-authorization-server"

0 commit comments

Comments
 (0)