Commit e27c99f
authored
Cache /.well-known/databricks-config lookups in the CLI (#5011)
## Why
Every CLI command (`databricks auth profiles`, `bundle validate`, every
workspace or account call) goes through `Config.EnsureResolved`, which
triggers an unauthenticated GET to
`{host}/.well-known/databricks-config` to populate host metadata. That
round trip is ~700ms against production and gets paid on every
invocation, doubling the latency of otherwise single-request commands.
## Changes
**Before:** every CLI invocation hits the well-known endpoint once (or
more when multiple configs get constructed).
**Now:** the first invocation populates a local disk cache under
`~/.cache/databricks/<version>/host-metadata/`; subsequent invocations
read from it. Failures are negatively cached for 60s (except for
`context.Canceled` / `context.DeadlineExceeded`, which are transient and
never cached).
The integration hooks into SDK `v0.128.0`'s
`config.DefaultHostMetadataResolverFactory` (added in
databricks/databricks-sdk-go#1636) via two pieces:
- `libs/hostmetadata/resolver.go`: `init()` registers a factory that
wraps `cfg.DefaultHostMetadataResolver()` in the caching resolver.
`NewResolver(fetch)` remains the unit-testable primary API.
- `main.go`: a blank import of `libs/hostmetadata` triggers that
`init()` at startup, so every `*config.Config` the CLI constructs now
and in the future picks up the cached lookup automatically. No per-site
wiring, no guardrail test.
Positive cache wraps the miss path, so the hit path is a single disk
read; negative cache is only consulted when positive misses.
`internal/testutil/env.go` pins `DATABRICKS_CACHE_DIR` to a temp dir in
test cleanup so tests don't leak cache files into `HOME`.
### Collateral cleanups
- `libs/cache/file_cache.go`: drop the `failed to stat cache file` debug
log when the file is simply missing (`fs.ErrNotExist`). It was pure
noise (the next line, `cache miss, computing`, conveys the same info)
and its OS-specific error text diverged between Unix (`no such file or
directory`) and Windows (`The system cannot find the file specified.`),
breaking cross-platform acceptance goldens. Genuine stat failures
(permission, corruption) still log.
- `libs/testdiff/replacement.go`: `devVersionRegex` now accepts either
`+SHA` or `-SHA` after `0.0.0-dev`. `build.GetSanitizedVersion()` swaps
`+` to `-` for filesystem safety when the version is used in cache
paths, and the old regex only covered the `+` form.
## Test plan
- [x] `make checks` clean
- [x] `make lint` clean (0 issues)
- [x] `go test ./libs/hostmetadata/... -race` clean (factory-installed
assertion + cache hit + fetch error + cancellation-not-cached + host
isolation + end-to-end integration)
- [x] `go test ./cmd/root/... ./bundle/config/... ./cmd/auth/...
./libs/auth/... -race` clean
- [x] End-to-end acceptance test `acceptance/auth/host-metadata-cache/`
asserts exactly ONE `/.well-known/databricks-config` GET across two
`auth profiles` invocations sharing a `DATABRICKS_CACHE_DIR`
- [x] Existing acceptance tests regenerated: fewer well-known GETs in
`out.requests.txt` (caching works), new `[Local Cache]` debug lines in
cache/telemetry tests, two `Warn: Failed to resolve host metadata` lines
removed (intentional: the resolver returns `(nil, nil)` on fetch errors,
which is how the SDK interprets "no metadata available"), stat-not-found
lines removed (see Collateral cleanups)
### Live validation against dogfood (from previous push)
Built locally and ran `databricks -p e2-dogfood current-user me` with
and without a warm cache:
| Scenario | Elapsed well-known time | Cache log output |
|---|---|---|
| Cold cache (fresh `DATABRICKS_CACHE_DIR`) | ~713ms fetch | `cache
miss, computing` -> `GET /.well-known/databricks-config` -> `computed
and stored result` |
| Warm cache (second invocation) | ~1ms | single `[Local Cache] cache
hit` line |
Net per-command savings: ~700ms, matching the Why.1 parent 488e86a commit e27c99f
37 files changed
Lines changed: 438 additions & 108 deletions
File tree
- acceptance
- auth
- bundle_and_profile
- credentials/unified-host
- host-metadata-cache
- bundle
- resources/volumes/change-schema-name
- run/inline-script/databricks-cli
- profile-is-passed/from_flag
- target-is-passed
- default
- from_flag
- cache
- clear
- simple
- cmd
- auth/profiles
- workspace/apps
- telemetry
- failure
- partial-success
- skipped
- success
- timeout
- workspace
- lakeview/publish
- repos
- create_with_provider
- delete_by_path
- get_errors
- update
- internal/testutil
- libs
- cache
- hostmetadata
- testdiff
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | 12 | | |
17 | 13 | | |
18 | 14 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | 25 | | |
35 | 26 | | |
36 | 27 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
Lines changed: 0 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
5 | 1 | | |
6 | 2 | | |
7 | 3 | | |
8 | 4 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | 5 | | |
14 | 6 | | |
15 | 7 | | |
| |||
Lines changed: 0 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | 19 | | |
24 | 20 | | |
25 | 21 | | |
| |||
0 commit comments