fix(auth/m2m): remove double-caching to enable proactive token refresh (#1550)

Fixes a double-caching bug in M2M OAuth that caused the proactive async
token refresh to have no effect, resulting in bursts of HTTP 401 or 403
errors at each token rotation boundary (~every hour).

Closes #1549.

## Why

`M2mCredentials.Configure` previously called
`clientcredentials.Config.TokenSource(ctx)`, which returns an
`oauth2.ReuseTokenSource`. That source was then passed to
`refreshableVisitor`, which wraps it in a `cachedTokenSource`. The
resulting stack is:

```
cachedTokenSource          ← Databricks async cache (proactive, T-20min)
    └── authTokenSource    ← discards context
            └── oauth2.ReuseTokenSource  ← Go stdlib cache, expiryDelta=10s
                    └── clientcredentials.tokenSource  ← HTTP /token endpoint
```

When `cachedTokenSource` triggers its async refresh at T−20 min, it
calls through to `ReuseTokenSource.Token()`. Because the token still has
20 minutes of life, `ReuseTokenSource` considers it valid and returns
the cached token without an HTTP call. The async refresh fires
repeatedly (with an accelerating schedule) but each call hits the same
inner cache — until only ~10 s remain, at which point
`ReuseTokenSource`'s `expiryDelta` window is crossed and a real network
call is finally made.

Any request that receives the about-to-expire token and whose round-trip
to Databricks completes after the token's expiry time gets an auth
error. In production this manifests as a burst of errors at precisely
one-hour intervals, correlated with pod startup times.

**Note on status codes**: the exact HTTP status code returned for an
expired M2M OAuth token varies by workspace and request type. A
completely invalid token returns 401 on all endpoints. A naturally
expired token has been observed to return 403 `"Invalid Token"` on both
management endpoints (`GET /api/2.0/serving-endpoints/{name}`) and
inference endpoints (`POST /serving-endpoints/{name}/invocations`).
Local workarounds that intercept auth errors at the transport layer need
to handle both 401 and 403.

## What changed

### Interface changes

None.

### Behavioral changes

- M2M OAuth token refresh now makes a real HTTP call to the token
endpoint at T−20 min (the intended proactive window) rather than at T−10
s.
- Callers will see one additional token endpoint call per hour per
process (the proactive refresh), offset by eliminating the burst of auth
errors at expiry.

### Internal changes

- `auth_m2m.go`: replaced `clientcredentials.Config.TokenSource(ctx)`
(which returns a `ReuseTokenSource`) with a `TokenSourceFn` closure that
calls `ccfg.Token(ctx)` directly. `cachedTokenSource` is now the sole
caching layer.
- `cache_test.go`: added `reuseTokenSource` test helper and two new
tests:
- `TestCachedTokenSource_AsyncRefreshBlockedByInnerCache` — documents
the double-caching behaviour using a mock clock.
- `TestCachedTokenSource_AsyncRefreshWithDirectSource` — verifies that a
direct (non-caching) inner source triggers an HTTP fetch at the
proactive window.
- `NEXT_CHANGELOG.md`: updated.

## How is this tested?

- `TestCachedTokenSource_AsyncRefreshBlockedByInnerCache`: uses a fake
clock and a controlled `reuseTokenSource` helper to confirm that inner
caching delays the HTTP fetch to T−10 s rather than T−20 min.
- `TestCachedTokenSource_AsyncRefreshWithDirectSource`: uses the same
fake clock with a direct token source to confirm the fetch occurs at
T−20 min after the fix.
- Both tests operate at the token-source level using a mock clock — they
count real HTTP fetches to the token endpoint rather than checking the
status code returned by Databricks API calls. This means they validate
the fix regardless of whether the downstream auth failure manifests as
401 or 403.
- Existing `TestM2mHappyFlow` and `TestM2mHappyFlowForAccount` continue
to pass.

---------

Signed-off-by: Alex Ausch <alex@ausch.name>
Signed-off-by: Ubuntu <renaud.hartert@databricks.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Renaud Hartert <renaud.hartert@databricks.com>

Author: 3 people

NEXT_CHANGELOG.md

@@ -15,12 +15,14 @@
 
  * Fix double-caching of OAuth tokens in Azure client secret credentials ([#1549](https://github.com/databricks/databricks-sdk-go/issues/1549)).
  * Disable async token refresh for GCP credential providers to avoid wasted refresh attempts caused by double-caching with Google's internal `oauth2.ReuseTokenSource` ([#1549](https://github.com/databricks/databricks-sdk-go/issues/1549)).
+ * Fixed double-caching in M2M OAuth that prevented the proactive async token refresh from reaching the HTTP endpoint until ~10s before expiry, causing bursts of 401 errors at token rotation boundaries ([#1549](https://github.com/databricks/databricks-sdk-go/issues/1549)).
 
 ### Documentation
 
 ### Internal Changes
 
  * Normalize internal token sources on `auth.TokenSource` for proper context propagation ([#1577](https://github.com/databricks/databricks-sdk-go/pull/1577)).
+ * Fix `TestAzureGithubOIDCCredentials` hang caused by missing `HTTPTransport` stub: `EnsureResolved` now calls `resolveHostMetadata`, which makes a real network request when no transport is set ([#1550](https://github.com/databricks/databricks-sdk-go/pull/1550)).
  * Bump golang.org/x/crypto from 0.21.0 to 0.45.0 in /examples/slog ([#1566](https://github.com/databricks/databricks-sdk-go/pull/1566)).
  * Bump golang.org/x/net from 0.23.0 to 0.33.0 in /examples/slog ([#1127](https://github.com/databricks/databricks-sdk-go/pull/1127)).
  * Bump golang.org/x/oauth2 from 0.20.0 to 0.27.0 ([#1563](https://github.com/databricks/databricks-sdk-go/pull/1563)).

config/auth_azure_github_oidc_test.go

@@ -49,6 +49,7 @@ func TestAzureGithubOIDCCredentials(t *testing.T) {
 				AzureTenantID:              "test-tenant-id",
 				ActionsIDTokenRequestURL:   "http://endpoint.com/test?version=1",
 				ActionsIDTokenRequestToken: "token-1337",
+				HTTPTransport:              fixtures.MappingTransport{},
 			},
 			wantErrPrefix: errPrefix("github-oidc-azure auth: not configured"),
 		},
@@ -79,6 +80,7 @@ func TestAzureGithubOIDCCredentials(t *testing.T) {
 				AzureClientID:              "test-client-id",
 				ActionsIDTokenRequestURL:   "http://endpoint.com/test?version=1",
 				ActionsIDTokenRequestToken: "token-1337",
+				HTTPTransport:              fixtures.MappingTransport{},
 			},
 			wantErrPrefix: errPrefix("github-oidc-azure auth: not configured"),
 		},
@@ -94,6 +96,7 @@ func TestAzureGithubOIDCCredentials(t *testing.T) {
 				AzureTenantID:            "test-tenant-id",
 				Host:                     "http://host.com/test",
 				ActionsIDTokenRequestURL: "http://endpoint.com/test?version=1",
+				HTTPTransport:            fixtures.MappingTransport{},
 			},
 			wantErrPrefix: errPrefix("github-oidc-azure auth: not configured"),
 		},
@@ -109,6 +112,7 @@ func TestAzureGithubOIDCCredentials(t *testing.T) {
 				AzureTenantID:              "test-tenant-id",
 				Host:                       "http://host.com/test",
 				ActionsIDTokenRequestToken: "token-1337",
+				HTTPTransport:              fixtures.MappingTransport{},
 			},
 			wantErrPrefix: errPrefix("github-oidc-azure auth: not configured"),
 		},

config/auth_m2m.go

@@ -9,7 +9,6 @@ import (
 
 	"github.com/databricks/databricks-sdk-go/config/credentials"
 	"github.com/databricks/databricks-sdk-go/config/experimental/auth"
-	"github.com/databricks/databricks-sdk-go/config/experimental/auth/authconv"
 	"github.com/databricks/databricks-sdk-go/logger"
 )
 
@@ -28,16 +27,29 @@ func (c M2mCredentials) Configure(ctx context.Context, cfg *Config) (credentials
 		return nil, fmt.Errorf("oidc: %w", err)
 	}
 	logger.Debugf(ctx, "Generating Databricks OAuth token for Service Principal (%s)", cfg.ClientID)
-	ts := (&clientcredentials.Config{
+	ccfg := &clientcredentials.Config{
 		ClientID:     cfg.ClientID,
 		ClientSecret: cfg.ClientSecret,
 		AuthStyle:    oauth2.AuthStyleInHeader,
 		TokenURL:     endpoints.TokenEndpoint,
 		Scopes:       cfg.GetScopes(),
-	}).TokenSource(ctx)
+	}
 
-	authTs := authconv.AuthTokenSource(ts)
+	// Use a direct (non-caching) token source so that cachedTokenSource is the
+	// single cache layer. clientcredentials.Config.TokenSource returns an
+	// oauth2.ReuseTokenSource which adds a second cache with a 10 s expiryDelta.
+	// With double-caching the async refresh in cachedTokenSource calls through to
+	// ReuseTokenSource, which returns its own cached token without making an HTTP
+	// request until only ~10 s remain — defeating the proactive 20-min refresh
+	// window and causing a burst of 401s at token expiry.
+	// See https://github.com/databricks/databricks-sdk-go/issues/1549.
+	//
+	// ctx is captured from Configure; the oauth2 HTTP client is bound to it via
+	// InContextForOAuth2 and must be used for all token requests.
+	directTS := auth.TokenSourceFn(func(_ context.Context) (*oauth2.Token, error) {
+		return ccfg.Token(ctx)
+	})
 	return credentials.NewOAuthCredentialsProviderFromTokenSource(
-		auth.NewCachedTokenSource(authTs, cacheOptions(cfg)...),
+		auth.NewCachedTokenSource(directTS, cacheOptions(cfg)...),
 	), nil
 }

config/auth_m2m_test.go

@@ -1,11 +1,14 @@
 package config
 
 import (
+	"context"
 	"errors"
 	"net/http"
 	"net/url"
+	"sync/atomic"
 	"testing"
 
+	"github.com/databricks/databricks-sdk-go/config/credentials"
 	"github.com/databricks/databricks-sdk-go/credentials/u2m"
 	"github.com/databricks/databricks-sdk-go/httpclient/fixtures"
 	"golang.org/x/oauth2"
@@ -96,6 +99,83 @@ func TestM2mNotSupported(t *testing.T) {
 	}
 }
 
+// TestM2mCredentials_DirectTokenSource verifies that M2mCredentials.Configure
+// plumbs a direct token source (ccfg.Token) through cachedTokenSource rather
+// than wrapping clientcredentials.Config.TokenSource, which returns an
+// oauth2.ReuseTokenSource that adds a second cache layer. With double-caching,
+// the proactive async refresh in cachedTokenSource is silently suppressed until
+// ~10 s before expiry, causing bursts of 401 errors at token rotation boundaries.
+// See https://github.com/databricks/databricks-sdk-go/issues/1549.
+func TestM2mCredentials_DirectTokenSource(t *testing.T) {
+	var tokenCalls int32
+	transport := &postCountingTransport{
+		calls: &tokenCalls,
+		inner: fixtures.MappingTransport{
+			"GET /oidc/.well-known/oauth-authorization-server": {
+				Response: u2m.OAuthAuthorizationServer{
+					TokenEndpoint: "https://localhost/token",
+				},
+			},
+			"POST /token": {
+				Response: oauth2.Token{
+					TokenType:   "Bearer",
+					AccessToken: "test-token",
+				},
+			},
+		},
+	}
+
+	cfg := &Config{
+		Host:          "a",
+		ClientID:      "b",
+		ClientSecret:  "c",
+		AuthType:      "oauth-m2m",
+		ConfigFile:    "/dev/null",
+		HTTPTransport: transport,
+	}
+
+	err := cfg.EnsureResolved()
+	if err != nil {
+		t.Fatalf("EnsureResolved(): %v", err)
+	}
+
+	ctx := cfg.refreshClient.InContextForOAuth2(cfg.refreshCtx)
+	provider, err := M2mCredentials{}.Configure(ctx, cfg)
+	if err != nil {
+		t.Fatalf("Configure(): %v", err)
+	}
+
+	oauthProvider := provider.(credentials.OAuthCredentialsProvider)
+
+	// Token() goes through cachedTokenSource, which fetches once and caches.
+	// Verify the endpoint is reached (not short-circuited by an inner cache).
+	tok, err := oauthProvider.Token(context.Background())
+	if err != nil {
+		t.Fatalf("Token(): %v", err)
+	}
+	if tok.AccessToken == "" {
+		t.Fatalf("Token(): empty access token")
+	}
+
+	if got := int(atomic.LoadInt32(&tokenCalls)); got != 1 {
+		t.Errorf("token endpoint calls = %d, want 1", got)
+	}
+}
+
+type postCountingTransport struct {
+	calls *int32
+	inner http.RoundTripper
+}
+
+func (t *postCountingTransport) RoundTrip(req *http.Request) (*http.Response, error) {
+	if req.Method == "POST" {
+		atomic.AddInt32(t.calls, 1)
+	}
+	return t.inner.RoundTrip(req)
+}
+
+func (t *postCountingTransport) SkipRetryOnIO() bool { return true }
+
 func TestM2M_Scopes(t *testing.T) {
 	tests := []struct {
 		name   string