Kubernetes Secrets Backend fails to reload `/var/run/secrets/kubernetes.io/serviceaccount/token`

[jd-hatzenbuhler]

Describe the bug
When using the local cluster config for the backend configuration, the plugin fails to reload the token once it expires.

To Reproduce
Steps to reproduce the behavior:

• Create a Kubernetes Secrets Backend without the service_account_jwt
• Create a role for the Kubernetes Secrets Engine on this backend
• Generate a credentials for the role - it will work
• Wait until the /var/run/secrets/kubernetes.io/serviceaccount/token expires
• Generate a credentials for the role - it will not work

Expected behavior
The Kubernetes Secrets Backend will reload the /var/run/secrets/kubernetes.io/serviceaccount/token every minute

The issue is in the backend caching that will not reload the client see path_creds.go
The code should changed from

if client != nil {
	return client, nil
}

to:

if client != nil && !b.isNeedReload(ctx, s) {
	return client, nil
}

where isNeedReload check if we need to reload the /var/run/secrets/kubernetes.io/serviceaccount/token

[makaiver]

Root Cause

getClient() caches b.client and returns it on all subsequent calls without ever re-reading config. newClient() bakes the JWT into rest.Config.BearerToken as a static string. So once the projected service account token expires (~1h), the cached client holds a stale token and all K8s API calls fail with 401.

The CachingFileReader infrastructure (localSATokenReader, configured with jwtReloadPeriod = 1min) already handles re-reading the token from disk — but configWithDynamicValues() is never called again because getClient() short-circuits at the cache check.

Fix

Move the configWithDynamicValues() call before the cache check, and compare the current token against the one used to build the cached client. The client is only rebuilt when the token actually changes (once per token rotation, typically ~1h). Between rotations, CachingFileReader.ReadFile() returns the same cached bytes, the string comparison matches, and the existing client is reused — zero overhead.

This matches how the sibling auth plugin (vault-plugin-auth-kubernetes) works: its loadConfig() is called on every login request, relying on the same CachingFileReader mechanism.

Changes (2 files, ~6 lines):

backend.go — add lastToken field and clear it in reset():

 type backend struct {
 	*framework.Backend
 	lock   sync.Mutex
 	client *client
+	// lastToken is the JWT used to construct the cached client.
+	// Used to detect when the token has been rotated by CachingFileReader.
+	lastToken string
 
 	localSATokenReader *fileutil.CachingFileReader
 	localCACertReader  *fileutil.CachingFileReader
 }

 func (b *backend) reset() {
 	b.lock.Lock()
 	defer b.lock.Unlock()
 	b.client = nil
+	b.lastToken = ""
 }

path_creds.go — fetch config before the cache check:

 func (b *backend) getClient(ctx context.Context, s logical.Storage) (*client, error) {
 	b.lock.Lock()
 	defer b.lock.Unlock()
 
-	client := b.client
-	if client != nil {
-		return client, nil
-	}
-
 	config, err := b.configWithDynamicValues(ctx, s)
 	if err != nil {
 		return nil, err
 	}
-
-	if b.client == nil && config == nil {
+	if config == nil {
 		config = new(kubeConfig)
 	}
 
+	// Reuse the cached client if the token hasn't changed.
+	// CachingFileReader returns the same bytes until jwtReloadPeriod
+	// expires, so this is a no-op for most calls.
+	if b.client != nil && b.lastToken == config.ServiceAccountJwt {
+		return b.client, nil
+	}
+
 	b.client, err = newClient(config)
 	if err != nil {
 		return nil, err
 	}
+	b.lastToken = config.ServiceAccountJwt
 
 	return b.client, nil
 }

[gnadaban]

Hi @tvoran , would you kindly take a look at this? Over the past weeks we've had a bunch of pages that I traced back to this exact issue.

It would mean the world if this was fixed so we don't have to hack around the problem.