Skip to content

Cache Azure user delegation key for SAS signing#781

Open
emilk wants to merge 2 commits into
apache:mainfrom
emilk:emilk/cache-azure-user-delegation-key
Open

Cache Azure user delegation key for SAS signing#781
emilk wants to merge 2 commits into
apache:mainfrom
emilk:emilk/cache-azure-user-delegation-key

Conversation

@emilk

@emilk emilk commented Jun 25, 2026

Copy link
Copy Markdown

Rationale for this change

AzureClient::signer() calls get_user_delegation_key() — a GetUserDelegationKey network round-trip (POST /?restype=service&comp=userdelegationkey) — on every signed_url / signed_urls call. A user delegation key is reusable until its expiry, so re-fetching it per request is wasteful and, under load, gets throttled by Azure (HTTP 503 ServerBusy).

Notably the other backends do not make an uncached network call while signing: the AWS signer signs locally from its cached credential, and GCP caches its signing credentials. Azure is the outlier — it makes a GetUserDelegationKey call on top of the already-cached AAD token.

We hit this in production: a workload issuing many presigned-URL requests against Azure Blob drove ~100k GetUserDelegationKey POSTs in 2h (~840/min), ~35% of which returned 503. The retry client backed off 10× (~10s) and then surfaced the error to the caller.

What changes are included in this PR?

Reuse the existing TokenCache<T> to cache the user delegation key on AzureClient:

  • A long-lived key (12h) is fetched once and reused to sign many short-lived SAS tokens.
  • TokenCache only returns a key with more than its min_ttl (2h here) remaining, so any SAS no longer than that is guaranteed to expire before its key (a SAS must not outlive the key it is signed with). The rare longer-lived SAS fetch a dedicated key.
  • The cache entry expires when the key Azure actually granted does (SignedExpiry), not when we requested.
  • Inherits TokenCache’s refresh-race double-check and fetch backoff.

TokenCache::with_min_ttl is extended to the azure-base feature so the Azure client can configure the cache (previously gated to aws-base/gcp-base).

The signed SAS tokens themselves are unchanged (still caller-specified expires_in).

Are there any user-facing changes?

No public API changes. Presigned-URL generation against Azure now performs far fewer GetUserDelegationKey requests.

Tests

Added azure::client::tests::test_delegation_key_expiry for the granted-vs-requested expiry parsing. The caching/refresh machinery itself is covered by the existing client::token tests.

Related issues / PRs

No existing issue tracks this specifically. Related:

@emilk emilk force-pushed the emilk/cache-azure-user-delegation-key branch from 3aa759a to 2690d70 Compare June 25, 2026 02:24
@emilk emilk marked this pull request as ready for review June 25, 2026 02:26
Comment thread src/azure/client.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants