Fix background-thread queue auth and error recovery#174
Open
elprans wants to merge 1 commit into
Open
Conversation
Queue auth on Vercel comes from per-request OIDC headers, but SDK work can run on threads that never had a request context: the queue SDK's lease renewal thread, and Celery's embedded worker loop and ETA timers. Instead of capturing and replaying header-context snapshots at every such call site, remember the freshest live request OIDC token process-wide in `vercel-oidc` and fall back to it, until its `exp` claim passes, whenever the ambient context has no token. Lease renewal primes the fallback on start, while the originating delivery's headers are still visible, so the renewal thread has credentials before its first extension fires. This removes the header-context replay machinery: `LeaseRenewal` no longer captures a context for the renewal worker thread, the Celery broker drops per-delivery context tracking, and the unreleased `context_aware_thread`/`get_headers_context` helpers are dropped from `vercel-headers`. Requeue paths now use the queue client's `retry_after()` instead of raw `extend_lease()`, gaining transient-failure retries and tolerance of an already-missing lease. List `TokenResolutionError`, `UnauthorizedError`, and `CommunicationError` in the Celery transports' `connection_errors` so a momentary token gap, an expired fallback token, or a network failure restarts the consumer instead of killing the worker with an unrecoverable error, which previously wedged the function instance into endlessly retrying push deliveries.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Queue auth on Vercel comes from per-request OIDC headers, but SDK work
can run on threads that never had a request context: the queue SDK's
lease renewal thread, and Celery's embedded worker loop and ETA timers.
Instead of capturing and replaying header-context snapshots at every
such call site, remember the freshest live request OIDC token
process-wide in
vercel-oidcand fall back to it, until itsexpclaimpasses, whenever the ambient context has no token. Lease renewal primes
the fallback on start, while the originating delivery's headers are
still visible, so the renewal thread has credentials before its first
extension fires.
This removes the header-context replay machinery:
LeaseRenewalnolonger captures a context for the renewal worker thread, the Celery
broker drops per-delivery context tracking, and the unreleased
context_aware_thread/get_headers_contexthelpers are dropped fromvercel-headers.Requeue paths now use the queue client's
retry_after()instead of rawextend_lease(), gaining transient-failure retries and tolerance of analready-missing lease.
List
TokenResolutionError,UnauthorizedError, andCommunicationErrorin the Celery transports'connection_errorsso amomentary token gap, an expired fallback token, or a network failure
restarts the consumer instead of killing the worker with an
unrecoverable error, which previously wedged the function instance into
endlessly retrying push deliveries.