Skip to content

Fix background-thread queue auth and error recovery#174

Open
elprans wants to merge 1 commit into
elprans/queue-retriesfrom
elprans/oidc-fixes
Open

Fix background-thread queue auth and error recovery#174
elprans wants to merge 1 commit into
elprans/queue-retriesfrom
elprans/oidc-fixes

Conversation

@elprans

@elprans elprans commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Queue auth on Vercel comes from per-request OIDC headers, but SDK work
can run on threads that never had a request context: the queue SDK's
lease renewal thread, and Celery's embedded worker loop and ETA timers.
Instead of capturing and replaying header-context snapshots at every
such call site, remember the freshest live request OIDC token
process-wide in vercel-oidc and fall back to it, until its exp claim
passes, whenever the ambient context has no token. Lease renewal primes
the fallback on start, while the originating delivery's headers are
still visible, so the renewal thread has credentials before its first
extension fires.

This removes the header-context replay machinery: LeaseRenewal no
longer captures a context for the renewal worker thread, the Celery
broker drops per-delivery context tracking, and the unreleased
context_aware_thread/get_headers_context helpers are dropped from
vercel-headers.

Requeue paths now use the queue client's retry_after() instead of raw
extend_lease(), gaining transient-failure retries and tolerance of an
already-missing lease.

List TokenResolutionError, UnauthorizedError, and
CommunicationError in the Celery transports' connection_errors so a
momentary token gap, an expired fallback token, or a network failure
restarts the consumer instead of killing the worker with an
unrecoverable error, which previously wedged the function instance into
endlessly retrying push deliveries.

Queue auth on Vercel comes from per-request OIDC headers, but SDK work
can run on threads that never had a request context: the queue SDK's
lease renewal thread, and Celery's embedded worker loop and ETA timers.
Instead of capturing and replaying header-context snapshots at every
such call site, remember the freshest live request OIDC token
process-wide in `vercel-oidc` and fall back to it, until its `exp` claim
passes, whenever the ambient context has no token. Lease renewal primes
the fallback on start, while the originating delivery's headers are
still visible, so the renewal thread has credentials before its first
extension fires.

This removes the header-context replay machinery: `LeaseRenewal` no
longer captures a context for the renewal worker thread, the Celery
broker drops per-delivery context tracking, and the unreleased
`context_aware_thread`/`get_headers_context` helpers are dropped from
`vercel-headers`.

Requeue paths now use the queue client's `retry_after()` instead of raw
`extend_lease()`, gaining transient-failure retries and tolerance of an
already-missing lease.

List `TokenResolutionError`, `UnauthorizedError`, and
`CommunicationError` in the Celery transports' `connection_errors` so a
momentary token gap, an expired fallback token, or a network failure
restarts the consumer instead of killing the worker with an
unrecoverable error, which previously wedged the function instance into
endlessly retrying push deliveries.
@vercel

vercel Bot commented Jul 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
vercel-py Ready Ready Preview Jul 11, 2026 5:33am

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant