docs: proposal for scale-from-zero aware routing#2328
Open
mohitgurnani1 wants to merge 3 commits into
Open
Conversation
This adds a design proposal for scale-from-zero / hibernate-aware routing: deliver requests to backends that are scaled to zero or hibernated by retrying within a configurable budget and optionally firing a deduplicated wake-up endpoint, instead of returning an immediate 503. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mohit Gurnani <mohit.gurnani@nutanix.com>
dosubot
Bot
added
the
size:L
012 is taken by the semantic caching proposal (envoyproxy#2326). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mohit Gurnani <mohit.gurnani@nutanix.com>
nacx
reviewed
Jul 11, 2026
nacx
left a comment
nacx
left a comment
Member
There was a problem hiding this comment.
Super interesting feature.
Thanks for the proposal! I left some comments for discussion
| policy precedence — is that precedence rule acceptable? | ||
| 2. Retrying on 503 cannot distinguish a local `no_healthy_upstream` 503 from a genuine upstream 503. | ||
| For a backend explicitly opted into `coldStart` this seems acceptable (and bounded by `maxWait`); | ||
| should we instead scope 503 retries with response-flag/header-based gating? |
Member
There was a problem hiding this comment.
IMO, opting into coldStart is acceptable.
| should we instead scope 503 retries with response-flag/header-based gating? | ||
| 3. Wake firing: fire on every request arrival for a cold-start backend (deduplicated by `coolDown`), | ||
| or only after a failed first attempt? The former is proposed for simplicity; the latter avoids | ||
| wake calls when the backend is already warm at the cost of detecting failure first. |
Member
There was a problem hiding this comment.
I think we should avoid wake-up calls when the backend is already up.
| or only after a failed first attempt? The former is proposed for simplicity; the latter avoids | ||
| wake calls when the backend is already warm at the cost of detecting failure first. | ||
| 4. Naming: `coldStart` vs `scaleFromZero` vs `wakePolicy`? | ||
| 5. Is a per-route-rule override of `maxWait` needed in v1, or is per-backend sufficient? |
Member
There was a problem hiding this comment.
I think per-backend is the right place to start.
| wake calls when the backend is already warm at the cost of detecting failure first. | ||
| 4. Naming: `coldStart` vs `scaleFromZero` vs `wakePolicy`? | ||
| 5. Is a per-route-rule override of `maxWait` needed in v1, or is per-backend sufficient? | ||
| 6. Standalone `aigw`: inject the retry policy identically (proposed), or gate the feature? |
Member
There was a problem hiding this comment.
aigw should behave the same as k8s based to the extent that it is possible.
| ## Open Questions for Community Discussion | ||
|
|
||
| 1. Should the extension server inject the retry policy, or should the feature only document a | ||
| required user-attached `BackendTrafficPolicy` (status quo pattern)? Proposed: inject, with user |
Member
There was a problem hiding this comment.
This is the main discussion point, I think.
If we take other features, such as provider fallback, we ask users to create the BTP that configures the retries themselves. To keep API consistency across features, I'd say we could start by not injecting the BTP and asking users to configure it.
missBerg
added
the
area/routing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a design proposal for scale-from-zero / hibernate-aware routing: when an AIServiceBackend's upstream is scaled to zero replicas or hibernated, the gateway keeps the request in flight with a derived Envoy retry policy bounded by a configurable maxWait, optionally fires a deduplicated wake-up HTTP endpoint, and returns an OpenAI-shaped error with Retry-After only after the budget is exhausted, instead of surfacing an immediate connect failure or local 503.
The design deliberately composes existing primitives (Envoy route retry policies, the retry-safe upstream ext_proc filter, endpoint-priority failover) and introduces no request queue in the gateway. InferencePool backends are out of scope since the Gateway API Inference Extension Endpoint Picker Flow Control layer already queues requests during scale-from-zero; this targets the AIServiceBackend path which has no equivalent today.
Feasibility was validated with a local data-plane PoC (real Envoy + extproc + fake upstream, both ext_proc filters in path): a request issued while nothing was listening on the backend port was delivered with 200 (4.0s end to end) when a listener appeared 3 seconds later, and a never-waking backend produced a 503 only after the 9.4s retry budget, with the raw Envoy local reply body that the proposed terminal-error shaping replaces.
The proposal includes the API sketch (a new optional coldStart field on AIServiceBackendSpec), architecture, key design decisions, a phased implementation plan, out-of-scope items, and open questions for discussion.
Disclosure: this proposal was drafted with the assistance of generative AI (Claude); I have reviewed and own the content.
Related Issues/PRs (if applicable)
Related Issue: #2327
Special notes for reviewers (if applicable)
This is a docs-only PR (a single new file under docs/proposals/). Feedback is sought primarily on the open questions section: retry-policy injection vs documented BackendTrafficPolicy, the 503 retry trigger scope, wake firing heuristics, and naming.