docs: clarify QuotaPolicy and token-based rate limiting

[0YHR0]

Description

• clarify that both QuotaPolicy and usage-based rate limiting can enforce cumulative token budgets
• document how the two APIs differ in configuration, scope, token accounting, client budgets, and time windows
• add concrete guidance for choosing between the two approaches and explain the current one-day QuotaPolicy limit
• keep the cross-reference in the usage-based rate limiting guide consistent

Why

The existing comparison described usage-based rate limiting as controlling request velocity. However, it can also use response token metadata as its cost and enforce a cumulative token budget. This made it unclear when users should choose QuotaPolicy and whether both APIs share the same time-window restrictions.

The updated documentation distinguishes the APIs by configuration model and scope rather than by what they count.

User impact

Users can now choose the policy that matches their deployment needs, including backend- and model-scoped quota buckets, separate token-type limits, and longer monthly or yearly windows.

Validation

• git diff --check
• npm ci
• npm run build
  • the production site build completed successfully
  • Docusaurus reported pre-existing broken API-page anchor warnings unrelated to this change
• make precommit test was attempted with the official Go module proxy, but dependency downloads timed out during the tidy target in the current network environment

AI assistance

This documentation update was prepared with OpenAI Codex assistance. The statements were checked against the QuotaPolicy API and implementation in this repository and the Envoy Gateway API version used by the project.

Fixes #2340

[missBerg]

Thanks for taking this on — the old note was genuinely misleading (usage-based rate limiting does count tokens, not just "request velocity"), so removing that framing is a correct fix. 🙏

That said, I think the new text overcorrects. Saying the difference is "the API and scope used to configure that budget, not whether tokens or requests are counted" reduces the two features to configuration mechanics, and that's not what the implementation says. The two APIs have different intents:

• QuotaPolicy is a platform concern: it manages the gateway's upstream consumption budget — how many tokens the platform may spend against a provider backend/model, shared across all routes and consumers. It attaches to AIServiceBackend, is enforced after backend selection, and in Shared mode a request is denied only when all applicable buckets are exhausted. The API is also explicitly designed toward quota-aware routing (routing away from backends with exhausted budgets — see the TODO in api/v1alpha1/quota_policy.go).
• Usage-based rate limiting is a consumer concern: it governs how much each client may consume on a route, keyed by client descriptors, and denies as soon as any matched limit is exceeded.

Both count tokens and both return 429, but they answer different questions: "has the platform exhausted its budget for this provider/model?" vs. "has this client exceeded its allowance?"

I'd love to keep your comparison table (it's useful and the Month/Year point is accurate), but lead with the intent distinction and add the enforcement-semantics difference. I'll leave some inline suggestions.

[missBerg]

This is the part I'd most like to see reframed. "The difference is the API and scope used to configure that budget" is true mechanically but hides the intent.

Suggestion to highlight intent more:

:::note QuotaPolicy vs. usage-based rate limiting
`QuotaPolicy` manages the **platform's upstream consumption budget** — how many tokens
the gateway may spend against a provider backend and model, shared across all routes and
consumers. [Usage-based rate limiting](./usage-based-ratelimiting.md) governs **consumer
usage** — how much each client may consume on a route. Both count tokens and both return
`429`, but they answer different questions: "has *the platform* exhausted its budget for
this provider/model?" versus "has *this client* exceeded its allowance?" See
[Choosing a policy](#choosing-a-policy) for details.
:::

[0YHR0]

Updated the note to lead with the platform's upstream budget versus a client's route allowance. It now states that both can count tokens and return 429 while answering those distinct questions.

[missBerg]

"the same rate limit infrastructure" isn't quite accurate. Both build on the same kind of infrastructure (envoyproxy/ratelimit + Redis), but QuotaPolicy is enforced by a dedicated AI Gateway rate limit service — its own domain (ai-gateway-quota), its own HTTP filter injected in the upstream filter chain, and a separate deployment (controller.quotaRateLimitServiceAddr in the helm chart) — while usage-based rate limiting uses Envoy Gateway's global rate limit service. Different deployments, different operational footprint.

Suggest: "Both build on Envoy's global rate limiting (envoyproxy/ratelimit backed by Redis), but QuotaPolicy is enforced by a dedicated AI Gateway rate limit service, separate from Envoy Gateway's."

[0YHR0]

Updated this wording to distinguish the dedicated AI Gateway quota rate limit service from Envoy Gateway's global rate limit service, while retaining the shared Redis-backed rate-limit protocol context.

[missBerg]

Two suggestions for the table:

1. Add an "Enforcement" row — this is the most decision-relevant difference and it's currently missing:

| Enforcement | Denied only when **all** applicable quota buckets are exhausted (`Shared` mode); checked per selected backend, after routing | Denied as soon as **any** matched limit is exceeded; checked at the route, before backend selection |

2. The Scope row currently compares topology on one side ("Models served by one or more AIServiceBackend resources") with config mechanics on the other ("Envoy Gateway rate limit rules and their client selectors"). Suggest making both sides topological:

| Scope | Backend- and model-scoped — the budget follows the `AIServiceBackend` across every route that sends traffic to it | Route/Gateway-scoped — budgets are keyed by client descriptors on a route |

[0YHR0]

Added separate Scope and Enforcement rows. The table now describes backend/model scope versus route/Gateway scope and documents Shared-mode exhaustion semantics alongside matched usage-based limits.

[missBerg]

nit: the new titles say "token-based rate limiting" but the feature's doc page is titled "Usage-Based Rate Limiting" and the body text uses "usage-based rate limiting". Let's stick to one term ("usage-based rate limiting") to avoid readers thinking these are three different features.

[0YHR0]

Standardized the notes to use usage-based rate limiting so the terminology matches the feature page.

[0YHR0]

Thanks for the detailed review. I revised the comparison to lead with the platform-budget versus client-allowance distinction, added the enforcement semantics, clarified the separate quota service, and standardized the usage-based rate limiting terminology. The updates are in 25f1f95.

[missBerg]

@0YHR0 let me take a look at the new version :)