Skip to content

Commit 6e8e71c

Browse files
committed
Defer multi-tenancy and multi-protocol support.
Signed-off-by: usize <mofoster@redhat.com>
1 parent c519bdb commit 6e8e71c

1 file changed

Lines changed: 45 additions & 3 deletions

File tree

docs/plans/ai-gateway.md

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -309,9 +309,14 @@ so no coordination is needed.
309309
Requires Envoy Gateway's global rate limit service (Redis-backed) for
310310
cross-instance quota enforcement.
311311

312-
Per-user or per-tenant rate limiting (multi-tenant cost allocation) is
313-
a separate concern. If needed, a future policy CRD could attach to the
314-
Gateway to layer per-client quotas on top of per-model limits.
312+
The initial design treats the namespace as a single tenant — rate
313+
limit counters are scoped per model, not per client. Per-user or
314+
per-tenant rate limiting is a separate concern. Multi-tenancy could
315+
be addressed by extracting rate limiting into a separate policy CRD
316+
with precedence rules akin to [BackendTrafficPolicy][] — where
317+
more specific policies override broader defaults.
318+
319+
[BackendTrafficPolicy]: https://gateway.envoyproxy.io/docs/concepts/gateway_api_extensions/backend-traffic-policy/
315320

316321
### AIAccessPolicy
317322

@@ -505,6 +510,43 @@ proposal.
505510

506511
Not in scope for the initial implementation.
507512

513+
### Future: protocol extensibility
514+
515+
The provider `schema` field declares the protocol a backend speaks.
516+
The initial implementation supports inference schemas (OpenAI,
517+
AWSBedrock, etc.), but this maps directly to the [Backend protocol
518+
model][Proposal 10] emerging in WG AI Gateway — where protocol is a
519+
property of the destination, not the route.
520+
521+
The `models` list is inference-specific: it defines virtual model
522+
names, maps them to provider backends, and configures per-model
523+
failover and rate limits. Non-inference protocols like MCP don't have
524+
a model selection concept. Extending AIRoutingPolicy to support MCP
525+
providers would add protocol-appropriate routing entries alongside
526+
`models`, driven by the provider's schema:
527+
528+
```yaml
529+
providers:
530+
- name: ollama
531+
endpoint: http://ollama.svc:11434
532+
schema: OpenAI # inference — uses models[]
533+
- name: tool-server
534+
endpoint: http://tools.svc:8080
535+
schema: MCP # MCP — no model mapping needed
536+
537+
models: # inference-specific
538+
- name: qwen2.5:3b
539+
backends:
540+
- provider: ollama
541+
model: qwen2.5:3b
542+
```
543+
544+
The renderer would generate `AIGatewayRoute` for inference providers
545+
and `MCPRoute` for MCP providers. Credential injection, mTLS, and
546+
rate limiting apply uniformly regardless of protocol. The exact shape
547+
of non-inference routing entries is left to a future proposal once
548+
the WG AI Gateway Backend specification matures.
549+
508550
## Example: complete deployment
509551

510552
A platform admin sets up a Gateway with mTLS. A team lead configures

0 commit comments

Comments
 (0)