You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: assets/agw-docs/pages/agentgateway/llm/providers/multiple-endpoints.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ Configure access to multiple OpenAI API endpoints such as for chat completions,
4
4
5
5
To set up multiple LLM endpoints, use the `ai.routes` field in the `policies` section of the {{< reuse "agw-docs/snippets/backend.md" >}} resource. This field maps the API paths to supported route types. The keys are URL suffix matches, like `/v1/models`. The values are the route types, like `Completions` or `Passthrough`.
6
6
7
-
Alternatively, you can define the routes in a separate {{< reuse "agw-docs/snippets/policy.md" >}} resource attached to an {{< reuse "agw-docs/snippets/agentgateway.md" >}} {{< reuse "agw-docs/snippets/gateway-api.md" >}}. This approach decouples routing configuration from the backend definition, which can simplify management when you have multiple routes pointing to the same backend.
7
+
Alternatively, you can define the routes in a separate {{< reuse "agw-docs/snippets/policy.md" >}} resource attached to an {{< reuse "agw-docs/snippets/agentgateway.md" >}} `HTTPRoute`. This approach decouples routing configuration from the backend definition, which can simplify management when you have multiple routes pointing to the same backend.
8
8
9
9
-`Completions`: Parses the request, translates it to the LLM provider format, and fully processes it as an LLM request. This route type unlocks the full set of {{< reuse "agw-docs/snippets/agentgateway.md" >}} LLM features, such as tokenization and token-based rate limiting, prompt guards, prompt enrichment, model aliasing, transformations, cost tracking, and detailed observability.
10
10
-`Detect`: Forwards the request as-is, but makes a best effort to extract the model and token counts so that a subset of policies still applies, specifically token-based rate limiting and telemetry. Guardrails and other request-shaping policies do not apply. Use this route type for endpoints with a format that cannot be automatically parsed by {{< reuse "agw-docs/snippets/agentgateway.md" >}}, but where you still want metrics and rate limiting.
0 commit comments