You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This guide walks administrators through the complete setup process for AI Gateway, from enabling LLM providers to configuring routing policies and MCP tool aggregation.
11
+
This guide walks administrators through the setup process for AI Gateway, from enabling LLM providers to configuring routing policies and MCP tool aggregation.
12
12
13
13
After completing this guide, you will be able to:
14
14
@@ -26,7 +26,7 @@ After completing this guide, you will be able to:
26
26
27
27
Providers represent upstream services (Anthropic, OpenAI, Google AI) and associated credentials. Providers are disabled by default and must be enabled explicitly by an administrator.
28
28
29
-
. In the Redpanda Cloud Console, navigate to *AI Gateway* → *Providers*.
29
+
. In the Redpanda Cloud Console, navigate to *Agentic AI* → *Providers*.
30
30
. Select a provider (for example, Anthropic).
31
31
. On the Configuration tab for the provider, click *Add configuration*.
32
32
. Enter your API Key for the provider.
@@ -43,17 +43,15 @@ The model catalog is the set of models made available through the gateway. Model
43
43
44
44
The infrastructure that serves the model differs based on the provider you select. For example, OpenAI has different reliability and availability metrics than Anthropic. When you consider all metrics, you can design your gateway to use different providers for different use cases.
45
45
46
-
. Navigate to *AI Gateway* → *Models*.
46
+
. Navigate to *Agentic AI* → *Models*.
47
47
. Review the list of available models from enabled providers.
48
-
. For each model you want to expose through gateways, toggle it to *Enabled*.
49
-
+
50
-
Common models to enable:
48
+
. For each model you want to expose through gateways, toggle it to *Enabled*. For example:
51
49
+
52
50
--
53
-
* `openai/gpt-4o` - OpenAI's most capable model
54
-
* `openai/gpt-4o-mini` - Cost-effective OpenAI model
55
-
* `anthropic/claude-sonnet-3.5` - Balanced Anthropic model
56
-
* `anthropic/claude-opus-4` - Anthropic's most capable model
51
+
* `openai/gpt-5.2`
52
+
* `openai/gpt-5.2-mini`
53
+
* `anthropic/claude-sonnet-4.5`
54
+
* `anthropic/claude-opus-4.6`
57
55
--
58
56
59
57
. Click *Save changes*.
@@ -62,14 +60,13 @@ Only enabled models will be accessible through gateways. You can enable or disab
62
60
63
61
=== Model naming convention
64
62
65
-
Model requests must use the `vendor/model_id` format in the model property of the request body. This format allows AI Gateway to route requests to the appropriate provider.
66
-
67
-
Examples:
63
+
Model requests must use the `vendor/model_id` format in the model property of the request body. This format allows AI Gateway to route requests to the appropriate provider. For example:
68
64
69
-
* `openai/gpt-4o`
70
-
* `anthropic/claude-sonnet-3.5`
71
-
* `openai/gpt-4o-mini`
65
+
* `openai/gpt-5.2`
66
+
* `anthropic/claude-sonnet-4.5`
67
+
* `openai/gpt-5.2-mini`
72
68
69
+
ifdef::ai-hub-available[]
73
70
== Choose a gateway mode
74
71
75
72
Before creating a gateway, decide which mode fits your needs.
@@ -102,12 +99,13 @@ For detailed comparison, see xref:ai-gateway/gateway-modes.adoc[].
102
99
103
100
* *AI Hub Mode*: See xref:ai-gateway/admin/configure-ai-hub.adoc[] for setup instructions
104
101
* *Custom Mode*: Continue with "Create a gateway" below for manual configuration
102
+
endif::[]
105
103
106
104
== Create a gateway
107
105
108
106
A gateway is a logical configuration boundary (policies + routing + observability) on top of a single deployment. It's a "virtual gateway" that you can create per team, environment (staging/production), product, or customer.
109
107
110
-
. Navigate to *AI Gateway* → *Gateways*.
108
+
. Navigate to *Agentic AI* → *Gateways*.
111
109
. Click *Create Gateway*.
112
110
. Configure the gateway:
113
111
+
@@ -126,11 +124,12 @@ TIP: A workspace is conceptually similar to a resource group in Redpanda streami
126
124
. After creation, note the following information:
127
125
+
128
126
--
129
-
* *Gateway ID*: Unique identifier (for example, `gw_abc123`) - users include this in the `rp-aigw-id` header
130
-
* *Gateway Endpoint*: Base URL for API requests (for example, `https://gw.ai.panda.com`)
127
+
* *Gateway endpoint*: URL for API requests (for example, `https://example/gateways/d633lffcc16s73ct95mg/v1`)
128
+
+
129
+
The gateway ID is embedded in the URL.
131
130
--
132
131
133
-
You'll share the Gateway ID and Endpoint with users who need to access this gateway.
132
+
You'll share the gateway endpoint with users who need to access this gateway.
134
133
135
134
== Configure LLM routing
136
135
@@ -188,7 +187,7 @@ Provider pools define which LLM providers handle requests, with support for prim
188
187
--
189
188
* *Name*: For example, `primary-anthropic`
190
189
* *Providers*: Select one or more providers (for example, Anthropic)
191
-
* *Models*: Choose which models to include (for example, `anthropic/claude-sonnet-3.5`)
190
+
* *Models*: Choose which models to include (for example, `anthropic/claude-sonnet-4.5`)
192
191
* *Load balancing*: If multiple providers are selected, choose distribution strategy (round-robin, weighted, etc.)
193
192
--
194
193
@@ -197,7 +196,7 @@ Provider pools define which LLM providers handle requests, with support for prim
197
196
--
198
197
* *Name*: For example, `fallback-openai`
199
198
* *Providers*: Select fallback provider (for example, OpenAI)
200
-
* *Models*: Choose fallback models (for example, `openai/gpt-4o`)
199
+
* *Models*: Choose fallback models (for example, `openai/gpt-5.2`)
201
200
* *Trigger conditions*: When to activate fallback:
202
201
** Rate limit exceeded (429 from primary)
203
202
** Timeout (primary provider slow)
@@ -215,8 +214,8 @@ Example CEL expression for tier-based routing:
215
214
[source,cel]
216
215
----
217
216
request.headers["x-user-tier"] == "premium"
218
-
? "anthropic/claude-opus-4"
219
-
: "anthropic/claude-sonnet-3.5"
217
+
? "anthropic/claude-opus-4.6"
218
+
: "anthropic/claude-sonnet-4.5"
220
219
----
221
220
222
221
. Click *Save routing configuration*.
@@ -227,49 +226,78 @@ TIP: Provider pool (UI) = Backend pool (API)
227
226
228
227
If a provider pool contains multiple providers, you can distribute traffic to balance load or optimize for cost/performance:
229
228
230
-
* *Round-robin*: Distribute evenly across all providers
231
-
* *Weighted*: Assign weights (for example, 80% to Anthropic, 20% to OpenAI)
232
-
* *Least latency*: Route to fastest provider based on recent performance
233
-
* *Cost-optimized*: Route to cheapest provider for each model
229
+
* Round-robin: Distribute evenly across all providers
230
+
* Weighted: Assign weights (for example, 80% to Anthropic, 20% to OpenAI)
231
+
* Least latency: Route to fastest provider based on recent performance
232
+
* Cost-optimized: Route to cheapest provider for each model
234
233
235
234
== Configure MCP tools (optional)
236
235
237
236
If your users will build glossterm:AI agent[,AI agents] that need access to glossterm:MCP tool[,tools] via glossterm:MCP[,Model Context Protocol (MCP)], configure MCP tool aggregation.
238
237
239
238
On the gateway details page, select the *MCP* tab to configure tool discovery and execution. The MCP proxy aggregates multiple glossterm:MCP server[,MCP servers], allowing agents to find and call tools through a single endpoint.
240
239
240
+
=== Configure MCP rate limits
241
+
242
+
Rate limits for MCP work the same way as LLM rate limits.
243
+
244
+
. In the *MCP* tab, locate the *Rate Limit* section.
245
+
. Click *Add rate limit*.
246
+
. Configure the maximum requests per second and optional burst allowance.
247
+
. Click *Save*.
248
+
241
249
=== Add MCP servers
242
250
243
-
. In the *MCP* tab, click *Add MCP server*.
251
+
. In the *MCP* tab, click *Create MCP Server*.
244
252
. Configure the server:
245
253
+
246
254
--
247
-
* *Server name*: Human-readable identifier (for example, `database-server`, `slack-server`)
248
-
* *Server URL*: Endpoint for the MCP server (for example, `https://mcp-database.example.com`)
249
-
* *Authentication*: Configure authentication if required (bearer token, API key, mTLS)
250
-
* *Enabled tools*: Select which tools from this server to expose (or *All tools*)
255
+
* *Server ID*: Unique identifier for this server
256
+
* *Display Name*: Human-readable name (for example, `database-server`, `slack-server`)
257
+
* *Server Address*: Endpoint URL for the MCP server (for example, `https://mcp-database.example.com`)
251
258
--
252
259
253
-
. Click *Test connection* to verify connectivity.
254
-
. Click *Save* to add the server to this gateway.
260
+
. Configure server settings:
261
+
+
262
+
--
263
+
* *Timeout (seconds)*: Maximum time to wait for a response from this server
264
+
* *Enabled*: Whether this server is active and accepting requests
265
+
* *Defer Loading Override*: Controls whether tools from this server are loaded upfront or on demand
266
+
+
267
+
[cols="1,2"]
268
+
|===
269
+
|Option |Description
255
270
256
-
Repeat for each MCP server you want to aggregate.
271
+
|Inherit from gateway
272
+
|Use the gateway-level deferred loading setting (default)
257
273
258
-
=== Configure deferred tool loading
274
+
|Enabled
275
+
|Always defer loading from this server. Agents receive only a search tool initially and query for specific tools when needed. This can reduce token usage by 80-90%.
259
276
260
-
Deferred tool loading dramatically reduces token costs by initially exposing only a search tool and orchestrator, rather than listing all available tools.
277
+
|Disabled
278
+
|Always load all tools from this server upfront.
279
+
|===
261
280
262
-
. In the *MCP* tab, locate *Deferred Loading*.
263
-
. Toggle *Enable deferred tool loading* to *On*.
264
-
. Configure behavior:
281
+
* *Forward OIDC Token Override*: Controls whether the client's OIDC token is forwarded to this MCP server
265
282
+
266
-
--
267
-
* *Initially expose*: Search tool + orchestrator only
268
-
* *Load on demand*: Tools are retrieved when agents query for them
269
-
* *Token savings*: Expect 80-90% reduction in token usage for tool definitions
283
+
[cols="1,2"]
284
+
|===
285
+
|Option |Description
286
+
287
+
|Inherit from gateway
288
+
|Use the gateway-level OIDC forwarding setting (default)
289
+
290
+
|Enabled
291
+
|Always forward the OIDC token to this server
292
+
293
+
|Disabled
294
+
|Never forward the OIDC token to this server
295
+
|===
270
296
--
271
297
272
-
. Click *Save*.
298
+
. Click *Save* to add the server to this gateway.
299
+
300
+
Repeat for each MCP server you want to aggregate.
273
301
274
302
See xref:ai-gateway/mcp-aggregation-guide.adoc[] for detailed information about MCP aggregation.
275
303
@@ -279,11 +307,24 @@ The MCP orchestrator is a built-in MCP server that enables programmatic tool cal
279
307
280
308
Example: A workflow requiring 47 file reads can be reduced from 49 round trips to just 1 round trip using the orchestrator.
281
309
282
-
The orchestrator is enabled by default when you enable MCP tools. You can configure:
310
+
The orchestrator is pre-configured when you initialize the MCP gateway. Its server configuration (Server ID, Display Name, Transport, Command, and Timeout) is system-managed and cannot be modified.
283
311
284
-
* *Execution timeout*: Maximum time for orchestrator workflows (for example, 30 seconds)
285
-
* *Memory limit*: Maximum memory for JavaScript execution (for example, 128MB)
286
-
* *Allowed operations*: Restrict which MCP tools can be called from orchestrator workflows
312
+
You can configure blocked tool patterns to prevent specific tools from being called through the orchestrator:
313
+
314
+
. In the *MCP* tab, select the orchestrator server to edit it.
315
+
. Under *Blocked Tools*, click *Add Pattern* to add glob patterns for tools that should be blocked from execution.
316
+
+
317
+
Example patterns:
318
+
+
319
+
--
320
+
* `server_id:*` - Block all tools from a specific server
321
+
* `*:dangerous_tool` - Block a specific tool across all servers
322
+
* `specific:tool` - Block a single tool on a specific server
323
+
--
324
+
+
325
+
NOTE: The orchestrator's own tools are blocked by default to prevent recursive execution.
326
+
327
+
. Click *Save*.
287
328
288
329
== Verify your setup
289
330
@@ -293,9 +334,8 @@ After completing the setup, verify that the gateway is working correctly:
0 commit comments