Skip to content

Commit 3397ef2

Browse files
artbergerartberger
authored
Merge pull request #750 from agentgateway/publish-semantic-routing-blog
blog: add semantic routing cost guide
2 parents dc0d334 + 8bf0554 commit 3397ef2

3 files changed

Lines changed: 315 additions & 0 deletions

File tree

‎assets/img/blog/cost-based-semantic-routing/semantic-routing-flow.svg‎

Lines changed: 75 additions & 0 deletions
Loading

‎content/blog/2026-07-17-semantic-routing-llm-costs.md‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
---
2+
title: "Reduce LLM Spend with Semantic Routing"
3+
category: "Deep Dive"
4+
toc: true
5+
publishDate: 2026-07-20
6+
author: "Daneyon Hansen"
7+
description: "Use agentgateway, vLLM Semantic Router, and catalog-priced telemetry to reduce routine coding-model spend without routing every request to the cheapest model."
8+
---
9+
10+
I spend a lot of time in Go and Rust, where a quick "one more thing" in a long
11+
coding chat can turn a tidy refactor into an hour of debugging. After the hard
12+
part is over, routine tests and follow-up questions can still burn
13+
premium-model credits. Sending every request to a cheaper model is not the
14+
answer: distributed-systems design, correctness work, and difficult debugging
15+
need more capable help.
16+
17+
This example combines [vLLM Semantic Router (vSR)](https://vllm-sr.ai/)
18+
with [agentgateway](https://agentgateway.dev/) to send routine Go and Rust
19+
coding tasks to `gpt-5.4-nano` and escalate complex correctness,
20+
distributed-systems, and deep-debugging work to `gpt-5.5`. The goal is a
21+
reproducible cost reduction without quietly routing everything to the cheapest
22+
tier.
23+
24+
## Make the Decision Observable
25+
26+
{{< reuse-image src="img/blog/cost-based-semantic-routing/semantic-routing-flow.svg" alt="Flow diagram: a coding agent sends model auto to agentgateway, vLLM Semantic Router selects either GPT-5.4 nano for routine coding or GPT-5.5 for complex coding, and agentgateway records catalog-priced telemetry" caption="One model name in, the appropriate model tier out - with the routing decision and realized cost visible at the gateway" >}}
27+
28+
The client sends an OpenAI-compatible request with `model: "auto"`. An
29+
agentgateway policy sends it to vSR, which evaluates semantic, complexity,
30+
keyword, context, and structure signals before selecting a model. Agentgateway
31+
forwards the request and records the outcome. Cost is not a classifier input;
32+
it is measured after the request, keeping the policy explainable.
33+
34+
`auto` opts into that automatic selection. A client can instead request
35+
`gpt-5.4-nano` or `gpt-5.5` directly to force a tier and bypass automatic
36+
routing; the request still passes through agentgateway. Organizations
37+
that require automatic routing can use a [request-body
38+
transformation](https://agentgateway.dev/docs/kubernetes/latest/traffic-management/transformations/validate/)
39+
to rewrite every request to `auto`.
40+
41+
## Price the Outcome at the Gateway
42+
43+
My company needs prices, not just token counts, to make a budget decision.
44+
Agentgateway loads a model cost catalog and exposes realized request cost in
45+
metrics, logs, and traces. The [model cost catalog
46+
guide](https://agentgateway.dev/docs/kubernetes/latest/llm/cost-controls/costs/) explains how
47+
to generate and load the catalog with `agctl`.
48+
49+
For this evaluation, I use the integrated [OpenTelemetry stack](https://agentgateway.dev/docs/kubernetes/main/observability/otel-stack/).
50+
Prometheus, Loki, Tempo, the OpenTelemetry Collector, and Grafana let
51+
engineering and finance inspect the same model-selection event.
52+
53+
## A Small, Reproducible Evaluation
54+
55+
I built the [cost-based semantic routing demo](https://github.com/danehans/agentgateway-demos/tree/main/cost-based-semantic-routing)
56+
around a compact 50-prompt Go and Rust dataset. Half of the prompts are routine
57+
implementation or test work; the other half involve correctness or
58+
distributed-systems problems. I send every prompt through two lanes:
59+
60+
| Lane | Purpose |
61+
|---|---|
62+
| `routed` | vSR selects `gpt-5.4-nano` or `gpt-5.5`. |
63+
| `always_expensive` | Every request uses `gpt-5.5`; the cost baseline. |
64+
65+
Each run has an ID on every request, so its cost query excludes unrelated
66+
cluster traffic. I deliberately leave out a forced-cheap lane: the useful
67+
comparison is whether routing saves money while still escalating complex work.
68+
69+
Each prompt has an expected tier in the checked-in dataset: 25 routine prompts
70+
expect `gpt-5.4-nano`, and 25 complex prompts expect `gpt-5.5`. Expected-tier
71+
agreement is the share of requests where vSR selected that labelled tier. It
72+
checks the routing policy, not the quality of the model's answer.
73+
74+
In a clean run on July 16, 2026, all 100 primary requests completed with exact
75+
model-catalog lookups. vSR selected `gpt-5.4-nano` for 30 requests and
76+
`gpt-5.5` for 20. Catalog-priced agentgateway metrics reported `$0.591444` for
77+
the routed lane versus `$0.914840` for the always-expensive lane, a **35.4%
78+
cost reduction**. Routed p50 latency was **5.22 seconds**, compared with
79+
**7.55 seconds** for the always-expensive lane. P95 was 19.09 seconds for
80+
routed traffic and 19.61 seconds for the always-expensive lane.
81+
82+
![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260716T202619Z-chart.svg)
83+
84+
The chart reached **90% expected-tier agreement (45 of 50 prompts)**. All 25
85+
routine prompts stayed on nano, and 20 of 25 complex prompts escalated to
86+
GPT-5.5. This is not a claim of perfect model selection, but it shows the
87+
savings did not come from a blanket downgrade: 40% of routed requests still
88+
used the expensive model. It is a policy sanity check, not an answer-quality
89+
benchmark or a substitute for application task-success and user-feedback data.
90+
91+
The default dataset uses independent requests. For long-running coding agents,
92+
the optional cache-transition evaluation uses two ordered ten-turn Go and Rust
93+
conversations with a stable prefix. It reports provider-observed cache reads by
94+
model transition without mixing cache behavior into the primary evaluation.
95+
96+
## Try It
97+
98+
The demo automates the setup work I would otherwise repeat by hand. It uses the [agentgateway semantic-routing
99+
example](https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing),
100+
creates or reuses a kind cluster, and installs agentgateway, vSR, a model cost
101+
catalog, and the OpenTelemetry stack. It checks readiness, routing,
102+
catalog-priced metrics, logs, and traces before the primary requests run.
103+
104+
Before running it, review the [demo requirements](https://github.com/danehans/agentgateway-demos/tree/main/cost-based-semantic-routing#requirements):
105+
12 GiB of Docker memory, 30 GiB of free disk, the required CLIs, an OpenAI API
106+
key, and access to both models.
107+
108+
```bash
109+
git clone https://github.com/danehans/agentgateway-demos.git
110+
cd agentgateway-demos/cost-based-semantic-routing
111+
112+
export OPENAI_API_KEY='sk-...'
113+
./demo.sh all --yes
114+
```
115+
116+
The default run sends 106 billable requests: two routing probes, four smoke-test
117+
requests, and 100 primary requests. It writes request-level JSONL, an
118+
evaluation-scoped Prometheus report, and an SVG chart under `results/`.
119+
120+
When I want to change the policy, I edit the fetched vSR values, redeploy with
121+
`./demo.sh router`, and run `./demo.sh eval --yes` again.
122+
123+
## Use It From Codex
124+
125+
Codex lets me choose a model manually, but a user-level profile can instead
126+
send every request to a gateway such as agentgateway with the stable `auto`
127+
model name:
128+
129+
```toml
130+
# ~/.codex/agentgateway.config.toml
131+
model = "auto"
132+
model_provider = "agentgateway"
133+
134+
[model_providers.agentgateway]
135+
name = "Corporate agentgateway"
136+
base_url = "https://my.corp.agentgateway.com/v1"
137+
wire_api = "responses"
138+
```
139+
140+
I start Codex with `codex --profile agentgateway`. It sends OpenAI Responses
141+
API traffic to agentgateway; vSR selects the tier and agentgateway records the
142+
decision and realized cost.
143+
144+
Current Codex emits an `unknown model auto` fallback-metadata warning. Routing
145+
still works, but Codex warns that the fallback metadata may degrade behavior.
146+
147+
The profile keeps credentials out of the configuration file. Codex documents
148+
[custom model providers](https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers)
149+
and user-level [configuration profiles](https://learn.chatgpt.com/docs/config-file/config-advanced#profiles).
150+
151+
## From Example to Rollout
152+
153+
Semantic routing is not a magic cost switch. Start with an explainable policy,
154+
compare it with always using the expensive model, confirm it uses both tiers,
155+
and tune it against task completion, feedback, retries, and escalation rates.
156+
Agentgateway supplies the accounting and observability; vSR makes the semantic
157+
decision.
158+
159+
This example is intentionally static: it uses configured keywords, semantic
160+
candidates, weights, and thresholds rather than learning from application
161+
outcomes. Those signals need periodic review as traffic changes. The
162+
agentgateway community and vSR project are continuing work to make semantic
163+
routing easier to tune, operate, and measure against real application traffic,
164+
so stay tuned.
165+
166+
## What's Next
167+
168+
After validating a tiering policy, use [agentgateway cost controls](https://agentgateway.dev/docs/kubernetes/main/llm/cost-controls/)
169+
to attribute usage with virtual keys, observe realized spend by model and
170+
consumer, and enforce budget or spend limits.

‎static/img/blog/cost-based-semantic-routing/20260716T202619Z-chart.svg‎

Lines changed: 70 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)