From f4aa0e1eb56a77841b96e4bd2da452270e7d85bb Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Mon, 13 Jul 2026 15:42:13 -0700 Subject: [PATCH 01/15] blog: reduce LLM spend with semantic routing Signed-off-by: Daneyon Hansen --- .../semantic-routing-flow.svg | 75 +++++++++ .../2026-07-10-semantic-routing-llm-costs.md | 150 ++++++++++++++++++ .../20260713T184930Z-chart.svg | 56 +++++++ 3 files changed, 281 insertions(+) create mode 100644 assets/img/blog/cost-based-semantic-routing/semantic-routing-flow.svg create mode 100644 content/blog/2026-07-10-semantic-routing-llm-costs.md create mode 100644 static/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg diff --git a/assets/img/blog/cost-based-semantic-routing/semantic-routing-flow.svg b/assets/img/blog/cost-based-semantic-routing/semantic-routing-flow.svg new file mode 100644 index 000000000..bfc2381d7 --- /dev/null +++ b/assets/img/blog/cost-based-semantic-routing/semantic-routing-flow.svg @@ -0,0 +1,75 @@ + + Cost-based semantic routing with agentgateway and vLLM Semantic Router + A coding agent sends model auto to agentgateway. vLLM Semantic Router selects either GPT-5.4 nano for routine coding or GPT-5.5 for complex coding. Agentgateway records catalog-priced telemetry. + + + + + + + + + + + + + + + + + + + + + + + + + + + Coding agent + model: auto + + + + + + agentgateway + routes the selected model + records the routing outcome + + OpenAI-compatible + + + + + + vSR semantic policy + selects the model tier + + + + + + + Routine coding + gpt-5.4-nano + lower-cost model + + + + + + Complex coding + gpt-5.5 + higher-capability model + + + + + + + Catalog-priced telemetry + model selection, tokens, cost, latency + Prometheus, logs, traces, Grafana + diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-10-semantic-routing-llm-costs.md new file mode 100644 index 000000000..7f5e86280 --- /dev/null +++ b/content/blog/2026-07-10-semantic-routing-llm-costs.md @@ -0,0 +1,150 @@ +--- +title: "Reduce LLM Spend With Semantic Routing You Can Measure" +category: "Deep Dive" +toc: false +publishDate: 2026-07-10 +author: "Daneyon Hansen" +description: "Use agentgateway, vLLM Semantic Router, and catalog-priced telemetry to reduce routine coding-model spend without routing every request to the cheapest model." +--- + +I spend a lot of time in Go and Rust, where a quick "one more thing" in a +long coding chat can turn a tidy refactor into an hour of debugging. I do not +always remember to step down from the strongest model after the hard part is +over. A few routine tests, configuration edits, and follow-up questions later, +I have spent premium-model credits on work that did not need them. + +Sending every request to a cheaper model is not the answer either. The +distributed-systems design, correctness work, and difficult debugging that +show up in those same conversations are exactly where I want more capable help. +I want the client to stay simple while the routing policy makes that choice. + +This example combines [vLLM Semantic Router (vSR)](https://vllm-semantic-router.com/) +with [agentgateway](https://agentgateway.dev/) to send routine Go and Rust +coding tasks to `gpt-5.4-nano` and escalate more complex correctness, +distributed-systems, and deep-debugging work to `gpt-5.5`. I am not trying to +claim perfect routing accuracy. I want a reproducible way to show lower spend +without quietly routing every request to the cheapest tier. + +## Make the Decision Observable + +{{< reuse-image src="img/blog/cost-based-semantic-routing/semantic-routing-flow.svg" alt="Flow diagram: a coding agent sends model auto to agentgateway, vLLM Semantic Router selects either GPT-5.4 nano for routine coding or GPT-5.5 for complex coding, and agentgateway records catalog-priced telemetry" caption="One model name in, the appropriate model tier out - with the routing decision and realized cost visible at the gateway" >}} + +I keep the client side boring: it sends an OpenAI-compatible request with +`model: "auto"`. An agentgateway policy sends the full request body to vSR. +vSR evaluates semantic, complexity, keyword, context, and structure signals, +then returns the selected model. Agentgateway forwards the request and records +what happened. + +I do not ask vSR to use historic request cost as a classifier input. Cost is an +outcome that agentgateway measures after the request. That keeps the policy +explainable: the routing configuration says which work merits the stronger +model, while the gateway shows what that decision actually cost. + +## Price the Outcome at the Gateway + +Token counts are useful, but my company needs prices to make a budget decision. +Agentgateway loads a model cost catalog, calculates realized request cost, and +exposes the data in metrics, logs, and traces. The [model cost catalog +guide](https://agentgateway.dev/docs/kubernetes/latest/llm/cost-controls/costs/) explains how +to generate and load the catalog with `agctl`. + +For this evaluation, I use the integrated [OpenTelemetry stack](https://agentgateway.dev/docs/kubernetes/main/observability/otel-stack/). +It adds Prometheus, Loki, Tempo, the OpenTelemetry Collector, and Grafana so +engineering and finance can inspect the same model-selection event: + +- Prometheus carries token, duration, catalog-lookup, and cost signals. +- Logs and traces retain the selected model, request ID, latency, and cost. +- Grafana puts spend next to the signals needed to explain a routing decision. + +## A Small, Reproducible Evaluation + +I built the [cost-based semantic routing demo](https://github.com/danehans/agentgateway-demos/tree/main/cost-based-semantic-routing) +around a compact 24-prompt Go and Rust dataset. Half of the prompts are routine +implementation or test work; the other half involve correctness or +distributed-systems problems. I send every prompt through two lanes: + +| Lane | Purpose | +|---|---| +| `routed` | vSR selects `gpt-5.4-nano` or `gpt-5.5`. | +| `always_expensive` | Every request uses `gpt-5.5`; the cost baseline. | + +Each evaluation run has an ID attached to every request. The runner uses it in +its cost queries, which keeps unrelated traffic in the cluster out of the +result. + +I deliberately leave out a forced-cheap lane. It would make the comparison +larger, but it would not answer the question I care about: does routing save +money while still escalating the work that deserves the expensive model? + +In a clean run on July 13, 2026, all 48 primary requests completed with exact +model-catalog lookups. vSR selected `gpt-5.4-nano` for 16 requests and +`gpt-5.5` for 8. Catalog-priced agentgateway metrics reported `$0.231896` for +the routed lane versus `$0.405800` for the always-expensive lane, a **42.9% +cost reduction**. Routed p50 latency was **3.96 seconds**, compared with +**11.02 seconds** for the always-expensive lane. P95 was similar at 19.48 and +20.12 seconds because the requests that still use GPT-5.5 naturally determine +the tail. + +![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg) + +The chart gives me the guardrail I need. It reached 83.3% tier agreement on the +checked-in coding sample and sent 8 of the 12 prompts labelled complex to +GPT-5.5. This is not a claim of perfect model selection. I read it as +evidence that the savings did not come from a blanket downgrade: one-third of +the routed requests still used the expensive model, including most of the +complex sample. + +That is the useful scope for this example. It keeps higher-risk work on a more +capable model by policy and makes the exceptions visible. It is not a general +answer-quality benchmark or a substitute for an application's own task-success +and user-feedback measures. + +## Try It + +I made the demo do the setup work that I would otherwise forget or repeat by +hand. Behind the scenes, it uses the merged [agentgateway semantic-routing +example](https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing), +creates or reuses a kind cluster, and installs agentgateway, vSR, a model cost +catalog, and the OpenTelemetry stack. Before the primary requests run, it +checks readiness, routing, catalog-priced metrics, logs, and traces. + +```bash +git clone https://github.com/danehans/agentgateway-demos.git +cd agentgateway-demos/cost-based-semantic-routing + +export OPENAI_API_KEY='sk-...' +./demo.sh all --yes +``` + +The default run sends 54 billable requests: two routing probes, four smoke-test +requests, and 48 primary requests. It writes request-level JSONL, an +evaluation-scoped Prometheus report, and an SVG chart under `results/`. + +When I want to change the policy, I edit the fetched vSR values, redeploy with +`./demo.sh router`, and run `./demo.sh eval --yes` again. + +## From Example to Rollout + +I would not treat semantic routing as a magic cost switch. I would start with +an explainable policy and compare it against sending every request to the +expensive model. Then I would confirm that the policy uses both tiers and +protects complex work, and tune it against the signals that matter to the +application: task completion, feedback, retries, and escalation rates. + +Agentgateway gives me the accounting and observability plane; vSR makes the +semantic decision. Together, they let me cut routine-model spend without +hiding a blanket downgrade behind an attractive cost chart. + +## What's Next + +I see cost-based semantic routing as one part of an LLM cost-management +practice. After validating a tiering policy, I can use +[agentgateway cost controls](https://agentgateway.dev/docs/kubernetes/main/llm/cost-controls/) +to attribute usage with virtual keys, observe realized spend by model and +consumer, and enforce budget or spend limits where I need them. That turns a +routing result into an operating model for keeping AI spend visible and under +control. + +The agentgateway community is continuing to improve cost-based routing and the +tools around it. Future work will make it easier to tune, operate, and measure +model-tier decisions as real application traffic evolves. diff --git a/static/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg b/static/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg new file mode 100644 index 000000000..6c57eb505 --- /dev/null +++ b/static/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg @@ -0,0 +1,56 @@ + + Semantic routing experiment results + Semantic routing spend, routing agreement, model mix, and latency for run 20260713T184930Z. + + + Semantic routing experiment + Run 20260713T184930Z | Catalog-priced agentgateway metrics + + + SPEND REDUCTION + 42.9% + Routed versus always expensive + + ROUTING AGREEMENT + 83.3% + Expected tiers in this coding sample + + ROUTED P50 LATENCY + 3.96 s + -64.1% versus always expensive + + COST PER DEMO RUN (USD) + Semantic routing$0.2319Always expensive$0.4058 + + + ROUTED MODEL MIX + + + 67% lower-cost + 16 gpt-5.4-nano | 8 gpt-5.5 + + + COMPLEX-PROMPT ESCALATION + + + 67% to GPT-5.5 + 8/12 complex prompts sent to GPT-5.5 + + + END-TO-END LATENCY + 3.96 s p50 + 19.48 s p95 semantic routing + Always expensive: 11.02 s p50, 20.12 s p95 + + Agreement uses expected tiers in the checked-in coding sample; it is a routing sanity check, not answer scoring. + From e1ce9855a4c68b0189b432d26f0c193d57109669 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Tue, 14 Jul 2026 10:06:07 -0700 Subject: [PATCH 02/15] blog: document Codex semantic routing Signed-off-by: Daneyon Hansen --- .../2026-07-10-semantic-routing-llm-costs.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-10-semantic-routing-llm-costs.md index 7f5e86280..0b2336296 100644 --- a/content/blog/2026-07-10-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-10-semantic-routing-llm-costs.md @@ -123,6 +123,37 @@ evaluation-scoped Prometheus report, and an SVG chart under `results/`. When I want to change the policy, I edit the fetched vSR values, redeploy with `./demo.sh router`, and run `./demo.sh eval --yes` again. +## Use It From Codex + +Codex lets me choose a model manually, but that is one more decision to keep in +my head during a long Go or Rust task. A user-level Codex profile can instead +send every request to the corporate gateway with the stable `auto` model name: + +```toml +# ~/.codex/agentgateway.config.toml +model = "auto" +model_provider = "agentgateway" + +[model_providers.agentgateway] +name = "Corporate agentgateway" +base_url = "https://my.corp.agentgateway.com/v1" +wire_api = "responses" +env_key = "OPENAI_API_KEY" +``` + +With `OPENAI_API_KEY` set to my OpenAI API key, I start Codex with +`codex --profile agentgateway`. The client sends OpenAI Responses API traffic +to agentgateway; vSR selects the actual tier and agentgateway records the +decision and realized cost. I validated this path with a routine table-driven +Go test request, which selected `gpt-5.4-nano`, and an advanced leader-election +request, which selected `gpt-5.5`. + +The profile keeps credentials out of the configuration file. A company that +needs to prevent direct model selection can enforce that separately at the +gateway; `auto` is the supported path for this routing policy. Codex documents +[custom model providers](https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers) +and user-level [configuration profiles](https://learn.chatgpt.com/docs/config-file/config-advanced#profiles). + ## From Example to Rollout I would not treat semantic routing as a magic cost switch. I would start with From 7755e3daf01f1f52e1cffbff3151eb6343b123ca Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Tue, 14 Jul 2026 10:25:34 -0700 Subject: [PATCH 03/15] blog: clarify routing evaluation Signed-off-by: Daneyon Hansen --- content/blog/2026-07-10-semantic-routing-llm-costs.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-10-semantic-routing-llm-costs.md index 0b2336296..15a0d42af 100644 --- a/content/blog/2026-07-10-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-10-semantic-routing-llm-costs.md @@ -35,8 +35,8 @@ vSR evaluates semantic, complexity, keyword, context, and structure signals, then returns the selected model. Agentgateway forwards the request and records what happened. -I do not ask vSR to use historic request cost as a classifier input. Cost is an -outcome that agentgateway measures after the request. That keeps the policy +vSR does not use historic request cost as a classifier input. Instead, cost is +an outcome that agentgateway measures after the request. That keeps the policy explainable: the routing configuration says which work merits the stronger model, while the gateway shows what that decision actually cost. @@ -89,10 +89,9 @@ the tail. The chart gives me the guardrail I need. It reached 83.3% tier agreement on the checked-in coding sample and sent 8 of the 12 prompts labelled complex to -GPT-5.5. This is not a claim of perfect model selection. I read it as -evidence that the savings did not come from a blanket downgrade: one-third of -the routed requests still used the expensive model, including most of the -complex sample. +GPT-5.5. This does not claim perfect model selection. It does show that the +savings did not come from a blanket downgrade: one-third of routed requests +still used the expensive model, including most complex prompts in the sample. That is the useful scope for this example. It keeps higher-risk work on a more capable model by policy and makes the exceptions visible. It is not a general From c527661e7cece994cbbcf94263b62336c3f919da Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Tue, 14 Jul 2026 10:26:23 -0700 Subject: [PATCH 04/15] blog: clarify demo dependency Signed-off-by: Daneyon Hansen --- content/blog/2026-07-10-semantic-routing-llm-costs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-10-semantic-routing-llm-costs.md index 15a0d42af..4bdf626d3 100644 --- a/content/blog/2026-07-10-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-10-semantic-routing-llm-costs.md @@ -101,7 +101,7 @@ and user-feedback measures. ## Try It I made the demo do the setup work that I would otherwise forget or repeat by -hand. Behind the scenes, it uses the merged [agentgateway semantic-routing +hand. Behind the scenes, it uses the [agentgateway semantic-routing example](https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing), creates or reuses a kind cluster, and installs agentgateway, vSR, a model cost catalog, and the OpenTelemetry stack. Before the primary requests run, it From 8739e95bb3fec93172bc22e0a1a3dbf0515461e9 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Tue, 14 Jul 2026 10:27:44 -0700 Subject: [PATCH 05/15] blog: generalize Codex workflow Signed-off-by: Daneyon Hansen --- content/blog/2026-07-10-semantic-routing-llm-costs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-10-semantic-routing-llm-costs.md index 4bdf626d3..6caa0f79c 100644 --- a/content/blog/2026-07-10-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-10-semantic-routing-llm-costs.md @@ -125,7 +125,7 @@ When I want to change the policy, I edit the fetched vSR values, redeploy with ## Use It From Codex Codex lets me choose a model manually, but that is one more decision to keep in -my head during a long Go or Rust task. A user-level Codex profile can instead +my head during a long development task. A user-level Codex profile can instead send every request to the corporate gateway with the stable `auto` model name: ```toml From e9bdd7921d3822a6819a75614181d61f0ebeb9d8 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Tue, 14 Jul 2026 10:28:41 -0700 Subject: [PATCH 06/15] blog: clarify Codex gateway setup Signed-off-by: Daneyon Hansen --- content/blog/2026-07-10-semantic-routing-llm-costs.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-10-semantic-routing-llm-costs.md index 6caa0f79c..0c1761328 100644 --- a/content/blog/2026-07-10-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-10-semantic-routing-llm-costs.md @@ -126,7 +126,8 @@ When I want to change the policy, I edit the fetched vSR values, redeploy with Codex lets me choose a model manually, but that is one more decision to keep in my head during a long development task. A user-level Codex profile can instead -send every request to the corporate gateway with the stable `auto` model name: +send every request to the gateway, e.g. agentgateway, with the stable `auto` +model name: ```toml # ~/.codex/agentgateway.config.toml From 2145d392f014b6f8965b4fc4bf658b9ca173c8ba Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Tue, 14 Jul 2026 12:00:59 -0700 Subject: [PATCH 07/15] blog: publish latest routing evaluation Signed-off-by: Daneyon Hansen --- .../2026-07-10-semantic-routing-llm-costs.md | 14 ++++++------ ...Z-chart.svg => 20260714T182035Z-chart.svg} | 22 +++++++++---------- 2 files changed, 18 insertions(+), 18 deletions(-) rename static/img/blog/cost-based-semantic-routing/{20260713T184930Z-chart.svg => 20260714T182035Z-chart.svg} (80%) diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-10-semantic-routing-llm-costs.md index 0c1761328..7e79b9e92 100644 --- a/content/blog/2026-07-10-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-10-semantic-routing-llm-costs.md @@ -76,16 +76,16 @@ I deliberately leave out a forced-cheap lane. It would make the comparison larger, but it would not answer the question I care about: does routing save money while still escalating the work that deserves the expensive model? -In a clean run on July 13, 2026, all 48 primary requests completed with exact +In a clean run on July 14, 2026, all 48 primary requests completed with exact model-catalog lookups. vSR selected `gpt-5.4-nano` for 16 requests and -`gpt-5.5` for 8. Catalog-priced agentgateway metrics reported `$0.231896` for -the routed lane versus `$0.405800` for the always-expensive lane, a **42.9% -cost reduction**. Routed p50 latency was **3.96 seconds**, compared with -**11.02 seconds** for the always-expensive lane. P95 was similar at 19.48 and -20.12 seconds because the requests that still use GPT-5.5 naturally determine +`gpt-5.5` for 8. Catalog-priced agentgateway metrics reported `$0.227359` for +the routed lane versus `$0.417020` for the always-expensive lane, a **45.5% +cost reduction**. Routed p50 latency was **4.85 seconds**, compared with +**7.55 seconds** for the always-expensive lane. P95 was similar at 19.24 and +20.83 seconds because the requests that still use GPT-5.5 naturally determine the tail. -![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg) +![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg) The chart gives me the guardrail I need. It reached 83.3% tier agreement on the checked-in coding sample and sent 8 of the 12 prompts labelled complex to diff --git a/static/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg b/static/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg similarity index 80% rename from static/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg rename to static/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg index 6c57eb505..d61191e71 100644 --- a/static/img/blog/cost-based-semantic-routing/20260713T184930Z-chart.svg +++ b/static/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg @@ -1,6 +1,6 @@ - Semantic routing experiment results - Semantic routing spend, routing agreement, model mix, and latency for run 20260713T184930Z. + Semantic routing evaluation results + Semantic routing spend, routing agreement, model mix, and latency for run 20260714T182035Z. - Semantic routing experiment - Run 20260713T184930Z | Catalog-priced agentgateway metrics + Semantic routing evaluation + Run 20260714T182035Z | Catalog-priced agentgateway metrics SPEND REDUCTION - 42.9% + 45.5% Routed versus always expensive ROUTING AGREEMENT @@ -26,11 +26,11 @@ Expected tiers in this coding sample ROUTED P50 LATENCY - 3.96 s - -64.1% versus always expensive + 4.85 s + -35.7% versus always expensive COST PER DEMO RUN (USD) - Semantic routing$0.2319Always expensive$0.4058 + Semantic routing$0.2274Always expensive$0.4170 ROUTED MODEL MIX @@ -48,9 +48,9 @@ END-TO-END LATENCY - 3.96 s p50 - 19.48 s p95 semantic routing - Always expensive: 11.02 s p50, 20.12 s p95 + 4.85 s p50 + 19.24 s p95 semantic routing + Always expensive: 7.55 s p50, 20.83 s p95 Agreement uses expected tiers in the checked-in coding sample; it is a routing sanity check, not answer scoring. From 3f1978b732ea5dd37e6c222a2a8e6b83de4cd1f5 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Thu, 16 Jul 2026 12:03:58 -0700 Subject: [PATCH 08/15] blog: refresh routing evaluation Signed-off-by: Daneyon Hansen --- .../2026-07-10-semantic-routing-llm-costs.md | 52 +++++++------- .../20260716T181919Z-chart.svg | 70 +++++++++++++++++++ 2 files changed, 98 insertions(+), 24 deletions(-) create mode 100644 static/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-10-semantic-routing-llm-costs.md index 7e79b9e92..f4de47d9c 100644 --- a/content/blog/2026-07-10-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-10-semantic-routing-llm-costs.md @@ -59,7 +59,7 @@ engineering and finance can inspect the same model-selection event: ## A Small, Reproducible Evaluation I built the [cost-based semantic routing demo](https://github.com/danehans/agentgateway-demos/tree/main/cost-based-semantic-routing) -around a compact 24-prompt Go and Rust dataset. Half of the prompts are routine +around a compact 50-prompt Go and Rust dataset. Half of the prompts are routine implementation or test work; the other half involve correctness or distributed-systems problems. I send every prompt through two lanes: @@ -76,28 +76,34 @@ I deliberately leave out a forced-cheap lane. It would make the comparison larger, but it would not answer the question I care about: does routing save money while still escalating the work that deserves the expensive model? -In a clean run on July 14, 2026, all 48 primary requests completed with exact -model-catalog lookups. vSR selected `gpt-5.4-nano` for 16 requests and -`gpt-5.5` for 8. Catalog-priced agentgateway metrics reported `$0.227359` for -the routed lane versus `$0.417020` for the always-expensive lane, a **45.5% -cost reduction**. Routed p50 latency was **4.85 seconds**, compared with -**7.55 seconds** for the always-expensive lane. P95 was similar at 19.24 and -20.83 seconds because the requests that still use GPT-5.5 naturally determine -the tail. +In a clean run on July 16, 2026, all 100 primary requests completed with exact +model-catalog lookups. vSR selected `gpt-5.4-nano` for 41 requests and +`gpt-5.5` for 9. Catalog-priced agentgateway metrics reported `$0.280899` for +the routed lane versus `$0.915050` for the always-expensive lane, a **69.3% +cost reduction**. Routed p50 latency was **4.13 seconds**, compared with +**7.45 seconds** for the always-expensive lane. P95 was 15.96 seconds for +routed traffic and 17.89 seconds for the always-expensive lane. -![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg) +![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg) -The chart gives me the guardrail I need. It reached 83.3% tier agreement on the -checked-in coding sample and sent 8 of the 12 prompts labelled complex to -GPT-5.5. This does not claim perfect model selection. It does show that the -savings did not come from a blanket downgrade: one-third of routed requests -still used the expensive model, including most complex prompts in the sample. +The chart gives me the guardrail I need. It reached 68% tier agreement on the +checked-in coding sample and sent 9 of the 25 prompts labelled complex to +GPT-5.5. This is not a claim of perfect model selection. It does show that the +savings did not come from a blanket downgrade: 18% of routed requests still +used the expensive model, including explicitly complex prompts in the sample. That is the useful scope for this example. It keeps higher-risk work on a more capable model by policy and makes the exceptions visible. It is not a general answer-quality benchmark or a substitute for an application's own task-success and user-feedback measures. +The default dataset uses independent requests. For long-running coding agents, +the demo also offers an optional cache-transition evaluation: two ordered +ten-turn Go and Rust conversations with a stable prefix. It reports +provider-observed cache reads by model transition, which makes the cost of +switching tiers visible without mixing cache behavior into the primary routing +evaluation. + ## Try It I made the demo do the setup work that I would otherwise forget or repeat by @@ -115,8 +121,8 @@ export OPENAI_API_KEY='sk-...' ./demo.sh all --yes ``` -The default run sends 54 billable requests: two routing probes, four smoke-test -requests, and 48 primary requests. It writes request-level JSONL, an +The default run sends 106 billable requests: two routing probes, four smoke-test +requests, and 100 primary requests. It writes request-level JSONL, an evaluation-scoped Prometheus report, and an SVG chart under `results/`. When I want to change the policy, I edit the fetched vSR values, redeploy with @@ -138,15 +144,13 @@ model_provider = "agentgateway" name = "Corporate agentgateway" base_url = "https://my.corp.agentgateway.com/v1" wire_api = "responses" -env_key = "OPENAI_API_KEY" ``` -With `OPENAI_API_KEY` set to my OpenAI API key, I start Codex with -`codex --profile agentgateway`. The client sends OpenAI Responses API traffic -to agentgateway; vSR selects the actual tier and agentgateway records the -decision and realized cost. I validated this path with a routine table-driven -Go test request, which selected `gpt-5.4-nano`, and an advanced leader-election -request, which selected `gpt-5.5`. +I start Codex with `codex --profile agentgateway`. The client sends OpenAI +Responses API traffic to agentgateway; vSR selects the actual tier and +agentgateway records the decision and realized cost. I validated this path with +a routine table-driven Go test request, which selected `gpt-5.4-nano`, and an +advanced leader-election request, which selected `gpt-5.5`. The profile keeps credentials out of the configuration file. A company that needs to prevent direct model selection can enforce that separately at the diff --git a/static/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg b/static/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg new file mode 100644 index 000000000..4aefc6568 --- /dev/null +++ b/static/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg @@ -0,0 +1,70 @@ + + Semantic routing evaluation results + Semantic routing spend, routing agreement, model mix, latency, and cache transitions for run 20260716T181919Z. + + + Semantic routing evaluation + Run 20260716T181919Z | Catalog-priced agentgateway metrics + + + SPEND REDUCTION + 69.3% + Routed versus always expensive + + ROUTING AGREEMENT + 68.0% + Expected tiers in this coding sample + + ROUTED P50 LATENCY + 4.13 s + -44.6% versus always expensive + + COST PER DEMO RUN (USD) + Semantic routing$0.2809Always expensive$0.9151 + + + ROUTED MODEL MIX + + + 82% lower-cost + 41 gpt-5.4-nano | 9 gpt-5.5 + + + COMPLEX-PROMPT ESCALATION + + + 36% to GPT-5.5 + 9/25 complex prompts sent to GPT-5.5 + + + END-TO-END LATENCY + 4.13 s p50 + 15.96 s p95 semantic routing + Always expensive: 7.45 s p50, 17.89 s p95 + + + CONVERSATION CACHE TRANSITIONS + 0 switches + 0 ordered continuation turns + + + PROVIDER CACHE READS + 0 tokens + + + CACHE READS BY TRANSITION + No ordered multi-turn transitions in this run + + Agreement uses expected tiers in the checked-in coding sample; cache tokens are reported by the upstream provider. + From 61d8ef2a7345315f7585c1efa00bb7fad6262b86 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Thu, 16 Jul 2026 12:04:15 -0700 Subject: [PATCH 09/15] blog: replace routing chart Signed-off-by: Daneyon Hansen --- .../20260714T182035Z-chart.svg | 56 ------------------- 1 file changed, 56 deletions(-) delete mode 100644 static/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg diff --git a/static/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg b/static/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg deleted file mode 100644 index d61191e71..000000000 --- a/static/img/blog/cost-based-semantic-routing/20260714T182035Z-chart.svg +++ /dev/null @@ -1,56 +0,0 @@ - - Semantic routing evaluation results - Semantic routing spend, routing agreement, model mix, and latency for run 20260714T182035Z. - - - Semantic routing evaluation - Run 20260714T182035Z | Catalog-priced agentgateway metrics - - - SPEND REDUCTION - 45.5% - Routed versus always expensive - - ROUTING AGREEMENT - 83.3% - Expected tiers in this coding sample - - ROUTED P50 LATENCY - 4.85 s - -35.7% versus always expensive - - COST PER DEMO RUN (USD) - Semantic routing$0.2274Always expensive$0.4170 - - - ROUTED MODEL MIX - - - 67% lower-cost - 16 gpt-5.4-nano | 8 gpt-5.5 - - - COMPLEX-PROMPT ESCALATION - - - 67% to GPT-5.5 - 8/12 complex prompts sent to GPT-5.5 - - - END-TO-END LATENCY - 4.85 s p50 - 19.24 s p95 semantic routing - Always expensive: 7.55 s p50, 20.83 s p95 - - Agreement uses expected tiers in the checked-in coding sample; it is a routing sanity check, not answer scoring. - From a435b116eb461880ac34decc4787d338c03a1539 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Thu, 16 Jul 2026 12:15:26 -0700 Subject: [PATCH 10/15] blog: update semantic routing publication date Signed-off-by: Daneyon Hansen --- ...ng-llm-costs.md => 2026-07-17-semantic-routing-llm-costs.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename content/blog/{2026-07-10-semantic-routing-llm-costs.md => 2026-07-17-semantic-routing-llm-costs.md} (99%) diff --git a/content/blog/2026-07-10-semantic-routing-llm-costs.md b/content/blog/2026-07-17-semantic-routing-llm-costs.md similarity index 99% rename from content/blog/2026-07-10-semantic-routing-llm-costs.md rename to content/blog/2026-07-17-semantic-routing-llm-costs.md index f4de47d9c..e3da9dda8 100644 --- a/content/blog/2026-07-10-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-17-semantic-routing-llm-costs.md @@ -2,7 +2,7 @@ title: "Reduce LLM Spend With Semantic Routing You Can Measure" category: "Deep Dive" toc: false -publishDate: 2026-07-10 +publishDate: 2026-07-17 author: "Daneyon Hansen" description: "Use agentgateway, vLLM Semantic Router, and catalog-priced telemetry to reduce routine coding-model spend without routing every request to the cheapest model." --- From 23341ec20ae5f239b13fba882f25671d679ff0e4 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Thu, 16 Jul 2026 12:20:39 -0700 Subject: [PATCH 11/15] blog: clarify routing limitations Signed-off-by: Daneyon Hansen --- content/blog/2026-07-17-semantic-routing-llm-costs.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/content/blog/2026-07-17-semantic-routing-llm-costs.md b/content/blog/2026-07-17-semantic-routing-llm-costs.md index e3da9dda8..7fdc8a0f2 100644 --- a/content/blog/2026-07-17-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-17-semantic-routing-llm-costs.md @@ -170,6 +170,12 @@ Agentgateway gives me the accounting and observability plane; vSR makes the semantic decision. Together, they let me cut routine-model spend without hiding a blanket downgrade behind an attractive cost chart. +This example is intentionally static: it uses configured keywords, semantic +candidates, weights, and thresholds rather than learning from application +outcomes. Those signals need periodic review as traffic changes. The +agentgateway community and vSR project are continuing work to make semantic +routing easier to tune, operate, and measure against real application traffic. + ## What's Next I see cost-based semantic routing as one part of an LLM cost-management @@ -179,7 +185,3 @@ to attribute usage with virtual keys, observe realized spend by model and consumer, and enforce budget or spend limits where I need them. That turns a routing result into an operating model for keeping AI spend visible and under control. - -The agentgateway community is continuing to improve cost-based routing and the -tools around it. Future work will make it easier to tune, operate, and measure -model-tier decisions as real application traffic evolves. From 2ebc1beffbd6bf6109f9aea68b9ee3bec16f2c19 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Thu, 16 Jul 2026 12:27:55 -0700 Subject: [PATCH 12/15] blog: tighten semantic routing post Signed-off-by: Daneyon Hansen --- .../2026-07-17-semantic-routing-llm-costs.md | 135 +++++++----------- 1 file changed, 50 insertions(+), 85 deletions(-) diff --git a/content/blog/2026-07-17-semantic-routing-llm-costs.md b/content/blog/2026-07-17-semantic-routing-llm-costs.md index 7fdc8a0f2..3a463553f 100644 --- a/content/blog/2026-07-17-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-17-semantic-routing-llm-costs.md @@ -7,54 +7,41 @@ author: "Daneyon Hansen" description: "Use agentgateway, vLLM Semantic Router, and catalog-priced telemetry to reduce routine coding-model spend without routing every request to the cheapest model." --- -I spend a lot of time in Go and Rust, where a quick "one more thing" in a -long coding chat can turn a tidy refactor into an hour of debugging. I do not -always remember to step down from the strongest model after the hard part is -over. A few routine tests, configuration edits, and follow-up questions later, -I have spent premium-model credits on work that did not need them. - -Sending every request to a cheaper model is not the answer either. The -distributed-systems design, correctness work, and difficult debugging that -show up in those same conversations are exactly where I want more capable help. -I want the client to stay simple while the routing policy makes that choice. +I spend a lot of time in Go and Rust, where a quick "one more thing" in a long +coding chat can turn a tidy refactor into an hour of debugging. After the hard +part is over, routine tests and follow-up questions can still burn +premium-model credits. Sending every request to a cheaper model is not the +answer: distributed-systems design, correctness work, and difficult debugging +need more capable help. This example combines [vLLM Semantic Router (vSR)](https://vllm-semantic-router.com/) with [agentgateway](https://agentgateway.dev/) to send routine Go and Rust -coding tasks to `gpt-5.4-nano` and escalate more complex correctness, -distributed-systems, and deep-debugging work to `gpt-5.5`. I am not trying to -claim perfect routing accuracy. I want a reproducible way to show lower spend -without quietly routing every request to the cheapest tier. +coding tasks to `gpt-5.4-nano` and escalate complex correctness, +distributed-systems, and deep-debugging work to `gpt-5.5`. The goal is a +reproducible cost reduction without quietly routing everything to the cheapest +tier. ## Make the Decision Observable {{< reuse-image src="img/blog/cost-based-semantic-routing/semantic-routing-flow.svg" alt="Flow diagram: a coding agent sends model auto to agentgateway, vLLM Semantic Router selects either GPT-5.4 nano for routine coding or GPT-5.5 for complex coding, and agentgateway records catalog-priced telemetry" caption="One model name in, the appropriate model tier out - with the routing decision and realized cost visible at the gateway" >}} -I keep the client side boring: it sends an OpenAI-compatible request with -`model: "auto"`. An agentgateway policy sends the full request body to vSR. -vSR evaluates semantic, complexity, keyword, context, and structure signals, -then returns the selected model. Agentgateway forwards the request and records -what happened. - -vSR does not use historic request cost as a classifier input. Instead, cost is -an outcome that agentgateway measures after the request. That keeps the policy -explainable: the routing configuration says which work merits the stronger -model, while the gateway shows what that decision actually cost. +The client sends an OpenAI-compatible request with `model: "auto"`. An +agentgateway policy sends it to vSR, which evaluates semantic, complexity, +keyword, context, and structure signals before selecting a model. Agentgateway +forwards the request and records the outcome. Cost is not a classifier input; +it is measured after the request, keeping the policy explainable. ## Price the Outcome at the Gateway -Token counts are useful, but my company needs prices to make a budget decision. -Agentgateway loads a model cost catalog, calculates realized request cost, and -exposes the data in metrics, logs, and traces. The [model cost catalog +My company needs prices, not just token counts, to make a budget decision. +Agentgateway loads a model cost catalog and exposes realized request cost in +metrics, logs, and traces. The [model cost catalog guide](https://agentgateway.dev/docs/kubernetes/latest/llm/cost-controls/costs/) explains how to generate and load the catalog with `agctl`. For this evaluation, I use the integrated [OpenTelemetry stack](https://agentgateway.dev/docs/kubernetes/main/observability/otel-stack/). -It adds Prometheus, Loki, Tempo, the OpenTelemetry Collector, and Grafana so -engineering and finance can inspect the same model-selection event: - -- Prometheus carries token, duration, catalog-lookup, and cost signals. -- Logs and traces retain the selected model, request ID, latency, and cost. -- Grafana puts spend next to the signals needed to explain a routing decision. +Prometheus, Loki, Tempo, the OpenTelemetry Collector, and Grafana let +engineering and finance inspect the same model-selection event. ## A Small, Reproducible Evaluation @@ -68,13 +55,9 @@ distributed-systems problems. I send every prompt through two lanes: | `routed` | vSR selects `gpt-5.4-nano` or `gpt-5.5`. | | `always_expensive` | Every request uses `gpt-5.5`; the cost baseline. | -Each evaluation run has an ID attached to every request. The runner uses it in -its cost queries, which keeps unrelated traffic in the cluster out of the -result. - -I deliberately leave out a forced-cheap lane. It would make the comparison -larger, but it would not answer the question I care about: does routing save -money while still escalating the work that deserves the expensive model? +Each run has an ID on every request, so its cost query excludes unrelated +cluster traffic. I deliberately leave out a forced-cheap lane: the useful +comparison is whether routing saves money while still escalating complex work. In a clean run on July 16, 2026, all 100 primary requests completed with exact model-catalog lookups. vSR selected `gpt-5.4-nano` for 41 requests and @@ -86,32 +69,24 @@ routed traffic and 17.89 seconds for the always-expensive lane. ![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg) -The chart gives me the guardrail I need. It reached 68% tier agreement on the -checked-in coding sample and sent 9 of the 25 prompts labelled complex to -GPT-5.5. This is not a claim of perfect model selection. It does show that the +The chart reached 68% tier agreement and sent 9 of 25 prompts labelled complex +to GPT-5.5. This is not a claim of perfect model selection, but it shows the savings did not come from a blanket downgrade: 18% of routed requests still -used the expensive model, including explicitly complex prompts in the sample. - -That is the useful scope for this example. It keeps higher-risk work on a more -capable model by policy and makes the exceptions visible. It is not a general -answer-quality benchmark or a substitute for an application's own task-success -and user-feedback measures. +used the expensive model. It is a policy sanity check, not an answer-quality +benchmark or a substitute for application task-success and user-feedback data. The default dataset uses independent requests. For long-running coding agents, -the demo also offers an optional cache-transition evaluation: two ordered -ten-turn Go and Rust conversations with a stable prefix. It reports -provider-observed cache reads by model transition, which makes the cost of -switching tiers visible without mixing cache behavior into the primary routing -evaluation. +the optional cache-transition evaluation uses two ordered ten-turn Go and Rust +conversations with a stable prefix. It reports provider-observed cache reads by +model transition without mixing cache behavior into the primary evaluation. ## Try It -I made the demo do the setup work that I would otherwise forget or repeat by -hand. Behind the scenes, it uses the [agentgateway semantic-routing +The demo automates the setup work I would otherwise repeat by hand. It uses the [agentgateway semantic-routing example](https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing), creates or reuses a kind cluster, and installs agentgateway, vSR, a model cost -catalog, and the OpenTelemetry stack. Before the primary requests run, it -checks readiness, routing, catalog-priced metrics, logs, and traces. +catalog, and the OpenTelemetry stack. It checks readiness, routing, +catalog-priced metrics, logs, and traces before the primary requests run. ```bash git clone https://github.com/danehans/agentgateway-demos.git @@ -130,9 +105,8 @@ When I want to change the policy, I edit the fetched vSR values, redeploy with ## Use It From Codex -Codex lets me choose a model manually, but that is one more decision to keep in -my head during a long development task. A user-level Codex profile can instead -send every request to the gateway, e.g. agentgateway, with the stable `auto` +Codex lets me choose a model manually, but a user-level profile can instead +send every request to a gateway such as agentgateway with the stable `auto` model name: ```toml @@ -146,42 +120,33 @@ base_url = "https://my.corp.agentgateway.com/v1" wire_api = "responses" ``` -I start Codex with `codex --profile agentgateway`. The client sends OpenAI -Responses API traffic to agentgateway; vSR selects the actual tier and -agentgateway records the decision and realized cost. I validated this path with -a routine table-driven Go test request, which selected `gpt-5.4-nano`, and an -advanced leader-election request, which selected `gpt-5.5`. +I start Codex with `codex --profile agentgateway`. It sends OpenAI Responses +API traffic to agentgateway; vSR selects the tier and agentgateway records the +decision and realized cost. I validated routine Go test traffic to nano and an +advanced leader-election request to GPT-5.5. -The profile keeps credentials out of the configuration file. A company that -needs to prevent direct model selection can enforce that separately at the -gateway; `auto` is the supported path for this routing policy. Codex documents +The profile keeps credentials out of the configuration file. A company can +separately enforce `auto` at the gateway. Codex documents [custom model providers](https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers) and user-level [configuration profiles](https://learn.chatgpt.com/docs/config-file/config-advanced#profiles). ## From Example to Rollout -I would not treat semantic routing as a magic cost switch. I would start with -an explainable policy and compare it against sending every request to the -expensive model. Then I would confirm that the policy uses both tiers and -protects complex work, and tune it against the signals that matter to the -application: task completion, feedback, retries, and escalation rates. - -Agentgateway gives me the accounting and observability plane; vSR makes the -semantic decision. Together, they let me cut routine-model spend without -hiding a blanket downgrade behind an attractive cost chart. +Semantic routing is not a magic cost switch. Start with an explainable policy, +compare it with always using the expensive model, confirm it uses both tiers, +and tune it against task completion, feedback, retries, and escalation rates. +Agentgateway supplies the accounting and observability; vSR makes the semantic +decision. This example is intentionally static: it uses configured keywords, semantic candidates, weights, and thresholds rather than learning from application outcomes. Those signals need periodic review as traffic changes. The agentgateway community and vSR project are continuing work to make semantic -routing easier to tune, operate, and measure against real application traffic. +routing easier to tune, operate, and measure against real application traffic, +so stay tuned. ## What's Next -I see cost-based semantic routing as one part of an LLM cost-management -practice. After validating a tiering policy, I can use -[agentgateway cost controls](https://agentgateway.dev/docs/kubernetes/main/llm/cost-controls/) +After validating a tiering policy, use [agentgateway cost controls](https://agentgateway.dev/docs/kubernetes/main/llm/cost-controls/) to attribute usage with virtual keys, observe realized spend by model and -consumer, and enforce budget or spend limits where I need them. That turns a -routing result into an operating model for keeping AI spend visible and under -control. +consumer, and enforce budget or spend limits. From 9b1f5d5ffd0ceccbb653692390c82aeaf684c334 Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Thu, 16 Jul 2026 12:31:01 -0700 Subject: [PATCH 13/15] blog: document model overrides Signed-off-by: Daneyon Hansen --- content/blog/2026-07-17-semantic-routing-llm-costs.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/content/blog/2026-07-17-semantic-routing-llm-costs.md b/content/blog/2026-07-17-semantic-routing-llm-costs.md index 3a463553f..5ee688abf 100644 --- a/content/blog/2026-07-17-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-17-semantic-routing-llm-costs.md @@ -31,6 +31,13 @@ keyword, context, and structure signals before selecting a model. Agentgateway forwards the request and records the outcome. Cost is not a classifier input; it is measured after the request, keeping the policy explainable. +`auto` opts into that automatic selection. A client can instead request +`gpt-5.4-nano` or `gpt-5.5` directly to force a tier and bypass automatic +routing; the request still passes through vSR and agentgateway. Organizations +that require the policy can [validate and reject unsupported model +values](https://agentgateway.dev/docs/kubernetes/latest/traffic-management/transformations/validate/), +including anything other than `auto`. + ## Price the Outcome at the Gateway My company needs prices, not just token counts, to make a budget decision. @@ -125,8 +132,7 @@ API traffic to agentgateway; vSR selects the tier and agentgateway records the decision and realized cost. I validated routine Go test traffic to nano and an advanced leader-election request to GPT-5.5. -The profile keeps credentials out of the configuration file. A company can -separately enforce `auto` at the gateway. Codex documents +The profile keeps credentials out of the configuration file. Codex documents [custom model providers](https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers) and user-level [configuration profiles](https://learn.chatgpt.com/docs/config-file/config-advanced#profiles). From d84d1367369735b800a1fcea9b90cb6d53a262fc Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Thu, 16 Jul 2026 12:57:58 -0700 Subject: [PATCH 14/15] blog: clarify direct model routing Signed-off-by: Daneyon Hansen --- content/blog/2026-07-17-semantic-routing-llm-costs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2026-07-17-semantic-routing-llm-costs.md b/content/blog/2026-07-17-semantic-routing-llm-costs.md index 5ee688abf..6f8d38dff 100644 --- a/content/blog/2026-07-17-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-17-semantic-routing-llm-costs.md @@ -33,7 +33,7 @@ it is measured after the request, keeping the policy explainable. `auto` opts into that automatic selection. A client can instead request `gpt-5.4-nano` or `gpt-5.5` directly to force a tier and bypass automatic -routing; the request still passes through vSR and agentgateway. Organizations +routing; the request still passes through agentgateway. Organizations that require the policy can [validate and reject unsupported model values](https://agentgateway.dev/docs/kubernetes/latest/traffic-management/transformations/validate/), including anything other than `auto`. From 51e0d7b125ddd0af388aedda4e53095579b3963d Mon Sep 17 00:00:00 2001 From: Daneyon Hansen Date: Thu, 16 Jul 2026 13:00:53 -0700 Subject: [PATCH 15/15] blog: refresh semantic routing results Signed-off-by: Daneyon Hansen --- .../2026-07-17-semantic-routing-llm-costs.md | 18 +++++------ ...Z-chart.svg => 20260716T193923Z-chart.svg} | 32 +++++++++---------- 2 files changed, 25 insertions(+), 25 deletions(-) rename static/img/blog/cost-based-semantic-routing/{20260716T181919Z-chart.svg => 20260716T193923Z-chart.svg} (77%) diff --git a/content/blog/2026-07-17-semantic-routing-llm-costs.md b/content/blog/2026-07-17-semantic-routing-llm-costs.md index 6f8d38dff..7e88e9f42 100644 --- a/content/blog/2026-07-17-semantic-routing-llm-costs.md +++ b/content/blog/2026-07-17-semantic-routing-llm-costs.md @@ -67,18 +67,18 @@ cluster traffic. I deliberately leave out a forced-cheap lane: the useful comparison is whether routing saves money while still escalating complex work. In a clean run on July 16, 2026, all 100 primary requests completed with exact -model-catalog lookups. vSR selected `gpt-5.4-nano` for 41 requests and -`gpt-5.5` for 9. Catalog-priced agentgateway metrics reported `$0.280899` for -the routed lane versus `$0.915050` for the always-expensive lane, a **69.3% -cost reduction**. Routed p50 latency was **4.13 seconds**, compared with -**7.45 seconds** for the always-expensive lane. P95 was 15.96 seconds for -routed traffic and 17.89 seconds for the always-expensive lane. +model-catalog lookups. vSR selected `gpt-5.4-nano` for 38 requests and +`gpt-5.5` for 12. Catalog-priced agentgateway metrics reported `$0.375954` for +the routed lane versus `$0.916940` for the always-expensive lane, a **59.0% +cost reduction**. Routed p50 latency was **5.58 seconds**, compared with +**7.75 seconds** for the always-expensive lane. P95 was 17.18 seconds for +routed traffic and 19.18 seconds for the always-expensive lane. -![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg) +![Catalog-priced semantic-routing result](/img/blog/cost-based-semantic-routing/20260716T193923Z-chart.svg) -The chart reached 68% tier agreement and sent 9 of 25 prompts labelled complex +The chart reached 74% tier agreement and sent 12 of 25 prompts labelled complex to GPT-5.5. This is not a claim of perfect model selection, but it shows the -savings did not come from a blanket downgrade: 18% of routed requests still +savings did not come from a blanket downgrade: 24% of routed requests still used the expensive model. It is a policy sanity check, not an answer-quality benchmark or a substitute for application task-success and user-feedback data. diff --git a/static/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg b/static/img/blog/cost-based-semantic-routing/20260716T193923Z-chart.svg similarity index 77% rename from static/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg rename to static/img/blog/cost-based-semantic-routing/20260716T193923Z-chart.svg index 4aefc6568..cc6431b33 100644 --- a/static/img/blog/cost-based-semantic-routing/20260716T181919Z-chart.svg +++ b/static/img/blog/cost-based-semantic-routing/20260716T193923Z-chart.svg @@ -1,6 +1,6 @@ Semantic routing evaluation results - Semantic routing spend, routing agreement, model mix, latency, and cache transitions for run 20260716T181919Z. + Semantic routing spend, routing agreement, model mix, latency, and cache transitions for run 20260716T193923Z. Semantic routing evaluation - Run 20260716T181919Z | Catalog-priced agentgateway metrics + Run 20260716T193923Z | Catalog-priced agentgateway metrics SPEND REDUCTION - 69.3% + 59.0% Routed versus always expensive ROUTING AGREEMENT - 68.0% + 74.0% Expected tiers in this coding sample ROUTED P50 LATENCY - 4.13 s - -44.6% versus always expensive + 5.58 s + -28.0% versus always expensive COST PER DEMO RUN (USD) - Semantic routing$0.2809Always expensive$0.9151 + Semantic routing$0.3760Always expensive$0.9169 ROUTED MODEL MIX - - 82% lower-cost - 41 gpt-5.4-nano | 9 gpt-5.5 + + 76% lower-cost + 38 gpt-5.4-nano | 12 gpt-5.5 COMPLEX-PROMPT ESCALATION - - 36% to GPT-5.5 - 9/25 complex prompts sent to GPT-5.5 + + 48% to GPT-5.5 + 12/25 complex prompts sent to GPT-5.5 END-TO-END LATENCY - 4.13 s p50 - 15.96 s p95 semantic routing - Always expensive: 7.45 s p50, 17.89 s p95 + 5.58 s p50 + 17.18 s p95 semantic routing + Always expensive: 7.75 s p50, 19.18 s p95 CONVERSATION CACHE TRANSITIONS