Skip to content

benchmarking.md

Latest commit

 

History

History
114 lines (90 loc) · 5.57 KB

benchmarking.md

File metadata and controls

114 lines (90 loc) · 5.57 KB
Raw

Benchmarking Lynkr

benchmark-tier-routing.js (repo root) is both a comparative benchmark (Lynkr vs LiteLLM vs Portkey on identical workloads) and a routing regression harness — its correctness assertions encode real production incidents so they can't silently return.

Quick start

# Gateway must be running (standalone or via wrap)
node index.js &          # or: lynkr wrap claude in another terminal

node benchmark-tier-routing.js

# Routing-only head-to-head (skips compression/cache scenarios; judges
# EVERY proxy on the same acceptable-tier sets, not just Lynkr):
MODE=routing node benchmark-tier-routing.js

# Repeat each scenario to catch non-deterministic routers (a scenario
# passes only if every run passes); restrict to one proxy with ONLY:
MODE=routing RUNS=3 ONLY=LiteLLM node benchmark-tier-routing.js

Competitors are optional; unreachable proxies are reported as SKIPPED (proxy not reachable) and excluded from cost comparisons — never counted as $0.00.

# LiteLLM head-to-head against its Auto Router v2 complexity router
# (litellm >= 1.94): tier targets mirror Lynkr's TIER_* config, and each
# tier deployment carries an explicit model_info.id so the benchmark can
# read the tier decision from the x-litellm-model-id header.
litellm --port 8082 --config litellm-autorouter-v2.yaml       # heuristic (default)
litellm --port 8082 --config litellm-autorouter-v2-llm.yaml   # LLM classifier

# Legacy load-balancing comparison (cost heuristic tier inference):
litellm --port 8082 --config your-litellm-config.yaml

# Portkey (needs a real ANTHROPIC_API_KEY — OAuth tokens won't work)
docker run -d -p 8083:8787 portkeyai/gateway

The 19 scenarios

Group IDs What it measures
Feature economics S1, T1, T2, H1, L1, L2, SC1, SC2, R1 tier routing, tool-schema stripping, history compression, TOON JSON compression, semantic cache
Routing regressions F1, F2, RS1, SR1, A1 force-cloud phrases, path-risk, reminder-injection immunity, suggestion-mode side requests, autonomous→REASONING
Session behaviour P1, P2 fingerprint pins a SIMPLE opener, then the pin escapes mid-session when the real task arrives
Envelope invariance IV1, IV2 the same ask bare vs wrapped in tool schemas + fat system-reminders must land the same tier (WS7 anchor scoring)
Cache correctness SC3 the cache must not serve an answer to a different question

MODE=routing runs the 11 scenarios that carry an acceptable tier set (S1, R1, F1, F2, RS1, SR1, A1, P1, P2, IV1, IV2) and nothing else.

Reading the output

Per-scenario rows report the routing decision (Tier column, from X-Lynkr-* headers) and the served model (from the response body — they differ when tier-fallback rescued a failed upstream, flagged SERVED-VIA-FALLBACK). Saved is signed: negative values expose proxy overhead (e.g. system-prompt injection on small requests) instead of hiding it.

ROUTING CORRECTNESS (Lynkr) is the regression scoreboard: Lynkr's strict per-incident expectations (expectTier). Any ✗ … ← REGRESSION means a routing change re-broke a fixed incident. Current baseline: 12/12 (11 tier assertions + SC3's cache guard). With RUNS>1 a scenario passes only if every run passes.

ROUTING SCOREBOARD (all proxies) judges every proxy that exposes a real tier decision against the same acceptable sets — broader than expectTier (e.g. R1 accepts COMPLEX or REASONING) so it's fair to routers with different tier philosophies. Lynkr's tier comes from X-Lynkr-Tier; LiteLLM's from x-litellm-model-id when running an Auto Router v2 config. IV2 additionally requires the same tier as IV1. Reference run (2026-07-15, same backends): Lynkr 11/11, LiteLLM v1.94 heuristic 4/11, LiteLLM + GPT-5.2 LLM classifier 6–8/11 (non-deterministic) — details in BENCHMARK_REPORT.md addendum.

Cost tables price local models (ollama/minimax/llama/qwen) at $0 and bill cache hits as zero ([CACHE-HIT], detected via the lynkr_semantic_cache response marker). The extrapolation section is a scenario-mix multiplication, not a traffic model — treat it as directional.

Hermeticity (why reruns are trustworthy)

Server-side state persists between runs: session pins live 6h and the semantic cache retains answers. The harness compensates:

  • a per-run nonce is embedded in the stateful scenarios (P1/P2 and the SC family), so each run gets fresh fingerprints and cache keys;
  • every non-cache scenario sends x-lynkr-no-cache: true, so run #2's feature measurements aren't silently served from run #1's cache;
  • SC3's assertion is similarity-aware: a hit at ≥0.97 similarity is a prior run's identical question (correct behaviour, passes); a hit below that matched a different question — the real false positive — and fails with the similarity printed.

Two consecutive runs must both score 12/12 (or pass RUNS=2 in one invocation — each run gets a fresh nonce automatically). If run #2 diverges, the benchmark found state leakage — that's a finding, not noise.

Known limitations

  • Upper tiers serve via fallback in scripted runs. azure-anthropic requires the OAuth token that only Claude Code supplies, so COMPLEX/ REASONING decisions get served by the fallback chain. Routing decisions are still asserted; served-quality comparisons need the wrap.
  • Nine-figure extrapolations from 19 requests are directional. For publishable numbers, loop the mix 50–100× (RUNS=50).
  • The token estimator (chars/4) is approximate; treat Saved as relative, not invoice-grade.