feat: add standalone KVCache model ranking page#18
Open
stmatengss wants to merge 8 commits into
Open
Conversation
Add per-turn analysis methods to Python MultiAgentHeuristicConfig class (per_turn_reusable_private_tokens, per_turn_content_hit_rates, per_turn_working_set_tokens) and equivalent JS methods on the web calculator's MultiAgentHeuristic class. Include a new "Per-Turn Analysis" Chart.js bar+line chart showing how cache hit rate and reusable private tokens evolve across conversation turns. Add test vectors and parity checks (232 total, up from 178). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds an interactive model ranking table that compares all preset models by their KV cache size requirements for a configurable sequence length. Features: - Extracted model registry (MODELS) as single source of truth - Ranking table with sortable columns (KV/Token, KV/Token/Rank, Total KV) - Configurable sequence length input (default: 8192 tokens) - Highlights currently selected model in the ranking table - Auto-updates when preset or sequence length changes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements the rank-models subcommand for comparing models by KV cache size requirements. Supports multiple sequence lengths and outputs in markdown or CSV format. Usage: kvcache-upper-bound rank-models --sequence-length 8192 kvcache-upper-bound rank-models --sequence-length 4096 --sequence-length 32768 --format csv kvcache-upper-bound rank-models --sequence-length 8192 --output ranking.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Independent HTML page focused solely on model ranking - Dark theme matching existing dashboard - Preset buttons for sequence lengths (1K-128K tokens) - Sortable table with architecture, deployment, and memory metrics - Shows per-token and total KV cache sizes with per-rank breakdown - Includes 6 preset models (Qwen3-27B, Llama3.1-70B/8B, Qwen2.5-72B, Mistral-7B, DeepSeek-V3-671B)
- Remove hardcoded tp_size/pp_size from model definitions - Add input fields for TP and PP parallelism settings (default: 1) - Table updates automatically when TP/PP values change - All models now share the same parallelism configuration
- Added all models from kvcache.ai/tools/kv-cache-size-calculator/ - New families: DeepSeek V4, GLM-5, Kimi K2.5/2.6, Qwen3.5/3.6, Gemma 4, Cohere, MiMo, MiniMax - Added family filter dropdown for easier navigation - Models with unknown parameter count show "N/A" for weight columns - MLA models use kv_lora_rank as effective head_dim with n_kv_heads=1 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaced all 41 parameter_count: null values with concrete numbers from model names and HuggingFace configs: - DeepSeek V4 Pro/Flash: 671B, V3.2: 685B - GLM-5/5.1/5.2: ~756B (estimated) - Kimi K2.5/K2.6: ~1T (estimated) - Qwen3.6: 27B, 35B-A3B - Qwen3.5: 397B, 122B, 35B, 27B, 9B, 4B, 2B, 0.8B - Qwen3: 32B, 14B, 8B, 4B, 1.7B, 0.6B - Qwen2.5: 32B, 14B, Coder-32B - Gemma 4: E2B(2B), E4B(4B), 26B-A4B, 31B - Cohere: Command R(35B), R+(104B), R7B(7B), A(111B), A+(111B) - MiMo: V2.5(7B), V2.5-Pro(14B) (estimated) - MiniMax: M2/M2.1/M2.5/M2.7 (~230B MoE each)
Critical fixes from code review:
1. MLA n_kv_heads consistency (16x KV overestimate):
- Fixed calculator.html: DeepSeek V2/V3/V3.1, Kimi K2 from 16 → 1
- Fixed CLI main.py: DeepSeek V3 from 16 → 1
- Fixed ranking.html: DeepSeek V2/V3.1, Kimi K2 from 16 → 1
- MLA stores compressed latent (kv_lora_rank=512), not decompressed
heads. n_kv_heads=1, head_dim=512 correctly models the KV size.
2. Hybrid model kv_cache_layer_count (up to 12x overestimate):
- Set kv_cache_layer_count for 10 Qwen3.5/3.6 hybrid-linear models
- Linear attention layers use recurrent state, NOT KV cache
- Only full-attention layers count toward KV cache size
- Sliding-window layers (Gemma 4, Cohere, MiMo) DO store KV cache,
so those correctly remain kv_cache_layer_count=null (all layers)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
website/ranking.htmlpage focused solely on comparing model KV cache requirementsFeatures
Sequence Length Presets:
Sortable Table Columns:
Models Included:
Use Case
Quick reference/lookup tool for comparing KV cache size requirements across popular LLM models at different sequence lengths.
Test Plan
website/ranking.htmldirectly in browserImplementation Notes
index.htmlandcalculator.html🤖 Generated with Claude Code