diff --git a/docs/superpowers/specs/2026-05-26-enhanced-web-calculator-design.md b/docs/superpowers/specs/2026-05-26-enhanced-web-calculator-design.md new file mode 100644 index 0000000..97ca645 --- /dev/null +++ b/docs/superpowers/specs/2026-05-26-enhanced-web-calculator-design.md @@ -0,0 +1,248 @@ +# Enhanced KVCache Web Calculator — Design Spec + +## Overview + +Redesign the web calculator (`website/calculator.html`) into a comprehensive, guided KVCache sizing tool that serves platform architects, ML engineers, and non-technical stakeholders equally well. All computation remains fully client-side (no backend). + +## Goals + +- Let users go from "I don't know where to start" to "I have a capacity plan" in under 60 seconds via the Quick-Start Wizard +- Provide pre-built model profiles for 10+ popular models so users never need to look up layer counts +- Enable side-by-side "what-if" scenario comparison with delta highlighting +- Add TPS-vs-machines curves and sensitivity analysis beyond the current hit-rate chart +- Allow exporting and sharing results (markdown, PNG, JSON, URL hash) +- Improve input validation, documentation (tooltips), and responsive layout + +## Non-Goals + +- No server-side computation or APIs +- No trace-based oracle analysis (future scope) +- No multi-language / i18n for this iteration +- No database or user accounts + +--- + +## Component 1: Model Preset Library + +### Data Structure + +```javascript +const MODEL_LIBRARY = [ + { + id: "qwen3-27b", + family: "Qwen", + name: "Qwen3-27B (MLA)", + params: { + n_layers: 64, kv_cache_layer_count: 16, n_kv_heads: 4, + head_dim: 256, dtype_bytes: 2, parameter_count: 27781419504, + weight_dtype_bytes: 2, block_size: 16 + }, + notes: "Multi-head Latent Attention — only 16 KV layers" + }, + // ... 10+ models +]; +``` + +### Models to Include + + +| Family | Model | Params (B) | KV Heads | Head Dim | KV Layers | Architecture Note | +| -------- | ---------------- | ---------- | -------- | -------- | --------- | ---------------------------------------------------------------------------------- | +| Qwen | Qwen3-27B | 27.8 | 4 | 256 | 16 | MLA | +| Qwen | Qwen2.5-72B | 72.7 | 8 | 128 | 80 | GQA | +| Qwen | Qwen2.5-7B | 7.6 | 4 | 128 | 28 | GQA | +| Llama | Llama 3.1 70B | 70.6 | 8 | 128 | 80 | GQA | +| Llama | Llama 3.1 8B | 8.0 | 8 | 128 | 32 | GQA | +| Llama | Llama 3.3 70B | 70.6 | 8 | 128 | 80 | GQA | +| DeepSeek | DeepSeek-V2 236B | 236 | 128 | 128 | 60 | MLA + MoE (compressed KV — use effective n_kv_heads=16, head_dim=512 for kv_bytes) | +| DeepSeek | DeepSeek-V3 671B | 671 | 128 | 128 | 61 | MLA + MoE (compressed KV — same note) | +| Mistral | Mixtral 8x22B | 141 | 8 | 128 | 56 | MoE, GQA | +| Mistral | Mistral 7B | 7.2 | 8 | 128 | 32 | GQA | +| GLM | GLM-4 9B | 9.4 | 2 | 128 | 40 | MQA | +| Yi | Yi-1.5 34B | 34.4 | 8 | 128 | 60 | GQA | + + +### UX + +- Dropdown at top of Model Profile section, grouped by family +- Type-ahead filter for quick search +- Selecting a model fills all model profile fields +- Fields remain editable after preset selection (badge shows "Modified" if changed) +- "Custom" option leaves fields empty for manual entry + +--- + +## Component 2: Scenario Comparison Mode + +### Interaction Flow + +1. User clicks "Compare" button (top of results panel) +2. Current configuration snapshots as **Scenario A** (locked, grayed form or collapsed summary) +3. Form becomes **Scenario B** (editable, highlighted border) +4. Results panel splits into two columns: A (frozen) | B (live-updating) +5. Delta row below each metric shows the difference (green = improvement, red = regression) +6. User can click "Swap" to make B the new A, or "Exit Compare" to return to single mode + +### Delta Display + +``` +┌─────────────────────────┬─────────────────────────┐ +│ Scenario A (Base) │ Scenario B (What-if) │ +├─────────────────────────┼─────────────────────────┤ +│ Strict Hit: 80.30% │ Strict Hit: 87.45% │ +│ │ Δ +7.15pp ▲ │ +│ LRU Hit: 79.26% │ LRU Hit: 84.12% │ +│ │ Δ +4.86pp ▲ │ +│ Min Machines: 4 │ Min Machines: 2 │ +│ │ Δ -2 ▼ (better) │ +└─────────────────────────┴─────────────────────────┘ +``` + +### What to Compare + +- All summary metrics (hit rates, TPS gains, saturation capacities) +- Tier table rows (matched by tier label) +- Chart overlay (both scenarios on same chart, A as dashed line) + +--- + +## Component 3: Quick-Start Wizard + +### Step 1 — Model Selection + +- Shows the model preset library as a card grid (logo/icon per family) +- Click to select; shows key stats (param count, KV bytes/token) +- "Skip — I'll configure manually" link at bottom + +### Step 2 — Hardware Configuration + +- **GPU type** dropdown: H20 (96GB) / A100 (80GB) / A100 (40GB) / H100 (80GB) / L40S (48GB) +- **GPU count**: number input (1–128) +- **Auto-suggest TP/PP**: based on model size vs. GPU memory, recommend a TP/PP split + - Rule: if model_weights_gb > 0.8 × gpu_memory_gb, suggest TP = ceil(model_weights_gb / (0.7 × gpu_memory_gb)) +- User can override TP/PP + +### Step 3 — Workload Template + +- **Template** dropdown with presets: + - **Multi-agent chat** (default): concurrent_agents=2048, shared_prefix=4096, avg_turns=8, avg_new_tokens=4096, private_window=65536 + - **RAG pipeline**: concurrent_agents=512, shared_prefix=8192, avg_turns=2, avg_new_tokens=2048, private_window=16384 + - **Code assistant**: concurrent_agents=1024, shared_prefix=16384, avg_turns=6, avg_new_tokens=8192, private_window=32768 + - **Light chatbot**: concurrent_agents=4096, shared_prefix=2048, avg_turns=4, avg_new_tokens=1024, private_window=8192 + - **Custom**: all fields editable +- Each template shows a 1-line description + +### Completion + +- "Calculate" button closes wizard, opens full calculator with all fields pre-filled +- Results appear immediately +- Wizard state not persistent — can be re-opened from a "Guided Setup" button + +--- + +## Component 4: Enhanced Charts + +### Chart A: Hit Rate vs. Capacity (existing, enhanced) + +- Keep current line chart +- Add: shaded area between strict-prefix and LRU (shows "policy gap") +- Add: vertical annotation line at current HBM capacity +- In comparison mode: overlay Scenario A as dashed lines + +### Chart B: TPS vs. Machine Count (new) + +- X-axis: number of machines (1 to 2× current) +- Y-axis: cluster TPS capacity +- Lines: strict-prefix policy, LRU policy +- Horizontal dashed line: target TPS +- Intersection point highlighted: "minimum machines needed" + +### Chart C: Parameter Sensitivity (new) + +- Bar chart or tornado chart +- Shows: if each parameter changes by ±20%, how much does HBM hit rate change? +- Parameters tested: concurrent_agents, shared_prefix, avg_turns, private_window, zipf_s, lru_efficiency +- Helps users understand which lever matters most + +--- + +## Component 5: Export & Share + +### Export Options + +1. **Copy as Markdown** — button copies the results table + summary as a markdown-formatted text block to clipboard +2. **Download Chart (PNG)** — per-chart download button using Chart.js `toBase64Image()` +3. **Download Results (JSON)** — full analysis output including all tier rows, summary, and input config +4. **Share via URL** — encode full configuration in URL hash: + ``` + calculator.html#config=eyJtb2RlbF9wcm9maWxlIjp7...} + ``` + - Base64-encoded JSON of all form parameters + - Loading page with hash auto-fills all fields and runs calculation + - Length limit: if config > 2KB encoded, offer "Copy shareable link" that uses compression (lz-string) + +### UI Placement + +- Export toolbar above results panel: `[📋 Markdown] [📊 PNG] [📄 JSON] [🔗 Share]` +- Tooltip explains each on hover + +--- + +## Component 6: UX Polish + +### Input Validation + + +| Rule | Display | +| ------------------------------------------------------------ | ------------------------------------------------------ | +| tp_size × pp_size must divide evenly into n_kv_heads | Red border + "TP×PP must divide n_kv_heads" | +| accelerator_count must be divisible by cards_per_machine | Red border + "GPU count must be N × cards_per_machine" | +| Numeric fields must be > 0 | Red border + "Must be positive" | +| parameter_count unrealistically small (< 1B for > 32 layers) | Yellow warning | + + +### Tooltips + +Every input field gets a `title` attribute with a 1-line explanation: + +- `concurrent_agents`: "Number of simultaneous agent sessions sharing the cache pool" +- `shared_prefix_tokens`: "System prompt tokens shared identically across all agents" +- etc. + +### Responsive Layout + +- ≥1200px: side-by-side (form left, results right) — current layout +- 800–1200px: stacked (form top, results bottom), form sections in 2-column grid +- <800px: single column, accordion for form sections + +### Dark/Light Mode + +- Toggle button in header (sun/moon icon) +- Uses CSS custom properties (already in place) — just swap the `:root` values +- Persist preference in `localStorage` + +--- + +## File Structure + +All changes remain in `website/calculator.html` (single file, no build step). The embedded JS grows but stays under 2000 LOC total. If it exceeds this, split into: + +- `website/calculator.html` (HTML + CSS) +- `website/js/engine.js` (calculation engine) +- `website/js/models.js` (preset library) +- `website/js/ui.js` (interactivity) + +For now, keep everything in one file for simplicity. + +--- + +## Implementation Priority + +1. Model Preset Library (immediate value, low complexity) +2. Quick-Start Wizard (onboarding, medium complexity) +3. Enhanced Charts — TPS vs. Machines (high value, low complexity) +4. Export & Share (medium value, low complexity) +5. Scenario Comparison (high value, high complexity) +6. UX Polish (ongoing, can be incremental) +7. Sensitivity Chart (nice-to-have, medium complexity) + diff --git a/openspec/changes/add-web-ui/.openspec.yaml b/openspec/changes/add-web-ui/.openspec.yaml new file mode 100644 index 0000000..9e883bf --- /dev/null +++ b/openspec/changes/add-web-ui/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-05-25 diff --git a/openspec/changes/add-web-ui/design.md b/openspec/changes/add-web-ui/design.md new file mode 100644 index 0000000..58e226f --- /dev/null +++ b/openspec/changes/add-web-ui/design.md @@ -0,0 +1,67 @@ +## Context + +The KVCache Upper Bound Oracle has a Python-based CLI that runs heuristic analysis for multi-agent KVCache sizing. There is already a `website/index.html` results dashboard that visualizes pre-computed JSON outputs using Chart.js. The goal is to add an interactive calculator page that performs the same `estimate-multi-agent` analysis entirely in the browser, requiring no backend. + +The core calculation (`_hit_rate_for_capacity_tokens`, `_shape_fraction`, `_generalized_harmonic`, TPS gain) is pure math with no I/O dependencies — ideal for client-side porting. + +## Goals / Non-Goals + +**Goals:** +- Fully client-side calculator: zero backend, works from any static hosting (GitHub Pages, file://) +- Parity with `estimate-multi-agent` CLI output for hit rates, TPS gains, and capacity planning +- Preset configurations loadable from dropdown (matching `configs/public_multi_agent_*.json`) +- All parameters editable with immediate recalculation +- Visual output: results table + Chart.js charts for hit rate curves + +**Non-Goals:** +- Porting the trace-based `analyze-buckets` command (requires large JSONL data files) +- Porting the `calibrate-multi-agent` command (requires trace I/O) +- Server-side rendering or API endpoints +- Mobile-first design (desktop-first, responsive as bonus) +- i18n for the calculator page (English only for v1) + +## Decisions + +### 1. Single HTML file with embedded JS/CSS + +**Choice**: One self-contained `website/calculator.html` file. + +**Rationale**: Matches the existing `website/index.html` pattern. No build step, no bundler, trivially deployable. The calculation logic is ~200 lines of JS — not enough to warrant modules. + +**Alternatives considered**: +- Separate `.js`/`.css` files: adds complexity for marginal benefit at this scale +- React/Vue SPA: massive overkill for a form + table + chart + +### 2. Port only the heuristic math to JavaScript + +**Choice**: Re-implement `_hit_rate_for_capacity_tokens`, `_shape_fraction`, `_generalized_harmonic`, `tps_gain`, `cluster_capacity_tps`, and the saturation/capacity helpers. + +**Rationale**: These are pure functions with clear inputs/outputs. The Python code is the reference implementation; the JS port can be validated against known config outputs. + +**Alternatives considered**: +- Pyodide (Python in WASM): 10MB+ download, slow startup — unacceptable for an interactive calculator +- WebAssembly compiled from Python: complex toolchain, same startup penalty + +### 3. Preset configs embedded as JSON objects in the HTML + +**Choice**: Embed 2-3 preset configs (matching `configs/public_multi_agent_*.json`) as JS objects inside the calculator page. + +**Rationale**: Avoids fetch/CORS issues when opening from `file://`. Keeps the page fully self-contained. Configs are small (<2KB each). + +### 4. Chart.js for visualization (same as existing dashboard) + +**Choice**: Use Chart.js (already loaded by the existing dashboard via CDN). + +**Rationale**: Consistency with existing page. Already proven to work. Line chart showing hit rate vs. capacity GB is the primary visualization. + +### 5. Real-time recalculation on parameter change + +**Choice**: Debounced recalculation (100ms) triggered on any input change. + +**Rationale**: The calculation is O(n) where n = zipf_population_blocks (max ~4096 iterations for harmonic sum). This completes in <5ms on any modern device — no need for web workers or lazy evaluation. + +## Risks / Trade-offs + +- **Numerical drift**: JS uses float64 exclusively vs Python's float64. Risk is minimal for this use case (hit rates are reported to 2-4 decimal places). → Mitigation: validate JS output against Python for all preset configs. +- **Maintenance burden**: Two implementations of the same math (Python + JS). → Mitigation: the JS port is small (~200 LOC), and the Python reference is authoritative. Add a comment in the JS referencing the Python source file. +- **Preset staleness**: If configs change, the embedded presets in HTML diverge. → Mitigation: document which config files each preset mirrors. diff --git a/openspec/changes/add-web-ui/proposal.md b/openspec/changes/add-web-ui/proposal.md new file mode 100644 index 0000000..d9845bd --- /dev/null +++ b/openspec/changes/add-web-ui/proposal.md @@ -0,0 +1,26 @@ +## Why + +The KVCache Upper Bound Oracle currently requires Python installation and CLI usage to estimate cache hit rates and capacity planning. A browser-based interactive calculator would let users experiment with parameters (model profile, deployment config, multi-agent heuristic settings) instantly without any backend or installation — making the tool accessible to product managers, architects, and anyone evaluating KVCache sizing. + +## What Changes + +- Add a new interactive web page (`website/calculator.html`) that runs the full `estimate-multi-agent` heuristic analysis entirely in the browser using JavaScript +- Port the core calculation logic (hit rate curves, TPS gain estimation, capacity planning) from Python to JavaScript +- Provide preset configurations (e.g., Qwen3-27B on H20) selectable via dropdown, with all parameters editable +- Display results as interactive tables and charts (hit rate vs. capacity, TPS gains per tier) +- Link the new calculator page from the existing results dashboard + +## Capabilities + +### New Capabilities +- `browser-calculator`: Interactive browser-based KVCache heuristic calculator that ports the `estimate-multi-agent` analysis to client-side JavaScript, with parameter controls and visual result display + +### Modified Capabilities + + +## Impact + +- **New files**: `website/calculator.html` (single-page app with embedded JS/CSS) +- **Modified files**: `website/index.html` (add navigation link to calculator) +- **Dependencies**: Chart.js (already used by existing dashboard), no server-side dependencies +- **No breaking changes**: existing CLI and Python library behavior is unchanged diff --git a/openspec/changes/add-web-ui/specs/browser-calculator/spec.md b/openspec/changes/add-web-ui/specs/browser-calculator/spec.md new file mode 100644 index 0000000..78277c8 --- /dev/null +++ b/openspec/changes/add-web-ui/specs/browser-calculator/spec.md @@ -0,0 +1,105 @@ +## ADDED Requirements + +### Requirement: Interactive parameter input form +The calculator page SHALL present a form with all parameters needed for multi-agent heuristic analysis, grouped into logical sections: Model Profile, Multi-Agent Heuristic, Deployment Configuration, and Analysis Settings. + +#### Scenario: Page loads with default preset +- **WHEN** user opens the calculator page +- **THEN** all form fields SHALL be populated with the first preset configuration (Qwen3-27B on H20) + +#### Scenario: User selects a different preset +- **WHEN** user selects a preset from the configuration dropdown +- **THEN** all form fields SHALL update to reflect that preset's values +- **AND** results SHALL recalculate immediately + +#### Scenario: User modifies a parameter +- **WHEN** user changes any numeric input field +- **THEN** the results SHALL recalculate within 200ms of the last keystroke + +### Requirement: Model Profile parameters +The form SHALL include editable fields for: n_layers, kv_cache_layer_count, n_kv_heads, head_dim, dtype_bytes, parameter_count, weight_dtype_bytes, tp_size, pp_size, block_size. + +#### Scenario: All model profile fields present +- **WHEN** user views the Model Profile section +- **THEN** all fields listed above SHALL be visible and editable as numeric inputs + +### Requirement: Multi-Agent Heuristic parameters +The form SHALL include editable fields for: concurrent_agents, shared_prefix_tokens, avg_new_tokens_per_turn, avg_turns_per_session, private_window_tokens, curve_mode (dropdown: linear/power_law_fit/zipf_harmonic), zipf_s, zipf_population_blocks, lru_like efficiency, strict_prefix_upper_bound efficiency. + +#### Scenario: Curve mode selection affects visible fields +- **WHEN** user selects curve_mode "zipf_harmonic" +- **THEN** zipf_s and zipf_population_blocks fields SHALL be visible +- **WHEN** user selects curve_mode "linear" +- **THEN** zipf_s and zipf_population_blocks fields MAY be hidden or grayed out + +### Requirement: Deployment Configuration parameters +The form SHALL include editable fields for: label, accelerator_count, cards_per_machine, machine_spec, gpu_memory_gb_per_card, total_tps, total_tps_unit (dropdown), baseline_per_card_tps, planning_target_total_tps, and at least one extra_capacity_tier with label and kv_gb_per_machine. + +#### Scenario: Add extra capacity tier +- **WHEN** user clicks "Add Tier" button +- **THEN** a new tier row SHALL appear with label and kv_gb_per_machine fields + +#### Scenario: Remove extra capacity tier +- **WHEN** user clicks remove button on a tier row +- **THEN** that tier SHALL be removed and results recalculated + +### Requirement: Prefill savings alpha parameter +The form SHALL include an editable field for prefill_savings_alpha with default value 0.8. + +#### Scenario: Alpha value affects TPS gain +- **WHEN** user sets prefill_savings_alpha to 0.5 +- **THEN** all TPS gain values in results SHALL reflect the updated alpha + +### Requirement: Calculation parity with Python implementation +The JavaScript calculator SHALL produce hit rate values within 0.001 absolute tolerance of the Python `analyze_multi_agent_heuristic` function for identical inputs. + +#### Scenario: Preset config produces matching results +- **WHEN** calculator runs with the embedded Qwen3-27B preset +- **THEN** strict_prefix_hit_rate and lru_like_hit_rate SHALL match Python CLI output within 0.001 + +#### Scenario: Generalized harmonic sum correctness +- **WHEN** calculating zipf_harmonic curve with zipf_s=1.3 and population_blocks=4096 +- **THEN** the generalized harmonic partial sums SHALL match Python's `_generalized_harmonic` within float64 precision + +### Requirement: Results table display +The calculator SHALL display a results table showing per-tier rows with columns: Tier Label, Total KV GB, Total KV Tokens, Strict Prefix Hit Rate, LRU-like Hit Rate, Content Hit Rate, Strict Prefix TPS Gain, LRU-like TPS Gain, and bottleneck indicator. + +#### Scenario: Multiple tiers displayed +- **WHEN** deployment has HBM tier plus 2 extra capacity tiers +- **THEN** results table SHALL show 3 rows (HBM, tier1, tier2) with all columns populated + +#### Scenario: Hit rates displayed as percentages +- **WHEN** results are calculated +- **THEN** hit rates SHALL be displayed as percentages with 2 decimal places (e.g., "85.32%") + +### Requirement: Hit rate chart visualization +The calculator SHALL display a Chart.js line chart plotting hit rate (y-axis, 0-100%) versus total KV capacity in GB (x-axis) for both strict_prefix and lru_like policies. + +#### Scenario: Chart updates on recalculation +- **WHEN** user changes any parameter +- **THEN** the chart SHALL update to reflect new hit rate curves + +#### Scenario: Content ceiling line shown +- **WHEN** chart renders +- **THEN** a horizontal dashed line SHALL indicate the content_hit_rate ceiling + +### Requirement: Summary metrics panel +The calculator SHALL display key summary metrics: content_hit_rate, working_set_tokens, saturation_capacity_gb (strict and LRU), average_request_tokens, and kv_bytes_per_token. + +#### Scenario: Summary updates on recalculation +- **WHEN** any parameter changes +- **THEN** summary metrics SHALL reflect the recalculated values + +### Requirement: No server dependency +The calculator SHALL run entirely in the browser with no backend API calls, server-side processing, or database access. All computation MUST happen in client-side JavaScript. + +#### Scenario: Works from file:// protocol +- **WHEN** user opens calculator.html directly from filesystem +- **THEN** all functionality SHALL work (presets load, calculation runs, chart renders) + +### Requirement: Navigation integration +The existing dashboard page (`website/index.html`) SHALL include a visible link/button to the calculator page. + +#### Scenario: User navigates from dashboard to calculator +- **WHEN** user is on the results dashboard +- **THEN** a navigation element (link or button) SHALL be visible that opens calculator.html diff --git a/openspec/changes/add-web-ui/tasks.md b/openspec/changes/add-web-ui/tasks.md new file mode 100644 index 0000000..1a5c783 --- /dev/null +++ b/openspec/changes/add-web-ui/tasks.md @@ -0,0 +1,39 @@ +## 1. Core Calculation Engine (JavaScript) + +- [x] 1.1 Implement `generalizedHarmonic(n, s)` function matching Python's `_generalized_harmonic` +- [x] 1.2 Implement `shapeFraction(ratio, curveShape)` supporting linear, power_law_fit, and zipf_harmonic modes +- [x] 1.3 Implement `hitRateForCapacityTokens(capacityTokens, heuristic, efficiency)` matching Python logic +- [x] 1.4 Implement helper functions: `gbToTokens`, `tokensToGb`, `kvBytesPerToken`, `tpsGain`, `estimatedTotalTps`, `clusterCapacityTps` +- [x] 1.5 Implement `MultiAgentHeuristic` class with methods: `averageReusablePrivateTokensPerAgent`, `averageRequestTokens`, `contentHitRate`, `totalWorkingSetTokens`, `strictSaturationCapacityTokens`, `policySaturationCapacityTokens` +- [x] 1.6 Implement `analyzeMultiAgentHeuristic(config)` that produces tier rows and scenario summaries +- [x] 1.7 Validate JS output against Python preset config output (Qwen3-27B) to confirm parity within 0.001 + +## 2. Page Structure and Styling + +- [x] 2.1 Create `website/calculator.html` with HTML boilerplate, Chart.js CDN, and CSS matching existing dashboard theme (dark mode, same color tokens) +- [x] 2.2 Build page layout: header with nav link back to dashboard, main content with form panel and results panel side-by-side +- [x] 2.3 Style form sections (Model Profile, Multi-Agent Heuristic, Deployment, Analysis Settings) with card styling matching existing dashboard + +## 3. Parameter Input Form + +- [x] 3.1 Add preset configuration dropdown with embedded JSON presets (Qwen3-27B default, Qwen3-27B-H20) +- [x] 3.2 Build Model Profile form section: n_layers, kv_cache_layer_count, n_kv_heads, head_dim, dtype_bytes, parameter_count, weight_dtype_bytes, tp_size, pp_size, block_size +- [x] 3.3 Build Multi-Agent Heuristic form section: concurrent_agents, shared_prefix_tokens, avg_new_tokens_per_turn, avg_turns_per_session, private_window_tokens, curve_mode dropdown, zipf_s, zipf_population_blocks, policy efficiencies +- [x] 3.4 Build Deployment form section: label, accelerator_count, cards_per_machine, machine_spec, gpu_memory_gb_per_card, total_tps, total_tps_unit, baseline_per_card_tps, planning_target_total_tps +- [x] 3.5 Build extra capacity tiers UI with dynamic add/remove buttons +- [x] 3.6 Add prefill_savings_alpha input field with default 0.8 + +## 4. Results Display + +- [x] 4.1 Build summary metrics panel showing: content_hit_rate, working_set_tokens, saturation capacities, avg_request_tokens, kv_bytes_per_token +- [x] 4.2 Build results table with columns: Tier, Total KV GB, KV Tokens, Strict Prefix Hit Rate, LRU Hit Rate, Content Ceiling, TPS Gain (Strict), TPS Gain (LRU), Bottleneck +- [x] 4.3 Implement Chart.js line chart: hit rate vs. capacity GB with strict_prefix and lru_like lines plus content ceiling dashed line +- [x] 4.4 Add TPS capacity planning results (estimated total TPS, min cards/machines for target) + +## 5. Interactivity and Integration + +- [x] 5.1 Wire up debounced recalculation (100ms) on any form input change event +- [x] 5.2 Implement preset loading: populate all form fields when dropdown changes +- [x] 5.3 Handle curve_mode toggle: show/hide zipf-specific fields based on selection +- [x] 5.4 Add navigation link from existing `website/index.html` dashboard to calculator page +- [x] 5.5 Test full flow: load page, switch presets, modify parameters, verify chart and table update correctly diff --git a/openspec/config.yaml b/openspec/config.yaml new file mode 100644 index 0000000..392946c --- /dev/null +++ b/openspec/config.yaml @@ -0,0 +1,20 @@ +schema: spec-driven + +# Project context (optional) +# This is shown to AI when creating artifacts. +# Add your tech stack, conventions, style guides, domain knowledge, etc. +# Example: +# context: | +# Tech stack: TypeScript, React, Node.js +# We use conventional commits +# Domain: e-commerce platform + +# Per-artifact rules (optional) +# Add custom rules for specific artifacts. +# Example: +# rules: +# proposal: +# - Keep proposals under 500 words +# - Always include a "Non-goals" section +# tasks: +# - Break tasks into chunks of max 2 hours diff --git a/scripts/generate_test_vectors.py b/scripts/generate_test_vectors.py new file mode 100755 index 0000000..0f5d8a9 --- /dev/null +++ b/scripts/generate_test_vectors.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""Generate test vectors for JS parity validation. + +Requires Python 3.11+ (same as the main package). +Run: python3 scripts/generate_test_vectors.py + +Outputs: tests/test_vectors_heuristic.json +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +from kvcache_upper_bound.heuristic import ( + analyze_multi_agent_heuristic, + load_multi_agent_heuristic_config, +) + + +def extract_vector(config_path: str, preset_name: str) -> dict: + config = load_multi_agent_heuristic_config(config_path) + result = analyze_multi_agent_heuristic(config) + + summary = result.scenario_summaries[0] + tier_rows = [ + { + "tier_label": row.tier_label, + "total_kv_gb": row.total_kv_gb, + "total_kv_tokens": row.total_kv_tokens, + "strict_prefix_hit_rate": row.strict_prefix_hit_rate, + "lru_like_hit_rate": row.lru_like_hit_rate, + "content_hit_rate": row.content_hit_rate, + "bottleneck": row.current_bottleneck, + "strict_tps_gain": row.strict_prefix_tps_gain, + "lru_tps_gain": row.lru_like_tps_gain, + "strict_estimated_total_tps": row.strict_prefix_estimated_total_tps, + "lru_estimated_total_tps": row.lru_like_estimated_total_tps, + "strict_cluster_capacity_tps": row.strict_prefix_current_cluster_capacity_tps, + "strict_min_cards": row.strict_prefix_min_card_count_for_target_total_tps, + "strict_min_machines": row.strict_prefix_min_machine_count_for_target_total_tps, + "lru_cluster_capacity_tps": row.lru_like_current_cluster_capacity_tps, + "lru_min_cards": row.lru_like_min_card_count_for_target_total_tps, + "lru_min_machines": row.lru_like_min_machine_count_for_target_total_tps, + } + for row in result.tier_rows + ] + + return { + "name": preset_name, + "config_path": config_path, + "summary": { + "content_hit_rate": summary.content_hit_rate, + "average_request_tokens": summary.average_request_tokens, + "total_working_set_tokens": summary.total_working_set_tokens, + "avg_reusable_private_tokens_per_agent": summary.avg_reusable_private_tokens_per_agent, + "hbm_kv_gb_per_card": summary.hbm_kv_gb_per_card, + "strict_prefix_saturation_capacity_gb": summary.strict_prefix_saturation_capacity_gb, + "lru_like_saturation_capacity_gb": summary.lru_like_saturation_capacity_gb, + "hbm_strict_prefix_hit_rate": summary.hbm_strict_prefix_hit_rate, + "hbm_lru_like_hit_rate": summary.hbm_lru_like_hit_rate, + }, + "tier_rows": tier_rows, + } + + +def main(): + configs = [ + ("configs/public_multi_agent_qwen3_5_27b.json", "qwen3-27b-8xh20"), + ("configs/public_multi_agent_qwen3_5_27b_1x1_h20.json", "qwen3-27b-1xh20"), + ] + + vectors = [] + for config_path, name in configs: + print(f"Generating vector: {name} ({config_path})") + vectors.append(extract_vector(config_path, name)) + + output_path = Path(__file__).resolve().parent.parent / "tests" / "test_vectors_heuristic.json" + output_path.write_text(json.dumps(vectors, indent=2) + "\n") + print(f"\nWritten {len(vectors)} test vectors to {output_path}") + + +if __name__ == "__main__": + main() diff --git a/scripts/validate_web.sh b/scripts/validate_web.sh new file mode 100755 index 0000000..eb9cdf9 --- /dev/null +++ b/scripts/validate_web.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# Validate the web calculator's JS engine against Python-generated test vectors. +# Usage: ./scripts/validate_web.sh +# +# To regenerate test vectors (requires Python 3.11+): +# python3 scripts/generate_test_vectors.py + +set -e +cd "$(dirname "$0")/.." + +echo "Running JS parity validation..." +node tests/validate_js_parity.mjs diff --git a/tests/test_vectors_heuristic.json b/tests/test_vectors_heuristic.json new file mode 100644 index 0000000..bb3562e --- /dev/null +++ b/tests/test_vectors_heuristic.json @@ -0,0 +1,256 @@ +[ + { + "name": "qwen3-27b-8xh20", + "summary": { + "content_hit_rate": 0.8181818181818182, + "average_request_tokens": 22528.0, + "total_working_set_tokens": 29364224.0, + "avg_reusable_private_tokens_per_agent": 14336.0, + "hbm_kv_gb_per_card": 89.53163421526551, + "strict_prefix_saturation_capacity_gb": 1792.25, + "lru_like_saturation_capacity_gb": 2986.916666666667, + "hbm_strict_prefix_hit_rate": 0.8030240690907123, + "hbm_lru_like_hit_rate": 0.7925829610824396 + }, + "tier_rows": [ + { + "tier_label": "HBM", + "total_kv_gb": 716.2530737221241, + "total_kv_tokens": 11735090.359863281, + "strict_prefix_hit_rate": 0.8030240690907123, + "lru_like_hit_rate": 0.7925829610824396, + "content_hit_rate": 0.8181818181818182, + "bottleneck": "Capacity", + "strict_tps_gain": 2.796571165380455, + "lru_tps_gain": 2.7327359797484196, + "strict_estimated_total_tps": 22.37256932304364, + "lru_estimated_total_tps": 21.861887837987357, + "strict_cluster_capacity_tps": 22.37256932304364, + "strict_min_cards": 8, + "strict_min_machines": 1, + "lru_cluster_capacity_tps": 21.861887837987357, + "lru_min_cards": 8, + "lru_min_machines": 1 + }, + { + "tier_label": "HBM+1T per Machine", + "total_kv_gb": 1740.253073722124, + "total_kv_tokens": 28512306.35986328, + "strict_prefix_hit_rate": 0.8177569143402311, + "lru_like_hit_rate": 0.8097647414691859, + "content_hit_rate": 0.8181818181818182, + "bottleneck": "Capacity", + "strict_tps_gain": 2.891891256263869, + "lru_tps_gain": 2.839390929685172, + "strict_estimated_total_tps": 23.135130050110952, + "lru_estimated_total_tps": 22.715127437481375, + "strict_cluster_capacity_tps": 23.135130050110952, + "strict_min_cards": 8, + "strict_min_machines": 1, + "lru_cluster_capacity_tps": 22.715127437481375, + "lru_min_cards": 8, + "lru_min_machines": 1 + }, + { + "tier_label": "HBM+10T per Machine", + "total_kv_gb": 10956.253073722124, + "total_kv_tokens": 179507250.35986328, + "strict_prefix_hit_rate": 0.8181818181818182, + "lru_like_hit_rate": 0.8181818181818182, + "content_hit_rate": 0.8181818181818182, + "bottleneck": "None", + "strict_tps_gain": 2.894736842105264, + "lru_tps_gain": 2.894736842105264, + "strict_estimated_total_tps": 23.157894736842113, + "lru_estimated_total_tps": 23.157894736842113, + "strict_cluster_capacity_tps": 23.157894736842113, + "strict_min_cards": 8, + "strict_min_machines": 1, + "lru_cluster_capacity_tps": 23.157894736842113, + "lru_min_cards": 8, + "lru_min_machines": 1 + } + ] + }, + { + "name": "qwen3-27b-1xh20", + "summary": { + "content_hit_rate": 0.8181818181818182, + "average_request_tokens": 22528.0, + "total_working_set_tokens": 29364224.0, + "avg_reusable_private_tokens_per_agent": 14336.0, + "hbm_kv_gb_per_card": 44.2530737221241, + "strict_prefix_saturation_capacity_gb": 1792.25, + "lru_like_saturation_capacity_gb": 2986.916666666667, + "hbm_strict_prefix_hit_rate": 0.7205355232805011, + "hbm_lru_like_hit_rate": 0.6966110065110587 + }, + "tier_rows": [ + { + "tier_label": "HBM", + "total_kv_gb": 44.2530737221241, + "total_kv_tokens": 725042.3598632812, + "strict_prefix_hit_rate": 0.7205355232805011, + "lru_like_hit_rate": 0.6966110065110587, + "content_hit_rate": 0.8181818181818182, + "bottleneck": "Capacity", + "strict_tps_gain": 2.360876045442853, + "lru_tps_gain": 2.2588089295364338, + "strict_estimated_total_tps": 2.360876045442853, + "lru_estimated_total_tps": 2.2588089295364338, + "strict_cluster_capacity_tps": 2.360876045442853, + "strict_min_cards": 4, + "strict_min_machines": 4, + "lru_cluster_capacity_tps": 2.2588089295364338, + "lru_min_cards": 4, + "lru_min_machines": 4 + }, + { + "tier_label": "HBM+1T per Machine", + "total_kv_gb": 1068.253073722124, + "total_kv_tokens": 17502258.35986328, + "strict_prefix_hit_rate": 0.8101479237150163, + "lru_like_hit_rate": 0.8008902630303357, + "content_hit_rate": 0.8181818181818182, + "bottleneck": "Capacity", + "strict_tps_gain": 2.8418644980775656, + "lru_tps_gain": 2.7832841221263327, + "strict_estimated_total_tps": 2.8418644980775656, + "lru_estimated_total_tps": 2.7832841221263327, + "strict_cluster_capacity_tps": 2.8418644980775656, + "strict_min_cards": 3, + "strict_min_machines": 3, + "lru_cluster_capacity_tps": 2.7832841221263327, + "lru_min_cards": 3, + "lru_min_machines": 3 + }, + { + "tier_label": "HBM+10T per Machine", + "total_kv_gb": 10284.253073722124, + "total_kv_tokens": 168497202.35986328, + "strict_prefix_hit_rate": 0.8181818181818182, + "lru_like_hit_rate": 0.8181818181818182, + "content_hit_rate": 0.8181818181818182, + "bottleneck": "None", + "strict_tps_gain": 2.894736842105264, + "lru_tps_gain": 2.894736842105264, + "strict_estimated_total_tps": 2.894736842105264, + "lru_estimated_total_tps": 2.894736842105264, + "strict_cluster_capacity_tps": 2.894736842105264, + "strict_min_cards": 3, + "strict_min_machines": 3, + "lru_cluster_capacity_tps": 2.894736842105264, + "lru_min_cards": 3, + "lru_min_machines": 3 + } + ] + }, + { + "name": "edge-linear-small", + "summary": { + "content_hit_rate": 0.7777777777777778, + "average_request_tokens": 4608.0, + "total_working_set_tokens": 788480.0, + "avg_reusable_private_tokens_per_agent": 1536.0, + "hbm_kv_gb_per_card": 66.9614839553833, + "strict_prefix_saturation_capacity_gb": 96.25, + "lru_like_saturation_capacity_gb": 137.39285714285714, + "hbm_strict_prefix_hit_rate": 0.6760815415117476, + "hbm_lru_like_hit_rate": 0.6065904123915566 + }, + "tier_rows": [ + { + "tier_label": "HBM", + "total_kv_gb": 66.9614839553833, + "total_kv_tokens": 548548.4765625, + "strict_prefix_hit_rate": 0.6760815415117476, + "lru_like_hit_rate": 0.6065904123915566, + "content_hit_rate": 0.7777777777777778, + "bottleneck": "Capacity", + "strict_tps_gain": 1.89845930575028, + "lru_tps_gain": 1.737961583602824, + "strict_estimated_total_tps": 3.79691861150056, + "lru_estimated_total_tps": 3.475923167205648, + "strict_cluster_capacity_tps": 3.79691861150056, + "strict_min_cards": 4, + "strict_min_machines": 4, + "lru_cluster_capacity_tps": 3.475923167205648, + "lru_min_cards": 4, + "lru_min_machines": 4 + }, + { + "tier_label": "HBM+512G", + "total_kv_gb": 578.9614839553833, + "total_kv_tokens": 4742852.4765625, + "strict_prefix_hit_rate": 0.7777777777777778, + "lru_like_hit_rate": 0.7777777777777778, + "content_hit_rate": 0.7777777777777778, + "bottleneck": "None", + "strict_tps_gain": 2.195121951219512, + "lru_tps_gain": 2.195121951219512, + "strict_estimated_total_tps": 4.390243902439024, + "lru_estimated_total_tps": 4.390243902439024, + "strict_cluster_capacity_tps": 4.390243902439024, + "strict_min_cards": 4, + "strict_min_machines": 4, + "lru_cluster_capacity_tps": 4.390243902439024, + "lru_min_cards": 4, + "lru_min_machines": 4 + } + ] + }, + { + "name": "edge-saturated", + "summary": { + "content_hit_rate": 0.8333333333333334, + "average_request_tokens": 12288.0, + "total_working_set_tokens": 139264.0, + "avg_reusable_private_tokens_per_agent": 2048.0, + "hbm_kv_gb_per_card": 89.53163421526551, + "strict_prefix_saturation_capacity_gb": 8.5, + "lru_like_saturation_capacity_gb": 10.5, + "hbm_strict_prefix_hit_rate": 0.8333333333333334, + "hbm_lru_like_hit_rate": 0.8333333333333334 + }, + "tier_rows": [ + { + "tier_label": "HBM", + "total_kv_gb": 2865.0122948884964, + "total_kv_tokens": 46940361.439453125, + "strict_prefix_hit_rate": 0.8333333333333334, + "lru_like_hit_rate": 0.8333333333333334, + "content_hit_rate": 0.8333333333333334, + "bottleneck": "None", + "strict_tps_gain": 3.000000000000001, + "lru_tps_gain": 3.000000000000001, + "strict_estimated_total_tps": 96.00000000000003, + "lru_estimated_total_tps": 96.00000000000003, + "strict_cluster_capacity_tps": 96.00000000000003, + "strict_min_cards": 16, + "strict_min_machines": 2, + "lru_cluster_capacity_tps": 96.00000000000003, + "lru_min_cards": 16, + "lru_min_machines": 2 + }, + { + "tier_label": "HBM+2T", + "total_kv_gb": 11057.012294888496, + "total_kv_tokens": 181158089.43945312, + "strict_prefix_hit_rate": 0.8333333333333334, + "lru_like_hit_rate": 0.8333333333333334, + "content_hit_rate": 0.8333333333333334, + "bottleneck": "None", + "strict_tps_gain": 3.000000000000001, + "lru_tps_gain": 3.000000000000001, + "strict_estimated_total_tps": 96.00000000000003, + "lru_estimated_total_tps": 96.00000000000003, + "strict_cluster_capacity_tps": 96.00000000000003, + "strict_min_cards": 16, + "strict_min_machines": 2, + "lru_cluster_capacity_tps": 96.00000000000003, + "lru_min_cards": 16, + "lru_min_machines": 2 + } + ] + } +] diff --git a/tests/validate_js_parity.mjs b/tests/validate_js_parity.mjs new file mode 100644 index 0000000..c44fd8b --- /dev/null +++ b/tests/validate_js_parity.mjs @@ -0,0 +1,168 @@ +#!/usr/bin/env node +/** + * Validates the JavaScript calculator engine against Python-generated test vectors. + * + * Usage: node tests/validate_js_parity.mjs + * + * Extracts the calculation engine from website/calculator.html, + * runs it against test vectors in tests/test_vectors_heuristic.json, + * and reports any divergence beyond tolerance (0.001 for rates, 0.01 for GB). + */ +import { readFileSync } from "fs"; +import { createContext, Script } from "vm"; +import { fileURLToPath } from "url"; +import { dirname, join } from "path"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const ROOT = join(__dirname, ".."); + +// Extract the JS engine portion from calculator.html (everything between + + + + +
+
+ +
Interactive Multi-Agent Heuristic Estimator
+
+ +
+ + +
+
+
+ +
+
+ +
+ +
+ + +
+
Model Preset
+
+
+ + +
+
+ + +
+
+
+ + +
+
Configuration Preset
+
+
+ + +
+
+
+ + +
+
Model Profile
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + +
+
Multi-Agent Heuristic
+
+
+
+
+
+
+
+ + +
+
+
+
+
+
+
+ + +
+
Deployment
+
+
+
+
+
+
+
+ + +
+
+
+
+
+
+ + +
+
Extra Capacity Tiers
+
+ +
+ + +
+
Analysis Settings
+
+
+
+
+
+ +
+ +
+
+ + + + + +
+
+ + + + + +
+
Summary Metrics
+
+
+ + +
+
Hit Rate vs. KV Capacity
+
+ +
+
+ + +
+
TPS vs. Machine Count
+
+ +
+
+ + +
+
Parameter Sensitivity
+
+
+ + +
+
Tier Analysis
+
+ + + +
+
+
+ + +
+
TPS Capacity Planning
+
+ + + +
+
+
+
+
+ + + + diff --git a/website/index.html b/website/index.html index 2e70a5b..34e293b 100644 --- a/website/index.html +++ b/website/index.html @@ -302,6 +302,7 @@
Results Dashboard
+