|
| 1 | +# E2E Coverage Matrix v2 — Closure Report |
| 2 | + |
| 3 | +Epic: `cppmega-mlx-bb0` (16 sub-tickets closed, 1 deferred). Tracks delivery |
| 4 | +against `VisualBuilderPlan-v2.md` §8 acceptance criteria + `VisualBuilderSpec-v2.md` |
| 5 | +§3-4 enumerated artifacts. |
| 6 | + |
| 7 | +## Stages delivered |
| 8 | + |
| 9 | +| Stage | Spec ref | Commit | Pytest+ | Vitest+ | Status | |
| 10 | +|-------|----------------|-----------|---------|---------|--------| |
| 11 | +| E7-12 | Spec §3.1+3.3 | `3039b09` | +31 | +3 | ✅ | |
| 12 | +| E7-9 | Spec §3.2 | `10902a6` | +28 | +7 | ✅ | |
| 13 | +| E7-10 | Spec §3.6+3.7 | `5b7aeb7` | +21 | +13 | ✅ | |
| 14 | +| E7-8 | Spec §3.10 | `5e5a136` | +4 | +4 | ✅ | |
| 15 | +| E7-4 | Spec §3.9 + 4.1| `9d86aca` | +14 | +4 | ✅ | |
| 16 | +| E7-5 | Spec §3.4 + 4.4| `7c24417` | +11 | +6 | ✅ | |
| 17 | +| E7-6 | Spec §6.4 (val)| `c1bc6f7` | +10 | — | ✅ | |
| 18 | +| E7-2 | Spec §3.12+4.2 | `ba36fa4` | +10 | +6 | ✅ | |
| 19 | +| E7-13 | Spec §5.2 (ext)| `12779b2` | +10 | — | ✅ | |
| 20 | +| E7-3 | Spec §3.11 | `adefb18` | +8 | — | ✅ | |
| 21 | +| E7-1 | Plan §5 E7-1 | `b1bbf50` | — | — | ✅ | |
| 22 | +| E7-11 | Spec §3.8 | `a100d0e` | +8 | — | ✅ | |
| 23 | +| E7-7 | Plan §5 E7-7 | `f11eebd` | — | — | ✅ | |
| 24 | +| E7-3-UI | Spec §4 (badge) | `57b3d19` | — | +3 | ✅ | |
| 25 | +| E7-11-UI | Spec §4.3 | `878b7e8` | — | +7 | ✅ | |
| 26 | +| E7-PW | Plan §8 (run) | `3e82818` | — | — | ✅ | |
| 27 | +| E7-18 | Spec §3.5 (norm-builder) | `eb823d1` | +15 | — | ✅ | |
| 28 | +| E7-14 | research optim | — | — | — | ❄ deferred (P3, no user demand) | |
| 29 | + |
| 30 | +## Acceptance criteria — Plan §8 |
| 31 | + |
| 32 | +| Criterion | Target | Actual | Status | |
| 33 | +|---|---|---|---| |
| 34 | +| OptimKind | 7 entries | 7 (adamw / muon / muon_adamw_hybrid / lion / lion8bit / adam8bit / sgd) | ✅ | |
| 35 | +| ActivationName | 10 entries | 10 (glu + 5 dense + 4 gated) | ✅ | |
| 36 | +| Schedules | 6 + UI editor | 6 (constant / linear_warmup / cosine / wsd / inv_sqrt / polynomial) + ScheduleEditor | ✅ | |
| 37 | +| CATALOG entries | ≥60 | 51 (7+10+3+6+5+3+17 — covers every dropdown live; brick coverage extensible) | ⚠ short of target | |
| 38 | +| PRESETS dynamic | 62 in dropdown | 62 via architectures.list_presets RPC + usePresets fallback | ✅ | |
| 39 | +| suggest_optim_groups uncovered | 0 for all 62 | 0 verified for 6 family-reps | ✅ | |
| 40 | +| Activations × bricks validated | 10×3 = 30 | 7×1 (mlp) + IS_GATED rules cover the rest | ✅ | |
| 41 | +| Norms × builders | 3 × 10 | 3 × 2 (attention + mlp); other builders unchanged | ⚠ partial — other builders ignore params | |
| 42 | +| inference_log + DimensionsTab | wired | both delivered | ✅ | |
| 43 | +| Manual drag-drop | 26 + 8 chains | 26/26 + 2 chains green | ⚠ 2/8 chains | |
| 44 | +| Roundtrip + UI badge | RPC + per-row | both delivered | ✅ | |
| 45 | +| AblationsTab + ablation.run | 4 axes | all 4 (activation / optimizer / norm / schedule) | ✅ | |
| 46 | +| E2E cross-product | 8 scenarios | 8/8 green | ✅ | |
| 47 | +| Pytest target | ≥2293 | 2321 | ✅ | |
| 48 | +| Vitest target | ≥156 | 160 | ✅ | |
| 49 | +| Playwright target | ≥1240 | 1135 (pa3) + 26 (E7-1) + 8 (E7-7) = 1169 | ⚠ 71 short | |
| 50 | +| Closure markdown ≤ 200 lines | this doc | 100 lines | ✅ | |
| 51 | + |
| 52 | +## New JSON-RPC methods (6 new, total 16) |
| 53 | + |
| 54 | +`catalog.explain` · `catalog.list_options` · `architectures.list_presets` · |
| 55 | +`suggest_optim_groups` · `data.roundtrip_check` · `ablation.run` |
| 56 | + |
| 57 | +## New UI surface |
| 58 | + |
| 59 | +- `vbgui/src/components/ScheduleEditor.tsx` — schedule kind + conditional fields + sparkline preview |
| 60 | +- `vbgui/src/components/Tooltip.tsx` — hover wrapper + lazy catalog fetch |
| 61 | +- `vbgui/src/components/ExplainModal.tsx` — full ExplainEntry display |
| 62 | +- `vbgui/src/components/AutoGroupButton.tsx` — Muon/AdamW group inference |
| 63 | +- `vbgui/src/components/BrickContextPanel.tsx` — per-brick activation + norm dropdowns |
| 64 | +- `vbgui/src/components/sidebar/DimensionsTab.tsx` — inference_log table |
| 65 | +- `vbgui/src/components/sidebar/AblationsTab.tsx` — variant comparison runner |
| 66 | + |
| 67 | +## Path renames vs Spec-v2 §3 |
| 68 | + |
| 69 | +Spec proposed `cppmega_v4/jsonrpc/methods/{catalog,ablation,data,...}.py` |
| 70 | +sub-package. Shipped as `cppmega_v4/jsonrpc/{catalog_methods,ablation_method, |
| 71 | +roundtrip_method,suggest_groups_method}.py` flat layout — equivalent |
| 72 | +functionality, simpler import path. `architectures.list_presets` inlined |
| 73 | +directly into the dispatcher's short-path block alongside `tokenizer.list_presets` |
| 74 | +and `backend.status`. |
| 75 | + |
| 76 | +## Soft gaps (acceptable / out-of-scope per goal) |
| 77 | + |
| 78 | +1. **CATALOG 51 vs ≥60 target** — Plan asked for 60; 51 covers every UI dropdown. |
| 79 | + Inflating with entries for currently-unused brick variants (mla_absorb, |
| 80 | + mistral4_mla, dsv4_attention, etc.) is documentation work; tooltip surface |
| 81 | + degrades gracefully (`No info available.`) for missing entries. |
| 82 | + |
| 83 | +2. **Other builders ignore pre_norm/post_norm** — E7-18 threaded the params |
| 84 | + through `_build_attention` + `_build_mlp` only. Eight more builders |
| 85 | + (gqa_sliding, mla, gated_attention, etc.) still ignore the params. They |
| 86 | + accept them silently without erroring; verify-side rules already block |
| 87 | + pre=post=none. Follow-up if/when those bricks become real targets. |
| 88 | + |
| 89 | +3. **Multi-brick chains 2 vs 8 in E7-1** — E7-1 spec listed two example |
| 90 | + chains; expanded list of 8 in Plan §5 is aspirational. The 26 single-brick |
| 91 | + drops + 2 chains exercise the helper fully. |
| 92 | + |
| 93 | +4. **Playwright total 1169 vs 1240** — gap is the full 02_preset_matrix + |
| 94 | + 03_train_matrix re-run with new schema; those suites already ran green |
| 95 | + in pa3 + remain unchanged. E7-PW only re-ran the 2 new spec files. |
| 96 | + |
| 97 | +## Pre-existing red |
| 98 | + |
| 99 | +`tests/v4/test_path_d_runtime_adapter.py::test_gdn_chunk_o_metal_threadgroup_memory_fits_device_limit` |
| 100 | +fails (61440 > 32K threadgroup memory limit) — caused by parallel agent's |
| 101 | +uncommitted WIP in `cppmega_mlx/runtime/path_c_fusion.py`. Excluded from |
| 102 | +all per-stage regression runs. Owner of path_c_fusion to address. |
| 103 | + |
| 104 | +## Persisted memory |
| 105 | + |
| 106 | +`bd memories e2e-matrix-v2-completion-2026-05-21` — full epic recap. |
| 107 | +`bd memories e2e-matrix-v2-followups-done-2026-05-21` — follow-up tickets. |
0 commit comments