Skip to content

Commit 1de2e8d

Browse files
kmbandyclaude
andcommitted
docs(sp2.5): multi-bracket results matrix + final-review pointer
Consolidates op-level/e2e-parity/perf/footprint for all three head_dim brackets (64/128/256) into one table, carries forward every task's Minor findings for the final whole-branch review, and points that review at 2d96f28..HEAD (SP2 + SP2.5). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZRfPpL8XFzk1hep9MMg9P
1 parent 7f7fc7d commit 1de2e8d

1 file changed

Lines changed: 50 additions & 3 deletions

File tree

docs/superpowers/plans/2026-06-30-sp2-multibracket-paged-turbo-vulkan.md

Lines changed: 50 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -699,9 +699,9 @@ The diagnostic tooling (`examples/pagedattn-dump/`, `examples/pagedattn-repro/`)
699699
**Files:**
700700
- Modify: `docs/superpowers/plans/2026-06-30-sp2-multibracket-paged-turbo-vulkan.md`, `.superpowers/sdd/progress.md`
701701

702-
- [ ] **Step 1: Write the consolidated bracket matrix** (head_dim 64/128/256 × {op-level CUDA-equivalence, e2e PPL parity, perf}) into this plan's Results section, with the carried Minor findings from each task for the final whole-branch review to triage.
703-
- [ ] **Step 2: Mark SP2.5 complete in the ledger** with the commit range and point the final whole-branch review at `2d96f287b..HEAD` (covers SP2 + SP2.5).
704-
- [ ] **Step 3: Commit.**
702+
- [x] **Step 1: Write the consolidated bracket matrix** (head_dim 64/128/256 × {op-level CUDA-equivalence, e2e PPL parity, perf}) into this plan's Results section, with the carried Minor findings from each task for the final whole-branch review to triage.
703+
- [x] **Step 2: Mark SP2.5 complete in the ledger** with the commit range and point the final whole-branch review at `2d96f287b..HEAD` (covers SP2 + SP2.5).
704+
- [x] **Step 3: Commit.**
705705

706706
```bash
707707
git add docs/superpowers/plans/2026-06-30-sp2-multibracket-paged-turbo-vulkan.md .superpowers/sdd/progress.md
@@ -710,6 +710,53 @@ git commit -m "docs(sp2.5): multi-bracket results matrix + final-review pointer"
710710

711711
---
712712

713+
## Results — Consolidated bracket matrix (Task 6)
714+
715+
All three native head_dim brackets for `GGML_OP_PAGED_ATTN_MT` on Vulkan, one row per bracket. "Op-level"
716+
= `tests/test-paged-attn-vk.cpp` dual-backend harness (Vulkan0 vs CUDA0, synthetic/host-prefilled inputs).
717+
"e2e PPL parity" = full `wiki.test.raw` `llama-perplexity`, Vulkan0 vs CUDA0, combined-stderr sigma.
718+
"Perf" = `llama-bench -p 512 -n 128`, Vulkan0, paged path confirmed active via verbose load log.
719+
720+
| head_dim | Model used for e2e | Cache type(s) | Op-level (Vulkan0 vs CUDA0) | e2e PPL parity (σ, full corpus) | Perf (pp512 / tg128, Vulkan0) | Footprint vs F16 |
721+
| --- | --- | --- | --- | --- | --- | --- |
722+
| 128 | LFM2.5-8B-A1B (padded fallback, `GGML_PAGED_TURBO4_64=0`) | F16, turbo4_0 | PASS (SP2 Tasks 3-5, tol 2e-3/5e-2) | ~2.6σ (pre-existing, see note below — not a hd128/turbo4_0 defect) | pp512 546.73±6.50, tg128 81.73±0.80 | turbo4_0 ≈ 68.3 B/head (½ of F16) |
723+
| 256 | Qwen3.5-4B-UD-Q4_K_XL | F16, turbo4_0 (2 blocks/head) | PASS (Task 1, tol 2e-3/5e-2) | **0.22σ** (10.1259±0.07238 vs 10.1484±0.07257) | pp512 280.55±4.07, tg128 34.98±0.29 | turbo4_0 ≈ 68.3 B/head (¼ of F16 at this head_dim) |
724+
| 64 | LFM2.5-8B-A1B (native) | turbo4_64, turbo4_64_ol/_ol8/_ol12, Q8_0 | PASS, 38/38 (Tasks 3-4 + outlier work, tol 5e-2) | turbo4_64 baseline **~2.4-2.6σ** (root-caused, see below); **turbo4_64_ol8 0.45σ — shipped fix** | pp512 553.69±2.71, tg128 87.44±2.00 (native turbo4_64, at parity/better than padded-128) | turbo4_64 = 34.1 B/head (**½** of padded-128 turbo4_0); turbo4_64_ol8 = 46 B/head (+35% vs turbo4_64, still 33% under F16) |
725+
726+
**Note on the hd64/hd128 PPL gap (turbo4_64 baseline and padded-128, both ~2.4-2.6σ):** this is NOT a
727+
head_dim-64 or turbo4_64-specific defect — it is a real, root-caused property of LFM2.5 specifically
728+
(ordinary cross-vendor floating-point noise, amplified by softmax's sensitivity to small head_dim, further
729+
amplified by 4-bit nearest-centroid quantization interacting with LFM2.5's real massive-activation outlier
730+
channels). It reproduces identically on the already-merged, unmodified padded-128 path (zero turbo4_64 code
731+
involved), and is fully explained and fixed for the turbo4_64 case (see "Post-Task-5 investigation" above,
732+
`.superpowers/sdd/paged-attn-layer-divergence-report.md`). **`turbo4_64_ol8` is the recommended cache type
733+
for any head_dim-64 hybrid-attention model wanting tight cross-backend parity**; the hd128 padded-fallback
734+
path is legacy/interim (superseded by native turbo4_64 for real head_dim-64 models) and carries no
735+
independent fix in this branch — it is out of scope, flagged for the final whole-branch review as a known,
736+
explained, non-blocking characteristic.
737+
738+
**Minor findings carried from every task, for the final whole-branch review to triage:**
739+
- Task 1 (turbo_wht.comp, SP1): `gs=32` comment in a push-constant describes an unreachable code path (supports_op only admits {64,128}). Cosmetic.
740+
- Task 2 (cpy_f32_turbo4_0.comp, SP1): CUDA `TURBO_MID_4BIT` thresholds differ from the CPU oracle's hardcoded midpoints by ~1e-6 at idx 0,6,8,14 — correct for CUDA-parity, watch-item only.
741+
- Task 2 (SP1): `wg_denoms={1,1,1}` deserves a clarifying comment; unused `param1`/`param2` push-const fields; no non-contiguous-source CPY test.
742+
- Task 3 (types.glsl, SP1): `block_turbo4_0_packed16` lacks a comment noting it's FA-only; asymmetric FA combos (K=turbo4_0, V=q4_0) untested; only kv=96 tested for turbo4_0 FA.
743+
- SP2 Task 3: `supports_op` omits an explicit `n_heads % n_kv_heads == 0` check (CUDA asserts it); multi-seq / invalid-block paths verified by reading, not tested; dead `q_reg`/`acc_o` MAX_VEC slots at HS=128.
744+
- SP2 Task 4: scatter dispatch grid omits `*N_QBLK` on the y-dim (correct only because SP2 pinned head_dim==128 — now genuinely exercised by SP2.5's hd256/hd64 brackets, which DO set N_QBLK correctly per Task 1/3 changes — reviewer should confirm the grid fix actually landed for those brackets, not just re-flag the original note). Comment-numbering swap in the scatter shader (cosmetic). Scatter-readback oracle only validates K, not V (adequate, no independent V check).
745+
- SP2 Task 5: `paged_attn_decode.comp` reads an unused `q_len` local; `compare_paged_case` doesn't check CUDA op support explicitly; CUDA's internal q_len==1 decode-vs-tile routing unverified from the diff; multi-seq (n_seq>1) decode untested.
746+
- SP2.5 Task 3+4 (turbo4_64): two cosmetic-only notes (see task log), no Important/Critical.
747+
- SP2.5 outlier work (turbo4_64_ol/_ol8/_ol12, Q8_0): CPU-side type-trait stubs are unsupported/abort-only (matches the existing turbo4_64 CPU-stub convention — hot path is CUDA/Vulkan-only); worth a final-review sanity check that this convention is acceptable for merge (it was never in question for turbo4_64 itself, extending it to 4 more types is the only new surface).
748+
749+
**Diagnostic-only files, deliberately left uncommitted (not for merge):** `examples/pagedattn-dump/`,
750+
`examples/pagedattn-repro/`, the `examples/CMakeLists.txt` `add_subdirectory` lines for them, and the small
751+
`ggml_format_name` instrumentation hunk in `src/llama-graph.cpp`. Safe to discard, or the final whole-branch
752+
review can decide whether any of this tooling is worth keeping for future similar investigations.
753+
754+
**Final whole-branch review scope:** `2d96f287b..HEAD` (SP2 + SP2.5 combined — SP2 established the paged
755+
op and the hd128 bracket, SP2.5 generalized to hd256 and added the native hd64 turbo4_64 bracket plus the
756+
outlier-channel fix). This is the range for `superpowers:requesting-code-review`'s final reviewer dispatch.
757+
758+
---
759+
713760
## Self-Review notes
714761

715762
- **Spec coverage:** Feature 1 (hd256 generalization) = Tasks 1-2; Feature 2 (turbo4_64) = Tasks 3-5; results/parity per bracket = Tasks 2/5/6. F16 test variant covered (hd256 F16 case Task 1; hd128 F16 pre-existing). All three brackets get op-level CUDA-equivalence; hd256 + hd64 get e2e; hd128 e2e is the existing padded-LFM2.5 run (noted in spec).

0 commit comments

Comments
 (0)