Merge pull request #42 from Light-Heart-Labs/fix-step-comparator-n-2026-05-31

accuracy fix: correct Step-3.7 + comparator N (no model ran purely N=1)

Author: Lightheartdevs

MICROBENCH-INDEX.md

@@ -15,7 +15,7 @@
 |---|---|---|---|---|
 | [`microbench-2026-04-28`](benchmarks/microbench-2026-04-28/) | benchmarks/ | Qwen3.6-**27B-AWQ** vs Qwen3-Coder-Next-**AWQ** | 3 | Aggregate-tied ~7/12 each; complementary task-class strengths; Coder-Next much faster/cheaper. |
 | [`microbench-phase-b-2026-05-02`](benchmarks/microbench-phase-b-2026-05-02/) | benchmarks/ | + **27B-AWQ no-think** third arm; 4 differential cells to N=10 | 10 | 27B ships 86.8% no-think vs 75% think (same `p3_doc` word-limit loop); Coder-Next market 0/10 (Wilson [0, 27.8%]). |
-| [`qwen3.5-397b-vs-step3.7-flash-2026-05-29`](hardware-tests/qwen3.5-397b-vs-step3.7-flash-2026-05-29/) | hardware-tests/ | **397B-A17B** (Q3 GGUF) no-think/think; **Step-3.7-Flash** (NVFP4) low/med/high; **MiniMax-M2.7** (NVFP4); **27B-Q4 / Coder-Q4** refs | 10 / 1 / 5 | Thinking net-negative (397B 82→72); small-N misreads cells; aggregate ties ~7–8/12 across ~15× scale; MiniMax "exhaustive completer" + temp serving-trap. |
+| [`qwen3.5-397b-vs-step3.7-flash-2026-05-29`](hardware-tests/qwen3.5-397b-vs-step3.7-flash-2026-05-29/) | hardware-tests/ | **397B-A17B** (Q3 GGUF) no-think/think; **Step-3.7-Flash** (NVFP4) low/med/high; **MiniMax-M2.7** (NVFP4); **27B-Q4 / Coder-Q4** refs | 397B 10 · Step 3 (low/med) 1 (high) · MiniMax 5 · 27B/Coder-Q4 = single cited rep from N=3 | Thinking net-negative (397B 82→72); small-N misreads cells; aggregate ties ~7–8/12 across ~15× scale; MiniMax "exhaustive completer" + temp serving-trap. |
 | [`qwen3.6-27b-fp8-microbench-2026-05-31`](hardware-tests/qwen3.6-27b-fp8-microbench-2026-05-31/) | hardware-tests/ | Qwen3.6-**27B-FP8** no-think/think | 5 | Thinking net-negative (35/60 vs 29/60); `p2_triage` 0/5 think vs 5/5 no-think; FP8 serving stable where Q8 failed. |
 
 ## The four "27B"s — don't conflate them

hardware-tests/qwen3.5-397b-vs-step3.7-flash-2026-05-29/findings-minimax-m2.7.md

@@ -3,8 +3,9 @@
 MiniMax-M2.7 (230B-A10B MoE), served as **NVFP4 on vLLM tensor-parallel (TP=2)**, run through the MMBT
 12-family agentic microbench. Added to the [397B vs Step-3.7-Flash entry](findings.md) as a fifth model.
 
-**N=5** per cell (60 cells). Comparators in the main entry are N=10 (397B) / N=1 (Step, 27B-Q4, Coder-Q4) —
-so MiniMax is **N=5, an asymmetry to read with the same caution this entry already documents for small N.**
+**N=5** per cell (60 cells). Comparators in the main entry are N=10 (397B) / N=3 at low-med, N=1 at high
+(Step-3.7) / a single cited representative from N=3 (27B-Q4, Coder-Q4) — so MiniMax at **N=5** sits in the
+middle of that asymmetry, to read with the same caution this entry documents for small N.
 
 ## TL;DR — two findings, the first is the bigger one
 

hardware-tests/qwen3.5-397b-vs-step3.7-flash-2026-05-29/findings.md

@@ -26,7 +26,7 @@ were serving failures** (35B: 36/36 HTTP-400; 27B-Q8: 23/36 token-runaway + 8/36
   `../step3.7-flash-nvfp4-dual-blackwell-2026-05-28/`. Cross-engine + cross-quant: **"best-as-each-ships,"
   not a clean precision study.**
 
-## Scorecard (N=10 for 397B; Step N=1/level; 27B+Coder Q4 N=1)
+## Scorecard (N=10 for 397B; Step N=3 at low/med, N=1 at high; 27B+Coder Q4 = single cited rep from N=3)
 
 | task | 397B no-think | 397B think | Step low/med/high | 27B-Q4 (ref) | Coder-Q4 (ref) |
 |---|:--:|:--:|:--:|:--:|:--:|
@@ -51,8 +51,10 @@ artifact. \* `p3_market` is STRUCTURAL_PASS (citation validity is hand-graded).
 graded by the same flat-vs-nested `phase1_grade.py` bug fixed in this entry, so the original published
 values are self-flagged as untrustworthy (p1_testwrite especially is likely a guaranteed-FAIL artifact).
 Their **p2/p3 cells are unaffected** by that bug and stay (used in the cross-model section below). The
-397B/Step p1 cells use the *fixed* grader and are valid. Step N=1/level, 27B/Coder Q4 N=1 (p2/p3 only) —
-directional; only 397B is N=10. Full per-replicate stability table + finish-reason audit in
+397B/Step p1 cells use the *fixed* grader and are valid. Step is **N=3 at low/medium, N=1 at high** (p2/p3
+cells; the p1 cells are N=1 at every level); the 27B/Coder Q4 columns are a **single cited representative
+from the N=3 [microbench-2026-04-28](../../benchmarks/microbench-2026-04-28/)** (p2/p3 only) — directional;
+only 397B is N=10. Full per-replicate stability table + finish-reason audit in
 [findings-n10.md](findings-n10.md).
 
 ### Grading-correctness fix (post-review, 2026-05-29)
@@ -100,7 +102,9 @@ A review caught that `phase1_grade.py` read flat keys (`coverage_pct`, `ruff_iss
    majority. **Two reasons not to over-read this as "scale doesn't matter":** (a) **cross-quant** — this is
    397B at *aggressive Q3_K_XL* (llama.cpp) vs an ~11B-active model at *NVFP4* (vLLM); it's
    "scale-at-3-bit vs small-at-4-bit," not a clean scale axis; (b) **N-asymmetry** — only 397B is N=10;
-   Step is N=1/level and 27B/Coder are N=1, and this very entry proves N=1 misreads the open-ended cells.
+   Step is N=3 at low/med (N=1 at high) and the 27B/Coder columns are a single cited representative, and
+   this very entry proves **small-N misreads the open-ended cells** (p3_market flips 1/3 at N=3 → 8/10 at
+   N=10 — even N=3 is unsafe here, let alone a single representative).
    "Turn thinking on" is therefore a per-task decision, not a default — but "scale ties" itself is the
    most caveated claim here, not the headline.
 

hardware-tests/qwen3.5-397b-vs-step3.7-flash-2026-05-29/manifest.json

@@ -94,7 +94,7 @@
     "minimax_m2.7_power": "NOT reliably captured for this run (continuous per-sample logger output missing/untagged). Earlier-cited TP=2 peak figures (~896W/1089W/64%/'crosses 1000W') are WITHDRAWN as unverifiable. Only per-cell instantaneous nvidia-smi snapshots survive (N=60): combined median 612W, max 703W, balanced ~313W/~300W per GPU, 0/60 over 1000W -- but these undersample decode peaks and are not comparable to 397B's continuous-sample figures. Claim is architectural (TP loads both GPUs simultaneously), not a measured peak."
   },
   "caveats": [
-    "N=10 per cell for 397B (Step N=1/level; 27B/Coder Q4 N=1). Open-ended cells p3_market/p3_pm/p3_doc are high-variance — see n_variance.",
+    "N=10 per cell for 397B. Step-3.7 is N=3 at low/medium and N=1 at high on the p2/p3 cells (p1 cells N=1 at every level). The 27B/Coder Q4 columns are a single cited representative from the N=3 microbench-2026-04-28, NOT N=1 runs. Open-ended cells p3_market/p3_pm/p3_doc are high-variance — see n_variance.",
     "Cross-engine (llama.cpp vs vLLM) AND cross-quant (Q3_K_XL vs NVFP4) vs Step-3.7-Flash: 'best-as-each-ships', NOT a clean precision study.",
     "Some graders are binary and can fail high-quality output on format/length (see p3_writing); pair with QUALITATIVE.md.",
     "MiniMax-M2.7 (5th model) runs at card-mandated temp=1.0/top_p=0.95/top_k=40 (NOT the temp=0.3 cohort) and N=5 (not N=10); a deliberate, documented deviation. Its continuous power telemetry was not captured this run — see minimax_m2.7_power.",