Merge pull request #28 from Light-Heart-Labs/add-qwen3.5-397b-microbench-2026-05-29

Qwen3.5-397B-A17B microbench (N=1) + GGUF/thinking tooling support

Author: Lightheartdevs

.gitignore

@@ -1,2 +1,8 @@
 # Bytecode in shipped inputs (these get re-generated when the agent runs)
 tooling/inputs/**/__pycache__/
+
+# Transient run artifacts — raw per-run logs and agent workspaces are not published
+tooling/logs/
+tooling/workspace/
+/logs/
+__pycache__/

hardware-tests/qwen3.5-397b-vs-step3.7-flash-2026-05-29/QUALITATIVE.md

@@ -0,0 +1,101 @@
+# Qwen3.5-397B-A17B vs Step-3.7-Flash — qualitative differences
+
+**Status: N=1 / provisional. Both 397B arms complete (no-think 8/12, think 7/12); Flash low/med/high complete.**
+This doc is deliberately *not* a pass/fail scorecard. Pass/fail ties (397B no-think 8/12
+vs Flash 7–8/12) hide the differences that matter — those live here. Every claim cites the
+cell/file it came from so it's reproducible. See SCORECARD/findings for the quantitative table.
+
+Models:
+- **397B** = Qwen3.5-397B-A17B, UD-Q3_K_XL GGUF, llama.cpp b9014, pipeline (`-sm layer`), ctx 131072, no-think arm unless stated.
+- **Flash** = Step-3.7-Flash-NVFP4, 201B MoE / ~11B active, vLLM, native CUTLASS FP4, reasoning levels low/med/high.
+- Cross-engine + cross-quant: "best-as-each-ships," NOT a clean precision study.
+
+## 1. Token / iteration economy — the sharpest split
+Per-cell from `logs/<cell>/transcript.jsonl` (`iter`, `completion_tokens`, tool calls, `wall_s`):
+
+| task | 397B no-think | Step-3.7 medium |
+|---|---|---|
+| p1_bugfix (both PASS) | 110 iters, 29.7k ctok, 463s | **333 iters, 107k ctok, 1222s** |
+| p2_extract (both PASS) | 10 iters, 3.7k ctok | **3 iters, 2.0k ctok** |
+| p2_ci | 42i / 6.4k | 28i / 13.2k |
+| p3_doc | 20i / 12.6k | 11i / 9.9k |
+| p3_business | 19i / 14.4k | 10i / 8.0k |
+| p3_market | 75i / 17.6k | 96i / 28.4k |
+
+**Read:** on the hard open-ended coding task both PASS, but Flash-medium burns ~3.6× the tokens
+and 3× the iterations of 397B no-think. On the *trivial* grounded task (extraction) it inverts —
+Flash is surgical (3 iters) where 397B plods (10). Flash's reasoning is a double-edged sword:
+crisp when the task is well-bounded, flaily when it's open-ended. 397B no-think is steadier across
+the difficulty range. Both hold ~1 tool call/turn (no thrashing).
+
+## 2. Same conclusions, different packaging
+`p3_business` (Borealis acquisition review, both PASS): both independently recommended **HOLD** and
+cited the *same* core issues (burn-rate/runway math, thin customer validation, unsubstantiated
+synergies, opaque valuation) — quality parity on the judgment. The form differs:
+- **397B is scaffold-heavy:** 15 concerns in 3 severity tiers, two ADRs (incl. a concern-prioritization
+  framework), a navigation README, per-deliverable "omissions" decision docs. (`logs/p3_business_397b-nothink_v1`, done_summary + workspace)
+- **Flash is economical:** same substance as tighter flowing prose, fewer artifacts. (`logs/p3_business_step3p7-medium_v1`)
+
+397B over-documents (useful if you want an audit trail, unprompted); Flash says it once and moves on.
+
+## 3. Failure-mode texture (from grade.json sub-scores, not just verdict)
+- **397B `p3_pm` FAIL = under-recall, not hallucination.** workstream_recall 6/6, milestone_recall 5/5,
+  decision_recall 3/4, but **risk_recall 2/6** in a clipped 373-word output. It drops items when terse;
+  it does not fabricate. Benign failure signature. (`logs/p3_pm_397b-nothink_v1/grade.json`)
+- **397B `p3_writing` FAIL ≈ grader strictness, not bad output.** The legal_summary deliverable is
+  accurate and audience-aware (correct incident window, tiered impact: 4 automation-failure / ~24
+  enterprise / ~11,400 general accounts, defensible case-by-case credit recommendation) and it wrote
+  ADRs documenting deliberate per-audience omissions. The binary grader rejected it anyway — real
+  quality runs ahead of pass rate here. Ties to the known binary-grader-misses-quality caveat.
+  (`logs/p3_writing_397b-nothink_v1` workspace)
+- **397B `p1_testwrite` (think) FAIL = a *rule* violation hiding real competence.** After a grader-bug
+  fix (see findings.md), the corrected metrics show think-mode wrote tests reaching **99% coverage / 153
+  passing** — strong, capable test-writing. It FAILs only because it edited `logalyzer/` production code,
+  violating the task's "only /tests/ may differ" rule (`logalyzer_unchanged: False`). The prior grader bug
+  reported `cov=0` and made this look like a flat incapacity ("coverage never improves"). Lesson: a broken
+  metric doesn't just mis-score — it **invents the wrong story about why**. The pass/fail bit (FAIL) was
+  right by accident; everything it implied about the model was wrong. (`logs/p1_testwrite_397b-think_v1/grade.json`)
+
+## 4. Does thinking help 397B? No — net −1, and the loss is revealing
+**397B no-think 8/12 vs 397B think 7/12.** Eleven of twelve cells are identical between modes; reasoning
+changed exactly one outcome — and made it *worse*:
+
+| flip | no-think | think | cause |
+|---|---|---|---|
+| **p3_doc** | PASS (692w) | **FAIL (721w)** | identical content, verbosity blew the limit |
+
+`p3_doc` think captured **all 8/8 facts** (`fact_coverage 1.0`), same as no-think — but wrote 721 words
+against a 700-word limit (`within_word_limit: False`) where no-think landed at 692. Thinking did not make
+it less accurate; it made it **less disciplined about the length constraint**, amplifying 397B's existing
+over-documentation tendency (§2). It also spent more turns getting there (35 vs 20).
+(`logs/p3_doc_397b-{nothink,think}_v1/grade.json`) — a clean case of why pass/fail alone misleads: the
+think output is arguably equal in substance and failed on form.
+
+Everywhere else thinking was **inert**: same PASS/FAIL, just more tokens and turns. On this suite,
+reasoning bought 397B nothing.
+
+**Reasoning shape:** 397B thinks in short targeted bursts (`p1_bugfix` think: 16 of 126 turns carry a
+substantial think block, median ~73 reasoning tokens/turn), not long monologues — but uses more turns
+than no-think on the same task (126 vs 110).
+
+**No runaways, either mode.** All 12 think cells finished `done_signal` (no max_tokens/length failures) —
+including `p3_market` (STRUCTURAL_PASS, 56 iters). Contrast Flash, which **ran away on `p3_market` at low
+effort** (hit max_tokens). 397B is runaway-resistant in both modes; Flash's runaway risk is concentrated
+at low reasoning effort. This is a real reliability edge for 397B.
+
+## 5. Integration cost (a "messy model" finding in itself)
+- Flash (vLLM) ran the harness out of the box once launched.
+- 397B (llama.cpp) needed two fixes: **`--reasoning-format none`** (default extracts CoT into
+  `reasoning_content`, leaving `content` empty → the agent loop reads a thinking turn as "done" and dies
+  at iter ~3; the no-think smoke could not catch this) and a **sandbox-cleanup workaround** (non-sudo
+  `rm` fails on root-owned workspace leftovers; sandbox containers not force-removed on abnormal exit →
+  re-run name collisions). Both are harness/engine-integration bugs, not model quality — but they're
+  exactly the "messy" friction MMBT exists to document. PR should fix both in the harness.
+
+## Net take (provisional, no-think only)
+397B no-think is the steady, over-documenting, high-prose-quality one whose misses are omissions; Flash
+is the fast, terse, reasoning-driven one — brilliant when bounded, flaily when not. They agree on
+substance more than the scorecard's "tie" suggests. Flash is the cheaper/faster way to the same band
+(~99 vs ~71 tok/s, one engine vs both GPUs); 397B's case is narrow (long-form synthesis reliability,
+one stable setting, no effort-tuning). The 397B-think arm is the apples-to-apples test against Flash's
+reasoning modes — pending.

hardware-tests/qwen3.5-397b-vs-step3.7-flash-2026-05-29/README.md

@@ -0,0 +1,53 @@
+# Qwen3.5-397B-A17B on 2× RTX PRO 6000 Blackwell — microbench N=1 (+ Step-3.7-Flash comparison)
+
+A large dense-MoE (397B total / ~17B active) run as a GGUF on llama.cpp, benched through the MMBT
+12-family agentic microbench in two reasoning modes (no-think / think) and compared against the
+Step-3.7-Flash-NVFP4 entry on the same box.
+
+**Provisional, N=1** — one replicate per cell. An N=3 re-run is queued; treat numbers as directional.
+
+## TL;DR
+- **397B no-think 8/12, think 7/12; Step-3.7-Flash 7–8/12.** Aggregate ties across a ~15× param range.
+- **Thinking didn't help 397B** (net −1) — the one regression is a verbosity-driven word-limit overrun at
+  identical fact coverage, not a reasoning failure.
+- **397B never ran away** (both arms) where Flash did at low effort — a real reliability edge.
+- **Flash is the cheaper/faster default** (~99 vs ~71 tok/s, one engine vs both GPUs); 397B's case is
+  narrow (long-form synthesis reliability, one stable setting).
+- The substance is qualitative — **read [QUALITATIVE.md](QUALITATIVE.md).**
+
+## Files
+- [findings.md](findings.md) — scorecard + headline findings.
+- [QUALITATIVE.md](QUALITATIVE.md) — behavioral analysis beyond pass/fail (token economy, packaging,
+  failure-mode texture, reasoning shape), every claim cited to a cell/file.
+- [manifest.json](manifest.json) — models, quant, engine, launch flags, run inventory, dates.
+
+## Reproduce
+```bash
+# 1. Serve the model (GGUF on llama.cpp). NOTE --reasoning-format none is REQUIRED for the think arm:
+#    the default extracts chain-of-thought into reasoning_content, leaving content empty, which the
+#    agentic harness reads as an early stop.
+docker run -d --name llama-397b --gpus all --shm-size 16g \
+  -v $HOME/models:/models:ro -p 127.0.0.1:8001:8000 \
+  ghcr.io/ggml-org/llama.cpp:server-cuda-b9014 \
+  -m /models/unsloth-Qwen3.5-397B-A17B-GGUF/UD-Q3_K_XL/Qwen3.5-397B-A17B-UD-Q3_K_XL-00001-of-00005.gguf \
+  -a qwen3.5-397b-a17b \
+  -ngl 999 -sm layer -fa on -c 131072 -b 2048 -np 1 --jinja --reasoning-format none \
+  --host 0.0.0.0 --port 8000
+
+# 2. Smoke-gate (thinking off, declare the served context window):
+bash tooling/scripts/smoke_test.sh qwen3.5-397b-a17b 8001 smoke397 "" off 131072
+
+# 3. Both arms (label encodes the mode; --max-model-len matches the served -c):
+bash tooling/scripts/run_microbench.sh qwen3.5-397b-a17b 8001 397b-nothink 1 "" off 131072
+bash tooling/scripts/run_microbench.sh qwen3.5-397b-a17b 8001 397b-think   1 "" on  131072
+
+# 4. Grade + summarize each label:
+bash tooling/scripts/grade_microbench.sh 397b-nothink && bash tooling/scripts/summarize.sh 397b-nothink
+bash tooling/scripts/grade_microbench.sh 397b-think   && bash tooling/scripts/summarize.sh 397b-think
+```
+
+## Hardware / environment
+2× NVIDIA RTX PRO 6000 Blackwell Workstation (sm_120, PCIe — no NVLink), TR PRO 7965WX, 252 GB RAM.
+llama.cpp pipeline parallel (`-sm layer`); tensor-parallel (`-sm row`) is ~45% slower on decode on
+this PCIe-only topology. Single-stream decode ~71–74 tok/s at Q3_K_XL. Power caps lifted (both GPUs
+600 W) for the run.

hardware-tests/qwen3.5-397b-vs-step3.7-flash-2026-05-29/findings.md

@@ -0,0 +1,84 @@
+# Qwen3.5-397B-A17B (GGUF, llama.cpp) — microbench N=1, vs Step-3.7-Flash
+
+**Provisional / N=1.** One replicate per cell — directional, not statistically settled. An N=3
+re-run is queued. Pair this with [QUALITATIVE.md](QUALITATIVE.md): the pass/fail table below ties
+across models, and the differences that matter are qualitative.
+
+## Setup
+- **Model:** Qwen3.5-397B-A17B, unsloth UD-Q3_K_XL GGUF (~167 GB on disk, 5 shards).
+- **Engine:** llama.cpp `ghcr.io/ggml-org/llama.cpp:server-cuda-b9014`, pipeline parallel (`-sm layer`),
+  `-ngl 999 -fa on -c 131072 -b 2048 -np 1 --jinja --reasoning-format none`, 2× RTX PRO 6000 Blackwell.
+- **Two arms:** `enable_thinking` off (no-think) and on (think), via the harness `--thinking {on,off}`
+  flag (sends `chat_template_kwargs.enable_thinking`).
+- **Comparison:** Step-3.7-Flash-NVFP4 (vLLM, native CUTLASS FP4) low/med/high — see
+  `../step3.7-flash-nvfp4-dual-blackwell-2026-05-28/`. Cross-engine + cross-quant: **"best-as-each-ships,"
+  not a clean precision study.**
+
+## Scorecard (N=1)
+
+| task | 397B no-think | 397B think | Step low/med/high | 27B (ref N=3) | Coder (ref N=3) |
+|---|:--:|:--:|:--:|:--:|:--:|
+| p1_bugfix | ✓ | ✓ | ✓/✓/✓ | 3/3 | 2/3 |
+| p1_testwrite † | ✗ | ✗ | ✗/✗/✗ | 0/3 † | 0/3 † |
+| p1_refactor † | ✗ | ✗ | ✓/✗/✗ | 0/3 † | 0/3 † |
+| p2_extract | ✓ | ✓ | ✓/✓/✓ | 3/3 | 3/3 |
+| p2_ci | ✓ | ✓ | ✓/✓/✓ | 3/3 | 3/3 |
+| p2_hallucination | ✓ | ✓ | ✓/✓/✓ | 3/3 | 1/3 |
+| p2_triage | ✓ | ✓ | ~/✓/✓ | 3/3 | 3/3 |
+| p3_doc | ✓ | **✗** | ~/✓/✓ | 0/3 | 2/3 |
+| p3_business | ✓ | ✓ | ✓/~/✗ | 2/3 | 3/3 |
+| p3_market * | ✓ | ✓ | **✗**/✓/✓ | 3/3 * | 0/3 |
+| p3_writing | ✗ | ✗ | ✗/~/✗ | 0/3 | 2/3 |
+| p3_pm | ✗ | ✗ | ✗/~/✓ | 0/3 | 1/3 |
+| **Total** | **8/12** | **7/12** | **7 / 8 / 8** | ~7/12 | ~7/12 |
+
+† `p1_refactor` fails on structure (no `output/` subpackage created), not the model's competence at the
+core edit. `p1_testwrite` — see the grading-correctness note below; the earlier "task-design" framing was
+partly a grader artifact. \* `p3_market` is graded STRUCTURAL_PASS (citation validity is a hand-grading dimension).
+
+### Grading-correctness fix (post-review, 2026-05-29)
+A review caught that `phase1_grade.py` read flat keys (`coverage_pct`, `ruff_issues`, `benchmark_s`) while
+`code_task_grader.py` writes nested ones (`coverage.line_coverage_pct`, `ruff.issue_count`,
+`benchmark.elapsed_s`). Effect: `p1_bugfix`'s ruff/benchmark gates were silently always-true, and
+`p1_testwrite`'s coverage gate was always-false. Fixed and **all phase-1 cells regraded**. Outcome:
+- **Totals unchanged (8/12 / 7/12)** — but now *trustworthy*, not coincidental.
+- `p1_bugfix` PASS is now genuinely validated: ruff 2→0 and benchmark **11.2s→0.537s** (the planted O(n²)
+  fix) are real and pass — they were previously ignored.
+- `p1_testwrite` still FAILs, but the **reason flips**: think-mode actually achieved **99% coverage / 153
+  passing tests** (the broken grader reported `cov=0` and hid it); it fails only on `logalyzer_unchanged`
+  (it edited production code, violating the "only /tests/ may differ" rule). The model is *capable* here —
+  the task constraint, not incapacity, is what fails it. The inherited † "task-design" footnote on testwrite
+  is misleading and should be re-examined for the published 27B/Coder cells too.
+- ⚠️ **The 27B / Coder reference columns in the scorecard above predate this fix.** Their phase-1
+  (bug-fixing / test-writing) numbers came from the same buggy grader, so they may be wrong — testwrite
+  especially is likely a guaranteed-FAIL artifact. **Historical phase-1 scores may need regrading; see
+  tracking issue #29.** Treat the reference columns' p1_* cells as provisional until that lands.
+
+## Headline findings
+
+1. **Scale doesn't move the aggregate.** A 397B-param model lands in the *same 7–8/12 band* as a 27B,
+   a ~30B coder, and an ~11B-active Flash. The interesting signal is per-task and qualitative, not the total.
+
+2. **Thinking is net −1 for 397B on this suite** (8/12 → 7/12). 11 of 12 cells are identical between modes;
+   the lone flip is `p3_doc` PASS→FAIL — and it's instructive: *both* modes captured all 8/8 facts
+   (`fact_coverage 1.0`); think just wrote 721 words against a 700-word limit (no-think: 692). Reasoning
+   amplified 397B's verbosity and blew a hard constraint with identical content. Thinking was inert
+   everywhere else — more tokens and turns, same outcomes. **Reasoning bought 397B nothing here.**
+
+3. **397B is runaway-resistant; Flash is not (at low effort).** All 24 397B cells (both arms) finished
+   `done_signal` — zero max_tokens/length failures. Step-3.7-Flash **ran away on `p3_market` at low effort**
+   (hit max_tokens). 397B's reliability edge is real and mode-independent.
+
+4. **397B's distinctive lane is long-form synthesis** (`p3_doc`/`p3_business`/`p3_market` all pass in
+   no-think) where the 27B was weak — but it's the **slowest and most expensive** way to reach the shared
+   band (~71 tok/s spanning both GPUs at Q3, vs Flash ~99 tok/s on one engine). Flash is the better default;
+   397B earns its keep only where synthesis reliability and a single stable setting matter.
+
+5. **Integration tax (a "messy model" finding).** Flash (vLLM) ran the harness out of the box. 397B
+   (llama.cpp) needed `--reasoning-format none` (default extracts CoT into `reasoning_content`, leaving
+   `content` empty → agent loop reads a thinking turn as "done" and dies at iter ~3 — invisible to a
+   thinking-off smoke) plus harness cleanup fixes (non-sudo `rm` on root-owned sandbox/grader leftovers).
+   All fixed in `tooling/`; see commit history.
+
+See [QUALITATIVE.md](QUALITATIVE.md) for the behavioral analysis (token economy, packaging style,
+failure-mode texture, reasoning shape) with per-cell citations.