Skip to content

Commit 3aec6fc

Browse files
Michael Bradleyclaude
andcommitted
Address PR #25 review round 2: status drift + claim id typo
P2: Strix status text now aligned with the data (7/12 cells, ctx=16K gen=128 present). Fixed in: - README.md status table row - README.md headline at-a-glance (replaced "_ctx=16K cell pending_" with the actual decode_tps_at_ctx16k value 7.06) - findings.md § Status of this PR - manifest.json deferred_to_follow_up wording P2: findings.md "weight-shard SHAs are not pinned" stale paragraph rewritten to point readers at workloads/mlx-models.sha256 and give the shasum -c verify command. P3: AUDIT.md claim-id typo fixed — was hw.best-stack.strix-halo.rocm7-works, actual id is hw.best-stack.strix.rocm7-works. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent e39bcf4 commit 3aec6fc

4 files changed

Lines changed: 6 additions & 6 deletions

File tree

hardware-tests/best-stack-followup-2026-05-17/AUDIT.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ This bundle **can** support claims about:
2626

2727
- **MLX vs llama.cpp Metal** on the SAME M5 hardware for Qwen3.6 dense + MoE at ~8-bit quantization, single user. Decode and prefill numbers are directly comparable in tok/s, and the model is the same model concept at the same nominal precision.
2828
- **dream-server ROCm 7 vs llama.cpp Vulkan b9151** on the SAME Strix Halo hardware for Qwen3.6-27B-Q8 GGUF (byte-identical file), single user. Decode is comparable directly; prefill cost can be attributed to engine vintage because the bundled custom llama.cpp build is older than canonical's pin.
29-
- **ROCm 7 loads where ROCm 6.4.4 did not** for the canonical workload on Strix Halo. This narrows (does not invalidate) the canonical "ROCm broken" claim — see `[hw.best-stack.strix-halo.rocm7-works]` in `claims.yaml`.
29+
- **ROCm 7 loads where ROCm 6.4.4 did not** for the canonical workload on Strix Halo. This narrows (does not invalidate) the canonical "ROCm broken" claim — see `[hw.best-stack.strix.rocm7-works]` in `claims.yaml`.
3030

3131
This bundle **cannot** support claims about:
3232

hardware-tests/best-stack-followup-2026-05-17/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ This is an early publication while the grids are still running. **Read `findings
2323
|---|---:|---:|---|
2424
| `m5-mbp/qwen3.6-27b/mlx/` | 12 | 12 | complete |
2525
| `m5-mbp/qwen3.6-35b-a3b/mlx/` | 12 | 12 | complete |
26-
| `strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/` | 6 | 12 | preliminary (ctx≤4K cells; ctx=16K + 32K still running, will land in follow-up commit) |
26+
| `strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/` | 7 | 12 | preliminary (ctx≤4K + ctx=16K gen=128 present; ctx=16K gen=512/2048 and ctx=32K cells still running, will land in follow-up commits) |
2727

2828
Multi-user (`conc≥4`) cells are out of scope here exactly as in the canonical study; they would not change the buyer story this bundle settles.
2929

@@ -33,7 +33,7 @@ Multi-user (`conc≥4`) cells are out of scope here exactly as in the canonical
3333
|---|---|---|---:|---:|---:|
3434
| M5 Max MBP | Qwen3.6-27B | mlx | 773.2 | **17.78 ± 0.04** | 17.14 |
3535
| M5 Max MBP | Qwen3.6-35B-A3B | mlx | **4124.6** | **102.71 ± 0.66** | 95.73 |
36-
| EVO X2 (Strix Halo) | Qwen3.6-27B | dreamserver-rocm7 | 120.0 | 7.67 ± 0.003 | _ctx=16K cell pending_ |
36+
| EVO X2 (Strix Halo) | Qwen3.6-27B | dreamserver-rocm7 | 120.0 | 7.67 ± 0.003 | 7.06 |
3737

3838
vs the canonical-study numbers (`qwen3.6-q8-fleet-2026-05-17/aggregate/canonical-headline.csv`) for the same hardware on llama.cpp:
3939

hardware-tests/best-stack-followup-2026-05-17/findings.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
- **Shipped at snapshot:**
66
- M5 MLX 27B grid (12/12 conc=1 cells, complete)
77
- M5 MLX 35B-A3B grid (12/12 conc=1 cells, complete)
8-
- Strix dream-server ROCm 7 partial grid (6/12 conc=1 cells covering ctx=1024 + 4096; ctx=16K and 32K still running, will land in follow-up commits)
8+
- Strix dream-server ROCm 7 partial grid (7/12 conc=1 cells covering ctx=1024 + 4096 + ctx=16K gen=128; ctx=16K gen=512/2048 and ctx=32K cells still running, will land in follow-up commits)
99
- Engine identification + reproducibility bundle for both paths
1010
- **Preliminary:** Strix dream-server ROCm 7 cells at ctx≥16K. Same engine, just slower cells. Update as they land.
1111
- **Deferred** (see `manifest.json.deferred_to_follow_up`):
@@ -95,7 +95,7 @@ Either of those changes would individually disqualify these rows from the canoni
9595
## § Reproducibility
9696

9797
- Prompt corpus: same SHA as canonical (`9a27eba85a8da9443d7fcf74e281b011831806c4b24aaaada3915463d5c13cd8`), see `workloads/prompts.jsonl`.
98-
- MLX models: `mlx-community/Qwen3.6-27B-8bit` and `mlx-community/Qwen3.6-35B-A3B-8bit` (HuggingFace IDs; HF will resolve to current revisions; weight-shard SHAs are not pinned in this bundle).
98+
- MLX models: `mlx-community/Qwen3.6-27B-8bit` and `mlx-community/Qwen3.6-35B-A3B-8bit`. HuggingFace IDs are floating refs, so the per-shard SHAs of the exact bytes we ran against are pinned in `workloads/mlx-models.sha256` (every safetensors weight shard + config + tokenizer + index file). Run `shasum -a 256 -c workloads/mlx-models.sha256` from a parent dir containing both model dirs to verify before reproducing.
9999
- Strix model file: `Qwen3.6-27B-Q8_0.gguf` SHA `f93f517f...` — byte-identical to canonical study.
100100
- Driver scripts: `harness/lib/bench-cell-mlx.py`, `harness/lib/bench-cell-lemonade.py`, with grid runners and `canon-backfill.py` for adding MMBT-canonical schema fields. Source SHA in `harness/VENDORED-FROM-SHA.txt`.
101101
- Engine SHAs (where available):

hardware-tests/best-stack-followup-2026-05-17/manifest.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@
122122
"Power and thermals samplers were not run alongside this bundle's cells; chassis-class thermal data still lives in qwen3.6-q8-fleet-2026-05-17."
123123
],
124124
"deferred_to_follow_up": [
125-
"Strix dream-server-rocm7 ctx=16K and ctx=32K cells (running at snapshot — 6/12 cells in this commit)",
125+
"Strix dream-server-rocm7 remaining cells: ctx=16K gen=512/2048, ctx=32K all three gen lengths (7/12 cells in this commit; rest running at snapshot)",
126126
"True Lemonade SDK on Strix Linux with Lemonade's own --llamacpp vulkan bundled binary — separates the Lemonade-wrapper question from the dream-server-downstream-build question",
127127
"Lemonade on Windows + DirectML + INT4 OGA NPU path — the actual unique productized acceleration AMD is selling on Ryzen AI silicon (requires a Windows Strix Halo host we did not have)",
128128
"Vanilla llama.cpp b9151 against ROCm 7 — isolates whether the ROCm-loading fix is in the v7 runtime or in dream-server's downstream patches",

0 commit comments

Comments
 (0)