Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion hardware-tests/best-stack-followup-2026-05-17/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ This is an early publication while the grids are still running. **Read `findings
|---|---:|---:|---|
| `m5-mbp/qwen3.6-27b/mlx/` | 12 | 12 | complete |
| `m5-mbp/qwen3.6-35b-a3b/mlx/` | 12 | 12 | complete |
| `strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/` | 7 | 12 | preliminary (ctx≤4K + ctx=16K gen=128 present; ctx=16K gen=512/2048 and ctx=32K cells still running, will land in follow-up commits) |
| `strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/` | 8 | 12 | preliminary (ctx≤4K + ctx=16K gen=128/512 present; ctx=16K gen=2048 and ctx=32K cells still running, will land in follow-up commits) |

Multi-user (`conc≥4`) cells are out of scope here exactly as in the canonical study; they would not change the buyer story this bundle settles.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"host","model","backend","cells_present_conc1","peak_prefill_tps","at_cell_prefill","peak_decode_tps","peak_decode_tps_sd","at_cell_decode","decode_tps_at_ctx16k","decode_tps_at_ctx16k_sd","ttft_ms_at_ctx16k","engine"
"m5-mbp","qwen3.6-27b","mlx",12,773.164,"ctx04096_gen0128_conc1",17.776,0.0353,"ctx01024_gen0128_conc1",17.138,0.018,20397.0,"mlx"
"m5-mbp","qwen3.6-35b-a3b","mlx",12,4124.576,"ctx04096_gen2048_conc1",102.705,0.6559,"ctx01024_gen0128_conc1",95.728,0.558,4429.0,"mlx"
"strix-halo","qwen3.6-27b","dreamserver-llamacpp-rocm7",7,119.968,"ctx01024_gen0128_conc1",7.666,0.003,"ctx01024_gen0128_conc1",7.062,0.0019,185573.0,"dreamserver-llamacpp-rocm7"
"strix-halo","qwen3.6-27b","dreamserver-llamacpp-rocm7",8,119.968,"ctx01024_gen0128_conc1",7.666,0.003,"ctx01024_gen0128_conc1",7.062,0.0019,185573.0,"dreamserver-llamacpp-rocm7"
8 changes: 7 additions & 1 deletion hardware-tests/best-stack-followup-2026-05-17/findings.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
- **Shipped at snapshot:**
- M5 MLX 27B grid (12/12 conc=1 cells, complete)
- M5 MLX 35B-A3B grid (12/12 conc=1 cells, complete)
- Strix dream-server ROCm 7 partial grid (7/12 conc=1 cells covering ctx=1024 + 4096 + ctx=16K gen=128; ctx=16K gen=512/2048 and ctx=32K cells still running, will land in follow-up commits)
- Strix dream-server ROCm 7 partial grid (8/12 conc=1 cells covering ctx=1024 + 4096 + ctx=16K gen=128/512; ctx=16K gen=2048 and ctx=32K cells still running, will land in follow-up commits)
- Engine identification + reproducibility bundle for both paths
- **Preliminary:** Strix dream-server ROCm 7 cells at ctx≥16K. Same engine, just slower cells. Update as they land.
- **Deferred** (see `manifest.json.deferred_to_follow_up`):
Expand Down Expand Up @@ -70,6 +70,12 @@ This bundle exercises a different combination on the same hardware:
- The productized AMD Linux stack does not deliver speedups beyond upstream `llama.cpp` Vulkan. On the prefill axis it is significantly slower because of engine vintage.
- The Strix Halo NPU acceleration story (the actual unique-selling-point of Ryzen AI silicon) lives on the Windows + DirectML + INT4 OGA path through Lemonade's official `oga-load` backend, which we did not test here. That, not ROCm Linux, is what AMD is selling.

### Additional finding: 300 s server-side request ceiling

`ctx=16384 gen=2048` and the entire `ctx=32K` tier exceed a 300 s response timeout in the dream-server Lemonade Server stack on Strix Halo. All 10 batches at `ctx16384_gen2048_conc1` errored at exactly 300.0 s wall time per request, producing a `.error` marker (no `.done`). Estimated true request length at that cell: ~186 s prefill (84 tok/s × 15,603 prompt tokens) + ~293 s decode (2048 / 7 tok/s) ≈ 480 s, well over the 300 s ceiling. At `ctx=32K`, prefill alone is ~371 s, so all `ctx=32K` cells are expected to hit the same ceiling.

This is itself a buyer-relevant ceiling: under the **dream-server lemonade stack as-shipped on Strix Halo Linux**, single-user requests longer than ~5 min fail. Vanilla `llama-server` from `b9151` does not have this ceiling (the canonical study's Strix Vulkan ctx=32K gen=2048 cell completes in ~250 s and decode is ~7 tok/s × 2048 = 293 s). The ceiling appears to be a Lemonade Server / FastAPI default; not yet investigated whether tunable.

## § Engine identification — why this took some unwinding

We initially set up bench cells expecting to measure "Lemonade SDK on Strix" — Lemonade's own Vulkan-bundled binary against the same Q8 GGUF. The numbers came back close to canonical Vulkan and we wrote down "Lemonade ≈ vanilla Vulkan on Linux" as the headline.
Expand Down
14 changes: 8 additions & 6 deletions hardware-tests/best-stack-followup-2026-05-17/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -79,14 +79,16 @@
},
{
"path": "strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/",
"status": "appendix-preliminary",
"cells_present": 7,
"status": "appendix-partial-with-ceiling-finding",
"cells_present": 8,
"cells_with_error_marker": 2,
"cells_planned": 12,
"load_bearing_for": "Demonstrates that ROCm 7 (dream-server's bundled build) loads and serves Qwen3.6-27B-Q8 on Strix Halo where the canonical study's pinned llama.cpp b9151 + ROCm 6.4.4 segfaulted. Also shows that this path's decode matches vanilla Vulkan within noise while prefill is significantly slower because the bundled engine is at an older upstream commit than b9151. The ctx=16K cell adds a long-context data point: TTFT = 185.6 s (vs canonical Vulkan ttft ~30 s at the same cell)the older-engine prefill cost shows up most dramatically at long context.",
"load_bearing_for": "Demonstrates that ROCm 7 (dream-server's bundled build) loads and serves Qwen3.6-27B-Q8 on Strix Halo where the canonical study's pinned llama.cpp b9151 + ROCm 6.4.4 segfaulted. Also shows that this path's decode matches vanilla Vulkan within noise while prefill is significantly slower because the bundled engine is at an older upstream commit than b9151. The ctx=16K cells confirm the long-context behavior holds: TTFT ≈ 186 s at gen=128 (vs canonical Vulkan ttft ~30 s at the same cell), decode steady ~7.0 tok/s through gen=128/512. ctx=16K gen=2048 and the entire ctx=32K tier exceed a 300 s server-side response timeout in this dream-server lemonade stack — see known_gaps.",
"known_gaps": [
"ctx=16K gen=512 and gen=2048 cells still running (2 cells)",
"ctx=32K cells not yet started (3 cells)",
"When all 12 land they will arrive in follow-up commits to this bundle."
"ctx=16384_gen2048_conc1 has a .error marker (no .done). All 10 batches timed out at exactly 300.0 s wall time per request. Estimated request length at this cell: ~186 s prefill (84 tok/s × 15603 prompt tokens) + ~293 s decode (2048 / 7 tok/s) = ~480 s, well over the 300 s ceiling.",
"ctx=32768_gen0128_conc1 has a .error marker (no .done). Same 300 s ceiling: prefill at ctx=32K is ~371 s (84 tok/s × 31186 prompt tokens), already over the ceiling before any decode begins.",
"ctx=32K gen=512 and gen=2048 cells are running at snapshot and are expected to fail the same way (will be pulled in as .error cells if they do).",
"The 300 s timeout appears to be a Lemonade Server / FastAPI default; not yet investigated whether it is tunable. Documented as a buyer-relevant ceiling: under the dream-server lemonade stack on Strix Halo Linux, single-user requests longer than ~5 min fail."
]
}
],
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{"batch": 0, "wall_s": 258.707440838, "aggregate_decode_tps": 1.9790694784098197, "per_slot_decode_tps_mean": 7.01277745858835}
{"batch": 1, "wall_s": 258.674228979, "aggregate_decode_tps": 1.9793235763024764, "per_slot_decode_tps_mean": 7.012576617878127}
{"batch": 2, "wall_s": 258.690663903, "aggregate_decode_tps": 1.9791978275334368, "per_slot_decode_tps_mean": 7.011209266671468}
{"batch": 3, "wall_s": 258.659795771, "aggregate_decode_tps": 1.9794340224921942, "per_slot_decode_tps_mean": 7.010876416241539}
{"batch": 4, "wall_s": 258.694397446, "aggregate_decode_tps": 1.9791692632496038, "per_slot_decode_tps_mean": 7.011952173775019}
{"batch": 5, "wall_s": 258.706087298, "aggregate_decode_tps": 1.9790798328229293, "per_slot_decode_tps_mean": 7.01147916081578}
{"batch": 6, "wall_s": 258.690643987, "aggregate_decode_tps": 1.979197979907343, "per_slot_decode_tps_mean": 7.010627015144488}
{"batch": 7, "wall_s": 258.71764173, "aggregate_decode_tps": 1.9789914463364182, "per_slot_decode_tps_mean": 7.010806432387516}
{"batch": 8, "wall_s": 258.676614034, "aggregate_decode_tps": 1.9793053265058727, "per_slot_decode_tps_mean": 7.0122631334210155}
{"batch": 9, "wall_s": 258.657727259, "aggregate_decode_tps": 1.9794498522262296, "per_slot_decode_tps_mean": 7.011407916728575}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
batch 0: wall=258.71s agg=1.98 tok/s
batch 1: wall=258.67s agg=1.98 tok/s
batch 2: wall=258.69s agg=1.98 tok/s
batch 3: wall=258.66s agg=1.98 tok/s
batch 4: wall=258.69s agg=1.98 tok/s
batch 5: wall=258.71s agg=1.98 tok/s
batch 6: wall=258.69s agg=1.98 tok/s
batch 7: wall=258.72s agg=1.98 tok/s
batch 8: wall=258.68s agg=1.98 tok/s
batch 9: wall=258.66s agg=1.98 tok/s
cell done: per_slot_decode_mean=7.011327689398175
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
{
"ctx": 16384,
"gen": 512,
"conc": 1,
"n_batches": 10,
"warmup_batches": 2,
"seed": 42,
"started": "2026-05-17T17:08:52.009180+00:00",
"model": "extra.Qwen3.6-27B-Q8_0.gguf",
"engine": "dreamserver-llamacpp-rocm7",
"per_slot": {
"n": 8,
"n_total_including_warmup": 10,
"decode_tps_mean": 7.011327689398175,
"decode_tps_median": 7.011308591700022,
"decode_tps_min": 7.010627015144488,
"decode_tps_max": 7.0122631334210155,
"decode_tps_sd": 0.0005700552550571158,
"prefill_tps_mean": 84.04787338318525,
"ttft_ms_mean": 185644.199625,
"elapsed_s_mean": 258.686603290875,
"prompt_tokens_max": 15603,
"gen_tokens_total": 4096
},
"aggregate": {
"n_batches_total": 10,
"n_batches_body": 8,
"aggregate_decode_tps_mean": 1.9792281938842535,
"aggregate_decode_tps_median": 1.9791979037203897,
"batch_wall_s_mean": 258.6866964285
},
"cold_start": {
"wall_s": 258.707316844,
"decode_tps": 7.01277745858835,
"aggregate_decode_tps": 1.9790694784098197,
"per_slot_decode_tps_mean": 7.01277745858835
},
"inferences_path": "inferences.jsonl",
"batches_path": "batches.jsonl",
"engine_note": "dream-server bundled llama.cpp at /opt/llama-custom/llama-server linked against libamdhip64.so.7 (ROCm 7) + custom librocblas/libhipblaslt; version string b1 (ff5ef82). Fronted by Lemonade Server OpenAI-compatible API on port 11434."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{"id": "b0_s0", "t0_mono_ns": 80369224480872, "t1_mono_ns": 80627931797716, "wall_start": 1779037732.0093482, "elapsed_s": 258.707316844, "decode_tps": 7.01277745858835, "prefill_tps": 84.0449390438265, "ttft_ms": 185650.679, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 0}
{"id": "b1_s0", "t0_mono_ns": 80627932107733, "t1_mono_ns": 80886606243906, "wall_start": 1779037990.7169747, "elapsed_s": 258.674136173, "decode_tps": 7.012576617878127, "prefill_tps": 84.04784324497093, "ttft_ms": 185644.264, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 1}
{"id": "b2_s0", "t0_mono_ns": 80886606589392, "t1_mono_ns": 81145297157289, "wall_start": 1779038249.3914568, "elapsed_s": 258.690567897, "decode_tps": 7.011209266671468, "prefill_tps": 84.04627046950606, "ttft_ms": 185647.738, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 2}
{"id": "b3_s0", "t0_mono_ns": 81145297492238, "t1_mono_ns": 81403957205221, "wall_start": 1779038508.0823596, "elapsed_s": 258.659712983, "decode_tps": 7.010876416241539, "prefill_tps": 84.06299497556724, "ttft_ms": 185610.803, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 3}
{"id": "b4_s0", "t0_mono_ns": 81403957443502, "t1_mono_ns": 81662651755922, "wall_start": 1779038766.742311, "elapsed_s": 258.69431242, "decode_tps": 7.011952173775019, "prefill_tps": 84.04206944657486, "ttft_ms": 185657.018, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 4}
{"id": "b5_s0", "t0_mono_ns": 81662652103339, "t1_mono_ns": 81921358096381, "wall_start": 1779039025.4369707, "elapsed_s": 258.705993042, "decode_tps": 7.01147916081578, "prefill_tps": 84.03811914793437, "ttft_ms": 185665.745, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 5}
{"id": "b6_s0", "t0_mono_ns": 81921358444228, "t1_mono_ns": 82180048976249, "wall_start": 1779039284.1433115, "elapsed_s": 258.690532021, "decode_tps": 7.010627015144488, "prefill_tps": 84.04888681442971, "ttft_ms": 185641.959, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 6}
{"id": "b7_s0", "t0_mono_ns": 82180049327506, "t1_mono_ns": 82438766875899, "wall_start": 1779039542.8341947, "elapsed_s": 258.717548393, "decode_tps": 7.010806432387516, "prefill_tps": 84.03606787315775, "ttft_ms": 185670.277, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 7}
{"id": "b8_s0", "t0_mono_ns": 82438767235405, "t1_mono_ns": 82697443761743, "wall_start": 1779039801.5521026, "elapsed_s": 258.676526338, "decode_tps": 7.0122631334210155, "prefill_tps": 84.04776446897914, "ttft_ms": 185644.438, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 8}
{"id": "b9_s0", "t0_mono_ns": 82697444085790, "t1_mono_ns": 82956101719023, "wall_start": 1779040060.228953, "elapsed_s": 258.657633233, "decode_tps": 7.011407916728575, "prefill_tps": 84.06081386933285, "ttft_ms": 185615.619, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 9}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
bench-cell-failed
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{"batch": 0, "wall_s": 300.075433243, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 1, "wall_s": 299.992882535, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 2, "wall_s": 299.991695467, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 3, "wall_s": 299.991457546, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 4, "wall_s": 299.991208596, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 5, "wall_s": 299.990371522, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 6, "wall_s": 299.991232326, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 7, "wall_s": 299.991217108, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 8, "wall_s": 299.991020651, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
{"batch": 9, "wall_s": 299.990978366, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
batch 0: wall=300.08s agg=0.00 tok/s
batch 1: wall=299.99s agg=0.00 tok/s
batch 2: wall=299.99s agg=0.00 tok/s
batch 3: wall=299.99s agg=0.00 tok/s
batch 4: wall=299.99s agg=0.00 tok/s
batch 5: wall=299.99s agg=0.00 tok/s
batch 6: wall=299.99s agg=0.00 tok/s
batch 7: wall=299.99s agg=0.00 tok/s
batch 8: wall=299.99s agg=0.00 tok/s
batch 9: wall=299.99s agg=0.00 tok/s
cell errored: 10/10 bad inferences
Loading
Loading