Merge pull request #26 from Light-Heart-Labs/best-stack-followup-2026-05-17

Best stack followup 2026 05 17

Author: Lightheartdevs

hardware-tests/best-stack-followup-2026-05-17/README.md

@@ -23,7 +23,7 @@ This is an early publication while the grids are still running. **Read `findings
 |---|---:|---:|---|
 | `m5-mbp/qwen3.6-27b/mlx/` | 12 | 12 | complete |
 | `m5-mbp/qwen3.6-35b-a3b/mlx/` | 12 | 12 | complete |
-| `strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/` | 7 | 12 | preliminary (ctx≤4K + ctx=16K gen=128 present; ctx=16K gen=512/2048 and ctx=32K cells still running, will land in follow-up commits) |
+| `strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/` | 8 | 12 | preliminary (ctx≤4K + ctx=16K gen=128/512 present; ctx=16K gen=2048 and ctx=32K cells still running, will land in follow-up commits) |
 
 Multi-user (`conc≥4`) cells are out of scope here exactly as in the canonical study; they would not change the buyer story this bundle settles.
 

hardware-tests/best-stack-followup-2026-05-17/aggregate/headline.csv

@@ -1,4 +1,4 @@
 "host","model","backend","cells_present_conc1","peak_prefill_tps","at_cell_prefill","peak_decode_tps","peak_decode_tps_sd","at_cell_decode","decode_tps_at_ctx16k","decode_tps_at_ctx16k_sd","ttft_ms_at_ctx16k","engine"
 "m5-mbp","qwen3.6-27b","mlx",12,773.164,"ctx04096_gen0128_conc1",17.776,0.0353,"ctx01024_gen0128_conc1",17.138,0.018,20397.0,"mlx"
 "m5-mbp","qwen3.6-35b-a3b","mlx",12,4124.576,"ctx04096_gen2048_conc1",102.705,0.6559,"ctx01024_gen0128_conc1",95.728,0.558,4429.0,"mlx"
-"strix-halo","qwen3.6-27b","dreamserver-llamacpp-rocm7",7,119.968,"ctx01024_gen0128_conc1",7.666,0.003,"ctx01024_gen0128_conc1",7.062,0.0019,185573.0,"dreamserver-llamacpp-rocm7"
+"strix-halo","qwen3.6-27b","dreamserver-llamacpp-rocm7",8,119.968,"ctx01024_gen0128_conc1",7.666,0.003,"ctx01024_gen0128_conc1",7.062,0.0019,185573.0,"dreamserver-llamacpp-rocm7"

hardware-tests/best-stack-followup-2026-05-17/findings.md

@@ -5,7 +5,7 @@
 - **Shipped at snapshot:**
   - M5 MLX 27B grid (12/12 conc=1 cells, complete)
   - M5 MLX 35B-A3B grid (12/12 conc=1 cells, complete)
-  - Strix dream-server ROCm 7 partial grid (7/12 conc=1 cells covering ctx=1024 + 4096 + ctx=16K gen=128; ctx=16K gen=512/2048 and ctx=32K cells still running, will land in follow-up commits)
+  - Strix dream-server ROCm 7 partial grid (8/12 conc=1 cells covering ctx=1024 + 4096 + ctx=16K gen=128/512; ctx=16K gen=2048 and ctx=32K cells still running, will land in follow-up commits)
   - Engine identification + reproducibility bundle for both paths
 - **Preliminary:** Strix dream-server ROCm 7 cells at ctx≥16K. Same engine, just slower cells. Update as they land.
 - **Deferred** (see `manifest.json.deferred_to_follow_up`):
@@ -70,6 +70,12 @@ This bundle exercises a different combination on the same hardware:
 - The productized AMD Linux stack does not deliver speedups beyond upstream `llama.cpp` Vulkan. On the prefill axis it is significantly slower because of engine vintage.
 - The Strix Halo NPU acceleration story (the actual unique-selling-point of Ryzen AI silicon) lives on the Windows + DirectML + INT4 OGA path through Lemonade's official `oga-load` backend, which we did not test here. That, not ROCm Linux, is what AMD is selling.
 
+### Additional finding: 300 s server-side request ceiling
+
+`ctx=16384 gen=2048` and the entire `ctx=32K` tier exceed a 300 s response timeout in the dream-server Lemonade Server stack on Strix Halo. All 10 batches at `ctx16384_gen2048_conc1` errored at exactly 300.0 s wall time per request, producing a `.error` marker (no `.done`). Estimated true request length at that cell: ~186 s prefill (84 tok/s × 15,603 prompt tokens) + ~293 s decode (2048 / 7 tok/s) ≈ 480 s, well over the 300 s ceiling. At `ctx=32K`, prefill alone is ~371 s, so all `ctx=32K` cells are expected to hit the same ceiling.
+
+This is itself a buyer-relevant ceiling: under the **dream-server lemonade stack as-shipped on Strix Halo Linux**, single-user requests longer than ~5 min fail. Vanilla `llama-server` from `b9151` does not have this ceiling (the canonical study's Strix Vulkan ctx=32K gen=2048 cell completes in ~250 s and decode is ~7 tok/s × 2048 = 293 s). The ceiling appears to be a Lemonade Server / FastAPI default; not yet investigated whether tunable.
+
 ## § Engine identification — why this took some unwinding
 
 We initially set up bench cells expecting to measure "Lemonade SDK on Strix" — Lemonade's own Vulkan-bundled binary against the same Q8 GGUF. The numbers came back close to canonical Vulkan and we wrote down "Lemonade ≈ vanilla Vulkan on Linux" as the headline.

hardware-tests/best-stack-followup-2026-05-17/manifest.json

@@ -79,14 +79,16 @@
     },
     {
       "path": "strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/",
-      "status": "appendix-preliminary",
-      "cells_present": 7,
+      "status": "appendix-partial-with-ceiling-finding",
+      "cells_present": 8,
+      "cells_with_error_marker": 2,
       "cells_planned": 12,
-      "load_bearing_for": "Demonstrates that ROCm 7 (dream-server's bundled build) loads and serves Qwen3.6-27B-Q8 on Strix Halo where the canonical study's pinned llama.cpp b9151 + ROCm 6.4.4 segfaulted. Also shows that this path's decode matches vanilla Vulkan within noise while prefill is significantly slower because the bundled engine is at an older upstream commit than b9151. The ctx=16K cell adds a long-context data point: TTFT = 185.6 s (vs canonical Vulkan ttft ~30 s at the same cell) — the older-engine prefill cost shows up most dramatically at long context.",
+      "load_bearing_for": "Demonstrates that ROCm 7 (dream-server's bundled build) loads and serves Qwen3.6-27B-Q8 on Strix Halo where the canonical study's pinned llama.cpp b9151 + ROCm 6.4.4 segfaulted. Also shows that this path's decode matches vanilla Vulkan within noise while prefill is significantly slower because the bundled engine is at an older upstream commit than b9151. The ctx=16K cells confirm the long-context behavior holds: TTFT ≈ 186 s at gen=128 (vs canonical Vulkan ttft ~30 s at the same cell), decode steady ~7.0 tok/s through gen=128/512. ctx=16K gen=2048 and the entire ctx=32K tier exceed a 300 s server-side response timeout in this dream-server lemonade stack — see known_gaps.",
       "known_gaps": [
-        "ctx=16K gen=512 and gen=2048 cells still running (2 cells)",
-        "ctx=32K cells not yet started (3 cells)",
-        "When all 12 land they will arrive in follow-up commits to this bundle."
+        "ctx=16384_gen2048_conc1 has a .error marker (no .done). All 10 batches timed out at exactly 300.0 s wall time per request. Estimated request length at this cell: ~186 s prefill (84 tok/s × 15603 prompt tokens) + ~293 s decode (2048 / 7 tok/s) = ~480 s, well over the 300 s ceiling.",
+        "ctx=32768_gen0128_conc1 has a .error marker (no .done). Same 300 s ceiling: prefill at ctx=32K is ~371 s (84 tok/s × 31186 prompt tokens), already over the ceiling before any decode begins.",
+        "ctx=32K gen=512 and gen=2048 cells are running at snapshot and are expected to fail the same way (will be pulled in as .error cells if they do).",
+        "The 300 s timeout appears to be a Lemonade Server / FastAPI default; not yet investigated whether it is tunable. Documented as a buyer-relevant ceiling: under the dream-server lemonade stack on Strix Halo Linux, single-user requests longer than ~5 min fail."
       ]
     }
   ],

hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/.done

@@ -0,0 +1,10 @@
+{"batch": 0, "wall_s": 258.707440838, "aggregate_decode_tps": 1.9790694784098197, "per_slot_decode_tps_mean": 7.01277745858835}
+{"batch": 1, "wall_s": 258.674228979, "aggregate_decode_tps": 1.9793235763024764, "per_slot_decode_tps_mean": 7.012576617878127}
+{"batch": 2, "wall_s": 258.690663903, "aggregate_decode_tps": 1.9791978275334368, "per_slot_decode_tps_mean": 7.011209266671468}
+{"batch": 3, "wall_s": 258.659795771, "aggregate_decode_tps": 1.9794340224921942, "per_slot_decode_tps_mean": 7.010876416241539}
+{"batch": 4, "wall_s": 258.694397446, "aggregate_decode_tps": 1.9791692632496038, "per_slot_decode_tps_mean": 7.011952173775019}
+{"batch": 5, "wall_s": 258.706087298, "aggregate_decode_tps": 1.9790798328229293, "per_slot_decode_tps_mean": 7.01147916081578}
+{"batch": 6, "wall_s": 258.690643987, "aggregate_decode_tps": 1.979197979907343, "per_slot_decode_tps_mean": 7.010627015144488}
+{"batch": 7, "wall_s": 258.71764173, "aggregate_decode_tps": 1.9789914463364182, "per_slot_decode_tps_mean": 7.010806432387516}
+{"batch": 8, "wall_s": 258.676614034, "aggregate_decode_tps": 1.9793053265058727, "per_slot_decode_tps_mean": 7.0122631334210155}
+{"batch": 9, "wall_s": 258.657727259, "aggregate_decode_tps": 1.9794498522262296, "per_slot_decode_tps_mean": 7.011407916728575}

hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/batches.jsonl

@@ -0,0 +1,11 @@
+batch 0: wall=258.71s agg=1.98 tok/s
+batch 1: wall=258.67s agg=1.98 tok/s
+batch 2: wall=258.69s agg=1.98 tok/s
+batch 3: wall=258.66s agg=1.98 tok/s
+batch 4: wall=258.69s agg=1.98 tok/s
+batch 5: wall=258.71s agg=1.98 tok/s
+batch 6: wall=258.69s agg=1.98 tok/s
+batch 7: wall=258.72s agg=1.98 tok/s
+batch 8: wall=258.68s agg=1.98 tok/s
+batch 9: wall=258.66s agg=1.98 tok/s
+cell done: per_slot_decode_mean=7.011327689398175

hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/bench-cell.log

@@ -0,0 +1,41 @@
+{
+  "ctx": 16384,
+  "gen": 512,
+  "conc": 1,
+  "n_batches": 10,
+  "warmup_batches": 2,
+  "seed": 42,
+  "started": "2026-05-17T17:08:52.009180+00:00",
+  "model": "extra.Qwen3.6-27B-Q8_0.gguf",
+  "engine": "dreamserver-llamacpp-rocm7",
+  "per_slot": {
+    "n": 8,
+    "n_total_including_warmup": 10,
+    "decode_tps_mean": 7.011327689398175,
+    "decode_tps_median": 7.011308591700022,
+    "decode_tps_min": 7.010627015144488,
+    "decode_tps_max": 7.0122631334210155,
+    "decode_tps_sd": 0.0005700552550571158,
+    "prefill_tps_mean": 84.04787338318525,
+    "ttft_ms_mean": 185644.199625,
+    "elapsed_s_mean": 258.686603290875,
+    "prompt_tokens_max": 15603,
+    "gen_tokens_total": 4096
+  },
+  "aggregate": {
+    "n_batches_total": 10,
+    "n_batches_body": 8,
+    "aggregate_decode_tps_mean": 1.9792281938842535,
+    "aggregate_decode_tps_median": 1.9791979037203897,
+    "batch_wall_s_mean": 258.6866964285
+  },
+  "cold_start": {
+    "wall_s": 258.707316844,
+    "decode_tps": 7.01277745858835,
+    "aggregate_decode_tps": 1.9790694784098197,
+    "per_slot_decode_tps_mean": 7.01277745858835
+  },
+  "inferences_path": "inferences.jsonl",
+  "batches_path": "batches.jsonl",
+  "engine_note": "dream-server bundled llama.cpp at /opt/llama-custom/llama-server linked against libamdhip64.so.7 (ROCm 7) + custom librocblas/libhipblaslt; version string b1 (ff5ef82). Fronted by Lemonade Server OpenAI-compatible API on port 11434."
+}

hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/cell.json

@@ -0,0 +1,10 @@
+{"id": "b0_s0", "t0_mono_ns": 80369224480872, "t1_mono_ns": 80627931797716, "wall_start": 1779037732.0093482, "elapsed_s": 258.707316844, "decode_tps": 7.01277745858835, "prefill_tps": 84.0449390438265, "ttft_ms": 185650.679, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 0}
+{"id": "b1_s0", "t0_mono_ns": 80627932107733, "t1_mono_ns": 80886606243906, "wall_start": 1779037990.7169747, "elapsed_s": 258.674136173, "decode_tps": 7.012576617878127, "prefill_tps": 84.04784324497093, "ttft_ms": 185644.264, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 1}
+{"id": "b2_s0", "t0_mono_ns": 80886606589392, "t1_mono_ns": 81145297157289, "wall_start": 1779038249.3914568, "elapsed_s": 258.690567897, "decode_tps": 7.011209266671468, "prefill_tps": 84.04627046950606, "ttft_ms": 185647.738, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 2}
+{"id": "b3_s0", "t0_mono_ns": 81145297492238, "t1_mono_ns": 81403957205221, "wall_start": 1779038508.0823596, "elapsed_s": 258.659712983, "decode_tps": 7.010876416241539, "prefill_tps": 84.06299497556724, "ttft_ms": 185610.803, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 3}
+{"id": "b4_s0", "t0_mono_ns": 81403957443502, "t1_mono_ns": 81662651755922, "wall_start": 1779038766.742311, "elapsed_s": 258.69431242, "decode_tps": 7.011952173775019, "prefill_tps": 84.04206944657486, "ttft_ms": 185657.018, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 4}
+{"id": "b5_s0", "t0_mono_ns": 81662652103339, "t1_mono_ns": 81921358096381, "wall_start": 1779039025.4369707, "elapsed_s": 258.705993042, "decode_tps": 7.01147916081578, "prefill_tps": 84.03811914793437, "ttft_ms": 185665.745, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 5}
+{"id": "b6_s0", "t0_mono_ns": 81921358444228, "t1_mono_ns": 82180048976249, "wall_start": 1779039284.1433115, "elapsed_s": 258.690532021, "decode_tps": 7.010627015144488, "prefill_tps": 84.04888681442971, "ttft_ms": 185641.959, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 6}
+{"id": "b7_s0", "t0_mono_ns": 82180049327506, "t1_mono_ns": 82438766875899, "wall_start": 1779039542.8341947, "elapsed_s": 258.717548393, "decode_tps": 7.010806432387516, "prefill_tps": 84.03606787315775, "ttft_ms": 185670.277, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 7}
+{"id": "b8_s0", "t0_mono_ns": 82438767235405, "t1_mono_ns": 82697443761743, "wall_start": 1779039801.5521026, "elapsed_s": 258.676526338, "decode_tps": 7.0122631334210155, "prefill_tps": 84.04776446897914, "ttft_ms": 185644.438, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 8}
+{"id": "b9_s0", "t0_mono_ns": 82697444085790, "t1_mono_ns": 82956101719023, "wall_start": 1779040060.228953, "elapsed_s": 258.657633233, "decode_tps": 7.011407916728575, "prefill_tps": 84.06081386933285, "ttft_ms": 185615.619, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n   - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 9}

hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/inferences.jsonl

@@ -0,0 +1 @@
+bench-cell-failed