diff --git a/hardware-tests/best-stack-followup-2026-05-17/README.md b/hardware-tests/best-stack-followup-2026-05-17/README.md index 2e2e811c..23e6fb42 100644 --- a/hardware-tests/best-stack-followup-2026-05-17/README.md +++ b/hardware-tests/best-stack-followup-2026-05-17/README.md @@ -23,7 +23,7 @@ This is an early publication while the grids are still running. **Read `findings |---|---:|---:|---| | `m5-mbp/qwen3.6-27b/mlx/` | 12 | 12 | complete | | `m5-mbp/qwen3.6-35b-a3b/mlx/` | 12 | 12 | complete | -| `strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/` | 7 | 12 | preliminary (ctx≤4K + ctx=16K gen=128 present; ctx=16K gen=512/2048 and ctx=32K cells still running, will land in follow-up commits) | +| `strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/` | 8 | 12 | preliminary (ctx≤4K + ctx=16K gen=128/512 present; ctx=16K gen=2048 and ctx=32K cells still running, will land in follow-up commits) | Multi-user (`conc≥4`) cells are out of scope here exactly as in the canonical study; they would not change the buyer story this bundle settles. diff --git a/hardware-tests/best-stack-followup-2026-05-17/aggregate/headline.csv b/hardware-tests/best-stack-followup-2026-05-17/aggregate/headline.csv index b3b9e550..f61b6e62 100644 --- a/hardware-tests/best-stack-followup-2026-05-17/aggregate/headline.csv +++ b/hardware-tests/best-stack-followup-2026-05-17/aggregate/headline.csv @@ -1,4 +1,4 @@ "host","model","backend","cells_present_conc1","peak_prefill_tps","at_cell_prefill","peak_decode_tps","peak_decode_tps_sd","at_cell_decode","decode_tps_at_ctx16k","decode_tps_at_ctx16k_sd","ttft_ms_at_ctx16k","engine" "m5-mbp","qwen3.6-27b","mlx",12,773.164,"ctx04096_gen0128_conc1",17.776,0.0353,"ctx01024_gen0128_conc1",17.138,0.018,20397.0,"mlx" "m5-mbp","qwen3.6-35b-a3b","mlx",12,4124.576,"ctx04096_gen2048_conc1",102.705,0.6559,"ctx01024_gen0128_conc1",95.728,0.558,4429.0,"mlx" -"strix-halo","qwen3.6-27b","dreamserver-llamacpp-rocm7",7,119.968,"ctx01024_gen0128_conc1",7.666,0.003,"ctx01024_gen0128_conc1",7.062,0.0019,185573.0,"dreamserver-llamacpp-rocm7" +"strix-halo","qwen3.6-27b","dreamserver-llamacpp-rocm7",8,119.968,"ctx01024_gen0128_conc1",7.666,0.003,"ctx01024_gen0128_conc1",7.062,0.0019,185573.0,"dreamserver-llamacpp-rocm7" diff --git a/hardware-tests/best-stack-followup-2026-05-17/findings.md b/hardware-tests/best-stack-followup-2026-05-17/findings.md index c8ad1949..3b584586 100644 --- a/hardware-tests/best-stack-followup-2026-05-17/findings.md +++ b/hardware-tests/best-stack-followup-2026-05-17/findings.md @@ -5,7 +5,7 @@ - **Shipped at snapshot:** - M5 MLX 27B grid (12/12 conc=1 cells, complete) - M5 MLX 35B-A3B grid (12/12 conc=1 cells, complete) - - Strix dream-server ROCm 7 partial grid (7/12 conc=1 cells covering ctx=1024 + 4096 + ctx=16K gen=128; ctx=16K gen=512/2048 and ctx=32K cells still running, will land in follow-up commits) + - Strix dream-server ROCm 7 partial grid (8/12 conc=1 cells covering ctx=1024 + 4096 + ctx=16K gen=128/512; ctx=16K gen=2048 and ctx=32K cells still running, will land in follow-up commits) - Engine identification + reproducibility bundle for both paths - **Preliminary:** Strix dream-server ROCm 7 cells at ctx≥16K. Same engine, just slower cells. Update as they land. - **Deferred** (see `manifest.json.deferred_to_follow_up`): @@ -70,6 +70,12 @@ This bundle exercises a different combination on the same hardware: - The productized AMD Linux stack does not deliver speedups beyond upstream `llama.cpp` Vulkan. On the prefill axis it is significantly slower because of engine vintage. - The Strix Halo NPU acceleration story (the actual unique-selling-point of Ryzen AI silicon) lives on the Windows + DirectML + INT4 OGA path through Lemonade's official `oga-load` backend, which we did not test here. That, not ROCm Linux, is what AMD is selling. +### Additional finding: 300 s server-side request ceiling + +`ctx=16384 gen=2048` and the entire `ctx=32K` tier exceed a 300 s response timeout in the dream-server Lemonade Server stack on Strix Halo. All 10 batches at `ctx16384_gen2048_conc1` errored at exactly 300.0 s wall time per request, producing a `.error` marker (no `.done`). Estimated true request length at that cell: ~186 s prefill (84 tok/s × 15,603 prompt tokens) + ~293 s decode (2048 / 7 tok/s) ≈ 480 s, well over the 300 s ceiling. At `ctx=32K`, prefill alone is ~371 s, so all `ctx=32K` cells are expected to hit the same ceiling. + +This is itself a buyer-relevant ceiling: under the **dream-server lemonade stack as-shipped on Strix Halo Linux**, single-user requests longer than ~5 min fail. Vanilla `llama-server` from `b9151` does not have this ceiling (the canonical study's Strix Vulkan ctx=32K gen=2048 cell completes in ~250 s and decode is ~7 tok/s × 2048 = 293 s). The ceiling appears to be a Lemonade Server / FastAPI default; not yet investigated whether tunable. + ## § Engine identification — why this took some unwinding We initially set up bench cells expecting to measure "Lemonade SDK on Strix" — Lemonade's own Vulkan-bundled binary against the same Q8 GGUF. The numbers came back close to canonical Vulkan and we wrote down "Lemonade ≈ vanilla Vulkan on Linux" as the headline. diff --git a/hardware-tests/best-stack-followup-2026-05-17/manifest.json b/hardware-tests/best-stack-followup-2026-05-17/manifest.json index 08353d7b..27939c16 100644 --- a/hardware-tests/best-stack-followup-2026-05-17/manifest.json +++ b/hardware-tests/best-stack-followup-2026-05-17/manifest.json @@ -79,14 +79,16 @@ }, { "path": "strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/", - "status": "appendix-preliminary", - "cells_present": 7, + "status": "appendix-partial-with-ceiling-finding", + "cells_present": 8, + "cells_with_error_marker": 2, "cells_planned": 12, - "load_bearing_for": "Demonstrates that ROCm 7 (dream-server's bundled build) loads and serves Qwen3.6-27B-Q8 on Strix Halo where the canonical study's pinned llama.cpp b9151 + ROCm 6.4.4 segfaulted. Also shows that this path's decode matches vanilla Vulkan within noise while prefill is significantly slower because the bundled engine is at an older upstream commit than b9151. The ctx=16K cell adds a long-context data point: TTFT = 185.6 s (vs canonical Vulkan ttft ~30 s at the same cell) — the older-engine prefill cost shows up most dramatically at long context.", + "load_bearing_for": "Demonstrates that ROCm 7 (dream-server's bundled build) loads and serves Qwen3.6-27B-Q8 on Strix Halo where the canonical study's pinned llama.cpp b9151 + ROCm 6.4.4 segfaulted. Also shows that this path's decode matches vanilla Vulkan within noise while prefill is significantly slower because the bundled engine is at an older upstream commit than b9151. The ctx=16K cells confirm the long-context behavior holds: TTFT ≈ 186 s at gen=128 (vs canonical Vulkan ttft ~30 s at the same cell), decode steady ~7.0 tok/s through gen=128/512. ctx=16K gen=2048 and the entire ctx=32K tier exceed a 300 s server-side response timeout in this dream-server lemonade stack — see known_gaps.", "known_gaps": [ - "ctx=16K gen=512 and gen=2048 cells still running (2 cells)", - "ctx=32K cells not yet started (3 cells)", - "When all 12 land they will arrive in follow-up commits to this bundle." + "ctx=16384_gen2048_conc1 has a .error marker (no .done). All 10 batches timed out at exactly 300.0 s wall time per request. Estimated request length at this cell: ~186 s prefill (84 tok/s × 15603 prompt tokens) + ~293 s decode (2048 / 7 tok/s) = ~480 s, well over the 300 s ceiling.", + "ctx=32768_gen0128_conc1 has a .error marker (no .done). Same 300 s ceiling: prefill at ctx=32K is ~371 s (84 tok/s × 31186 prompt tokens), already over the ceiling before any decode begins.", + "ctx=32K gen=512 and gen=2048 cells are running at snapshot and are expected to fail the same way (will be pulled in as .error cells if they do).", + "The 300 s timeout appears to be a Lemonade Server / FastAPI default; not yet investigated whether it is tunable. Documented as a buyer-relevant ceiling: under the dream-server lemonade stack on Strix Halo Linux, single-user requests longer than ~5 min fail." ] } ], diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/.done b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/.done new file mode 100644 index 00000000..e69de29b diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/batches.jsonl b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/batches.jsonl new file mode 100644 index 00000000..1181ed59 --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/batches.jsonl @@ -0,0 +1,10 @@ +{"batch": 0, "wall_s": 258.707440838, "aggregate_decode_tps": 1.9790694784098197, "per_slot_decode_tps_mean": 7.01277745858835} +{"batch": 1, "wall_s": 258.674228979, "aggregate_decode_tps": 1.9793235763024764, "per_slot_decode_tps_mean": 7.012576617878127} +{"batch": 2, "wall_s": 258.690663903, "aggregate_decode_tps": 1.9791978275334368, "per_slot_decode_tps_mean": 7.011209266671468} +{"batch": 3, "wall_s": 258.659795771, "aggregate_decode_tps": 1.9794340224921942, "per_slot_decode_tps_mean": 7.010876416241539} +{"batch": 4, "wall_s": 258.694397446, "aggregate_decode_tps": 1.9791692632496038, "per_slot_decode_tps_mean": 7.011952173775019} +{"batch": 5, "wall_s": 258.706087298, "aggregate_decode_tps": 1.9790798328229293, "per_slot_decode_tps_mean": 7.01147916081578} +{"batch": 6, "wall_s": 258.690643987, "aggregate_decode_tps": 1.979197979907343, "per_slot_decode_tps_mean": 7.010627015144488} +{"batch": 7, "wall_s": 258.71764173, "aggregate_decode_tps": 1.9789914463364182, "per_slot_decode_tps_mean": 7.010806432387516} +{"batch": 8, "wall_s": 258.676614034, "aggregate_decode_tps": 1.9793053265058727, "per_slot_decode_tps_mean": 7.0122631334210155} +{"batch": 9, "wall_s": 258.657727259, "aggregate_decode_tps": 1.9794498522262296, "per_slot_decode_tps_mean": 7.011407916728575} diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/bench-cell.log b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/bench-cell.log new file mode 100644 index 00000000..d9792639 --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/bench-cell.log @@ -0,0 +1,11 @@ +batch 0: wall=258.71s agg=1.98 tok/s +batch 1: wall=258.67s agg=1.98 tok/s +batch 2: wall=258.69s agg=1.98 tok/s +batch 3: wall=258.66s agg=1.98 tok/s +batch 4: wall=258.69s agg=1.98 tok/s +batch 5: wall=258.71s agg=1.98 tok/s +batch 6: wall=258.69s agg=1.98 tok/s +batch 7: wall=258.72s agg=1.98 tok/s +batch 8: wall=258.68s agg=1.98 tok/s +batch 9: wall=258.66s agg=1.98 tok/s +cell done: per_slot_decode_mean=7.011327689398175 diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/cell.json b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/cell.json new file mode 100644 index 00000000..03d6aa81 --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/cell.json @@ -0,0 +1,41 @@ +{ + "ctx": 16384, + "gen": 512, + "conc": 1, + "n_batches": 10, + "warmup_batches": 2, + "seed": 42, + "started": "2026-05-17T17:08:52.009180+00:00", + "model": "extra.Qwen3.6-27B-Q8_0.gguf", + "engine": "dreamserver-llamacpp-rocm7", + "per_slot": { + "n": 8, + "n_total_including_warmup": 10, + "decode_tps_mean": 7.011327689398175, + "decode_tps_median": 7.011308591700022, + "decode_tps_min": 7.010627015144488, + "decode_tps_max": 7.0122631334210155, + "decode_tps_sd": 0.0005700552550571158, + "prefill_tps_mean": 84.04787338318525, + "ttft_ms_mean": 185644.199625, + "elapsed_s_mean": 258.686603290875, + "prompt_tokens_max": 15603, + "gen_tokens_total": 4096 + }, + "aggregate": { + "n_batches_total": 10, + "n_batches_body": 8, + "aggregate_decode_tps_mean": 1.9792281938842535, + "aggregate_decode_tps_median": 1.9791979037203897, + "batch_wall_s_mean": 258.6866964285 + }, + "cold_start": { + "wall_s": 258.707316844, + "decode_tps": 7.01277745858835, + "aggregate_decode_tps": 1.9790694784098197, + "per_slot_decode_tps_mean": 7.01277745858835 + }, + "inferences_path": "inferences.jsonl", + "batches_path": "batches.jsonl", + "engine_note": "dream-server bundled llama.cpp at /opt/llama-custom/llama-server linked against libamdhip64.so.7 (ROCm 7) + custom librocblas/libhipblaslt; version string b1 (ff5ef82). Fronted by Lemonade Server OpenAI-compatible API on port 11434." +} \ No newline at end of file diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/inferences.jsonl b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/inferences.jsonl new file mode 100644 index 00000000..0fdb3572 --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen0512_conc1/inferences.jsonl @@ -0,0 +1,10 @@ +{"id": "b0_s0", "t0_mono_ns": 80369224480872, "t1_mono_ns": 80627931797716, "wall_start": 1779037732.0093482, "elapsed_s": 258.707316844, "decode_tps": 7.01277745858835, "prefill_tps": 84.0449390438265, "ttft_ms": 185650.679, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 0} +{"id": "b1_s0", "t0_mono_ns": 80627932107733, "t1_mono_ns": 80886606243906, "wall_start": 1779037990.7169747, "elapsed_s": 258.674136173, "decode_tps": 7.012576617878127, "prefill_tps": 84.04784324497093, "ttft_ms": 185644.264, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 1} +{"id": "b2_s0", "t0_mono_ns": 80886606589392, "t1_mono_ns": 81145297157289, "wall_start": 1779038249.3914568, "elapsed_s": 258.690567897, "decode_tps": 7.011209266671468, "prefill_tps": 84.04627046950606, "ttft_ms": 185647.738, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 2} +{"id": "b3_s0", "t0_mono_ns": 81145297492238, "t1_mono_ns": 81403957205221, "wall_start": 1779038508.0823596, "elapsed_s": 258.659712983, "decode_tps": 7.010876416241539, "prefill_tps": 84.06299497556724, "ttft_ms": 185610.803, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 3} +{"id": "b4_s0", "t0_mono_ns": 81403957443502, "t1_mono_ns": 81662651755922, "wall_start": 1779038766.742311, "elapsed_s": 258.69431242, "decode_tps": 7.011952173775019, "prefill_tps": 84.04206944657486, "ttft_ms": 185657.018, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 4} +{"id": "b5_s0", "t0_mono_ns": 81662652103339, "t1_mono_ns": 81921358096381, "wall_start": 1779039025.4369707, "elapsed_s": 258.705993042, "decode_tps": 7.01147916081578, "prefill_tps": 84.03811914793437, "ttft_ms": 185665.745, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 5} +{"id": "b6_s0", "t0_mono_ns": 81921358444228, "t1_mono_ns": 82180048976249, "wall_start": 1779039284.1433115, "elapsed_s": 258.690532021, "decode_tps": 7.010627015144488, "prefill_tps": 84.04888681442971, "ttft_ms": 185641.959, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 6} +{"id": "b7_s0", "t0_mono_ns": 82180049327506, "t1_mono_ns": 82438766875899, "wall_start": 1779039542.8341947, "elapsed_s": 258.717548393, "decode_tps": 7.010806432387516, "prefill_tps": 84.03606787315775, "ttft_ms": 185670.277, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 7} +{"id": "b8_s0", "t0_mono_ns": 82438767235405, "t1_mono_ns": 82697443761743, "wall_start": 1779039801.5521026, "elapsed_s": 258.676526338, "decode_tps": 7.0122631334210155, "prefill_tps": 84.04776446897914, "ttft_ms": 185644.438, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 8} +{"id": "b9_s0", "t0_mono_ns": 82697444085790, "t1_mono_ns": 82956101719023, "wall_start": 1779040060.228953, "elapsed_s": 258.657633233, "decode_tps": 7.011407916728575, "prefill_tps": 84.06081386933285, "ttft_ms": 185615.619, "prompt_tokens": 15603, "gen_tokens": 512, "content_sha256": "3f853616a64eff452e5d66623e17a0a32367be7805838ddef43830c040242ef7", "content_preview": "\n\n\nHere's a thinking process:\n\n1. **Analyze User Input:**\n - **Input Text:** The opening paragraph of Jane Austen's *Pride and Prejudice*, repeated many times (looks like a copy-paste error or intentional repetition). The core text is just the famous opening lines and the first conversation between Mr. and Mrs. Bennet about Netherfield Park being let to Mr. Bingley.\n - **Task:** Summar", "content_len_chars": 2288, "slot": 0, "batch": 9} diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/.error b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/.error new file mode 100644 index 00000000..3635f66d --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/.error @@ -0,0 +1 @@ +bench-cell-failed diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/batches.jsonl b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/batches.jsonl new file mode 100644 index 00000000..9ef7d8d4 --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/batches.jsonl @@ -0,0 +1,10 @@ +{"batch": 0, "wall_s": 300.075433243, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 1, "wall_s": 299.992882535, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 2, "wall_s": 299.991695467, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 3, "wall_s": 299.991457546, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 4, "wall_s": 299.991208596, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 5, "wall_s": 299.990371522, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 6, "wall_s": 299.991232326, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 7, "wall_s": 299.991217108, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 8, "wall_s": 299.991020651, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 9, "wall_s": 299.990978366, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/bench-cell.log b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/bench-cell.log new file mode 100644 index 00000000..e9b9113a --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/bench-cell.log @@ -0,0 +1,11 @@ +batch 0: wall=300.08s agg=0.00 tok/s +batch 1: wall=299.99s agg=0.00 tok/s +batch 2: wall=299.99s agg=0.00 tok/s +batch 3: wall=299.99s agg=0.00 tok/s +batch 4: wall=299.99s agg=0.00 tok/s +batch 5: wall=299.99s agg=0.00 tok/s +batch 6: wall=299.99s agg=0.00 tok/s +batch 7: wall=299.99s agg=0.00 tok/s +batch 8: wall=299.99s agg=0.00 tok/s +batch 9: wall=299.99s agg=0.00 tok/s +cell errored: 10/10 bad inferences diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/cell.json b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/cell.json new file mode 100644 index 00000000..41c742db --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/cell.json @@ -0,0 +1,25 @@ +{ + "ctx": 16384, + "gen": 2048, + "conc": 1, + "n_batches": 10, + "warmup_batches": 2, + "seed": 42, + "started": "2026-05-17T17:51:58.960464+00:00", + "model": "extra.Qwen3.6-27B-Q8_0.gguf", + "engine": "dreamserver-llamacpp-rocm7", + "per_slot": { + "n": 0, + "decode_tps_mean": null + }, + "aggregate": { + "n_batches_total": 10, + "n_batches_body": 8, + "aggregate_decode_tps_mean": 0.0, + "aggregate_decode_tps_median": 0.0, + "batch_wall_s_mean": 299.99114769774997 + }, + "cold_start": null, + "inferences_path": "inferences.jsonl", + "batches_path": "batches.jsonl" +} \ No newline at end of file diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/inferences.jsonl b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/inferences.jsonl new file mode 100644 index 00000000..7aa025be --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx16384_gen2048_conc1/inferences.jsonl @@ -0,0 +1,10 @@ +{"id": "b0_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 82956175707908, "wall_start": 1779040318.9605749, "slot": 0, "batch": 0} +{"id": "b1_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 83256251442322, "wall_start": 1779040619.03631, "slot": 0, "batch": 1} +{"id": "b2_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 83556244539429, "wall_start": 1779040919.0294073, "slot": 0, "batch": 2} +{"id": "b3_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 83856236455728, "wall_start": 1779041219.0213232, "slot": 0, "batch": 3} +{"id": "b4_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 84156228081728, "wall_start": 1779041519.012949, "slot": 0, "batch": 4} +{"id": "b5_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 84456219532674, "wall_start": 1779041819.0044005, "slot": 0, "batch": 5} +{"id": "b6_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 84756210136837, "wall_start": 1779042118.9950047, "slot": 0, "batch": 6} +{"id": "b7_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 85056201577684, "wall_start": 1779042418.9864454, "slot": 0, "batch": 7} +{"id": "b8_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 85356192983216, "wall_start": 1779042718.9778507, "slot": 0, "batch": 8} +{"id": "b9_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 85656184295915, "wall_start": 1779043018.969164, "slot": 0, "batch": 9} diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/.error b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/.error new file mode 100644 index 00000000..3635f66d --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/.error @@ -0,0 +1 @@ +bench-cell-failed diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/batches.jsonl b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/batches.jsonl new file mode 100644 index 00000000..ef64f145 --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/batches.jsonl @@ -0,0 +1,10 @@ +{"batch": 0, "wall_s": 299.999894186, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 1, "wall_s": 299.981838443, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 2, "wall_s": 299.984941129, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 3, "wall_s": 299.986651595, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 4, "wall_s": 299.987288698, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 5, "wall_s": 299.988304431, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 6, "wall_s": 299.988440541, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 7, "wall_s": 299.99460058, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 8, "wall_s": 299.99497966, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} +{"batch": 9, "wall_s": 299.992449785, "aggregate_decode_tps": 0.0, "per_slot_decode_tps_mean": 0} diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/bench-cell.log b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/bench-cell.log new file mode 100644 index 00000000..01e4d724 --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/bench-cell.log @@ -0,0 +1,11 @@ +batch 0: wall=300.00s agg=0.00 tok/s +batch 1: wall=299.98s agg=0.00 tok/s +batch 2: wall=299.98s agg=0.00 tok/s +batch 3: wall=299.99s agg=0.00 tok/s +batch 4: wall=299.99s agg=0.00 tok/s +batch 5: wall=299.99s agg=0.00 tok/s +batch 6: wall=299.99s agg=0.00 tok/s +batch 7: wall=299.99s agg=0.00 tok/s +batch 8: wall=299.99s agg=0.00 tok/s +batch 9: wall=299.99s agg=0.00 tok/s +cell errored: 10/10 bad inferences diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/cell.json b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/cell.json new file mode 100644 index 00000000..bf497084 --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/cell.json @@ -0,0 +1,25 @@ +{ + "ctx": 32768, + "gen": 128, + "conc": 1, + "n_batches": 10, + "warmup_batches": 2, + "seed": 42, + "started": "2026-05-17T18:42:05.607907+00:00", + "model": "extra.Qwen3.6-27B-Q8_0.gguf", + "engine": "dreamserver-llamacpp-rocm7", + "per_slot": { + "n": 0, + "decode_tps_mean": null + }, + "aggregate": { + "n_batches_total": 10, + "n_batches_body": 8, + "aggregate_decode_tps_mean": 0.0, + "aggregate_decode_tps_median": 0.0, + "batch_wall_s_mean": 299.989707052375 + }, + "cold_start": null, + "inferences_path": "inferences.jsonl", + "batches_path": "batches.jsonl" +} \ No newline at end of file diff --git a/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/inferences.jsonl b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/inferences.jsonl new file mode 100644 index 00000000..c9a231cf --- /dev/null +++ b/hardware-tests/best-stack-followup-2026-05-17/strix-halo/qwen3.6-27b/dreamserver-llamacpp-rocm7/ctx32768_gen0128_conc1/inferences.jsonl @@ -0,0 +1,10 @@ +{"id": "b0_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 85962823199721, "wall_start": 1779043325.6080668, "slot": 0, "batch": 0} +{"id": "b1_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 86262823424671, "wall_start": 1779043625.608292, "slot": 0, "batch": 1} +{"id": "b2_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 86562805539858, "wall_start": 1779043925.5904071, "slot": 0, "batch": 2} +{"id": "b3_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 86862790757043, "wall_start": 1779044225.5756245, "slot": 0, "batch": 3} +{"id": "b4_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 87162777597478, "wall_start": 1779044525.5624647, "slot": 0, "batch": 4} +{"id": "b5_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 87462765089966, "wall_start": 1779044825.5499573, "slot": 0, "batch": 5} +{"id": "b6_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 87762753608127, "wall_start": 1779045125.5384755, "slot": 0, "batch": 6} +{"id": "b7_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 88062742332425, "wall_start": 1779045425.5271997, "slot": 0, "batch": 7} +{"id": "b8_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 88362737220779, "wall_start": 1779045725.5220883, "slot": 0, "batch": 8} +{"id": "b9_s0", "error": "HTTP Error 500: Internal Server Error", "t0_mono_ns": 88662732408323, "wall_start": 1779046025.5172758, "slot": 0, "batch": 9}