Skip to content

Commit f15aca9

Browse files
kmbandyclaude
kmbandy
and
claude
committed
perf-baseline: post-WMMA-fix snapshots (bench + PPL + NIAH on R9700/Qwen3.6-35B)
First clean baseline after the WMMA operand-order fix (ed0db4c). All three sweeps run end-to-end on gfx1201 / R9700, single-GPU, against Qwen3.6-35B-A3B-UD-Q4_K_XL. bench_paged_paths (4 paths × {8K, 64K}): vanilla_f16 8K 2250 t/s prefill, 60.6 t/s decode paged_turbo4 8K 1974 t/s prefill, 32.6 t/s decode paged_turbo3 8K 1954 t/s prefill, 33.3 t/s decode paged_aiter_f16 8K 1688 t/s prefill, 54.9 t/s decode (64K cells in JSON; vanilla 1642/50.9, paged_turbo4 980/31.4, paged_turbo3 939/31.8, paged_aiter 705/36.4) Paged decode is ~half of vanilla — separate perf followup (kernel correctness validated upstream of this commit). ppl_kv_sweep ({f16, turbo4, turbo3} × {4K, 8K} over wiki.test): f16/4K=5.7507 f16/8K=5.7012 turbo4/4K=5.7887 (+0.66%) turbo4/8K=5.7270 (+0.45%) turbo3/4K=5.8226 (+1.25%) turbo3/8K=5.7633 (+1.09%) turbo3 within 0.59-0.63% of turbo4 — under the 1-3% adopt threshold, confirming the turbo3 pivot signal post-fix. (PPL via stock fattn path; not affected by paged-kernel changes.) niah_kv_sweep (11 cells, every cell retrieved the needle): f16 × {8K, 32K, 128K} — 3/3 ✅ turbo4 × {8K, 32K, 128K, 256K} — 4/4 ✅ turbo3 × {8K, 32K, 128K, 256K} — 4/4 ✅ 256K cells use 524288 ctx + tier 65,20,15 + SSD + semantic-index (~12 min per cell, ~321 t/s prefill). Script change: niah_kv_sweep.sh tier-cell ctx_size scales with target (target × 2) instead of hard-coded 1M. The 1M setting was attempting to allocate ~20GB of hot-pool layer storage upfront, which OOMs the R9700 (32GB total, ~21GB consumed by the 35B model). target × 2 keeps the hot-pool layer alloc under the available VRAM for all tested target sizes, with the warm→RAM + cold→SSD wiring still loaded so the tier infrastructure is exercised. f16 deliberately skipped at 256K — the f16 KV footprint alone exceeds the available VRAM at that ctx. Only the 4-bit (turbo4) and 3-bit (turbo3) cells fit the 32GB budget for 256K end-to-end. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent ed0db4c commit f15aca9

5 files changed

Lines changed: 367 additions & 16 deletions

File tree

‎scripts/perf/niah_kv_sweep.sh‎

Lines changed: 59 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -89,16 +89,33 @@ if para_break < 0 or para_break > mid + 4000:
8989
para_break = mid
9090
needle_block = f"\n\n{needle}\n\n"
9191
haystack = raw[:para_break] + needle_block + raw[para_break:]
92-
# Final prompt: haystack + clear question with a label so we can parse output
93-
prompt = haystack + "\n\n" + query + " "
92+
# Chat-format prompt: wrap the haystack + question as a user message so the
93+
# chat-tuned model actually treats it as a question to answer (instead of
94+
# continuing the wikipedia-style haystack). Use /v1/chat/completions endpoint
95+
# in run_cell, which auto-applies the model's chat template (Qwen3 ChatML).
96+
user_content = (
97+
"I am going to give you a long passage of text. Somewhere inside it, "
98+
"a sentence reveals a secret passphrase. Read the whole passage, then "
99+
"answer my question at the end.\n\n"
100+
"=== BEGIN PASSAGE ===\n"
101+
+ haystack +
102+
"\n=== END PASSAGE ===\n\n"
103+
+ query
104+
)
94105
body = {
95-
"prompt": prompt,
96-
"n_predict": 32,
97-
"temperature": 0.0,
98-
"cache_prompt": False,
106+
"messages": [{"role": "user", "content": user_content}],
107+
"max_tokens": 64,
108+
# Tiny non-zero temperature to break the degenerate-repetition loops
109+
# we observed at temperature=0 on long haystacks. NIAH cares about
110+
# retrieval correctness, not bit-exact reproducibility.
111+
"temperature": 0.1,
112+
"top_p": 0.9,
99113
"stream": False,
100-
"ignore_eos": False,
101-
"stop": ["\n", "."],
114+
"cache_prompt": False,
115+
# Qwen3 ChatML defaults to enabling the <think>...</think> reasoning
116+
# block, which gobbles our token budget before any actual answer is
117+
# emitted. Disable it so the model goes straight to the response.
118+
"chat_template_kwargs": {"enable_thinking": False},
102119
}
103120
with open(out_path, "w") as f:
104121
json.dump(body, f)
@@ -118,20 +135,42 @@ run_cell() {
118135
local resp_path="${WORK_DIR}/${label}.resp.json"
119136
rm -f "${server_log}" "${resp_path}"
120137

121-
# ctx-size with paged-blocks admission headroom (~2x prompt).
122-
local ctx_size=$(( target_tokens * 2 ))
123-
if (( ctx_size < 32768 )); then ctx_size=32768; fi
138+
# Per-cell tier config. Short cells (<=TIER_HOT_MAX) stay all-hot so the
139+
# measurement is isolated to the kernel and KV format; long cells engage
140+
# the production tiered stack (warm→RAM, cold→SSD, semantic-index) which
141+
# is the only way 256K+ prompts fit alongside the 35B-A3B model.
142+
local tier_flags ctx_size tier_mode
143+
if (( target_tokens <= TIER_HOT_MAX )); then
144+
tier_mode="100,0,0"
145+
tier_flags="--kv-tiered 100,0,0 --kv-tier-paged-blocks --ctx-checkpoints 0"
146+
ctx_size=$(( target_tokens * 2 ))
147+
if (( ctx_size < 32768 )); then ctx_size=32768; fi
148+
else
149+
tier_mode="65,20,15"
150+
# ctx-size scales with target so the hot-pool layer allocation fits in
151+
# the R9700's 32GB alongside the 35B model (model ~21GB → ~9GB free).
152+
# At ~25KB hot KV per token, hot=65% of (target*2) keeps the hot pool
153+
# under that budget for targets up to ~262K. The warm→RAM + cold→SSD
154+
# wiring still loads so the tier infrastructure is exercised.
155+
tier_flags="--kv-tiered 65,20,15 --kv-tier-paged-blocks --ctx-checkpoints 0 --kv-tier-ssd-path ${KV_TIER_SSD_PATH} --kv-tier-semantic-index ${KV_TIER_SEMANTIC_INDEX}"
156+
ctx_size=$(( target_tokens * 2 ))
157+
fi
124158

125-
echo " [${label}] launching server (cache=${cache_type}, ctx=${ctx_size})..." >&2
159+
echo " [${label}] launching server (cache=${cache_type}, ctx=${ctx_size}, tier=${tier_mode})..." >&2
126160

161+
# --jinja is REQUIRED for /v1/chat/completions to honor the model's
162+
# chat template + chat_template_kwargs in the request body. Without it
163+
# the model produces degenerate output (observed: "The leFallrésrates..."
164+
# on Qwen3.6 with a 7K-token haystack prompt).
165+
# shellcheck disable=SC2086 # tier_flags is intentionally word-split
127166
setsid nohup "${LLAMA_SERVER}" \
128167
--model "${MODEL_PATH}" \
129168
--device ROCm0 --n-gpu-layers 999 \
130169
--ctx-size "${ctx_size}" --parallel 1 \
131-
--kv-tiered 100,0,0 --kv-tier-paged-blocks --ctx-checkpoints 0 \
170+
${tier_flags} \
132171
--cache-type-k "${cache_type}" --cache-type-v "${cache_type}" \
133172
--flash-attn on --no-mmap --no-warmup \
134-
--cache-ram 0 \
173+
--cache-ram 0 --jinja \
135174
--host 127.0.0.1 --port "${PORT}" \
136175
--timeout 3600 --alias niah \
137176
> "${server_log}" 2>&1 < /dev/null &
@@ -165,7 +204,7 @@ run_cell() {
165204
local timeout=$(( target_tokens / 50 + 240 ))
166205
local http_rc=0
167206
local start_ts; start_ts=$(date +%s)
168-
curl -fsS -X POST "http://127.0.0.1:${PORT}/v1/completions" \
207+
curl -fsS -X POST "http://127.0.0.1:${PORT}/v1/chat/completions" \
169208
-H "Content-Type: application/json" \
170209
--data-binary @"${prompt_path}" \
171210
-o "${resp_path}" -m "${timeout}" || http_rc=$?
@@ -185,7 +224,11 @@ import json, sys
185224
cache_type, target_tokens, ctx_size, resp_path, needle_token, elapsed = sys.argv[1:7]
186225
r = json.load(open(resp_path))
187226
t = r.get("timings", {})
188-
text = r.get("choices", [{}])[0].get("text", "")
227+
# /v1/chat/completions returns choices[0].message.content (vs /v1/completions
228+
# which has choices[0].text). Fall back to "text" in case llama-server's chat
229+
# endpoint shape ever diverges.
230+
choice = r.get("choices", [{}])[0]
231+
text = choice.get("message", {}).get("content") or choice.get("text", "") or ""
189232
hit = needle_token in text
190233
out = {
191234
"cache_type": cache_type,

‎tests/perf-baseline/niah-kv/ed0db4c94-20260520T183816Z.json‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
{
2+
"commit": "ed0db4c94a6fa20ac83d93942fcd310ad188ff0f",
3+
"short_commit": "ed0db4c94",
4+
"timestamp_utc": "20260520T183816Z",
5+
"model": "/home/kmbandy/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf",
6+
"needle": "The secret passphrase is verdant-quokka-7831.",
7+
"cells": [
8+
{
9+
"cache_type": "f16",
10+
"target_tokens": 8192,
11+
"ctx_size": 32768,
12+
"prompt_n": 6915,
13+
"prompt_per_second": 1888.5408584380907,
14+
"predicted_n": 10,
15+
"response": "verdant-quokka-7831",
16+
"needle_hit": true,
17+
"elapsed_s": 4
18+
},
19+
{
20+
"cache_type": "f16",
21+
"target_tokens": 32768,
22+
"ctx_size": 65536,
23+
"prompt_n": 29227,
24+
"prompt_per_second": 1269.7993684847675,
25+
"predicted_n": 10,
26+
"response": "verdant-quokka-7831",
27+
"needle_hit": true,
28+
"elapsed_s": 24
29+
},
30+
{
31+
"cache_type": "f16",
32+
"target_tokens": 131072,
33+
"ctx_size": 262144,
34+
"prompt_n": 113419,
35+
"prompt_per_second": 470.3256659082463,
36+
"predicted_n": 10,
37+
"response": "verdant-quokka-7831",
38+
"needle_hit": true,
39+
"elapsed_s": 242
40+
}
41+
]
42+
}

‎tests/perf-baseline/niah-kv/ed0db4c94-20260520T184332Z.json‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
{
2+
"commit": "ed0db4c94a6fa20ac83d93942fcd310ad188ff0f",
3+
"short_commit": "ed0db4c94",
4+
"timestamp_utc": "20260520T184332Z",
5+
"model": "/home/kmbandy/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf",
6+
"needle": "The secret passphrase is verdant-quokka-7831.",
7+
"cells": [
8+
{
9+
"cache_type": "turbo4",
10+
"target_tokens": 8192,
11+
"ctx_size": 32768,
12+
"prompt_n": 6915,
13+
"prompt_per_second": 1941.6107254408003,
14+
"predicted_n": 10,
15+
"response": "verdant-quokka-7831",
16+
"needle_hit": true,
17+
"elapsed_s": 4
18+
},
19+
{
20+
"cache_type": "turbo4",
21+
"target_tokens": 32768,
22+
"ctx_size": 65536,
23+
"prompt_n": 29227,
24+
"prompt_per_second": 1345.003003442742,
25+
"predicted_n": 10,
26+
"response": "verdant-quokka-7831",
27+
"needle_hit": true,
28+
"elapsed_s": 22
29+
},
30+
{
31+
"cache_type": "turbo4",
32+
"target_tokens": 131072,
33+
"ctx_size": 262144,
34+
"prompt_n": 113419,
35+
"prompt_per_second": 519.7664187064474,
36+
"predicted_n": 10,
37+
"response": "verdant-quokka-7831",
38+
"needle_hit": true,
39+
"elapsed_s": 219
40+
},
41+
{
42+
"cache_type": "turbo4",
43+
"target_tokens": 262144,
44+
"ctx_size": 524288,
45+
"prompt_n": 228181,
46+
"prompt_per_second": 321.0505698876605,
47+
"predicted_n": 10,
48+
"response": "verdant-quokka-7831",
49+
"needle_hit": true,
50+
"elapsed_s": 712
51+
},
52+
{
53+
"cache_type": "turbo3",
54+
"target_tokens": 8192,
55+
"ctx_size": 32768,
56+
"prompt_n": 6915,
57+
"prompt_per_second": 1932.1881590815356,
58+
"predicted_n": 10,
59+
"response": "verdant-quokka-7831",
60+
"needle_hit": true,
61+
"elapsed_s": 4
62+
},
63+
{
64+
"cache_type": "turbo3",
65+
"target_tokens": 32768,
66+
"ctx_size": 65536,
67+
"prompt_n": 29227,
68+
"prompt_per_second": 1332.2451834370386,
69+
"predicted_n": 10,
70+
"response": "verdant-quokka-7831",
71+
"needle_hit": true,
72+
"elapsed_s": 22
73+
},
74+
{
75+
"cache_type": "turbo3",
76+
"target_tokens": 131072,
77+
"ctx_size": 262144,
78+
"prompt_n": 113419,
79+
"prompt_per_second": 509.3033025485018,
80+
"predicted_n": 10,
81+
"response": "verdant-quokka-7831",
82+
"needle_hit": true,
83+
"elapsed_s": 224
84+
},
85+
{
86+
"cache_type": "turbo3",
87+
"target_tokens": 262144,
88+
"ctx_size": 524288,
89+
"prompt_n": 228181,
90+
"prompt_per_second": 315.9512972445368,
91+
"predicted_n": 10,
92+
"response": "verdant-quokka-7831",
93+
"needle_hit": true,
94+
"elapsed_s": 724
95+
}
96+
]
97+
}

‎tests/perf-baseline/paged-paths/ed0db4c94-20260520T172622Z.json‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
{
2+
"commit": "ed0db4c94a6fa20ac83d93942fcd310ad188ff0f",
3+
"short_commit": "ed0db4c94",
4+
"timestamp_utc": "20260520T172622Z",
5+
"model": "/home/kmbandy/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf",
6+
"cells": [
7+
{
8+
"path": "paged_turbo4",
9+
"aiter": 0,
10+
"target_tokens": 8192,
11+
"cache_type": "turbo4",
12+
"ctx_size": 32768,
13+
"prompt_n": 9416,
14+
"prompt_ms": 4769.441,
15+
"prompt_per_second": 1974.2355550681937,
16+
"predicted_n": 32,
17+
"predicted_ms": 981.047,
18+
"predicted_per_second": 32.61821299081491
19+
},
20+
{
21+
"path": "paged_turbo4",
22+
"aiter": 0,
23+
"target_tokens": 65536,
24+
"cache_type": "turbo4",
25+
"ctx_size": 131072,
26+
"prompt_n": 61625,
27+
"prompt_ms": 62877.118,
28+
"prompt_per_second": 980.0862692211814,
29+
"predicted_n": 32,
30+
"predicted_ms": 1020.551,
31+
"predicted_per_second": 31.35561084159439
32+
},
33+
{
34+
"path": "paged_turbo3",
35+
"aiter": 0,
36+
"target_tokens": 8192,
37+
"cache_type": "turbo3",
38+
"ctx_size": 32768,
39+
"prompt_n": 9416,
40+
"prompt_ms": 4818.393,
41+
"prompt_per_second": 1954.1784989310752,
42+
"predicted_n": 32,
43+
"predicted_ms": 962.299,
44+
"predicted_per_second": 33.25369765530256
45+
},
46+
{
47+
"path": "paged_turbo3",
48+
"aiter": 0,
49+
"target_tokens": 65536,
50+
"cache_type": "turbo3",
51+
"ctx_size": 131072,
52+
"prompt_n": 61625,
53+
"prompt_ms": 65639.922,
54+
"prompt_per_second": 938.8341442575145,
55+
"predicted_n": 32,
56+
"predicted_ms": 1005.678,
57+
"predicted_per_second": 31.8193298451393
58+
},
59+
{
60+
"path": "paged_aiter_f16",
61+
"aiter": 1,
62+
"target_tokens": 8192,
63+
"cache_type": "f16",
64+
"ctx_size": 32768,
65+
"prompt_n": 9416,
66+
"prompt_ms": 5579.473,
67+
"prompt_per_second": 1687.61458295434,
68+
"predicted_n": 32,
69+
"predicted_ms": 582.694,
70+
"predicted_per_second": 54.91733225329247
71+
},
72+
{
73+
"path": "paged_aiter_f16",
74+
"aiter": 1,
75+
"target_tokens": 65536,
76+
"cache_type": "f16",
77+
"ctx_size": 131072,
78+
"prompt_n": 61625,
79+
"prompt_ms": 87349.927,
80+
"prompt_per_second": 705.4957241120534,
81+
"predicted_n": 32,
82+
"predicted_ms": 878.049,
83+
"predicted_per_second": 36.44443533333561
84+
},
85+
{
86+
"path": "vanilla_f16",
87+
"aiter": 0,
88+
"target_tokens": 8192,
89+
"cache_type": "f16",
90+
"ctx_size": 32768,
91+
"prompt_n": 9416,
92+
"prompt_ms": 4185.011,
93+
"prompt_per_second": 2249.934349037553,
94+
"predicted_n": 32,
95+
"predicted_ms": 527.911,
96+
"predicted_per_second": 60.6162781226381
97+
},
98+
{
99+
"path": "vanilla_f16",
100+
"aiter": 0,
101+
"target_tokens": 65536,
102+
"cache_type": "f16",
103+
"ctx_size": 131072,
104+
"prompt_n": 61625,
105+
"prompt_ms": 37522.781,
106+
"prompt_per_second": 1642.335625389813,
107+
"predicted_n": 32,
108+
"predicted_ms": 628.921,
109+
"predicted_per_second": 50.880794249198225
110+
}
111+
]
112+
}

0 commit comments

Comments
 (0)