Skip to content

Commit bf87261

Browse files
kmbandyclaude
kmbandy
and
claude
committed
perf/bench: add vanilla_f16 axis + first locked baseline
bench_paged_paths.sh now runs three paths × two ctx sizes (six cells): - paged_turbo4 — --kv-tiered 100,0,0 + paged-blocks + TURBO4 KV (prod tier) - paged_aiter_f16 — same paged stack but MAD_USE_AITER=1 + F16 KV (MAD-199 gate) - vanilla_f16 — no --kv-tiered, no paged-blocks, F16 KV (mainline-comparable) The vanilla cell tracks how upstream/stock llama.cpp would behave on the same hardware. With future upstream syncs it gives a clean delta vs our tiered/paged work. Per the 2026-05-18 dev-process feedback, both paths must be benched after each commit; the vanilla row adds a third reference so we don't conflate stock F16 numbers (e.g. last night's "50 t/s decode") with paged-TURBO4 numbers. Also commits 3a7e61a-20260519T205603Z.json as the first locked baseline at HEAD (3a7e61a). On Qwen3.6-35B-A3B-UD-Q4_K_XL @ R9700 / gfx1201: - vanilla_f16 8K: 2250 prefill / 59.2 decode | 64K: 1672 / 48.0 - paged_aiter_f16 8K: 1689 / 54.4 | 64K: 706 / 35.3 - paged_turbo4 8K: 1948 / 32.0 | 64K: 952 / 30.6 Reproduces with: scripts/perf/bench_paged_paths.sh (N_PREDICT=128 used here). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 3a7e61a commit bf87261

2 files changed

Lines changed: 160 additions & 31 deletions

File tree

‎scripts/perf/bench_paged_paths.sh‎

Lines changed: 74 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,24 @@
11
#!/usr/bin/env bash
2-
# bench_paged_paths.sh — bench AITER on/off × prefill+decode × 8K/64K.
2+
# bench_paged_paths.sh — bench three attention paths × prefill+decode × 8K/64K.
33
#
44
# Mechanizes the "test both paged attention paths every commit" rule. Unlike
55
# profile_prefill.sh this does NOT use rocprof (which slows the GPU ~2x and
66
# hides production-curve regressions); it runs llama-server clean and reads
77
# timings off the /v1/completions response.
88
#
99
# Two axes:
10-
# aiter ∈ {0, 1} — MAD_USE_AITER=1 forces the AITER unified_attention
11-
# path. AITER currently asserts F16 KV (MAD-199), so
12-
# the AITER row uses --cache-type-k f16; the non-AITER
13-
# row uses the production TURBO4 cache type.
14-
# ctx ∈ {8K, 64K} — short prompt + long prompt, covers the bend in the
15-
# prefill curve where the tile kernel kicks in.
10+
# path ∈ {paged_turbo4, paged_aiter_f16, vanilla_f16}
11+
# — paged_turbo4: production-tier path
12+
# (--kv-tiered 100,0,0 --kv-tier-paged-blocks,
13+
# TURBO4 quantized KV, MAD_USE_AITER unset)
14+
# — paged_aiter_f16: AITER unified_attention
15+
# (--kv-tiered 100,0,0 --kv-tier-paged-blocks,
16+
# F16 KV per MAD-199, MAD_USE_AITER=1)
17+
# — vanilla_f16: stock llama.cpp F16 KV
18+
# (no --kv-tiered, no --kv-tier-paged-blocks)
19+
# tracks upstream / mainline-comparable perf.
20+
# ctx ∈ {8K, 64K} — short prompt + long prompt, covers the bend in
21+
# the prefill curve where the tile kernel kicks in.
1622
#
1723
# Output: tests/perf-baseline/paged-paths/<short_commit>-<UTC_timestamp>.json
1824
#
@@ -69,9 +75,19 @@ prompt_file_for() {
6975
import json, sys
7076
target_tokens, out_path, n_predict = sys.argv[1:4]
7177
target_tokens = int(target_tokens); n_predict = int(n_predict)
72-
# ~3.6 chars/token avg on Qwen for English prose. Pad a bit so we land at
73-
# or just above the target.
74-
char_target = int(target_tokens * 3.8)
78+
# Measured: Qwen BPE on our synthetic English corpus = ~4.78 chars/token.
79+
# Two competing constraints:
80+
# - For small targets (8K), the decode-gate at mt_pagedattn.cu:1028 requires
81+
# max_ctx_len >= 8192. Below that, decode falls to scalar fallback
82+
# (~32 t/s instead of ~50 t/s on Qwen3.6-35B-A3B). Need to overshoot.
83+
# - For large targets (64K), admission (MAD-141) needs prompt < ~0.5x ctx.
84+
# Overshooting drives the request past the hot pool and stalls.
85+
# So: aggressive overshoot at small target (clears the gate), undershoot at
86+
# large target (stays admissible). Both land far above the gate threshold.
87+
if target_tokens <= 16384:
88+
char_target = int(target_tokens * 5.5)
89+
else:
90+
char_target = int(target_tokens * 4.5)
7591
seed = (
7692
"The pagedattn benchmark needles its way through tiered KV cache, "
7793
"stitching tile kernels to flash-decode and watching where the bytes "
@@ -84,6 +100,11 @@ body = {
84100
"temperature": 0.0,
85101
"cache_prompt": False,
86102
"stream": False,
103+
# The synthetic prompt sometimes hits an early "stop" finish_reason after
104+
# ~2 generated tokens (e.g., \n\n). Force the full decode window so the
105+
# bench actually measures n_predict tokens of decode throughput.
106+
"ignore_eos": True,
107+
"stop": [],
87108
}
88109
with open(out_path, "w") as f:
89110
json.dump(body, f)
@@ -92,38 +113,59 @@ PY
92113
}
93114

94115
# ----------------------------------------------------------------------------
95-
# Run one (aiter, ctx) cell. Echoes a single JSON object to stdout.
116+
# Run one (path, ctx) cell. Echoes a single JSON object to stdout.
96117
# ----------------------------------------------------------------------------
97118
run_cell() {
98-
local aiter="$1" target_tokens="$2"
119+
local path="$1" target_tokens="$2"
99120
local prompt_path; prompt_path="$(prompt_file_for "${target_tokens}")"
100121

101-
local label="aiter${aiter}_ctx${target_tokens}"
122+
local label="${path}_ctx${target_tokens}"
102123
local server_log="${WORK_DIR}/${label}.server.log"
103124
local resp_path="${WORK_DIR}/${label}.resp.json"
104125
rm -f "${server_log}" "${resp_path}"
105126

106-
# AITER currently requires F16 KV (MAD-199). Non-AITER uses prod TURBO4.
107-
local cache_type
108-
if [[ "${aiter}" == "1" ]]; then
109-
cache_type="f16"
110-
else
111-
cache_type="turbo4"
112-
fi
113-
# ctx-size: must be >= target tokens + n_predict. Round up for headroom.
114-
local ctx_size=$(( target_tokens + N_PREDICT + 256 ))
115-
if (( ctx_size < 8192 )); then ctx_size=8192; fi
127+
# Per-path config: cache type, AITER toggle, KV-mode flags.
128+
local cache_type aiter_env tier_flags
129+
case "${path}" in
130+
paged_turbo4)
131+
cache_type="turbo4"; aiter_env="0"
132+
tier_flags="--kv-tiered 100,0,0 --kv-tier-paged-blocks --ctx-checkpoints 0"
133+
;;
134+
paged_aiter_f16)
135+
cache_type="f16"; aiter_env="1"
136+
tier_flags="--kv-tiered 100,0,0 --kv-tier-paged-blocks --ctx-checkpoints 0"
137+
;;
138+
vanilla_f16)
139+
# Stock llama.cpp: no tiered KV, no paged blocks. F16 cache.
140+
# This is the upstream-comparable baseline — what mainline
141+
# llama.cpp would do on the same hardware/model.
142+
cache_type="f16"; aiter_env="0"
143+
tier_flags=""
144+
;;
145+
*) echo "ERROR: unknown path '${path}'" >&2; return 1 ;;
146+
esac
147+
148+
# ctx-size: with paged + --kv-tiered 100,0,0 the entire KV pool is hot —
149+
# no tier to demote to — so MAD-141 admission needs ~2x prompt-size
150+
# headroom (a 1.5x ratio failed mid-prefill on the 64K cell). Same 2x
151+
# ratio is fine for vanilla_f16 (no admission machinery there, but
152+
# consistent ctx makes paths comparable). At 2x, F16 at 64K target ≈
153+
# 8GB KV which fits alongside the 35B Q4 weights on R9700. 32K floor
154+
# keeps the 8K cell on the same headroom curve.
155+
local ctx_size=$(( target_tokens * 2 ))
156+
if (( ctx_size < 32768 )); then ctx_size=32768; fi
116157

117-
echo " [${label}] launching server (cache=${cache_type}, ctx=${ctx_size})..." >&2
158+
echo " [${label}] launching server (cache=${cache_type}, ctx=${ctx_size}, aiter=${aiter_env})..." >&2
118159

119-
MAD_USE_AITER="${aiter}" setsid nohup "${LLAMA_SERVER}" \
160+
# shellcheck disable=SC2086 # tier_flags is intentionally word-split
161+
MAD_USE_AITER="${aiter_env}" setsid nohup "${LLAMA_SERVER}" \
120162
--model "${MODEL_PATH}" \
121163
--device ROCm0 --n-gpu-layers 999 \
122164
--ctx-size "${ctx_size}" --parallel 1 \
123-
--kv-tiered 100,0,0 \
165+
${tier_flags} \
124166
--cache-type-k "${cache_type}" --cache-type-v "${cache_type}" \
125167
--flash-attn on --no-mmap --no-warmup \
126-
--cache-ram 0 --ctx-checkpoints 0 \
168+
--cache-ram 0 \
127169
--host 127.0.0.1 --port "${PORT}" \
128170
--timeout 3600 --alias bench \
129171
> "${server_log}" 2>&1 < /dev/null &
@@ -180,12 +222,13 @@ run_cell() {
180222
return 1
181223
fi
182224

183-
python3 - "${aiter}" "${target_tokens}" "${cache_type}" "${ctx_size}" "${resp_path}" <<'PY'
225+
python3 - "${path}" "${aiter_env}" "${target_tokens}" "${cache_type}" "${ctx_size}" "${resp_path}" <<'PY'
184226
import json, sys
185-
aiter, target_tokens, cache_type, ctx_size, resp_path = sys.argv[1:6]
227+
path, aiter, target_tokens, cache_type, ctx_size, resp_path = sys.argv[1:7]
186228
r = json.load(open(resp_path))
187229
t = r.get("timings", {})
188230
out = {
231+
"path": path,
189232
"aiter": int(aiter),
190233
"target_tokens": int(target_tokens),
191234
"cache_type": cache_type,
@@ -209,9 +252,9 @@ echo " out=${OUT_PATH}" >&2
209252

210253
cells=()
211254
IFS=',' read -r -a CTX_ARR <<< "${CTX_SIZES}"
212-
for aiter in 0 1; do
255+
for path in paged_turbo4 paged_aiter_f16 vanilla_f16; do
213256
for ctx in "${CTX_ARR[@]}"; do
214-
cell_json="$(run_cell "${aiter}" "${ctx}")"
257+
cell_json="$(run_cell "${path}" "${ctx}")"
215258
cells+=("${cell_json}")
216259
done
217260
done

‎tests/perf-baseline/paged-paths/3a7e61a72-20260519T205603Z.json‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
{
2+
"commit": "3a7e61a72e20f88d0ed5ab007b15132d6ceb7ec5",
3+
"short_commit": "3a7e61a72",
4+
"timestamp_utc": "20260519T205603Z",
5+
"model": "/home/kmbandy/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf",
6+
"cells": [
7+
{
8+
"path": "paged_turbo4",
9+
"aiter": 0,
10+
"target_tokens": 8192,
11+
"cache_type": "turbo4",
12+
"ctx_size": 32768,
13+
"prompt_n": 9416,
14+
"prompt_ms": 4832.675,
15+
"prompt_per_second": 1948.403317003523,
16+
"predicted_n": 128,
17+
"predicted_ms": 4009.383,
18+
"predicted_per_second": 31.92511166930174
19+
},
20+
{
21+
"path": "paged_turbo4",
22+
"aiter": 0,
23+
"target_tokens": 65536,
24+
"cache_type": "turbo4",
25+
"ctx_size": 131072,
26+
"prompt_n": 61625,
27+
"prompt_ms": 64741.651,
28+
"prompt_per_second": 951.860186574482,
29+
"predicted_n": 128,
30+
"predicted_ms": 4187.654,
31+
"predicted_per_second": 30.56604007876486
32+
},
33+
{
34+
"path": "paged_aiter_f16",
35+
"aiter": 1,
36+
"target_tokens": 8192,
37+
"cache_type": "f16",
38+
"ctx_size": 32768,
39+
"prompt_n": 9416,
40+
"prompt_ms": 5575.539,
41+
"prompt_per_second": 1688.8053334395115,
42+
"predicted_n": 128,
43+
"predicted_ms": 2354.834,
44+
"predicted_per_second": 54.35627309610784
45+
},
46+
{
47+
"path": "paged_aiter_f16",
48+
"aiter": 1,
49+
"target_tokens": 65536,
50+
"cache_type": "f16",
51+
"ctx_size": 131072,
52+
"prompt_n": 61625,
53+
"prompt_ms": 87282.165,
54+
"prompt_per_second": 706.0434396878217,
55+
"predicted_n": 128,
56+
"predicted_ms": 3626.049,
57+
"predicted_per_second": 35.30012970039842
58+
},
59+
{
60+
"path": "vanilla_f16",
61+
"aiter": 0,
62+
"target_tokens": 8192,
63+
"cache_type": "f16",
64+
"ctx_size": 32768,
65+
"prompt_n": 9416,
66+
"prompt_ms": 4185.096,
67+
"prompt_per_second": 2249.888652494471,
68+
"predicted_n": 128,
69+
"predicted_ms": 2160.697,
70+
"predicted_per_second": 59.24014334263434
71+
},
72+
{
73+
"path": "vanilla_f16",
74+
"aiter": 0,
75+
"target_tokens": 65536,
76+
"cache_type": "f16",
77+
"ctx_size": 131072,
78+
"prompt_n": 61625,
79+
"prompt_ms": 36866.17,
80+
"prompt_per_second": 1671.5867148662312,
81+
"predicted_n": 128,
82+
"predicted_ms": 2665.361,
83+
"predicted_per_second": 48.02351351280371
84+
}
85+
]
86+
}

0 commit comments

Comments
 (0)