Commit 93a1e6a
fix(cuda): size paged-decode partials by total_q_tokens, not avg_q_len
launch_paged_attn_decode sizes the partials scratch and strides every
head/seq/chunk partial offset by max_q_len, which the dispatch set to
avg_q_len = total_q_tokens / num_seqs. With --parallel N and one active
request (1 active seq + N-1 idle slots) this floors to 0 (1/4 = 0), so:
* partials_n = n_heads * num_seqs * num_chunks * 0 * (HS+2) = 0 (empty buf)
* partial_chunk_base = (...) * max_q_len(0) * (HS+2) = 0 for ALL
heads/seqs/chunks -> every partial write collapses onto offset 0 of a
zero-sized buffer -> OOB + total collision -> garbage tokens.
Only surfaced under --parallel > 1 (decode path); --parallel 1 has
avg_q_len >= 1. Manifested as token-salad on LFM2.5 paged turbo4 at the
service's --parallel 4 (clean at --parallel 1); q8_0 only escaped by luck
of allocation layout (the OOB is UB).
Fix: max_q_len = total_q_tokens. The partials inner dim must be >= the
largest per-seq q_len; total_q_tokens is a safe upper bound (no seq exceeds
the batch total) and is already gate-capped to <= 8. The decode gate was
previously fixed to key off total_q_tokens for the same flooring reason;
this carries that correction through to the buffer sizing it forgot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>1 parent fb556b1 commit 93a1e6a
1 file changed
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1208 | 1208 | | |
1209 | 1209 | | |
1210 | 1210 | | |
1211 | | - | |
| 1211 | + | |
1212 | 1212 | | |
1213 | 1213 | | |
1214 | 1214 | | |
| |||
0 commit comments