Skip to content

Commit 93a1e6a

Browse files
kmbandyclaude
andcommitted
fix(cuda): size paged-decode partials by total_q_tokens, not avg_q_len
launch_paged_attn_decode sizes the partials scratch and strides every head/seq/chunk partial offset by max_q_len, which the dispatch set to avg_q_len = total_q_tokens / num_seqs. With --parallel N and one active request (1 active seq + N-1 idle slots) this floors to 0 (1/4 = 0), so: * partials_n = n_heads * num_seqs * num_chunks * 0 * (HS+2) = 0 (empty buf) * partial_chunk_base = (...) * max_q_len(0) * (HS+2) = 0 for ALL heads/seqs/chunks -> every partial write collapses onto offset 0 of a zero-sized buffer -> OOB + total collision -> garbage tokens. Only surfaced under --parallel > 1 (decode path); --parallel 1 has avg_q_len >= 1. Manifested as token-salad on LFM2.5 paged turbo4 at the service's --parallel 4 (clean at --parallel 1); q8_0 only escaped by luck of allocation layout (the OOB is UB). Fix: max_q_len = total_q_tokens. The partials inner dim must be >= the largest per-seq q_len; total_q_tokens is a safe upper bound (no seq exceeds the batch total) and is already gate-capped to <= 8. The decode gate was previously fixed to key off total_q_tokens for the same flooring reason; this carries that correction through to the buffer sizing it forgot. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent fb556b1 commit 93a1e6a

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

ggml/src/ggml-cuda/mt_pagedattn.cu

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1208,7 +1208,7 @@ void ggml_cuda_op_paged_attn_mt(ggml_backend_cuda_context & ctx, ggml_tensor * d
12081208
num_seqs, (int) k_cur->ne[2], n_kv_heads, stream);
12091209

12101210
const int num_chunks = paged_attn_decode_num_chunks(max_ctx_len);
1211-
const int max_q_len = avg_q_len; // see comment above
1211+
const int max_q_len = total_q_tokens; // partials inner stride: must be >= max per-seq q_len. avg_q_len = total/num_seqs floors to 0 with idle parallel slots (1 active + N idle), collapsing every head/seq/chunk partial offset to 0 -> OOB corruption. total_q_tokens (gate-capped <= 8) is a safe upper bound.
12121212
const size_t partials_n = (size_t) n_heads * (size_t) num_seqs
12131213
* (size_t) num_chunks * (size_t) max_q_len
12141214
* (size_t) (HS + 2);

0 commit comments

Comments
 (0)