Skip to content

Commit 6653f64

Browse files
kmbandyclaude
andcommitted
fix(context): keep n_outputs == n_tokens after n_seqs rounding for mean/rank pooling
graph_reserve() rounds n_tokens up to a multiple of n_seqs, but callers (e.g. the pp reserve via n_outputs_pp) compute n_outputs from the unrounded n_tokens. For embedding models with mean/rank pooling, build_pooling multiplies t_embd (reduced to [n_outputs,...] via out_ids) by inp_mean of shape [n_tokens, n_seqs], which requires n_outputs == n_tokens; the rounding broke that invariant and tripped GGML_ASSERT(ggml_can_mul_mat) during graph reserve (e.g. llama-embed / nemotron-embed at n_ubatch=1280, n_seqs=3 -> n_tokens rounded to 1281). Re-establish the invariant after rounding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 100bce6 commit 6653f64

1 file changed

Lines changed: 10 additions & 0 deletions

File tree

src/llama-context.cpp

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2287,6 +2287,16 @@ ggml_cgraph * llama_context::graph_reserve(
22872287
LLAMA_LOG_DEBUG("%s: making n_tokens a multiple of n_seqs - n_tokens = %u, n_seqs = %u, n_outputs = %u\n", __func__, n_tokens, n_seqs, n_outputs);
22882288
}
22892289

2290+
// For embedding models with mean/rank pooling, build_pooling multiplies t_embd (reduced to
2291+
// [n_outputs, ...] via out_ids) by inp_mean of shape [n_tokens, n_seqs], which requires
2292+
// n_outputs == n_tokens. Rounding n_tokens up to a multiple of n_seqs above can break that
2293+
// invariant (callers compute n_outputs from the unrounded n_tokens), so re-establish it here.
2294+
if (cparams.embeddings &&
2295+
(cparams.pooling_type == LLAMA_POOLING_TYPE_MEAN || cparams.pooling_type == LLAMA_POOLING_TYPE_RANK) &&
2296+
n_outputs != n_tokens) {
2297+
n_outputs = n_tokens;
2298+
}
2299+
22902300
ggml_backend_sched_reset(sched.get());
22912301

22922302
// when the scheduler is reset, we cannot reuse the old graph, so we reset the previous graph result to prevent that

0 commit comments

Comments
 (0)