Skip to content

Commit 8d078ea

Browse files
kmbandyclaude
andcommitted
fix(kv-paged): size paged K/V from first attention layer for hybrid models
The paged KV cache read n_head_kv/head_dim from layer 0, but hybrid models (e.g. LFM2.5 lfm2moe) have conv/recurrent layers (n_head_kv==0) interleaved with attention layers and layer 0 may be non-attention, tripping GGML_ASSERT(n_kv_heads > 0). Derive both from the first attention layer (n_head_kv > 0) instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 1f671b8 commit 8d078ea

1 file changed

Lines changed: 11 additions & 2 deletions

File tree

src/llama-kv-cache-paged.cpp

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,8 +108,17 @@ llama_kv_cache_paged::llama_kv_cache_paged(
108108
// stay default-constructed (k/v == nullptr) and never queried.
109109
const auto & hparams = model.hparams;
110110
const uint32_t n_layer = hparams.n_layer();
111-
const uint32_t head_dim = hparams.n_embd_head_v(/*il=*/0);
112-
const uint32_t n_kv_heads = hparams.n_head_kv(/*il=*/0);
111+
// Hybrid models (e.g. LFM2.5 lfm2moe) interleave conv/recurrent layers
112+
// (n_head_kv == 0) with attention layers; layer 0 may be non-attention.
113+
// Size the paged K/V storage from the first attention layer (n_head_kv > 0).
114+
uint32_t head_dim = 0, n_kv_heads = 0;
115+
for (uint32_t il = 0; il < n_layer; ++il) {
116+
if (hparams.n_head_kv(il) > 0) {
117+
n_kv_heads = hparams.n_head_kv(il);
118+
head_dim = hparams.n_embd_head_v(il);
119+
break;
120+
}
121+
}
113122

114123
GGML_ASSERT(head_dim > 0 && "head_dim must be > 0");
115124
GGML_ASSERT(n_kv_heads > 0 && "n_kv_heads must be > 0");

0 commit comments

Comments
 (0)