Skip to content

Commit f057cc9

Browse files
kmbandyclaude
andcommitted
fix(wp): keep tiny per-layer weights resident (don't page < 4KB)
DeepSeek V4 introduces tiny per-layer weights used in non-matmul ops (hyper-connection scale/base {3}/{hc_mix_dim}, attn sinks, expert-prob bias). The pager paged everything per-layer except a few named tensors, so these got a null buffer; a paged non-matmul leaf never gets a backend assigned and ggml_gallocr asserts (buffer_id >= 0) during sched_reserve, crashing any DeepSeek-V4 paged load. Add a 4KB floor to both the is_paged_weight predicate and the weight_page_infos registration so tiny weights stay resident. Standard {n_embd} norms are well above 4KB and remain paged (27B/LFM behavior unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZRfPpL8XFzk1hep9MMg9P
1 parent 978a720 commit f057cc9

1 file changed

Lines changed: 15 additions & 0 deletions

File tree

src/llama-model.cpp

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1597,12 +1597,21 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
15971597
// (token_embd, output_norm, output.weight) and non-weight tensors
15981598
// (RoPE freqs, positional embeddings, etc.) are NOT paged — they
15991599
// get real allocation + load_all_data via the normal path.
1600+
// Don't page tiny per-layer weights (RMS/norm scales, hyper-connection
1601+
// scale/base vectors, attention sinks, expert-prob biases). Paging a
1602+
// few-byte tensor is pointless (overhead >> payload) and a paged
1603+
// non-matmul leaf left with buffer==NULL never gets a backend
1604+
// assigned, tripping the gallocr buffer_id>=0 assert (hit on
1605+
// DeepSeek V4's hc_attn_scale {3}). Standard {n_embd} norms are well
1606+
// above this and stay paged, preserving existing behavior.
1607+
constexpr size_t WP_MIN_PAGED_BYTES = 4096;
16001608
auto is_paged_weight = [&](ggml_tensor * t) -> bool {
16011609
const char * n = ggml_get_name(t);
16021610
if (ml.get_weight(n) == nullptr) return false; // not a weight at all
16031611
if (std::strncmp(n, "token_embd", 10) == 0) return false;
16041612
if (std::strncmp(n, "output_norm", 11) == 0) return false;
16051613
if (std::strcmp (n, "output.weight") == 0) return false;
1614+
if (ggml_nbytes(t) < WP_MIN_PAGED_BYTES) return false; // tiny -> resident
16061615
return true; // it's a paged per-layer weight
16071616
};
16081617
const bool paging_on_device_buft =
@@ -1743,6 +1752,12 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
17431752
std::strcmp (n, "output.weight") == 0) {
17441753
continue;
17451754
}
1755+
// Tiny per-layer weights stay resident — see WP_MIN_PAGED_BYTES
1756+
// rationale above (paged non-matmul leaf gets no backend ->
1757+
// gallocr buffer_id assert; hit on DeepSeek V4 hc_attn_scale).
1758+
if (ggml_nbytes(t) < WP_MIN_PAGED_BYTES) {
1759+
continue;
1760+
}
17461761
}
17471762
llama_weight_page_info info;
17481763
info.name = ggml_get_name(t);

0 commit comments

Comments
 (0)