Skip to content

Commit 3faad72

Browse files
kmbandyclaude
andcommitted
fix(wp): keep tiny per-layer weights resident (don't page < 4KB)
DeepSeek V4 introduces tiny per-layer weights used in non-matmul ops (hyper-connection scale/base {3}/{hc_mix_dim}, attn sinks, expert-prob bias). The pager paged everything per-layer except a few named tensors, so these got a null buffer; a paged non-matmul leaf never gets a backend assigned and ggml_gallocr asserts (buffer_id >= 0) during sched_reserve, crashing any DeepSeek-V4 paged load. Add a 4KB floor to both the is_paged_weight predicate and the weight_page_infos registration so tiny weights stay resident. Standard {n_embd} norms are well above 4KB and remain paged (27B/LFM behavior unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZRfPpL8XFzk1hep9MMg9P
1 parent 978a720 commit 3faad72

1 file changed

Lines changed: 15 additions & 0 deletions

File tree

src/llama-model.cpp

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,14 @@
4141
#include <string>
4242
#include <vector>
4343

44+
// Weight paging: minimum tensor size to bother paging. Tiny per-layer weights
45+
// (norm/RMS scales, hyper-connection scale/base vectors, attention sinks,
46+
// expert-prob biases) stay resident — paging a few-byte tensor is pointless and
47+
// a paged non-matmul leaf left with buffer==NULL never gets a backend assigned,
48+
// tripping the gallocr buffer_id>=0 assert (hit on DeepSeek V4 hc_attn_scale
49+
// {3}). Standard {n_embd} norms are well above this and remain paged.
50+
static constexpr size_t WP_MIN_PAGED_BYTES = 4096;
51+
4452
static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params & params) {
4553
switch (arch) {
4654
case LLM_ARCH_LLAMA:
@@ -1603,6 +1611,7 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
16031611
if (std::strncmp(n, "token_embd", 10) == 0) return false;
16041612
if (std::strncmp(n, "output_norm", 11) == 0) return false;
16051613
if (std::strcmp (n, "output.weight") == 0) return false;
1614+
if (ggml_nbytes(t) < WP_MIN_PAGED_BYTES) return false; // tiny -> resident
16061615
return true; // it's a paged per-layer weight
16071616
};
16081617
const bool paging_on_device_buft =
@@ -1743,6 +1752,12 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
17431752
std::strcmp (n, "output.weight") == 0) {
17441753
continue;
17451754
}
1755+
// Tiny per-layer weights stay resident — see WP_MIN_PAGED_BYTES
1756+
// rationale above (paged non-matmul leaf gets no backend ->
1757+
// gallocr buffer_id assert; hit on DeepSeek V4 hc_attn_scale).
1758+
if (ggml_nbytes(t) < WP_MIN_PAGED_BYTES) {
1759+
continue;
1760+
}
17461761
}
17471762
llama_weight_page_info info;
17481763
info.name = ggml_get_name(t);

0 commit comments

Comments
 (0)