model: add openPangu-2.0-Flash (92B-A6B) with MLA-latent cache, DSA/SWA, mHC, and multi-head MTP#2065
Conversation
|
This seems quite similar to DeepSeek-V4? |
|
Yes, at least somewhat similar, I was interested in porting DP V4 to play around with the flash and this PR already helps map out some preliminary structures. |
|
That was my hope, since DeepSeek 4 is too large for me to work on and test without renting. I'm hoping to have CUDA ready today, and that enabling the other 2 MTP heads makes a difference. |
joelfarthing
force-pushed
the
filament/openpangu
branch
from
593e473 to
d5415e6
Compare
|
Rebased to main as of this comment, with three commits added. Two of them ( The mHC commit is a cheap cont working around CUDA binbcast silently misreading strided views; I'll report that separately since it is a backend issue, not arch-specific. With those two fixes, the branch runs on CUDA (#2045's CUB argsort and ARGSORT_THRESH fence cover the DSA top-k), so no CUDA follow-up PR is needed. The third commit ( Q4_K_M,
f16 is worth about 24% of CPU prefill by itself (halved bytes through the latent reads); layer-0 attention out vs the f32 oracle differs by at most 0.0114, and the spec byte-gate stays byte-identical under it. Offload is another 4.4x prefill on top. Decode stays bound by the 49 GiB of routed experts in RAM, so offload doesntt help TG for me. The 3.4K needle retrieves exactly on CUDA in both spec modes, and a 32K needle (31,941 prompt tokens, f16 cache, DSA selection and MTP spec all active) retrieves exactly on CPU. Two notes for anyone trying this: set Future work: MTP heads 2 and 3, and state serialization. I'll leave this PR-as is for review. |
|
Starting to look into this PR. Downloaded the quantized model from https://huggingface.co/ji-farthing/openPangu-2.0-Flash-ik-llama-GGUF. Before I go into more detail, a couple of questions:
Why are these tensors being ignored? Or asked differently, why are they there if they are being ignored?
|
|
Ran perplexity and a bunch of simple conversations with But then started
Didn't have the patience to wait until the end. The whole point of MLA, sparse attention, and all the other fancy stuff being used is to improve performance at large context lengths. Instead, we see a much more dramatic performance decline with context length than simple models using standard transformer attention. We also get to enjoy a massive compute buffer size - 25852.03 MiB for context of 32k and u-batch of 2k. |
|
I cannot have anything but Oh, I see. Selecting but it does seem to have an impact. The impact being ~20% performance reduction. |
|
Many thanks for looking at this. The answer to some of these questions is: I was unsure how much of a model's implementation you want in a single PR, and at over 1600 new LoC, I erred on the side of doing it in big pieces to ask for direction, rather than all at once.
The converter writes both the fused
Yes, this model ships three NextN layers (46-48) where GLM/DeepSeek-style MTP models ship one. They only load when an MTP stage is configured (
I established every correctness gate (byte-identity, f32 oracle comparisons, needle probes) at f32 first.
The "V cache" is a transposed copy of the same latent ( |
On the sweep-bench decline and the compute buffer: this implementation has memory sparsity but not compute sparsity yet. The DSA selection is applied as an additive mask on a manual full-width attention path, so the KQ and value GEMMs still run over every cached position at every layer and the selection only bounds which positions can contribute probability mass. On top of that, the lightning indexer scores every cached position before top-k, and per DSA layer that pre-selection chain materializes |
On |
|
I think it would be useful if you looked at the GLM-5 DSA efforts going on in parallel. There we had to deal with the same memory explosion related to the indexer. As far as I can tell, the indexer here is basically identical to the GLM-5 indexer. Can you explain a bit better about the attention sinks? Why do they need to be prepended, what they do, etc. It seems strange that someone would pick attention sinks that need to be prepended to the regular attention keys, which basically invalidates every flash attention implementation out there. |
I've been following the GLM-DSA work, and you're right that the indexers are near-identical by design; I lifted the CUB argsort from #2045 while it was in flight, and this differs only in head count (24 vs 32) and where q/k are tapped (q from the post-conv post-norm q_lora, k rms-normed without conv, rope on the first 64 dims). Both remedies carry over directly: the head-loop scoring from #2058/#2066 is exactly the chunking I meant above and bounds the compute buffer by the head count, and the #2068 gather of the n_top_k selected keys out of the latent cache is the "make the selection computational" item, with the nice property you showed of TG beating dense past ~8K. For openPangu the gathered set has to carry the 128 sinks alongside the selected keys, and the SWA layers are windowed anyway, but neither changes the shape. My plan was to adopt both once they settle on the GLM side rather than fork the approach here, but I'm happy to pull the head-loop into this PR now if you'd rather not merge with the current buffer behavior. On the sinks: Thanks again for the review and direction. One scheduling note: I'm traveling this weekend, so fixes from review may take a few days. |
I'll be on the mountains hiking. Much too hot to stay here. So, not much will happen between now and Monday. |
Add OpenPanguV2ForCausalLM conversion support (converter-only; runtime graph is Stage-2). Registers a new LLM_ARCH_OPENPANGU on the Python/gguf-py side: - gguf-py/constants.py: MODEL_ARCH.OPENPANGU + name, indexer KV keys, 22 new tensor enums (DSA indexer x4, MoME convs x3, param-sink x2, mHC/Hyper- Connections x12, block-post-norm), and the full MODEL_TENSORS list reusing the deepseek MLA + MoE + NextN bricks. - tensor_mapping.py: arch-specific block mappings that disambiguate the sandwich norms (post_attention/pre_mlp/post_mlp) and pin every Pangu-only tensor; non-block global mHC merge module. - convert_hf_to_gguf.py: OpenPanguV2Model (subclasses DeepseekV2Model) with set_gguf_parameters (MLA/MoE/indexer/mHC/param-sink/DSA+SWA metadata), modify_tensors (expert merge, kv_b split, no MTP skip), and the OpenPanguV2Tokenizer pre-tokenizer hash. Validated offline against the real 50-shard safetensors index: all 37,587 tensors map to a GGUF target (0 unmapped), and set_gguf_parameters reads only hparams present in config.json. No weights downloaded; no GPU. Pinned on the ik/dsa_loop_hadamard_blend DSA substrate.
…piles
New arch on main (DSA-decoupled). Declares openPangu-2.0-Flash to the runtime so
the model loads into memory; the compute graph is the next step.
- llama-arch.{h,cpp}: LLM_ARCH_OPENPANGU + name; 3 KV keys (mhc_num_stream,
mhc_recur_norm, param_sink_number); 18 tensor enums (mHC x12, MoME conv x3,
param-sink x2, block-post-norm).
- llama-model.cpp: OPENPANGU tensor-name block, strings matched to the converter.
- llama-model.h: layer + model struct fields (mHC / conv / sink / block-post / merge).
- llama-hparams.{h,cpp}: reader (MLA + MoE + sigmoid gate + indexer + mHC +
param-sink + NextN); n_layer_kv_from_start = n_layer - nextn (MTP skipped).
- llama-load-tensors.cpp: create_openpangu_tensors (GLM-DSA MLA/MoE base + Pangu
tensors; indexer loaded-but-unused for dense fallback); dispatch + is_mla_attn.
Builds clean (CPU-only libllama). Dense-fallback design: no DSA indexer / SWA
windowing / MTP for first generation (exact <=512 tokens). Graph is Stage-2b.
…ention order in spec
… (garbled)
First full forward pass of openPangu-2.0-Flash on ik_llama. Pipeline works end to
end: new LLM_ARCH_OPENPANGU loads the Q4 GGUF, the graph executes, and llama-cli
generates 40 tokens (EXIT=0). Output is currently garbled (tensor-layout bug to
debug), but the structure is proven.
graphs/build_openpangu.cpp: dense decompressed-MHA attention + 4-stream mHC
(Hyper-Connections) with 20-iter Sinkhorn + MoE(sigmoid+shared) + sandwich norms
+ entry stream-repeat/tail-merge + inp_out_ids selection.
Bring-up fixes to load+run:
- llama-vocab.cpp: register 'openpangu' pre-tokenizer (QWEN2 family)
- llama.cpp: OPENPANGU -> LLAMA_ROPE_TYPE_NORM (was defaulting to NONE=-1)
- llama-load-tensors.cpp: full wkv_b load; k_b/v_b as flattened 2D; block_post_norm
dim = S*H (10240); conv weights 2D {3,C}; mHC alpha/beta/gamma + param_sink +
merge params use bare (no-.weight) tensor names
- llama-model.cpp: OPENPANGU is NOT is_mla_attn (decompressed MHA, standard KV cache)
- graph loop bounded to base layers (skip NextN/MTP)
v0 deferrals (need conv-state cache / manual attention path, all documented):
MoME convs (passthrough), o_conv, param_sink. Next: fix the layout bug to coherence.
…E convs, param_sink Four correctness fixes on top of the end-to-end scaffold, verified checkpoint-by- checkpoint against a Python golden reference running on the GGUF's own dequantized weights (block-0 activations now match to rounding at full fidelity): - rope: NORM -> NEOX. Pangu config rope_interleave=false; the Infer source maps it as is_neox_style = not rope_interleave (rotary_mode='half'). - mHC Sinkhorn: the flat h_res block is torch-[r,c] row-major, so a bare ggml reshape lands column-fastest; the doubly-stochastic iteration ran transposed (Sinkhorn is not transpose-symmetric). One transpose at input fixes the whole chain including the mhc_post application. - MoME convs (qa/compresskv/o): were passthrough stubs. Implemented as out = x + causal_conv1d(x) (every Infer call site uses residual_connection=1; tap stats confirm the perturbation form). Taps cast f16->f32 for ggml_mul. Batch-local v0: exact for fresh-sequence prefill; decode steps miss the t-1/t-2 taps until a conv-state cache exists. - param_sink: 128 learned latent-KV entries prepended per layer via a manual attention path (kv_store + explicit soft_max over [sinks ++ cache]); huge effect at short context. o_conv now applied pre-o_proj on the same path. flash_attn forced off for OPENPANGU (FA kernel cannot see the sinks). - converter: add_bos_token=true (HF prepends <|pangu_text_start|> via the post-processor; the key was absent so ik dropped BOS). Greedy Q4_K_M smoke, chat template + <think>: coherent CoT reasoning and a correct answer. Layer-0 instrumentation (opg0_* names) kept for now.
Allocate a per-layer cache_s_l tensor for OPENPANGU base layers holding the last two pre-conv latents of the three MoME sites, packed [qa 2*1024 | compresskv 2*512 | o 2*6144] f32 (~60KB/layer). The conv helper reads the [C,2] history window (zeros at sequence start, kv_head==0), builds xx = [hist ++ x], and writes the last two columns back each ubatch — the concat naturally handles both prefill chaining and the T==1 shift. Read precedes write in graph order; the fixed-offset copy is graph-reuse safe. Verified: prefill anchors unchanged (bit-path identical, zero-history branch); -ub 1 token-by-token run matches the golden reference at t4 (qlora_conv 0.084, R_block 0.008 rel; attn_out 0.15 on one channel = f16 KV-cache rounding, washes out by post-norm); final logits differ from full-batch only by a common-mode shift that softmax cancels. Chat-template greedy smoke: think-block repetition is gone — clean structured CoT and correct answer. v0 limits documented in the helper: one state slot (single sequence); cache rewinds leave the state stale.
Wire the three NextN layers (46-48) into ik's MTP speculative framework (--spec-type mtp). v0 drafts with head 1 (layer 46), self-chained by the framework. - llama.cpp: add OPENPANGU to the cparams.mtp arch allowlist (it was silently zeroed, which left the target context without a logits buffer once the server enabled embeddings -> GGML_ASSERT(lctx.logits) in speculative_is_compat). - load-tensors: MTP layers carry no mHC tensors (tail_use_mhc=false in the reference) — create them only for base layers. nextn.* tensors were already wired by the Stage-1 probe. - build_openpangu: extract the attention sublayer into build_openpangu_attention (shared base/MTP); add build_openpangu_mtp: eh_proj(cat(enorm(embed), hnorm(prev_hidden))) -> one plain-residual Pangu block (sandwich norms, convs+param_sink, MoE+shexp, no mHC/block_post_norm) -> shared_head norm+head. MTP branch returns the draft graph when mtp_op_type != NONE; main graph keeps all-token outputs under cparams.mtp. MTP convs run batch-local (no conv-state slot) — affects acceptance only. A/B (Q4_K_M, CPU, greedy, 192-token chat CoT completion, warm back-to-back, medians of 3, bracketed B/A/B): no-spec: 2.44 t/s (2.34-2.86) --spec-type mtp:n_max=3: 4.23 / 4.49 t/s (brackets) => ~1.7-1.8x Draft acceptance 34% on CoT prose (46% on repetitive text); spec and no-spec greedy outputs are byte-identical. Headroom: conv-state for MTP drafts, n_max tuning, true 3-head chaining (spec_step_idx).
get_formated_timings() (the /completion path) omitted the speculative counters that get_timings() (the OAI path) already reports; add them, guarded by n_draft_total > 0 like the OAI path.
… decoding + MTP draft chaining
The v0 single-slot conv state held the last-2 pre-conv latents of the most
recent batch, so any speculative draft rejection left latents of REJECTED
positions in the state and every later decode ran with wrong t-1/t-2 taps
(3 conv sites x 46 layers). At 192-token greedy runs every spec config
diverged from no-spec, each differently (rejection-pattern dependent).
Replace it with a per-layer ring cache_s_l [n_lora_q+n_lora_kv+n_head*v_dim, 16]:
column pos%16 holds position pos's pre-conv latents ([qa|ckv|o] packed).
Invariant: reads target only positions before the first batch token, which
are committed, and committed latents depend only on the committed prefix -
rollback-safe by construction, no checkpointing. Writes cover the last
min(T,16) batch positions in <=2 contiguous cpy segments; the copy sources
are views of the [hist ++ x] concat so the history read is an ancestor of
every write (read-before-write by graph dependency).
The ring is also allocated for the NextN/MTP layers, so the draft head
chains real conv taps across WARMUP -> sequential DRAFT_GEN steps (was
batch-local zero-history per draft token).
graph_reuse is forced off for the arch: ring view offsets are position-
baked and the reuse patcher only updates the standard K/V-store copies.
Measured cost on the CPU server path: none visible. ggml_set_rows driven
by an input index tensor is the future reuse-safe shape.
Verified (Q4_K_M, CPU, greedy 192-tok chat-CoT, warm single process):
- no-spec output byte-identical to pre-ring build
- spec output byte-identical to no-spec below the n_predict cap, for all
of n_max in {1,2,3,4,6} x p_min in {0,0.3,0.6} (old build: all diverged)
- acceptance n3-p0: 33.9% -> 60.9%; n3-p0.3: 58.1% -> 68.9%
- TG medians: no-spec 3.19-3.32 t/s; mtp:n_max=3,p_min=0.3 6.97 t/s (~2.1x)
…tness past the dense fallback The dense fallback was exact only <=512 tokens (SWA window). This wires the real DSA/SWA hybrid schedule, self-contained from GGUF keys the converter already writes (openpangu.swa_layers + sliding_window_list; absent keys keep the old dense fallback): - SWA layers (30 base @512): the generic inp_KQ_mask_swa path, per-layer mask choice in the builder. The NextN/MTP layers are SWA @2048 in the checkpoint schedule; MTP graphs run in their own context, so the mask fill picks hparams.n_swa_mtp when built with an MTP op type. - DSA layers (16, every 3rd): lightning indexer implemented in-graph from the Infer reference semantics (jointfix _pangu_torch_calib): q_idx = wq_b on the post-conv post-norm q-lora latent (24x128), k_idx = rms-normed wk(x) shared across heads, both NEOX-roped on the FIRST n_rot channels; score = sum_g w_g * relu(q_g . k) in f32, causal-masked, exact top-k via argsort + ggml_set_rows scatter into a -1e30 base -> additive selection mask on the existing manual soft_max seam. Selection engages only when the causal window exceeds index_top_k (2048); below that the layer is exactly dense. - Indexer keys cached per position (cache_idx_l, f32 [128, kv_size], DSA layers only) with the same committed-position invariant as the conv-state ring, so speculative rollbacks stay safe. - param sinks remain outside both the window and the selection budget, matching the reference. Verified (Q4_K_M, CPU): - <=512 tokens: byte-identical to the dense build (96/160-token greedy) - indexer scores vs a GGUF-dequant golden reference at 2101 tokens: 1e-3 rel (f16 weight rounding); top-3 selection indices exact on all compared queries - >512 coherence clean; 3.4K-token needle retrieval through active selection (needle outside every SWA window, ~1300 positions pruned) answers exactly
…nt, 14x smaller cache, ~2.2x TG Store per position only [ckv_norm 512 | roped k_pe 64] (f32, k_l) plus the transposed 512-latent (f32, v_l, v_trans layout); per-head K/V are never materialized. q_nope is absorbed through attn_k_b (loaded 2D from the converter split for base layers; derived at load via llm_prepare_mla for the NextN layers - now guarded for layers without attention weights, e.g. the idle NextN heads 2/3). The value side is the latent itself, up-projected through attn_v_b after the weighted sum, matching the Infer _forward_dsa reference. param sinks are native latent-space entries, which removes the per-step full-cache concat+cast that dominated long-context decode. llama_state row sizes now come from llama_kv_k_row_embd/llama_kv_v_row_embd (arch-aware), fixing an out-of-bounds crash in the server prompt-cache save path (hparams-derived 9216-wide rows vs actual 576-wide latent rows). Verified (Q4_K_M, CPU): layer-0 attention output matches an f32 golden reference computed from the same GGUF weight encodings (~1e-2 on O(1) values); MTP spec output byte-identical to no-spec; 3.4K needle retrieval through active DSA selection exact under greedy. Output differs from the materialized build at the token level because attn_k_b/attn_v_b are independently quantized tensors - both are legitimate Q4-fidelity encodings. Perf (CPU, warm): no-spec TG 3.2-3.3 -> 6.9-7.1 t/s; mtp:n_max=3,p_min=0.3 -> 11.1 t/s (byte-exact, 67% acceptance); prefill 30.5 t/s at 3.4K; KV self size at 4K ctx: 5.5 GiB -> 391 MiB. Not yet supported on the latent cache: K-shift/defrag (context shifting) - unreached in current usage.
… dead weight/keys Post-audit hardening. The cache's position-indexed side state (MoME conv ring, DSA indexer keys) made several generic serving paths silently unsound; they are now fenced loudly instead of documented as unsupported: - s_l_position_ring flag on llama_kv_cache: the qnext-state predicate no longer claims the conv ring, so per-seq state save, seq_cp and the s_copy graph skip it - state save/restore refused for the arch at every llama_state_* entry (the ring and idx_l are not in the state format; restoring without them diverges silently) - K-shift/self-extend assert, defrag skips with a warning, server ctx_shift off via new llama_model_supports_ctx_shift() - single sequence enforced at context creation (n_seq_max > 1 refused) - server prompt-cache reuse limited to pure extension via new llama_model_supports_partial_kv_reuse(): mid-cache divergence reprocesses from scratch (the 16-column ring cannot rewind); multi-turn continuation stays fast - MTP draft length clamped to 13 via new llama_model_max_draft_tokens() so a rejected draft can never overwrite the ring columns the next decode reads - K/V cache types forced to f32 for the arch so the KV size log reports the truth - cache_size(): real latent-cache branch (was falling through to the ~14x larger materialized estimate used for offload planning) - unused fused wkv_b no longer loaded (TENSOR_SKIP; the graph runs entirely on the pre-split k_b/v_b), llm_prepare_mla openPangu special-case removed (it was a no-op) - stale v0 comments rewritten to describe the shipped graph; converter stops writing dead keys (dsa_layers, block_post_layernorm_idx) and the tokenizer pre-hash is registered in convert_hf_to_gguf_update.py Gates on this build: greedy spec output byte-identical to no-spec (EOS-terminated, sha-equal); 3.4K needle retrieved exactly; -np 2 / state save / n_max=20 / stale prefix reuse all refused or clamped with clear messages.
The ring, indexer and latent stores are addressed by absolute position through kv_head; the fences make append-only decode the only reachable mode, but the invariant was unchecked. Assert it at both graph entries (base and MTP) so any future cache plumbing that breaks it fails at build instead of corrupting output. Worst-case measurement builds pass pos = null and are exempt.
…reads strided views) h_pre is a row-slice view of the fused mixes tensor. The CPU mul handles the strides; the CUDA broadcast path reads the view as if contiguous, so token 0 mixes correctly and every later token gets h_post/h_res rows instead. First divergent node in the whole graph (oracle rel 0.36 at opg0_attn_mhcpre_x, fixed to 7.5e-5). Sibling views h_post/h_res were already cont-wrapped, which is why only h_pre was exposed.
…the 0*(-inf) NaN)
The selection-mask base and zeros were built by scaling the MASKED scores by
zero, but post-mask sc contains -inf and 0 * -inf = NaN. The CPU clamp launders
NaN back to -1e30 (fminf/fmaxf ignore NaN); the CUDA clamp propagates it, so
every DSA layer emitted NaN masks at n_kv > top_k and logits collapsed
(observed: eval-callback CLAMP sum -1.3e36 on CPU vs nan on CUDA, 11748 NaNs
downstream). Scale the pre-mask finite scores instead, which is correct on any
backend regardless of clamp NaN semantics. Also defensively cont the strided
KQ_mask slice feeding the score add (same strided-view kernel class as the mHC
h_pre fix; unproven here but cheap). Gates after fix: 2600-token probe coherent,
3.4K needle exact ('7391') with and without MTP speculation, PP ~120 t/s.
| // Position indexing makes speculative rollbacks safe: accepted positions' latents depend only | ||
| // on the committed prefix and stay valid; rejected columns are overwritten before any read. | ||
| // Positions < 0 (sequence start) read zero history — the exact reference behaviour. | ||
| static ggml_tensor * openpangu_causal_conv(ggml_context * ctx, ggml_cgraph * gf, |
There was a problem hiding this comment.
Without me following the final outcome of this operation, this cannot be modeled via ggml_ssm_conv?
There was a problem hiding this comment.
The conv is ggml_ssm_conv-shaped and its saved_steps per-step mode (as DeltaNet uses) could carry the MTP rollback. The ring predates that path; I can definitely look at refactoring onto ggml_ssm_conv.
The openpangu_swa_window_view reuse key stored n_kv/n_tokens/window/pad as int64_t, but these are bounded well under 2^31 (window/pad are uint32_t at source; n_kv/n_tokens <= context length). Narrow to int32_t/uint32_t and drop the widening casts. w_view/win_off stay int64_t: they feed ggml view dims/offsets. Per review.
| ? layer.param_sink_k_pe | ||
| : ggml_cast(ctx0, layer.param_sink_k_pe, GGML_TYPE_F32); // [64, NS] | ||
| ggml_tensor * sink_blk = ggml_concat(ctx0, s_ckv, s_kpe, 0); // [576, NS] | ||
| ggml_tensor * s_lat_t = ggml_cont(ctx0, ggml_transpose(ctx0, s_ckv)); // [NS, 512] |
There was a problem hiding this comment.
As far as I can tell, the result is always the exact same tensor (per layer). Why are we computing it again and again on each eval instead of computing it just once?
There was a problem hiding this comment.
You're right. sink_blk/s_lat_t are pure functions of the layer weights, so yes, recomputed here every eval. I'll pre-compute them once at load (same as wk_b) and reference the cached tensors.
|
There is way too much openPangu-specific stuff in @SamuelOliveirads Can you look into the MTP-specific changes? Thanks. |
| openpangu_register_cache_copy(lctx, il, OPENPANGU_COPY_K_CKV, cpy_kl, kl->nb[1]); | ||
| ggml_build_forward_expand(gf, cpy_kl); | ||
| } else { | ||
| ggml_tensor * kl_ckv = ggml_view_2d(ctx0, kl, kv_lora_rank, n_tokens, kl->nb[1], kv_head*kl->nb[1]); |
There was a problem hiding this comment.
Why are we doing this? Making two views and triggering two copies into the cache? The standard approach used everywhere else would be to do ggml_concat on the two parts and then have a single copy into the cache. If we did that, we wouldn't need the base_offset addition to the CacheCopy struct.
The per-layer attention-sink block (sink_blk [576,NS]) and its transposed latent (s_lat_t [NS,512]) are pure functions of the layer weights, yet were rebuilt every eval across all 49 layers (RMS-norm + cast + concat + transpose). Compute them once at load, mirroring the wk_b derived-weight precompute, and read the stored tensors in build_openpangu_attention. Numerically identical; removes per-token work at decode.
|
I observe that the
Is this the intended behavior? It doesn't seem logical. So, I reran the sweep with This looks like a bug. For PP the expected Or is this somehow made to work correctly via the mask, so that we are just wasting compute cycles but the result is correct. |
…ack checkpoint Migrate the MoME depthwise causal conv (three sites per attention sublayer: qa-lora, compressed-kv, attn-out) from the bespoke 16-column position-indexed ring onto the core ggml_ssm_conv op with a recurrent conv-state slot. Cache: s_l becomes [2*conv_col_ne, qnext_state_slots], holding the (d_conv-1)=2 history taps per channel for the three sites (float offsets 0 / 2*n_lora_q / 2*(n_lora_q+n_lora_kv)). Drops the conv_hist_idx / conv_write_idx graph inputs and their fill in llama_set_inputs; adds one single-sequence sq input for ggml_ssm_conv shared across the three sites and the MTP head. Speculative rollback: the position ring self-healed rejected draft columns by absolute position; a recurrent slot does not, since seq_rm is a no-op for recurrent state. openPangu is admitted at the three spec-checkpoint save/init gates so the whole-slot shadow checkpoint (gpu-fallback) snapshots the conv slot before drafting and restores it before the accepted-token replay. The restore path is already keyed on ckpt.valid, so no gate change is needed there. Per-step checkpoint mode is declined for openPangu, which has no SSM recurrent term, so auto mode resolves to the whole-slot shadow. Gated: non-spec needle unchanged; MTP-spec needle correct with healthy draft acceptance (rollback verified via the acceptance canary).
The non-quantized latent store split the [ckv | roped k_pe] row into two views and two cache copies, with a base_offset field on the CacheCopy struct to place the second one. Match the quantized path: concat the two parts and do one copy into the cache row. This drops the second cache-copy slot (OPENPANGU_COPY_K_KPE) and removes base_offset from CacheCopy entirely. Cache contents are unchanged: the concat writes the same [ckv 512 | k_pe 64] bytes to the same row. Gated on the needle for both the f16 latent path (the one that changed) and the q8 latent path, plus coherence.
…idx_l The DSA lightning indexer stored its per-position keys in an openPangu-only idx_l cache, parallel to the kr_l indexer cache GLM-DSA already uses. Both have the same storage contract: [indexer_head_size, kv_size], idx_type_k dtype, one row per KV cell, written at kv_head and read [dim, n_kv] from zero. openPangu now allocates its indexer keys into kr_l and shares the dsa_cache_copies graph-reuse fixup. The fixup patch is factored into a helper that both the generic path and the openPangu update_cache_copies branch call, so the openPangu indexer copy is repointed to the current kv_head on graph reuse like every other cache write. This drops the idx_l vector, its allocation and memory accounting, and the openPangu third cache-copy slot (now one latent copy per layer). Per-arch allocation predicates stay separate (GLM uses indexer_is_full, openPangu uses the window==0 DSA schedule); only the kr_l storage and the copy fixup are shared. openPangu keeps its no-shift/no-defrag/no-state-I/O behavior, and the GLM Hadamard/k-shift logic stays GLM-gated. Gated: needle correct on f16 and q8 latent caches and under MTP speculation (acceptance unchanged at 0.67), plus coherence.
…omments The ggml_ssm_conv refactor bakes the pos-0 conv-state reset into graph topology (a scale-by-zero node on the state view). A graph built at pos 0 could be reused at pos > 0 when the batch shape and padded n_kv match (a 1-token prompt followed by TG is the concrete case), zeroing the conv history on every reused decode. Admit openPangu at the existing reset_previous gate so pos-0 graphs are discarded from reuse, the same guard the qnext recurrent state relies on. Also retire the internal phase-plan comments the conv refactor left behind: they claimed the spec-checkpoint wiring had not landed in the commit that landed it, and misdescribed the s_l slot as awaiting rollback support. Gated: needle 8457 on f16 and q8 latent, MTP-spec needle (drafts fully accepted), coherence.
…onally Review follow-up (item 1 of the second review). The explicit/default distinction carried less than claimed: the latent K/V fallback was f16, which is already the -ctk/-ctv and API default, so distinguishing unset from set-to-the-default bought nothing, and the two bools were behaviorally redundant. The only load-bearing use was the indexer cache, where openPangu defaulted to f32 while -ictk defaults to f16. Gating the f16 indexer directly (needle on f16 and q8 latent paths, MTP speculation, coherence) shows no quality difference, so openPangu now takes the standard f16 indexer default and the f32 special case is gone. Default indexer cache memory halves (64 -> 32 MiB at c 8192). Removes type_k_explicit/type_v_explicit/idx_type_k_explicit from llama.h, the cparams/mparams plumbing, and common; the resolve helpers become plain unconditional validators, so -ctk q8_0 is honored and an unsupported type errors out at load instead of silently coercing. Gated: needle 8457 on the new f16-indexer default, on q8 latent with MTP speculation, and with -ictk f32 explicitly honored (64 MiB f32 buffer in the load log); -ctk q4_0 and -ictk q4_1 refused with a clear error.
The MTP framework's one-token draft shortcut caches a prediction one row past the accepted prefix during the accepted-token update, then skips re-decoding the last sampled token at the next draft round. A mask-addressed cache tolerates the resulting position gap; openPangu's position-addressed append-only cache (cell == position) does not: after a rollback the next draft decode lands one cell behind its position, and after a full acceptance the cache head sits one row ahead of the next draft base, either way tripping the kv_head == pos[0] invariant and aborting the server. The checkpoint admission in the conv refactor made this the standard openPangu speculative flow; the needle-first gates never generated enough draft rounds against a short prompt to reach it. Decline the shortcut re-seed for openPangu in mtp_accept_batch (restoring the drafting behavior all measured acceptance numbers were taken on) and trim rows at or beyond the draft base in mtp_speculative_gen_draft, so every draft decode stays position-contiguous with the cache head. Gated: the crashing flow (short prompt, 512-token spec generation, then a second request) completes with acceptance 0.60 prose / 0.87 code, matching the pre-checkpoint baseline profile; needle 8457 plus coherence on f16+spec and q8+spec.
|
Thanks again @ikawrakow, this was a useful pass. 1. Explicit cache type + 2. 3. 64-bit integers. The SWA reuse-key fields are now 32-bit. I kept only the ggml view dimensions and byte offsets as 4. Sinkhorn kernel. Left for a future fused op, as discussed. 5. 6. Sink precompute. 7. openPangu code outside the builder. I removed the ring inputs/API and the separate indexer cache/copy path; the final cleanup removed another 26 net lines from shared files. Outside the builder remain: cache allocation/fixups, masks, load-time sink derivation, position-strict guards, and the generic MTP controls because the graph or server path uses them. 8. Warm TG on 4070, Q4_K_M,
With the conv rewrite we see a small increase, +3.2 to +3.5%. Repeat context curve
After the known 2048 dip, TG stays between 10.64 and 10.75 t/s from 6K through 32K. The CUDA0 compute buffer is 2893 MiB and peak VRAM is 8717 MiB, both unchanged from before. 9. DSA/SWA KQ widths. Not a correctness bug: the mask preserves selection, and the full-width path is used only where gathering would cost more than it saves. The |
SamuelOliveirads
left a comment
SamuelOliveirads
left a comment
There was a problem hiding this comment.
When testing using the model you published I didn't find any issues with MTP for head=1 when using the server and I treated head>1 as part of 'in development' process so I ignored the performance regression here.
I didn't find any issues in the code or with the server benchmark, but the code doesn't work with the CLI, running there generate duplicate tokens regardless of the head count, could be good to address if you will expand for CLI or just disable it.
Another thing, you’ll probably want to leave mtp_heads marked as experimental in the argument description and set 1 as default, unless you already plan to implement the logic improvement that you proposed in previous comments.
Decode an already-emitted pending token when a draft cannot be used instead of sampling unchanged logits and duplicating output. Document single-head MTP as the default and multi-head modes as experimental.
Thanks for testing this and for catching the CLI path. Both points are addressed in the latest push. The duplication was a real CLI-only carry bug. Fixed in
I also updated the argument description, startup message, and speculative documentation so heads=1 remains the default, while heads>1 and heads=0 (all model heads) are explicitly marked experimental. Thanks again for the careful test. |
Bring the openPangu-2.0-Flash port up to date with current main (70ea152). Two files conflicted; resolutions: - docs/parameters.md: keep the MTP heads key and default/experimental contract; add main's dflash stage, cross_ctx key, and draft-model example. - examples/server/server-context.cpp: keep the partial-KV-reuse and MTP fresh-warmup reset guards; adopt ikawrakow#2105's prompt-cache cleanup and reusable-fraction behavior around them. Merged tree is byte-identical to the validated integration tree c3e9b73.
SamuelOliveirads
left a comment
SamuelOliveirads
left a comment
There was a problem hiding this comment.
At least for MTP path where I checked it looks good for me
ikawrakow
left a comment
ikawrakow
left a comment
There was a problem hiding this comment.
Wondering if the SWA logic for openPangu can be reused for other SWA models.
3 participants
openPangu-2.0-Flash is Huawei's 92B-A6B MoE trained to 512K context. No GGUF converter or runtime existed for it anywhere I could find; it stacks several mechanisms no other checkpoint has. This adds both, under a new
LLM_ARCH_OPENPANGUwith its own graph builder. The DeepSeek and GLM paths are untouched.Update (2026-07-08). This PR has advanced substantially since it was opened; see the re-review comment below for the current, measured state (every number there is on the current branch tip). The original description below is partly superseded — in particular the compute buffer (25.85 GiB → 3.4 GiB), the KV cache (V store dropped), the q8 latent/indexer cache (now real storage; unsupported types refused, not coerced), the attn_kv_b tensors (now dropped in the converter, not just skipped at load), and the Performance table (superseded by the sweep in the re-review). Chat-parser support was also added. Where the two differ, treat the re-review comment as authoritative.
Beyond stock MLA, the arch needs:
attn_k_b, values up-projected throughattn_v_bpost-sum. f32 by default; explicit-ctk f16 -ctv f16is accepted and halves it (BF16/quantized warn and fall back). Latent K/V runs 782 MiB f32 / 391 MiB f16 at 4K vs roughly 11.6 GiB materialized f32 (14x narrower rows); the indexer-key cache adds 32 MiB at 4K (scales with context) and the conv rings a constant 23 MiB. The fusedattn_kv_bhas no consumer and is skipped at load.--spec-type mtpframework, self-chained; draft convs chain state through the same ring.The cache is position-strict (rope baked into cached k_pe, ring and indexer keys addressed by absolute position), so everything that would silently violate it is fenced: FA and graph reuse forced off (sinks and the top-k mask need the manual soft_max path; the ring bakes position-dependent view offsets that the reuse patcher doesn't know),
n_seq_max > 1refused at context creation, allllama_state_*save/restore refused (ring and indexer keys aren't in the state format yet; restoring without them diverges silently), K-shift/self-extend assert, defrag skips with a warning, server ctx-shift disabled, unsupported cache-type requests coerced to f32 with a warning (f16 accepted; quantized V is rejected earlier by the generic no-FA check), MTP draft length clamped to 13 (a longer rejected draft could overwrite the ring columns the next decode reads), and server prompt-cache reuse limited to pure extension (mid-cache divergence reprocesses from scratch; multi-turn continuation keeps the fast path). The graph also asserts kv head == first batch position at build, so future cache plumbing that breaks position addressing fails loudly instead of corrupting output. Three new conservative-defaultllama_model_*capability predicates carry this.Shared-code footprint: behavior-visible for other models are the
/completiontimings gainingdraft_n/draft_n_accepted(parity with the OAI path) and a null-wkv_bguard inllm_prepare_mlafor layers with skipped attention weights. Behavior-neutral by construction: llama_state row sizes go through arch-aware helpers that reduce to the old expression elsewhere, and the SWA mask fill threads ann_swa_effequal ton_swaeverywhere but openPangu-with-MTP. The rest is arch-gated plumbing.Correctness. Validated against f32 references computed from the same GGUF-dequant weights the runtime reads (
llama-eval-callback, named checkpoints):quicksortis correct including edge cases,double-link-listoutput module passes its own 13 tests; no repetition, truncation, or template leakage on those or onbulgariaorsayap.The three oracle scripts are in this gist. The selection seam here is the same shape as the one in #2063, so the indexer oracle and the dense-equivalence and rollback gates should carry over if they're of any use there.
Performance (i7-11700K + RTX 4070 12 GB, Q4_K_M with convs/mHC/sinks/indexer kept f16; warm same-session medians of 3; PP = 3.4K needle prompt, TG = greedy ~900-token generation; spec =
--spec-type mtp:n_max=3,p_min=0.3):-ngl 999 -ot exps=CPU, f16 cache, no spec-ngl 999 -ot exps=CPU, f16 cache, MTP specThe f16 cache is worth about 24% of CPU prefill by itself; CUDA offload of attention + latent cache + indexer is another 4.4x prefill while the 49 GiB of routed experts stay in RAM (decode stays expert-bound, so offload does not help TG on this box). Spec acceptance tracks output structure:
bulgaria63.4%,quick-sort90.5%,double-link-list95.0%,sayap99.0%; p_min=0.3 beat every fixed n_max in a 3x6 sweep. With-ot exps=CPU, setGGML_CUDA_NO_PINNED=1or the 50 GiB host weight buffer gets pinned.Limits. Runs CPU-only or CUDA-offloaded as above; greedy spec-vs-no-spec output is byte-identical on CPU, while CUDA can flip near-tie tokens past ~1K context from batch-size-dependent kernel reduction order (same behavior as mainline CUDA). Single sequence. Longest locally validated context is 32K; at 512K the latent cache is about 55 GB f32 / 27.5 GB f16. MTP heads 2 and 3 load but are idle. State serialization is a natural follow-up (everything is position-indexed, so a full-tensor round-trip suffices). No rope_scaling in the config; 512K is native theta 6.4e6. The template needs
--jinjaand enables thinking by default under the kwarg namethinking(notenable_thinking).Conversion:
The custom-q overrides matter: the k=3 conv tensors crash block quantization, and the exotic small tensors are a rounding error of file size kept high-precision. A ready Q4_K_M produced by exactly these commands is at ji-farthing/openPangu-2.0-Flash-ik-llama-GGUF for anyone who wants to test without converting.