feat(kimi-k3): Kimi K3 support: MXFP4 to BF16 conversion and inference graph by worthant · Pull Request #58 · AtomicBot-ai/atomic-llama-cpp-turboquant

worthant · 2026-07-27T17:05:58Z

What this is

Support for Kimi K3 (moonshotai/Kimi-K3, 2.8T MoE, MXFP4 QAT) in llama.cpp: checkpoint conversion to BF16 GGUF plus a full inference graph for imatrix and quantization.

Note: this branch is cut from upstream ggml-org/llama.cpp master (0e4a036), which is about 136 commits ahead of our master, so the PR diff includes upstream commits. Merge by rebasing our master or cherry-pick the single feature commit.

Parts

Python (conversion):

conversion/base.py: dequant for compressed-tensors mxfp4-pack-quantized (E2M1 nibbles, low nibble = even element; E8M0 uint8 scale, group 32), verified against real Kimi-K3 shard 2 tensors
conversion/kimi_k3.py: KimiK3Model (subclass of KimiLinearModel): A_log slice to [:96] plus exp transform, vision tensors skipped, new hparams
gguf-py: arch kimi-k3, new tensors (attn_res_, ffn_res_, output_res_*, ffn_latent_norm, attn_gate), new KVs (situ betas, attn_res_block_size, kda gate_lower_bound)

C++ (graph):

src/models/kimi-k3.cpp: hybrid 69 KDA + 24 gated MLA (NoPE) with:
- KDA safe gate -5sigmoid(exp(A_log)(g_raw+dt_bias)) and a full-rank output gate
- MLA output gate sigmoid(g_proj(x)) before o_proj, MLA absorption KV cache
- Stable LatentMoE: router on hidden 7168, experts in 3584 latent space, RMSNorm AFTER the weighted sum
- AttnRes: residual-stream snapshot bank (layers 0,12,...,84), softmax mixtures over up to 9 rows before attn/FFN/output
LLM_FFN_SITU in llama-graph: 4*tanh(g/4)sigmoid(g) * 25tanh(up/25)

Verified

Synthetic mini K3 (4 layers, 8 experts, real kimi-k2 tiktoken): convert -> BF16 GGUF -> Q8_0 quantize -> generation, no errors
MXFP4 dequant checked against the real tensor layers.1.experts.0.w1 from HF (weight stats correct: +-0.125, std ~ 1/sqrt(3584))
Tokenizer: tiktoken.model is byte-identical to K2, so the kimi-k2 vocab path works unchanged

References

modeling_kimi_linear.py from the K3 HF repo, SGLang PR [Kimi] Support kimi-k3 sgl-project/sglang#32541, FLA kda ops, compressed-tensors mxfp4

* metal: fuse snake activation (mul, sin, sqr, mul, add) Mirror the CUDA, Vulkan and CPU snake fusion: same matcher on the naive 5-op chain, same F32 contract on a and inv_b, same F32/F16/BF16 kernel with F32 compute. Follows the Metal backend idioms: bf16 instantiation gated behind GGML_METAL_HAS_BF16 and concurrency ranges checked on the remaining chain nodes before encoding, as done by the bin fusion. Covered by the existing backend-agnostic SNAKE_FUSE tests. * metal: absorb snake fusion into ggml_metal_op_bin Extract the matcher to ggml_metal_op_can_fuse_snake, mirroring the Vulkan naming, and dispatch the fused path from ggml_metal_op_bin. The encode loop switch is back to a single call per case. Address review from ggerganov * metal: fix indentation in ggml_metal_op_can_fuse_snake

* ci : add HF_TOKEN to self-hosted workflows Pass the HF_TOKEN_CI repo secret as HF_TOKEN env var in the self-hosted build and server workflows. Fix the stale build.yml path reference. Assisted-by: pi:llama.cpp/Qwen3.6-27B * cont : add comment --------- Co-authored-by: ggerganov <ggerganov@users.noreply.github.com>

…l-org#25678) * cuda : relax tensor contiguity requirements for quantized concat * tests : add test cases for non-contiguous quantized concat * ggml : relax contiguity requirements for quantized concat --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

…25637) In MODEL mode, modelPropsCache is never populated: fetchModelProps call sites are gated on router-only state (isRouterMode checks, routerModels always empty), so supportsThinking always reads an empty chat template once a model is auto-selected. Read serverStore.props.chat_template directly in non-router mode, since the global /props already describes the single loaded model.

…-org#25705) read_file with append_loc emits "{n}\u2192 {line}". The space after the arrow is meant as a separator, but it is indistinguishable from real indentation. Models strip "{n}\u2192" yet keep the space, so the old_text passed to edit_file carries a phantom leading space and never matches (normalize_for_fuzzy_match trims trailing whitespace only, never leading). Drop the separator space so the arrow abuts content: stripping "{n}\u2192" now yields the exact line with its real indentation preserved, and the failure mode cannot occur by construction. Update the description example to match the new format.

* Fix crash with draft-simple * Fix tests for spec decoding

) * opencl: route `sub_group_shuffle_xor` to qcom ext when KHR ext is unavailable KHR `sub_group_shuffle_xor` is not defined by compiler when `cl_qcom_subgroup_shuffle` is present, causing certain FA kernels fail to build. Define the KHR shuffle_xor using the qcom extension. * opencl: skip FA kernels with mixed and quant types for A7x to avoid compiler crash

match cli and completion, which don't enforce it

…kernel + wmma kernel) (ggml-org#25545) * cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) * chore : remove indentation of #pragma unroll * cuda : remove unnecessary kernel template declarations * cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants. * cuda : relax MMA architecture requirements to Turing in lightning indexer implementation * chore : renamed variables * chore : rename ggml_cuda_op_lightning_indexer() to ggml_cuda_lightning_indexer() * chore : TODO for AMD rocWMMA * chore : whitespace formatting * chore : another variable rename to fix problems caused by shadowing * chore : yet another rename, this time uppercased all constants * cuda : added alignment checks for Q and K tensors in lightning indexer implementation --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

* feat: Add shimmer text animation for processing state indicators * feat: Redesign CollapsibleContentBlock component with improved UX * feat: Add conditional setting display support with dependsOn field * feat: Add showAgenticTurnStats setting for per-turn statistics * feat: Update ChatMessageAgenticContent with improved UI and new features * feat: Enhance file read tool UI/UX * feat: Refine styling of collapsible content and code preview blocks * feat: add terminal variant to CollapsibleContentBlock * feat: add built-in tools UI registry * feat: extract ChatMessageReasoningBlock and ChatMessageToolCallBlock * refactor: simplify ChatMessageAgenticContent to use extracted blocks * fix: correct markdown content block margin spacing * fix: reorganize SettingsChatFields layout and reset button positioning * fix: use direct map access in agentic store session methods * refactor: remove reasoning preview/throttle system from CollapsibleContentBlock * feat: add auto-scroll to reasoning block and remove showThoughtInProgress * feat: add ChatMessageToolCallDateTime component and support for new tool types * feat: improve auto-scroll reliability in reasoning block with RAF coalescing and MutationObserver * feat: show MCP server favicon for tools without a built-in icon * feat: add search-results parsing utilities and tests * feat: add ChatMessageToolCallSearchResults component * feat: integrate search results rendering into ChatMessageAgenticContent * feat: display tool call input alongside output in ChatMessageToolCallBlock * style: use muted foreground color in reasoning block content * chore: Format * feat: Refine reasoning block layout and make pending thoughts display configurable * feat: Stream tool call code blocks with auto-scroll and handle partial JSON * feat: add streaming permission gate infrastructure * feat: wire permission gate into the agentic loop * fix: bail out on abort and skip already-approved tool calls * fix: clear partial tool calls on abort and savePartialResponse * test: cover partial tool call cleanup end-to-end * refactor: Remove streaming permission gate logic * fix: Correct autoscroll and streaming gates for tool calls and reasoning blocks * refactor: Chat Message Assistant componentization * fix: Show health metadata for disabled MCP servers and promote connections on enable * fix: Inherit global enabled state for missing MCP per-chat overrides * refactor: Cleanup * refactor: Split ChatMessageToolCallBlock into dedicated components * feat: Add live streaming and auto-scroll for tool execution output * feat: Add line numbers and change markers to file edit diffs * chore: Formatting * feat: Add type definitions and utilities for recommended MCP servers * feat: Add recommended MCP servers configuration and storage key * feat: Add McpServerCardCompact component for recommended servers * feat: Add recommended servers section to Add New Server dialog * feat: Update McpServerForm to support authorization requirements * feat: Add select-none classes for text selection prevention * feat: Add recommended MCP server icon assets * refactor: Store dismissed MCP recommendations as a boolean flag * feat: Render tool results as JSON or Markdown based on detected content type * feat: UI improvement * feat: Render search block early and update heading to show execution state * fix: Prevent non-web-search tools from triggering the search UI block * refactor: Cleanup * refactor: Extract hardcoded icon size classes into shared constants * refactor: Extract hardcoded tool result separator into a shared constant * refactor: Tool Calls UI/logic * refactor: Cleanup * refactor: Cleanup * refactor: Cleanup

* opencl: exclude Adreno A7x from using Adreno MoE kernels Some compilers for A7x devices miscompile the repack kernels, corrupting the weights and causing MoE models to generate garbage output * opencl: exclude A6x and unknown Adreno from MoE weights repack

…ues for Adreno 850 GPU (ggml-org#25745) * opencl: workaround for A850 compiler compat * opencl: fix DX compiler version parsing and cleanup --------- Co-authored-by: Li He <lih@qti.qualcomm.com>

Assisted-by: pi:llama.cpp/Qwen3.6-27B

@8192

* CUDA: dedup MoE gate/up activation quantization (fp4) For MoE gate/up projections the src1 activation is broadcast across the routed experts (ne11 == 1), so ids_src1 maps every one of a token's n_expert_used slots to the same physical row. The MMQ path therefore re-quantized each token's activation n_expert_used times. For fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of once per expert. For NVFP4 a single quantize+scatter kernel (quantize_scatter_mmq_nvfp4) quantizes each token once and writes the resulting block_fp4_mmq straight to all n_expert_used slots, using an inverse token->compact-row map (build_tok2c). MXFP4, and GGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique rows then gather into the expert-sorted layout (gather_mmq_fp4_blocks). Both are bit-identical to the previous gather-then-quantize path (identical source data, deterministic per-block quantization), verified by test-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the default, gather, and per-expert paths) and by coherent end-to-end generation. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original per-expert path. Same-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192 (nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks): activation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused quantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The fused path avoids materializing and re-reading the 8x compact buffer, writing the expert copies directly from registers. * CUDA: bounds-check token ids in build_tok2c_kernel Guard against malformed ids_src1: skip out-of-range token ids (t < 0 or t >= n_tokens) and drop entries beyond n_expert_used per token instead of writing past the token's tok2c region. No behavior change for valid MoE routing data; test-backend-ops MUL_MAT_ID 790/790. * Refactor the code based on review comments - Removed previously added kernels that were not necessary anymore\ - Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows. * Adding q8_1 support for dedup and addressing review comments * Add pragma unrolls * Remove redundant cudaMemsetAsync call * Removing follow up redundancies --------- Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>

PR ggml-org#16308 set info.devices[id].integrated = false unconditionally for all CUDA/HIP devices as a workaround for corrupted output on Jetson Orin (ggml-org#15034). On HIP/ROCm the device's real hipDeviceProp_t.integrated flag is needed: with the cached field forced to false, supports_buft() refuses CUDA host buffers on AMD APU/UMA parts, while get_type() already reads prop.integrated (ggml-org#23007) — an inconsistency that breaks integrated-GPU host-buffer use on ROCm. Guard the workaround so it only applies to non-HIP (CUDA) builds and restore prop.integrated for HIP, keeping the Jetson workaround intact for CUDA. Fixes ggml-org#23977 Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>

Otherwise this gives lots of unnecessary warnings: W srv operator(): (CORS) skip non-localhost origin:

* support cuda virtual devices * disable NCCL path when virtual devices are used * label virtual devices in description; add GPUx2 server CI jobs * code refactor

…l-org#25733) * spec: fix dflash target tokenizer mismatch during conversion * fix ci ty check

)

…e graph New arch kimi-k3 (KimiK3ForConditionalGeneration, text_config kimi_linear): - compressed-tensors mxfp4-pack-quantized dequant (E2M1 nibbles low=even, E8M0 uint8 scale, group 32), verified against real Kimi-K3 shard tensors - KDA safe gate: g = lower_bound * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias)), full-rank output gate g_proj, A_log sliced to [:n_head] at conversion - gated MLA (NoPE): attn * sigmoid(g_proj(x)) before o_proj, q_lora_rank 1536 - Stable LatentMoE: router on full hidden, experts in 3584 latent space, RMSNorm after the weighted expert sum, then up-projection - AttnRes: residual-stream snapshot bank every attn_res_block_size layers, softmax mixtures before attention, before FFN and at model output - SiTU-GLU activation (LLM_FFN_SITU): beta*tanh(g/beta)*sigmoid(g) * lbeta*tanh(up/lbeta) Tested on a synthetic mini K3: convert -> BF16 GGUF -> Q8_0 -> generation.

worthant · 2026-07-27T17:20:13Z

Numerical parity verified against the Moonshot reference implementation (CPU, unmodified modeling_kimi_linear.py running on top of a fake fla package backed by flash-linear-attention's own naive_recurrent_kda torch reference).

Setup: synthetic mini K3 (4 layers: 3 KDA + 1 gated MLA, dense layer 0, 2 MoE layers with 8 MXFP4 experts, AttnRes block size 2, real kimi tiktoken vocab), prompt of 7 tokens, logits compared at every position over the full 163840 vocab.

pos	max abs diff	rms diff	corr	argmax match
0	0.00367	0.00083	0.99999	yes
1	0.00380	0.00082	0.99999	yes
2	0.00322	0.00073	0.99999	yes
3	0.00422	0.00084	0.99999	yes
4	0.00464	0.00106	0.99998	yes
5	0.00476	0.00095	0.99998	yes
6	0.00370	0.00077	0.99999	yes

Diff magnitude is bf16 weight rounding noise (llama.cpp: bf16 weights + f32 activations; reference: f32 upcast of the same bf16 weights).

Greedy continuation, 12 steps: llama.cpp used the cached path (KDA recurrent state + conv state + MLA absorption KV cache), reference ran cache-free full recompute. Token sequences are identical:
32495 161037 147832 133108 124684 114456 149045 79813 81528 51184 29676 110160

This covers all four K3 deltas numerically: SiTU-GLU, AttnRes mixtures, LatentMoE (router on full hidden, latent norm after weighted sum), KDA safe gate with full-rank output gate, and the gated NoPE MLA.

New kimik3 projector type, its graph builder and the mmproj converter. Also adds an optional clip.%s.attention.head_dim so build_vit stops deriving d_head from n_embd, which is wrong whenever a tower's qkv width differs from n_embd.

Graph node budget, LLM_TYPE for the 93-layer config, four hparams promoted to required, and an E8M0 NaN check during the MXFP4 repack. Details in the PR description.

Mirrors upstream PR 26185 commit cf11c4c. Without this the real checkpoint fails the n_expert assert at load; the synthetic test model (8 experts) never hit it.

…_MAP The vision converter class lives in kimivl.py but the mmproj module map had no K3 entry, so --mmproj failed with 'not supported'. The vision-tower commit added the class without wiring the lazy module map.

The patchmergerv2 post-norm tensor had no entry in tensor_mapping so the mmproj conversion aborted. The text converter inherited kimi-linear's tokenizer-eos override (163585 [EOS], document terminator) while K3's chat turn terminator is 163586 <|end_of_msg|>; chat generation never stopped at turn end. Existing GGUFs can be patched in place with gguf_set_metadata.py.

K3's assistant output is an XTML-ish tagged format built by the template's open_tag/close_tag macros. Two properties break generic parsing: 1. The generation prompt ends with open_tag('think'), so the completion starts inside the think section with no opening marker in the output (thinking_forced_open). 2. Only <|open|>/<|close|>/<|sep|>/<|end_of_msg|> are special tokens; tag names ("think", "response", "message") are ordinary text tokens. Adds common_chat_params_init_kimi_k3 (PEG_NATIVE) with detection on the marker trio, reasoning extraction, response unwrapping, and tool-call parsing of the tools/call/argument tag structure with argument types taken from the tool schema. Includes the K3 chat template fixture and 9 test-chat cases derived from real generations of the full 2.8T model. Verified end-to-end against Kimi-K3-Q2_K (GrEarl/Kimi-K3-GGUF) on 8x B200: content, reasoning_content, streaming deltas, and tool_calls all correct; finish_reason stop/tool_calls as appropriate.

Per-role message-start markers for token-level span splitting. User and assistant messages carry only the role attribute, so their full opener (through <|sep|>) is used; system and tool messages continue with more attributes (type=/tool=/index=), so those delimiters stop after the role's closing quote. Verified against the K3 tiktoken vocabulary that the closing quote is always a standalone token across all attribute variants, so the token-level prefix match stays exact.

…n.head_dim The mmproj converter writes the per-head dim because MoonViT-V2's qkv width (1536) is not derivable from the embedding length (1024); the writer method and KV constant were missing.

…s none Moonshot's tokenizer_config.json has no chat_template (K3 is API-first), so converted GGUFs were template-less and chat tools refused to start. Embed models/templates/Kimi-K3.jinja at conversion unless the checkpoint provides its own.

--chat-template already accepts MTMD; the file variant did not, which is inconvenient for models whose template only exists as a jinja file.

Active parameters per token: 69 KDA layers x 443.6M + 24 MLA x 232.2M + 92 MoE layers x 718.4M (16 of 896 experts, 2 shared, latent projections, router) + dense layer 0 FFN + lm_head = ~104B, matching Moonshot's published figure. The A50B label came in with the vision commits and understated it by half.

arthw and others added 30 commits July 15, 2026 10:31

sycl : support kernel type fp16 for conv2d_dw (ggml-org#25653)

ae9291e

sycl : fix get_rows Q2_K, Q4_K, Q5_K (ggml-org#25656)

d3fba0c

DeepseekV4: reduce graph splits (ggml-org#25702)

33a75f4

ggml: add f16 out_prod support for CPU and out_prod op for Vulkan (gg…

f955e39

…ml-org#23997)

Fix crash with draft-simple (ggml-org#25720)

956973c

* Fix crash with draft-simple * Fix tests for spec decoding

CUDA: tighter MMQ src1 buffer size for native fp4 (ggml-org#25613)

f6f12e4

tokenize : drop --stdin mutual-exclusion check (ggml-org#25672)

aff6eb6

match cli and completion, which don't enforce it

cuda: extract Q1_0 elements via __byte_perm (ggml-org#25628)

602f828

opencl: disable FA and MoE weights repack to work around compiler iss…

3dafb58

…ues for Adreno 850 GPU (ggml-org#25745) * opencl: workaround for A850 compiler compat * opencl: fix DX compiler version parsing and cleanup --------- Co-authored-by: Li He <lih@qti.qualcomm.com>

quant : allow using manual tensor types with --pure (ggml-org#25716)

56d6e9d

ci : add official website link to release notes (ggml-org#25728)

a320cbf

Assisted-by: pi:llama.cpp/Qwen3.6-27B

server: Ignore empty / non-existing Origin headers (ggml-org#25756)

8ee54c8

Otherwise this gives lots of unnecessary warnings: W srv operator(): (CORS) skip non-localhost origin:

Enable CUDA graphs on volta+turing (ggml-org#25749)

3f08ef2

CUDA: Support CUDA Virtual Devices (ggml-org#25228)

79bba02

* support cuda virtual devices * disable NCCL path when virtual devices are used * label virtual devices in description; add GPUx2 server CI jobs * code refactor

convert : fix dflash target tokenizer mismatch during conversion (ggm…

a55a8c5

…l-org#25733) * spec: fix dflash target tokenizer mismatch during conversion * fix ci ty check

server : allow text-only slot save/restore with mtmd (ggml-org#25076)

a8dc0e3

tests: actually exercise test-recurrent-state-rollback (ggml-org#25758

32e789f

)

vendor: update BoringSSL to 0.20260713.0 (ggml-org#25624)

86b719b

TP: fix Phi3, Bert, Plamo2/3, ChatGLM (ggml-org#25536)

2e1fd76

github-actions Bot added build devops jinja parser OpenCL Hexagon WebGPU OpenVINO CUDA conversion server/ui mtmd vendor labels Jul 27, 2026

worthant force-pushed the feat/kimi-k3-support branch from 8863f03 to b2f13d7 Compare July 27, 2026 17:13

worthant changed the title ~~🔨 feat(kimi-k3): Kimi K3 — MXFP4→BF16 конверсия и инференс-граф~~ feat(kimi-k3): Kimi K3 support: MXFP4 to BF16 conversion and inference graph Jul 27, 2026

worthant self-assigned this Jul 27, 2026

worthant marked this pull request as ready for review July 27, 2026 18:24

shimmyshimmer and others added 3 commits July 29, 2026 13:15

kimi-k3 : fixes for loading and running the full-size model

f55f4b0

Graph node budget, LLM_TYPE for the 93-layer config, four hparams promoted to required, and an E8M0 NaN check during the MXFP4 repack. Details in the PR description.

fix(kimi-k3): raise LLAMA_MAX_EXPERTS to 1024 for K3's 896 experts

4dfedb0

Mirrors upstream PR 26185 commit cf11c4c. Without this the real checkpoint fails the n_expert assert at load; the synthetic test model (8 experts) never hit it.

worthant force-pushed the feat/kimi-k3-support branch from 99e21f3 to 4dfedb0 Compare July 29, 2026 11:33

worthant and others added 8 commits July 29, 2026 15:13

fix(kimi-k3): register KimiK3ForConditionalGeneration in MMPROJ_MODEL…

d0e635a

…_MAP The vision converter class lives in kimivl.py but the mmproj module map had no K3 entry, so --mmproj failed with 'not supported'. The vision-tower commit added the class without wiring the lazy module map.

fix(kimi-k3): add add_vision_head_dim writer for clip.vision.attentio…

d70d259

…n.head_dim The mmproj converter writes the per-head dim because MoonViT-V2's qkv width (1536) is not derivable from the embedding length (1024); the writer method and KV constant were missing.

fix(mtmd): enable --chat-template-file for llama-mtmd-cli

5545e23

--chat-template already accepts MTMD; the file variant did not, which is inconvenient for models whose template only exists as a jinja file.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(kimi-k3): Kimi K3 support: MXFP4 to BF16 conversion and inference graph - #58

feat(kimi-k3): Kimi K3 support: MXFP4 to BF16 conversion and inference graph#58
worthant wants to merge 148 commits into
masterfrom
feat/kimi-k3-support

worthant commented Jul 27, 2026 •

edited

Loading

Uh oh!

worthant commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

worthant commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this is

Parts

Verified

References

Next

Uh oh!

worthant commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

worthant commented Jul 27, 2026 •

edited

Loading