feat(models): add Kimi K2 + K2.7 (text-only) support#3065
Draft
faresobeid wants to merge 1 commit into
Draft
Conversation
Kimi K2 reuses DeepSeek-V3's architecture verbatim (MLA + MoE hyperparameters, weight-key naming, even auto_map points at transformers' own DeepseekV3ForCausalLM), so this subclasses DeepseekV3Config directly and adds a plain (non-sparse) MLA attention layer (layers/mla.py) rather than redefining ~20 config fields. Reuses GLM-4 MoE's existing HF<->prime conversion functions as-is since the MoE layout (per-expert gate/up/down, shared-experts, e_score_correction_bias) matches exactly. K2.7 (and K2.5/K2.6) ship as a multimodal wrapper with the same text backbone, language_model.-prefixed. Loading one strips that prefix and drops the vision tower entirely — text-only, no VLM support. Verified via scripts/mini_moe.py's HF<->prime roundtrip check (max logits diff 1e-6, using transformers' own DeepseekV3ForCausalLM as the reference — Kimi's own checkpoints use this exact class). No real Kimi checkpoint was available to validate against directly. vLLM already serves real Kimi checkpoints natively (architectures=DeepseekV3ForCausalLM routes into its existing DeepSeek-V3 support), so no inference-side changes are needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
transformers'DeepseekV3ForCausalLMas theirauto_maptarget).KimiK2ConfigsubclassesDeepseekV3Configdirectly rather than redefining its ~20 MLA/MoE fields.layers/mla.py: plain (dense, non-absorbed) Multi-head Latent Attention, shared infrastructure any MLA family can build on.glm4_moe— the per-expert gate/up/down + shared-experts +e_score_correction_biaslayout matches exactly, no adaptation needed.language_model.-prefixed weights + a vision tower). Loading one strips the prefix and drops the vision tower entirely — text-only, no VLM/multimodal support.sami/kimi-k2draft (add kimi k2 #1761, closed unmerged) — same core design (subclassDeepseekV3Config, reuse GLM's MoE conversion), updated for currentmain(renamedkimi_k25→kimi_k2,fa4→flash_attention_4,get_cu_seqlens_from_position_ids,MoEArgs.fp8), plus the K2.7 text-extraction addition and a dedup pass on the two MLA attention variants.Verification
scripts/mini_moe.py --arch kimi_k2: HF↔prime roundtrip logits match to1e-6, usingtransformers' ownDeepseekV3ForCausalLMas the reference (real Kimi checkpoints use this exact class, so this validates the modeling code faithfully reimplements the same math).config.jsondeclaresarchitectures: ["DeepseekV3ForCausalLM"]and routes straight into vLLM's existing DeepSeek-V3 support.🤖 Generated with Claude Code