Fix deepseek indexer#1431
Merged
Merged
Conversation
Note that the default flipped to False. This is correct for DeepSeek v3.2, but not for GLM.
pcuenca
commented
Jun 24, 2026
Comment on lines
+336
to
+337
| if "enable_thinking" in kwargs: | ||
| kwargs["thinking_mode"] = "thinking" if kwargs.pop("enable_thinking") else "chat" |
Contributor
Author
There was a problem hiding this comment.
This is technically a different issue, but mlx_lm.generate did not run otherwise.
pcuenca
commented
Jun 24, 2026
Comment on lines
-74
to
+75
| traditional=True, | ||
| traditional=args.indexer_rope_interleave, |
Contributor
Author
There was a problem hiding this comment.
Warning: we are flipping the default to False here.
However, I think the only subclass is currently glm_moe_dsa and the property does exist for GLM 5 models:
- GLM 5 original, mlx
- GLM 5.1 mlx-community/GLM-5.1, mlx-community/GLM-5.1-MXFP4-Q8, mlx-community/GLM-5.1-DQ4plus-q8
- GLM 5.2 mlx (I ran the conversion from the original and the property was thus inherited).
EternaPeptix
added a commit
to EternaPeptix/mlx-lm
that referenced
this pull request
Jun 24, 2026
…n deepseek_v32 chat template
EternaPeptix
added a commit
to EternaPeptix/mlx-lm
that referenced
this pull request
Jun 24, 2026
angeloskath
approved these changes
Jun 24, 2026
angeloskath
left a comment
Member
There was a problem hiding this comment.
Thanks for fixing the super subtle bug!
Also
This description was written by a human 🙋♂️
Absolute king! Thanks for that.
000alen
added a commit
to velum-labs/mlx-lm
that referenced
this pull request
Jul 2, 2026
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
machiabeli
added a commit
to machiabeli/mlx-lm-1
that referenced
this pull request
Jul 4, 2026
…parse path) - indexer_rope_interleave defaults True for GLM-5.2 (the arch keeps the pre-ml-explore#1431 interleaved convention; validated against the real 743B mxfp4 checkpoint — engaged-DSA generation at 3k context is coherent and an independent C/Metal engine reproduces the top-k selection exactly with interleaved rope). - test_all_models entry with indexer_types=["full","shared","full","shared"] (exercises shared-layer construction + asymmetric CacheList at B=1/B=2). - test_glm_moe_dsa_shared_schedule: full layers own indexers, shared layers don't (checkpoint ships no weights for them), interleaved rope flag, CacheList(kv,indexer) vs CacheList(kv), and the ENGAGED path — prefill L > index_topk then a decode step threading prev_topk full->shared, finite outputs. The sparse (>topk) path is additionally cross-validated externally: a C reference matched this implementation's engaged logits at 1e-6 on full AND shared schedules (tiny config), and per-layer parity probes on the real checkpoint at 3015-token context selected identical top-2048 sets.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This description was written by a human 🙋♂️
Rope
traditionalwas changed for the Indexer here as part of #867. This fixed a very obvious long-sequence generation error with GLM 5: the model started producing gibberish after a while.I think that the indexer of Deepseek 3.2 requires
False, but usingTruedoes not result in immediately observed garbage output. Outputs remain coherent, but quality silently degrades for long sequences.I tested by generating 6 JavaScript implementations for different games using the prompts shown in the script below. We want generation to go beyond the 2048 sequence length where the indexer kicks in. In all 6 cases, the implementations from the current mlx-lm package are coherent, but the games are not playable. Using the patched version, all games are playable (with confusing visual imperfections in two of the games, but playable).
I found this issue while working on GLM 5.2. I think that silent degradation + superclass being used as the reference for other implementations is worth fixing as robustly as we can. Happy to run additional tests to confirm behavior!
Generation script
Covers 5 prompts, there's a sixth one I ran outside the script.
This PR is the counterpart of huggingface/transformers#46842 (in transformers, GLM appears to be wrong but Deepseek is correct).