Skip to content

Fix deepseek indexer#1431

Merged
angeloskath merged 4 commits into
ml-explore:mainfrom
pcuenca:fix-deepseek-indexer
Jun 24, 2026
Merged

Fix deepseek indexer#1431
angeloskath merged 4 commits into
ml-explore:mainfrom
pcuenca:fix-deepseek-indexer

Conversation

@pcuenca

@pcuenca pcuenca commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

This description was written by a human 🙋‍♂️

Rope traditional was changed for the Indexer here as part of #867. This fixed a very obvious long-sequence generation error with GLM 5: the model started producing gibberish after a while.

I think that the indexer of Deepseek 3.2 requires False, but using True does not result in immediately observed garbage output. Outputs remain coherent, but quality silently degrades for long sequences.

I tested by generating 6 JavaScript implementations for different games using the prompts shown in the script below. We want generation to go beyond the 2048 sequence length where the indexer kicks in. In all 6 cases, the implementations from the current mlx-lm package are coherent, but the games are not playable. Using the patched version, all games are playable (with confusing visual imperfections in two of the games, but playable).

I found this issue while working on GLM 5.2. I think that silent degradation + superclass being used as the reference for other implementations is worth fixing as robustly as we can. Happy to run additional tests to confirm behavior!

Generation script

Covers 5 prompts, there's a sixth one I ran outside the script.

#!/bin/bash

declare -a prompts=(
  "Write a complete, polished, single-file HTML+JavaScript implementation of Tetris, with comments. Include the 7 tetromino shapes, rotation, line clearing, scoring, increasing speed by level, and a next-piece preview."
  "Write a complete, polished, single-file HTML+JavaScript 2048 game, with comments. Arrow-key controls, tile sliding and merging, score, win at 2048, and game-over detection."
  "Write a complete, polished, single-file HTML+JavaScript implementation of Snake, with comments. Grid movement, growth on eating food, self- and wall-collision death, score, and a restart."
  "Write a complete, polished, single-file HTML+JavaScript implementation of Breakout/Arkanoid, with comments. A paddle, a grid of destructible bricks, ball physics with wall/paddle/brick bounces, lives, score, and level-clear."
  "Write a complete, polished, single-file HTML+JavaScript implementation of Minesweeper, with comments. Random mine placement, left-click reveal with flood-fill on zero-adjacency cells, right-click flagging, number hints, and win/lose detection."
)

typeset -i id=1
for p in "${prompts[@]}"
do
  for version in "mlx-lm" "mlx-lm.patched"
  do
    uv pip install -e "./$version"
    echo "[$version] Generating prompt $p"
    mlx_lm.generate --model models/mlx-community/DeepSeek-V3.2-4bit --prompt "$p" -m 16384 --temp 0 | tee ${id}-${version}.txt
  done
  let id=$id+1
done

This PR is the counterpart of huggingface/transformers#46842 (in transformers, GLM appears to be wrong but Deepseek is correct).

pcuenca added 3 commits June 24, 2026 13:53
Note that the default flipped to False. This is correct for DeepSeek v3.2, but not for GLM.
Comment thread mlx_lm/chat_templates/deepseek_v32.py Outdated
Comment on lines +336 to +337
if "enable_thinking" in kwargs:
kwargs["thinking_mode"] = "thinking" if kwargs.pop("enable_thinking") else "chat"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is technically a different issue, but mlx_lm.generate did not run otherwise.

Comment on lines -74 to +75
traditional=True,
traditional=args.indexer_rope_interleave,

@pcuenca pcuenca Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning: we are flipping the default to False here.

However, I think the only subclass is currently glm_moe_dsa and the property does exist for GLM 5 models:

EternaPeptix added a commit to EternaPeptix/mlx-lm that referenced this pull request Jun 24, 2026
EternaPeptix added a commit to EternaPeptix/mlx-lm that referenced this pull request Jun 24, 2026

@angeloskath angeloskath left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing the super subtle bug!

Also

This description was written by a human 🙋‍♂️

Absolute king! Thanks for that.

@angeloskath angeloskath merged commit 2ed2231 into ml-explore:main Jun 24, 2026
2 checks passed
@pcuenca pcuenca deleted the fix-deepseek-indexer branch June 25, 2026 10:48
000alen added a commit to velum-labs/mlx-lm that referenced this pull request Jul 2, 2026
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
machiabeli added a commit to machiabeli/mlx-lm-1 that referenced this pull request Jul 4, 2026
…parse path)

- indexer_rope_interleave defaults True for GLM-5.2 (the arch keeps the
  pre-ml-explore#1431 interleaved convention; validated against the real 743B mxfp4
  checkpoint — engaged-DSA generation at 3k context is coherent and an
  independent C/Metal engine reproduces the top-k selection exactly with
  interleaved rope).
- test_all_models entry with indexer_types=["full","shared","full","shared"]
  (exercises shared-layer construction + asymmetric CacheList at B=1/B=2).
- test_glm_moe_dsa_shared_schedule: full layers own indexers, shared layers
  don't (checkpoint ships no weights for them), interleaved rope flag,
  CacheList(kv,indexer) vs CacheList(kv), and the ENGAGED path — prefill
  L > index_topk then a decode step threading prev_topk full->shared,
  finite outputs. The sparse (>topk) path is additionally cross-validated
  externally: a C reference matched this implementation's engaged logits at
  1e-6 on full AND shared schedules (tiny config), and per-layer parity
  probes on the real checkpoint at 3015-token context selected identical
  top-2048 sets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants