Fix NewlineTokenizer registration for transformers >= 5.13#1465
Open
chandukona wants to merge 1 commit into
Open
Fix NewlineTokenizer registration for transformers >= 5.13#1465chandukona wants to merge 1 commit into
chandukona wants to merge 1 commit into
Conversation
chandukona
changed the title
This was referenced Jul 6, 2026
youssofal
added a commit
to youssofal/MTPLX
that referenced
this pull request
Jul 7, 2026
…ery fresh install at model load (#136, #135) transformers 5.13.0 changed AutoTokenizer.register to require a config class as the key; mlx-lm 0.31.x registers by name ("NewlineTokenizer"), so `import mlx_lm` raises AttributeError('str' has no '__module__') before any model can load. Every venv resolved after 5.13.0 shipped is dead on arrival — three independent reporters in 24h (#136 + 2 dupes, #135 is timing-consistent), and we hit the identical crash ourselves on 2026-07-03 and pinned the local venv manually (this makes the fix survive the app's fingerprint force-reinstall, which the manual pin did not). Marker-gated like the sibling mlx pins. Same content as PR #137 by @davidtai (verified: bisect 5.12.1 good / 5.13.0 bad matches our own finding). Wheel metadata verified: Requires-Dist: transformers<5.13 present in the built wheel. Drop the cap once mlx-lm registers the 5.13-compatible way (ml-explore/mlx-lm#1465).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1458
transformers 5.13.0 added an unguarded
key.__module__.startswith("transformers.")check in_LazyAutoMapping.register, so registering a tokenizer with a string key now raisesAttributeError: 'str' object has no attribute '__module__'at import time. mlx-lm registeredNewlineTokenizerunder the string"NewlineTokenizer", which older transformers versions tolerated because the key was never dereferenced.This PR passes the class itself as the registration key instead. Classes have
__module__, so registration succeeds on 5.13, and nothing else changes: name-based lookup (tokenizer_class_from_name) resolves"NewlineTokenizer"throughREGISTERED_TOKENIZER_CLASSES, which is keyed by the tokenizer's__name__regardless of the mapping key — so model repos with"tokenizer_class": "NewlineTokenizer"in theirtokenizer_config.jsonload exactly as before.Tested with transformers 5.13.0 and 5.7.0 (the minimum pinned version):
import mlx_lmsucceeds on both, andtokenizer_class_from_name("NewlineTokenizer")returns the class.If you'd prefer registering with a
PreTrainedConfigsubclass per the documented signature, happy to update — this version just keeps the diff to one line.