Skip to content

fix: pin transformers<5.13.0 to avoid AutoTokenizer.register breakage#1471

Open
rggammon wants to merge 1 commit into
ml-explore:mainfrom
rggammon:fix/transformers-5.13-register-compat
Open

fix: pin transformers<5.13.0 to avoid AutoTokenizer.register breakage#1471
rggammon wants to merge 1 commit into
ml-explore:mainfrom
rggammon:fix/transformers-5.13-register-compat

Conversation

@rggammon

@rggammon rggammon commented Jul 4, 2026

Copy link
Copy Markdown

Problem

transformers==5.13.0 introduced a breaking change in AutoTokenizer.register() that causes mlx-lm to fail at import time with Python 3.14 on macOS.

The error occurs in mlx_lm/tokenizer_utils.py at:
python AutoTokenizer.register("NewlineTokenizer", fast_tokenizer_class=NewlineTokenizer)

transformers 5.13.0 added a check in auto_factory.py that requires registered tokenizer classes to have module starting with 'transformers.'. Since NewlineTokenizer is defined in mlx-lm, not in transformers, this check fails.

Error:

File "...transformers/models/auto/auto_factory.py", line 680, in register
    if key.__module__.startswith("transformers."):

To reproduce:

pip install mlx-lm transformers==5.13.0
python -c "import mlx_lm"  # fails

Fix

Pin transformers<5.13.0 as a temporary fix while a proper solution (e.g. updating the AutoTokenizer.register call to be compatible with the new API) is investigated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant