Summary

In transformers v5, AutoTokenizer.from_pretrained("ibm-granite/granite-4.0-micro") resolves to GPT2Tokenizer, which constructs BPE from vocab.json/merges.txt. In v4, it resolved to GPT2TokenizerFast, which loaded the pre-built tokenizer.json. The two file sources produce different token IDs for strings containing numbers and punctuation.

This is a silent regression — the v5 GPT2Tokenizer inherits from TokenizersBackend (Rust-based), so tokenizer.is_fast returns True and the class appears to work correctly. But the token IDs differ from what LoRA adapters were trained with, causing downstream accuracy regressions.

Ask: regression test in transformers

A tokenizer regression test should be added to transformers to catch future changes that alter token IDs for existing models. The test should:

1. Assert AutoTokenizer.from_pretrained() returns a PreTrainedTokenizerFast instance
2. Assert tokenizer.encode() produces known-good token IDs for strings with numbers and punctuation (the divergent cases)
3. Run without GPU — tokenizer-only, no model loading

This would have caught the v5 regression immediately in CI instead of after a full eval cycle.

Token ID divergence

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-4.0-micro")

The regression introduced in v5.4.0 persists in v5.7.0 — all 20 divergent cases remain unfixed.

Why this happens

Transformers v4.57.6 — AutoTokenizer resolves to GPT2TokenizerFast, which loads tokenizer.json:

tokenization_gpt2_fast.py L24:

VOCAB_FILES_NAMES = {"vocab_file": "vocab.json", "merges_file": "merges.txt", "tokenizer_file": "tokenizer.json"}

The tokenizer_file is passed to PreTrainedTokenizerFast.__init__(), which loads the complete tokenizer pipeline from tokenizer.json.

Transformers v5.4.0+ (including v5.7.0) — tokenization_gpt2_fast.py is removed. AutoTokenizer resolves to GPT2Tokenizer, which constructs BPE from vocab.json/merges.txt:

tokenization_gpt2.py L25-28:

VOCAB_FILES_NAMES = {
    "vocab_file": "vocab.json",
    "merges_file": "merges.txt",
}

The BPE model is constructed inline at L54-63:

self._tokenizer = Tokenizer(
    BPE(
        vocab=self._vocab,
        merges=self._merges,
        dropout=None,
        continuing_subword_prefix="",
        end_of_word_suffix="",
        fuse_unk=False,
    )
)

No tokenizer_file/tokenizer.json reference exists in the v5 file. For the original openai-community/gpt2 model, vocab.json/merges.txt and tokenizer.json are consistent, so both paths produce the same IDs. For granite-4.0-micro, whose tokenizer.json encodes a different BPE configuration, they diverge.

Impact on downstream tasks

The LoRA adapters (ibm-granite/granitelib-rag-r1.0) were trained with tokenizer.json token IDs. When inference uses different IDs, the adapter's learned weights are applied to the wrong token positions. Three of six RAG intrinsics regressed:

The other three intrinsics (answerability, clarification, citations) were unaffected because their eval scripts already used PreTrainedTokenizerFast or their prompts did not exercise the divergent BPE merges.

Workaround

Use PreTrainedTokenizerFast directly, which loads tokenizer.json:

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("ibm-granite/granite-4.0-micro")

Environment

• Model: ibm-granite/granite-4.0-micro
• Transformers v4 baseline: 4.57.6 (uses GPT2TokenizerFast + tokenizer.json)
• Transformers v5 regression: 5.4.0 through 5.7.0 (uses GPT2Tokenizer + vocab.json/merges.txt)
• Python: 3.12