Fix added-token prefix space in SentencePiece fast tokenizers#45822

Open

qflen wants to merge 6 commits into

huggingface:mainfrom

qflen:fix-added-token-prefix-space

Commits on May 7, 2026

Fix added-token prefix space in SentencePiece fast tokenizers
qflen
committed
Merge remote-tracking branch 'upstream/main' into fix-added-token-prefix-space
qflen
committed

Commits on May 12, 2026

Simplify added-token prefix-space fix
qflen
committed

Commits on Jul 4, 2026

Commits on Jul 6, 2026

Merge branch 'main' into fix-added-token-prefix-space
itazap
authored