Fix added-token prefix space in SentencePiece fast tokenizers

[qflen]

Fixes #28218

When a user adds a token to a SentencePiece-style fast tokenizer (e.g. NLLB), the tokens-trie splits the input
at the new token and runs the Metaspace pre_tokenizer on each chunk.

The default prepend_scheme="always" then prefixes every chunk with ▁, so "abcdgym" tokenizes to ["abcd", "▁gym"] and decode(encode("abcdgym")) returns "abcd gym".

This PR mirrors what SpmConverter.pre_tokenizer already does for non-legacy spm tokenizers:
when a post-init, non-special add_tokens call lands on a standalone Metaspace(prepend_scheme="always"), the pre_tokenizer is swapped for one with prepend_scheme="first" so only the first chunk receives the prefix.

Sequence pipelines (WhitespaceSplit + Metaspace, used by T5, Pegasus, UDOP, LASR) are left untouched because they rely on "always" for whitespace handling.

With tokenizer.add_tokens(["abcd"]) applied to a fresh NLLB tokenizer:

Input	Before	After
"abcdgym"	["abcd", "▁gym"] → "abcd gym"	["abcd", "gym"] → "abcdgym"
"I like to walk abcdgym along the beach"	[..., "▁walk", "▁", "abcd", "▁gym", "▁along", ...] → "I like to walk abcd gym along the beach"	[..., "▁walk", "▁", "abcd", "gym", "▁along", ...] → "I like to walk abcdgym along the beach"
"abcd gym" (literal space already there)	["abcd", "▁gym"]	["abcd", "▁gym"] (unchanged)

Inputs without any added tokens are unchanged (e.g. "Hello world" → ["▁Hello", "▁world"] either way).

Code Agent Policy

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed or approved via a Github issue or the forum? Please add a link to it if that is the case.
  Issue: Tokenizer adds an additional space after the added token #28218
  (@ArthurZucker proposed mirroring the SpmConverter approach)
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

@ArthurZucker @itazap

---

AI assistance disclosure

I've used Opus 4.7 Max Effort for debugging and reviewed / double-checked every line.

• Coordination:
  Tokenizer adds an additional space after the added token #28218 (maintainer guidance to follow SpmConverter.pre_tokenizer

• Claimed issue in:
  Tokenizer adds an additional space after the added token #28218 (comment))

• Why this is not a duplicate:
  The only related open PR (SPLIT PR: add_prefix_space fix #31315) targets a different issue (SPMConverter does not always add the user defined symbol -> slow fast is thus not equivalent #30824, plumbing add_prefix_space through the fast tokenizer); it does not touch the post-init alignment of prepend_scheme for added tokens.

• Tests run:

  • make style and make typing and make fix-repo, all green, no extra diff
  • pytest tests/tokenization/test_tokenization_fast.py (11 passed, 3 skipped)
  • pytest tests/test_tokenization_common.py (18 passed, 3 skipped)
  • pytest tests/models/nllb/test_tokenization_nllb.py (61 passed, 4 skipped)
  • pytest tests/models/big_bird/test_tokenization_big_bird.py (100 passed, 4 skipped)
  • pytest tests/models/{reformer,t5,albert,udop,moshi,seamless_m4t}/test_tokenization_*.py (all green)
  • The new regression test test_added_token_does_not_get_metaspace_prefix was verified to fail on main
    (without the fix) and pass with the fix.

[Rocketknight1]

cc @ArthurZucker @itazap

[itazap]

okay but since "always" and "first" are equivalent for normal text (without added tokens), the root fix is just: in _get_prepend_scheme, return "first" instead of "always" when add_prefix_space=True. That would fix new conversions to fast ! we just need to handle it in _add_tokens for the cases when loading a .json from the hub but should be a bit simpler without new flags

[qflen]

Thanks for the feedback. Pushed changes.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: xlnet

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 28796948160:2
Result: failure | Jobs: 15 | Tests: 171,542 | Failures: 6 | Duration: 21h 34m

[itazap]

Thanks for working on this!

One concern: _tokenizer_json_is_byte_level fetches and fully parses tokenizer.json on every AutoTokenizer.from_pretrained where model_type is "llama" - this same file gets fetched and parsed again immediately after by convert_to_native_format. .

I'm wondering if we can instead do an existence check instead, like _has_tekken_tokenizer_file above it?

Llama1/2 (aentencepiece) always ships tokenizer.model I believe and Llama3+ never do

[qflen]

@itazap thanks, this is the #28218 metaspace prepend_scheme fix, but I can add the byte-level detection here too.

[qflen]

Actually, was your comment meant for #47017? I think this can be merged as is.