Route byte-level llama tokenizers to TokenizersBackend#47017

Open

subin9 wants to merge 2 commits into

huggingface:mainfrom

subin9:fix-llama3-bytelevel-tokenizer-routing

Commits on Jul 10, 2026

Route byte-level llama tokenizers to TokenizersBackend

rebel-sbpark
authored and
subin9
committed

Commits on Jul 12, 2026

Add regression test for byte-level llama repos shipping tokenizer.model
rebel-sbpark
committed