Skip to content

Route byte-level llama tokenizers to TokenizersBackend#47017

Open
subin9 wants to merge 2 commits into
huggingface:mainfrom
subin9:fix-llama3-bytelevel-tokenizer-routing
Open

Route byte-level llama tokenizers to TokenizersBackend#47017
subin9 wants to merge 2 commits into
huggingface:mainfrom
subin9:fix-llama3-bytelevel-tokenizer-routing

Add regression test for byte-level llama repos shipping tokenizer.model

5bbdf64
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
Get PR number / Get PR number
succeeded Jul 12, 2026 in 4s