Skip to content

Fix detokenizer for byte-level tokenizers with an SPM-style decoder (fixes #1041)#1329

Open
robertlangdonn wants to merge 1 commit into
ml-explore:mainfrom
robertlangdonn:fix-tekken-detokenizer-spaces
Open

Fix detokenizer for byte-level tokenizers with an SPM-style decoder (fixes #1041)#1329
robertlangdonn wants to merge 1 commit into
ml-explore:mainfrom
robertlangdonn:fix-tekken-detokenizer-spaces

Commits

Commits on Jul 11, 2026