Skip to content

Improve Unigram tokenization performance for long inputs#34

Open
bepsvpt wants to merge 1 commit into
huggingface:mainfrom
bepsvpt:fix/unigram-quadratic-tokenization
Open

Improve Unigram tokenization performance for long inputs#34
bepsvpt wants to merge 1 commit into
huggingface:mainfrom
bepsvpt:fix/unigram-quadratic-tokenization

Avoid quadratic suffix materialization in Unigram tokenization

184e887
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs