Skip to content

docs(bpe): cite Incremental BPE Tokenization paper #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gorzell merged 3 commits into from
Jun 17, 2026
Merged

docs(bpe): cite Incremental BPE Tokenization paper #130

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ebd4785
docs(bpe): cite Incremental BPE Tokenization paper
gorzell Jun 17, 2026
35e387b
docs(bpe): use inline-code complexity notation for consistency
gorzell Jun 17, 2026
75e4040
docs(bpe): note paper builds on our incremental algorithm
gorzell Jun 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions crates/bpe/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,3 +300,7 @@ This case is particularly challenging for tiktoken, which shows a quadratic grow
The Huggingface encoder scales better, but becomes slower and slower compared to our implementation as input size increases.

![worst-case encoding runtime comparison](./images/performance-worstcase.svg)

For a full runtime analysis of incremental BPE tokenization, see Jiang and Gong, ["Incremental BPE Tokenization"](https://arxiv.org/abs/2605.30813) (ICML 2026), which presents an algorithm with a worst-case `O(n log^2 t)` complexity (where `n` is the input length and `t` is the maximum token length).

@aneubeck aneubeck Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could write that this is paper is building on our incremental algorithm and takes it one step further by combining the aho-corasick-search with the compatibility test into one automaton.

@gorzell gorzell Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great context, thanks. Added a sentence noting their work builds on our incremental algorithm and combines the aho-corasick search with the compatibility test into a single automaton (75e4040).

Their work builds on our incremental algorithm and takes it one step further by combining the aho-corasick search with the compatibility test into a single automaton.
Their implementation is available at [ModelTC/mtc-inc-bpe](https://github.com/ModelTC/mtc-inc-bpe).