Skip to content

Commit ebd4785

Browse files
gorzellCopilot
andcommitted
docs(bpe): cite Incremental BPE Tokenization paper
Add a reference to Jiang and Gong, "Incremental BPE Tokenization" (ICML 2026), in the tokenizer comparison section of the bpe README, linking to the paper's full runtime analysis and the authors' implementation at ModelTC/mtc-inc-bpe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 2c3ff25 commit ebd4785

1 file changed

Lines changed: 3 additions & 0 deletions

File tree

crates/bpe/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -300,3 +300,6 @@ This case is particularly challenging for tiktoken, which shows a quadratic grow
300300
The Huggingface encoder scales better, but becomes slower and slower compared to our implementation as input size increases.
301301

302302
![worst-case encoding runtime comparison](./images/performance-worstcase.svg)
303+
304+
For a full runtime analysis of incremental BPE tokenization, see Jiang and Gong, ["Incremental BPE Tokenization"](https://arxiv.org/abs/2605.30813) (ICML 2026), which presents an algorithm with a worst-case $\mathcal{O}(n \log^2 t)$ complexity (where $n$ is the input length and $t$ is the maximum token length).
305+
Their implementation is available at [ModelTC/mtc-inc-bpe](https://github.com/ModelTC/mtc-inc-bpe).

0 commit comments

Comments
 (0)