File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77
88## [ Unreleased]
99
10+ ## [ 0.5.0] ( https://github.com/ModelTC/mtc-incremental-bpe/compare/v0.4.1...v0.5.0 ) - 2025-11-28
11+
12+ ### Added
13+
14+ - [ ** breaking** ] rename functions and expose more for ` IncBpeTokenization `
15+
16+ ### Fixed
17+
18+ - use the roots of the subtrees as indicators of parents, fix #19
19+ - expose priority of a token in normalized dict, fix panic when token id exceeded vocab size
20+ - use ` LinkedList ` for suffix chain
21+ - check token length explicitly
22+ - add ` non_exhaustive ` to errors
23+ - use u16 as ` skip_len `
24+
25+ ### Other
26+
27+ - use ` tinyvec ` replacing ` smallvec ` to reduce memory footprint
28+ - * (tests)* add tests on repeated characters
29+ - unify integer literals
30+
1031## [ 0.4.1] ( https://github.com/ModelTC/mtc-incremental-bpe/compare/v0.4.0...v0.4.1 ) - 2025-11-27
1132
1233### Other
Original file line number Diff line number Diff line change 11[workspace .package ]
2- version = " 0.4.1 "
2+ version = " 0.5.0 "
33edition = " 2024"
44license = " MIT OR Apache-2.0"
55description = " Incremental BPE tokenization for all prefixes"
You can’t perform that action at this time.
0 commit comments