Skip to content

Commit ea2092f

Browse files
github-actions[bot]ChieloNewctle
authored andcommitted
chore: release v0.5.0
1 parent 72e5ee3 commit ea2092f

2 files changed

Lines changed: 22 additions & 1 deletion

File tree

CHANGELOG.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
## [0.5.0](https://github.com/ModelTC/mtc-incremental-bpe/compare/v0.4.1...v0.5.0) - 2025-11-28
11+
12+
### Added
13+
14+
- [**breaking**] rename functions and expose more for `IncBpeTokenization`
15+
16+
### Fixed
17+
18+
- use the roots of the subtrees as indicators of parents, fix #19
19+
- expose priority of a token in normalized dict, fix panic when token id exceeded vocab size
20+
- use `LinkedList` for suffix chain
21+
- check token length explicitly
22+
- add `non_exhaustive` to errors
23+
- use u16 as `skip_len`
24+
25+
### Other
26+
27+
- use `tinyvec` replacing `smallvec` to reduce memory footprint
28+
- *(tests)* add tests on repeated characters
29+
- unify integer literals
30+
1031
## [0.4.1](https://github.com/ModelTC/mtc-incremental-bpe/compare/v0.4.0...v0.4.1) - 2025-11-27
1132

1233
### Other

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[workspace.package]
2-
version = "0.4.1"
2+
version = "0.5.0"
33
edition = "2024"
44
license = "MIT OR Apache-2.0"
55
description = "Incremental BPE tokenization for all prefixes"

0 commit comments

Comments
 (0)