Skip to content

Commit 2dad252

Browse files
Update tokenizers requirement from 0.22 to 0.23
Updates the requirements on [tokenizers](https://github.com/huggingface/tokenizers) to permit the latest version. - [Release notes](https://github.com/huggingface/tokenizers/releases) - [Changelog](https://github.com/huggingface/tokenizers/blob/main/RELEASE.md) - [Commits](huggingface/tokenizers@v0.22.0...v0.23.1) --- updated-dependencies: - dependency-name: tokenizers dependency-version: 0.23.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
1 parent eb7963e commit 2dad252

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

crates/bpe/benchmarks/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,4 +23,4 @@ bpe-openai = { path = "../../bpe-openai" }
2323
criterion = "0.8"
2424
rand = "0.10"
2525
tiktoken-rs = "0.9"
26-
tokenizers = { version = "0.22", features = ["http"] }
26+
tokenizers = { version = "0.23", features = ["http"] }

0 commit comments

Comments
 (0)