Update tokenizers submodule to the linear-time HF encode path#20472
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20472
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
There was a problem hiding this comment.
Pull request overview
Updates the extension/llm/tokenizers submodule to pick up a new linear-time HuggingFace tokenizer encode path, along with an encode-latency benchmark, to reduce long-prompt prefill tokenization time in the Gemma/Eagle runners while preserving tokenization outputs.
Changes:
- Bump
extension/llm/tokenizerssubmodule revision to include the new linear-timeHFTokenizer::encodeimplementation. - Pick up tokenizer-side performance improvements (notably around merge/normalization paths) and an encode-latency benchmark.
- Intended to improve prefill tokenization latency without changing token IDs or greedy outputs (per PR description).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Can you also include this commit too? meta-pytorch/tokenizers@c17438e Just merged now |
Bump extension/llm/tokenizers, picking up the linear-time HFTokenizer encode work (merge_all O(n log n), ReplaceNormalizer::normalize O(N) single forward pass) plus a targeted encode-latency benchmark. This cuts long-prompt prefill tokenization time in the gemma4 / eagle3 runners; token ids and greedy output are unchanged, verified e2e on the gemma4-31B target (identical 18-token encode + decode after the bump). ghstack-source-id: 525dc35 ghstack-comment-id: 4734208425 Pull-Request: #20349
digantdesai
force-pushed
the
tokenizer_nlon_pin_bump
branch
from
099f306 to
dfd3d2e
Compare
The tokenizers submodule bump (#20472) moved the fbsource//third-party/nlohmann-json dependency onto the tokenizers :headers target, which is reachable from the OSS buck2 CI build (multimodal_runner_lib -> text_token_generator -> tokenizers:headers). Since fbsource aliases to shim_et, this requires a shim_et//third-party/nlohmann-json package, which did not exist (only re2 had one). Add the shim, mirroring shim_et//third-party/re2, aliasing nlohmann-json to the header-only nlohmann_json target added to the tokenizers submodule (meta-pytorch/tokenizers#201), and bump the submodule pin to include it. Also wire the nlohmann_json external dep in env_interface.bzl to the same target so hf_tokenizer/tekken resolve under OSS buck instead of dropping the dep.
Bump extension/llm/tokenizers, picking up the linear-time HFTokenizer encode work (merge_all O(n log n), ReplaceNormalizer::normalize O(N) single forward pass) plus a targeted encode-latency benchmark. This cuts long-prompt prefill tokenization time in the gemma4 / eagle3 runners; token ids and greedy output are unchanged, verified e2e on the gemma4-31B target (identical 18-token encode + decode after the bump).