Skip to content

Commit dfd3d2e

Browse files
committed
Update tokenizers submodule to the linear-time HF encode path
Bump extension/llm/tokenizers, picking up the linear-time HFTokenizer encode work (merge_all O(n log n), ReplaceNormalizer::normalize O(N) single forward pass) plus a targeted encode-latency benchmark. This cuts long-prompt prefill tokenization time in the gemma4 / eagle3 runners; token ids and greedy output are unchanged, verified e2e on the gemma4-31B target (identical 18-token encode + decode after the bump). ghstack-source-id: 525dc35 ghstack-comment-id: 4734208425 Pull-Request: #20349
1 parent d9f3278 commit dfd3d2e

1 file changed

Lines changed: 1 addition & 1 deletion