Update tokenizers submodule to the linear-time HF encode path (pytorch#20472)

digantdesai · web-flow · commit 46c784f40651 · 2026-06-24T15:43:12.000-05:00
Bump extension/llm/tokenizers, picking up the linear-time HFTokenizer
encode work (merge_all O(n log n), ReplaceNormalizer::normalize O(N)
single forward pass) plus a targeted encode-latency benchmark. This cuts
long-prompt prefill tokenization time in the gemma4 / eagle3 runners;
token ids and greedy output are unchanged, verified e2e on the
gemma4-31B target (identical 18-token encode + decode after the bump).
diff --git a/extension/llm/tokenizers b/extension/llm/tokenizers
@@ -1 +1 @@
-Subproject commit 3f98e9903e4e9972e5371522d1b64bc7793c250b
+Subproject commit c17438e18855ffb31cf32ccf96c13c1483273df6
