Improve robustness for handling Python-authored regexes

[nico-martin]

Summary

• Fix JS regex compatibility for tokenizer patterns exported from Python/Rust tokenizers by sanitizing extraneous escapes before #, &, ~, ', and ".
• Add Unicode-aware normalization for shorthand classes in regex patterns so multilingual tokenizers behave as expected in JavaScript (\w, \W, \d, \D).
• Add a runtime warning when tokenizer regex contains \b or \B, since JavaScript boundary semantics can be ASCII-oriented.
• Add edge-case tests covering quote-class compatibility (['\\\"]) and multilingual \w+ matching (including Hebrew text).

Why

• Some tokenizer regex patterns that are valid in Python/Rust fail to compile in JavaScript Unicode regex mode due to invalid identity escapes.
• JavaScript shorthand character classes differ from Python/Rust behavior and can silently degrade multilingual tokenization quality.

Validation

• Ran npm test -- tests/edgeCases.test.ts.
• All tests in tests/edgeCases.test.ts pass, including the new compatibility checks.

[Copilot]

Pull request overview

This PR improves JavaScript compatibility for tokenizer regex patterns coming from Python/Rust by sanitizing problematic escapes, normalizing shorthand classes for Unicode behavior, and adding targeted edge-case tests.

Changes:

• Expand escape sanitization for certain identity escapes before compiling RegExp in JS.
• Normalize \w/\W/\d/\D into Unicode-aware equivalents for better multilingual tokenization behavior.
• Add an edge-case test to validate quote-class regex compilation and Unicode \w+ matching (Hebrew).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
src/utils/core.ts	Adjusts regex sanitization and introduces Unicode shorthand normalization + boundary warnings during pattern compilation.
tests/edgeCases.test.ts	Adds regression tests for quote-class compilation and multilingual \w+ matching.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[eyaler]

wow thanks @nico-martin
i dont currently have an enviroment to build and test ts - so i need to set that up.
some quick questions for now:

1. \W is not taken care of when in a character class? and also no warning is raised?
2. i get that the \b, \B is more difficult and perhaps less common, so i guess a warning could work for the first iteration - just not sure why this is mentioned in the context of compatibility. But this also opens a larger question: can we assume no one is depending on the current behavior for all these changes?
3. why \p{Nd} and not \p{N} as before?
4. it seems that the char class detection might be too simplistic. there could be a bare escaped \[ outside of char class. also there could be an escaped \] inside as char class, and also not sure if there are other special places where regex syntax allows square brackets
5. more generally, i wonder if there aren't existing regex normilizers that we could use, instead of building this from scratch? i imagine that are more edge cases to come in regard to both this parsing and the js/python differences.

[xenova]

friendly ping @nico-martin :)

[nico-martin]

Hi @eyaler,
This whole regex thing is quite complex, and to be honest, it's not really my favourite subject :D. So thanks a lot for your questions!

1. yes, this is a gap right now. \W is handled outside a character class, but inside a class it is left as \W, and there is no warning. so this should either be fixed, or at least warn if we leave it as-is.

2. \b / \B are mentioned because they are part of the same python/rust vs js regex mismatch, but this branch does not change them, only warns. about compatibilty: yes, this can technically change behavior for anyone depending on JS ascii-only \w, \d, etc. My thinking is that tokenizer regexes should match the source tokenizer semantics, but we should call this out as a behaviour change.

3. \p{Nd} is closer to \d. \p{N} means all unicode numbers, incl things that are not decimal digits. python/rust \d is decimal digits, so \p{Nd} seems like the better mapping here.

4. agreed, the char class detection is too simple. it just toggles on [ and ], so escaped \[ / \] can confuse it. this is ok-ish for the narrow fix, but before merging we should either add tests for these cases or use a real parser.

5. i checked npm and did not find a drop-in python/rust-to-js tokenizer regex normalizer. what exists is mostly js regex parsers/transformers:

   • @eslint-community/regexpp: maintained js regex parser/validator. probably the best fit if we want ast-aware parsing.
   • regexp-tree: js regex parser/transformer with ast/codegen, but seems less obviously maintained.
   • regexpu-core: transpiles modern js regex to older js. not useful for python/rust input.
   • regjsparser / regjsgen: lower-level parser + generator, still js-focused.
   • refa: powerful but probably way too much for this.
   • oniguruma-to-es: interesting, but converts oniguruma, not really the tokenizer dialect here.
   • regex: nice js regex helper, but not a converter.

   so i think there is no package that does the whole thing for us. best next step would be using @eslint-community/regexpp for parsing, and keep the actual tokenizer-specific rewrites ourselves. that would fix the bracket parsing without writing a whole parser.

[xenova]

thanks for tackling this!
Indeed, we'd like to avoid new dependencies, so finding a way to handle these cases in-library (at least for any tokenizer regex on the HF hub) will be the best approach imo.

For example, this PR (huggingface/huggingface.js#1418) which fixed jinja templates for top 100k templates on the HF hub was done using existing templates as test cases.

I'll write a script which does the same for regexes.

[xenova]

okay, it took a couple of hours to run, but after downloading ~32k tokenizer.json files, there only appear to be ~105 unique regexes? (this seems quite strange tbh... I'll choose to believe it for now)

[
  "",
  "\\{(?=[ \\w])",
  "@[\\w\\d]+",
  "\\p{N}{1,3}",
  "(?:\\r?\\n)+(?!\\r?\\n)",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\r\n]*|\\s*[\r\n]+|\\s+(?!\\S)|\\s+",
  "[\u201c\u201d\u201e\u201f\uff02]",
  "/(?=[ \\w])",
  "[\r\n]",
  "(\\[[^\\]]+]|Br?|Cl?|N|O|S|P|F|I|b|c|n|o|s|p|\\(|\\)|\\.|=|#|-|\\+|\\\\|\\/|:|~|@|\\?|>>?|\\*|\\$|\\%[0-9]{2}|[0-9])",
  "\u2015(?=[ \\w])",
  "[!\"#$%&'()*+,\\-./:;<=>?@\\[\\\\\\]^_`{|}~][A-Za-z]+|[^\r\n\\p{L}\\p{P}\\p{S}]?[\\p{L}\\p{M}]+| ?[\\p{P}\\p{S}]+[\r\n]*|\\s*[\r\n]+|\\s+(?!\\S)|\\s+",
  "\\s?\\p{L}+",
  "(\\[[^\\]]+]|Br?|Cl?|N|O|S|P|F|I|b|c|n|o|s|p|\\(|\\)|\\.|=|#|-|\\+|\\\\|\\/|:|~|@|\\?|>|\\*|\\$|\\%[0-9]{2}|[0-9])",
  "[\\(\\)\\[\\]\\{\\}]|([!\"\\#\\$%\\&'\\*\\+,\\-\\./:;<=>\\?\\\\\\^_`\\|\\~])\\1*",
  "[^\u0590-\u05ff\u0000-\u007f\u200c-\u203f\u20a0-\u20bf\u2200-\u22ff\u2150-\u218b\ufb00-\ufb4f]+",
  "\n",
  "[0-9][0-9][0-9]",
  "\\A\\p{Nd}{1,2}(?=\\p{Nd}{3}+\\z)",
  "\u201c(?=[ \\w])",
  "(http(s)?:\\/\\/.)?(www\\.)?[-a-zA-Z0-9@:%._\\+~#=]{2,256}\\.[a-z]{2,6}\\b([-a-zA-Z0-9@:%_\\+.~#?&//=]*)",
  "[\u4e00-\u9fa5\u0800-\u4e00\uac00-\ud7ff]+",
  "[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "[\\x00-\\x08\\x0b\\x0d-\\x1f\\x7f\\ufeff]",
  "\\\n(?=[ \\w])",
  " ",
  "\\((?=[ \\w])",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "www[a-zA-Z0-9_\\-?=%&/.~]+",
  "\u2039(?=[ \\w])",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?(?:\\p{L}\\p{M}*)+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "@(https?|ftp)://(-\\.)?([^\\s/?\\.#-]+\\.?)+(/[^\\s]*)?$@iS",
  "'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?+\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]++[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s+",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)| (?!\\h?\\p{N})\\h?(?:\\p{L}\\p{M}*)+|\\p{N}| ?[^\\s\\p{L}\\p{N}\\p{M}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "<\\|startoftext\\|>|<\\|endoftext\\|>|'s|'t|'re|'ve|'m|'ll|'d|[\\p{L}]+|[\\p{N}]|[^\\s\\p{L}\\p{N}]+",
  "\\s?[A-Za-z\u00b5\u00c0-\u00d6\u00d8-\u00f6\u00f8-\u01ba\u01bc-\u01bf\u01c4-\u0293\u0295-\u02af\u0370-\u0373\u0376\u0377\u037b-\u037d\u037f\u0386\u0388-\u038a\u038c\u038e-\u03a1\u03a3-\u03f5\u03f7-\u0481\u048a-\u052f\u0531-\u0556\u10a0-\u10c5\u13a0-\u13f5\u13f8-\u13fd\u1c90-\u1cba\u1cbd-\u1cbf\u1d00-\u1d2b\u1d6b-\u1d77\u1d79-\u1d9a\u1e00-\u1f15\u1f18-\u1f1d\u1f20-\u1f45\u1f48-\u1f4d\u1f50-\u1f57\u1f59\u1f5b\u1f5d\u1f5f-\u1f7d\u1f80-\u1fb4\u1fb6-\u1fbc\u1fbe\u1fc2-\u1fc4\u1fc6-\u1fcc\u1fd0-\u1fd3\u1fd6-\u1fdb\u1fe0-\u1fec\u1ff2-\u1ff4\u1ff6-\u1ffc\u2102\u2107\u210a-\u2113\u2115\u2119-\u211d\u2124\u2126\u2128\u212a-\u212d\u212f-\u2134\u2139\u213c-\u213f\u2145-\u2149\u214e\u2183\u2184\u2c00-\u2c7b\u2c7e-\u2ce4\u2ceb-\u2cee\u2cf2\u2cf3\ua640-\ua66d\ua680-\ua69b\ua722-\ua76f\ua771-\ua787\ua78b-\ua78e\uab70-\uabbf\ufb00-\ufb06\ufb13-\ufb17\uff21-\uff3a\uff41-\uff5a\ud801\udc00-\ud801\udc4f\ud801\udcb0-\ud801\udcd3\ud801\udcd8-\ud801\udcfb\ud803\udc80-\ud803\udcb2\ud803\udcc0-\ud803\udcf2\ud806\udca0-\ud806\udcdf\ud83a\udd00-\ud83a\udd43]+",
  "(?<=[\\u0980-\\u09ff]):",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "\u2019(?=[ \\w])",
  "(?i: ?(?:[jtmslndcy]['\u2019]|qu['\u2019]|jusqu['\u2019]|lorsqu['\u2019]|puisqu['\u2019]|quoiqu['\u2019]|aujourd['\u2019]hui)|['\u2019][stmd]|['\u2019][rv]e|['\u2019]ll)| ?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "\\s?\\p{P}+",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}+| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
  "\\p{Nd}{1,510}(?=(?>\\p{Nd}{510})*(?:\\P{Nd}|$))|\\G\\p{Nd}{510}",
  "http[s]?://[a-zA-Z0-9_\\-./~\\?=%&]+",
  "(?=http)[^\\s]+",
  "\n ",
  "(?<!\\n)^ ",
  "\\d",
  "[!\"\\#\\$%\\&'\\(\\)\\*\\+,\\-\\.:;=\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~]",
  " ?[^(\\s|[.,!?\u2026\u3002\uff0c\u3001\u0964\u06d4\u060c])]+",
  "\\s{2,}|[\\n\\r\\t]",
  "(\\W)?@usuario(\\W)",
  "<br [/](https://file+.vscode-resource.vscode-cdn.net/)>",
  "\u00ab(?=[ \\w])",
  "[\\w-]+@([\\w-]+\\.)+[\\w-]+",
  "\\[(?=[ \\w])",
  "\"(?=[ \\w])",
  "\\A\\p{Nd}{3}|\\G\\p{Nd}{3}",
  "[^ !\"$',.:;?A-Za-z\u00a1\u00ab\u00bb\u00bf\u00e6\u00e7\u00f0\u00f8\u0127\u014b\u0153\u01c0-\u01c3\u0250-\u0268\u026a-\u0276\u0278-\u027b\u027d\u027e\u0280-\u0284\u0288-\u0292\u0294\u0295\u0298\u0299\u029b-\u029d\u029f\u02a1\u02a2\u02a4\u02a7\u02b0-\u02b2\u02b4\u02b7\u02bc\u02c8\u02cc\u02d0\u02d1\u02de\u02e0\u02e4\u0329\u03b2\u03b8\u03c7\u1d7b\u2014\u201c\u201d\u2026\u2191-\u2193\u2197\u2198\u2c71]",
  "'(?:s|t|re|ve|ll|m|d)|\\b[A-Za-z](?:\\.[A-Za-z])+\\.?|\\d+\\.?\\d*[eE][+-]?\\d+|\\d+\\.\\d+|\\d+|==|!=|->|<=|>=|\\*\\*|//|[-+*/]=|[A-Za-z_][A-Za-z0-9_]*|\\w+|\\n+|[ \\t]+|[^\\s]",
  "://",
  "^",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s{512}(?!\\S)|\\s{256}(?!\\S)|\\s{128}(?!\\S)|\\s{64}(?!\\S)|\\s{32}(?!\\S)|\\s{16}(?!\\S)|\\s{8}(?!\\S)|\\s{4}(?!\\S)|\\s{1,2}(?!\\S)|\\s{1}",
  "<(?=[ \\w])",
  " {2,}",
  "[ \\u00A0]",
  "\\p{N}",
  "\\d{1,3}(?=(?:\\d{3})*\\b)",
  "\u201a(?=[ \\w])",
  "\\s?[!-/:-~\uff01-\uff0f\uff1a-\uff5e\u2018-\u201f\u3000-\u3002]+",
  "\\s+",
  "\\s+$",
  "(?<!\\n)^",
  "[\u4e00-\u9fa5\u3040-\u309f\u30a0-\u30ff]+",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1}| ?[^\\s\\p{L}\\p{N}\r\n]+|\\s*[\r\n]+|\\s+(?!\\S)|\\s+",
  "[a-zA-Z]+\\.com",
  "[\\r\\x00]",
  "(?=www)[^\\s]+",
  "\\x1F",
  "'s|'t|'re|'ve|'m|'ll|'d|[\\p{L}]+|[\\p{N}]|[^\\s\\p{L}\\p{N}]+",
  " ?[\uff01-\uff0f\uff1a-\uff5e\u2018-\u201f\u3000-\u3002\u300a\u300b\u300c\u300d\u3010\u3011]+",
  "^ ",
  "(\\[UNK\\]|[\\s\\S])",
  "\u2018(?=[ \\w])",
  "[UZOB]",
  "\\s+|\\w+|[^\\w\\s]+",
  "\u2581",
  "[\u2018\u2019\u02bc\u201b\uff07]",
  "'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s",
  "\u2013(?=[ \\w])",
  "\\s",
  "\\S+@\\S+",
  ".",
  "[^$;:,.!?\u2014\u2026\"()\u201c\u201d \u0303\u02a3\u02a5\u02a6\u02a8\u1d5d\uab67AIOQSTWY\u1d4aabcdefhijklmnopqrstuvwxyz\u0251\u0250\u0252\u00e6\u03b2\u0254\u0255\u00e7\u0256\u00f0\u02a4\u0259\u025a\u025b\u025c\u025f\u0261\u0265\u0268\u026a\u029d\u026f\u0270\u014b\u0273\u0272\u0274\u00f8\u0278\u03b8\u0153\u0279\u027e\u027b\u0281\u027d\u0282\u0283\u0288\u02a7\u028a\u028b\u028c\u0263\u0264\u03c7\u028e\u0292\u0294\u02c8\u02cc\u02d0\u02b0\u02b2\u2193\u2192\u2197\u2198\u1d7b]",
  "(\\W)url(\\W)",
  "'(?=[ \\w])",
  "(?:[\\u4E00-\\u9FFF\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\uf900-\\uFAFF\\uFF65-\\uFF9F\\u2F00-\\u2FDF]+|[\\u0E00-\\u0E7F]+|[\\u0E80-\\u0EFF]+|[\\u1780-\\u17FF]+|[\\u1000-\\u109F\\uAA60-\\uAA7F\\uA9E0-\\uA9FF]+|[\\uAC00-\\uD7AF\\u1100-\\u11FF]+)",
  "[\\n\\r\\t]",
  "\\\t(?=[ \\w])",
  "\u2014(?=[ \\w])",
  "[0-9]",
  "</?[^>]+>",
  "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?(?:\\p{L}\\p{M}*(?: \\p{L}\\p{M}*)*)+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]?|\\s*[\\r\\n]|\\s+(?!\\S)|\\s+"
]

maybe check if these are all parsed correctly (and match python implementation)

[xenova]

and tbh, this could work well to make PROBLEMATIC_REGEX_MAP obsolete... it's quite hacky (and a lot of hard-coding for specific tokenizer regex patterns) 😅

[xenova]

Snuck a full Fable 5 review in before it was shut down 😆 comment attached below

---

Correctness fixes (all 8, in src/utils/core.ts)

1. Line anchors: dropped the m flag; ^/$ are now rewritten to \n-only lookarounds, matching oniguruma (JS multiline would also anchor at \r/U+2028/U+2029).
2. Stacked quantifiers: X{n}+ / X{n,m}+ now wrap any atom as (?:X{n})+ — literal, class, or group, not just \p{..} — via atom-tracking in the new scanner. {,n} is also normalized to {0,n}.
3. (?i:...) folding: ranges fold correctly ([a-f] → [a-fA-F]), class-initial hyphens no longer suppress folding ([-y] folds y), group syntax like (? is copied verbatim, nested (?i: groups expand recursively, and \p{...} is consumed atomically (this last one fixed a latent crash where (?i:\w) produced \p{lL}).
4. \h/\H: now hex-digit classes (oniguruma/Ruby semantics), not horizontal whitespace.
5. Escape parity: the scanner consumes escape pairs atomically, so \ + newline can never be corrupted; identity-escape stripping is now general (any punctuation JS+u rejects) instead of a hardcoded TODO list.
6. Word characters: probing revealed onig uses two sets — standalone \w/\W/\b/\B/\p{Word} = Alphabetic|M|Nd|Pc plus Latin-1 ctype extras (²³¹¼½¾), while in-class shorthands use the pure property set. Both are now exact, including \b boundaries.
7. \G: the misleading lookbehind approximation is gone; \G is dropped (its continuation semantics are inexpressible) and the divergence is pinned by knownDivergence tests instead of hidden.
8. \Z: no longer accepts \r\n (onig allows only a single final \n).

Bonus fixes that fell out of the probes: \s/\S → \p{White_Space} (JS \s wrongly adds U+FEFF and misses U+0085), positive classes with \W/\H are lifted into alternations ([\W_] → (?:[_]|[^…word…])), POSIX classes ([[:alpha:]]), \x{...} → \u{...}, \p{Word}, \a/\e, and literal {/}/stray ] (onig-legal, previously fatal JS syntax errors).

---

[xenova]

did you notice any download corruptions? 👀

[nico-martin]

Yes, the npm run test in github actions often fails because of multiple invalid responses.

FAIL tests/memoryLeaks.test.ts (212 MB heap size)
  Memory leak tests
    ✕ should not leak memory (6 ms)

  ● Memory leak tests › should not leak memory

    Failed to fetch https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/resolve/main/tokenizer_config.json: 429 Too Many Requests

      35 |     const response = await fetch(url);
      36 |     if (!response.ok) {
    > 37 |       throw new Error(`Failed to fetch ${url}: ${response.status} ${response.statusText}`);
         |             ^
      38 |     }
      39 |     return await response.json();
      40 |   };

One example:
https://github.com/huggingface/tokenizers.js/actions/runs/27579303033/job/81583399599

So I tried multiple things to make the tests more robust. But I wasn't able actually find a robust solution..

[xenova]

back to green :) I'd say final reviews are in order now.

[xenova]

I think it's ready! @nico-martin feel free to do a final review and merge when you're happy.