Skip to content

Commit 2777d68

Browse files
apocryphxclaude
andcommitted
expectedDivergences: reclassify 2 T5 entries from pending-investigation to huggingface#356
Verified locally on a combined branch (huggingface#354huggingface#355huggingface#356 ⊕ this PR): all three T5 divergences are fixed by huggingface#356's scalar-iteration switch. Two were originally listed as `fixedBy: 0` ("Unigram TM trademark / VS-16 segmentation" and "Unigram ZWJ-after-text edge"); the combined-branch run fired cleanup hints for both, alongside the emoji-keycap-and-flags hint. Root cause is the same as the keycap case: a vocab-relevant scalar (TM glyph U+2122, ZWJ U+200D) is hidden inside a grapheme cluster that the old `Character`-based Unigram lattice never decomposed. Moving the iteration unit to `Unicode.Scalar` exposes it. `expectedDivergences` count drops from 10 pending-investigation entries to 8 (5 TinyLlama Metaspace whitespace + 3 Qwen2.5 BPE merge-ordering). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 00cb4cf commit 2777d68

1 file changed

Lines changed: 6 additions & 9 deletions

File tree

Tests/TokenizersTests/MultilingualConformanceTests.swift

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -142,10 +142,13 @@ private let expectedDivergences: Set<ExpectedDivergence> = [
142142
//
143143
// Bug 3 (#356): `UnigramTokenizer` + `TokenLattice` switch from `Character`
144144
// (grapheme cluster) iteration to `Unicode.Scalar`. Affects T5-style Unigram
145-
// on inputs where a vocab-relevant scalar is hidden inside a grapheme cluster
146-
// (digit inside `1️⃣`).
145+
// wherever a vocab-relevant scalar is hidden inside a grapheme cluster: the
146+
// digit inside `1️⃣`; the `TM` (U+2122) + VS-16 (U+FE0F) cluster in `™️`;
147+
// the ZWJ (U+200D) sitting between two text characters in escape sequences.
147148
//
148-
.init(modelId: "google-t5/t5-small", inputId: "emoji-keycap-and-flags", fixedBy: 356),
149+
.init(modelId: "google-t5/t5-small", inputId: "emoji-keycap-and-flags", fixedBy: 356),
150+
.init(modelId: "google-t5/t5-small", inputId: "multiscript-legacy-symbols", fixedBy: 356),
151+
.init(modelId: "google-t5/t5-small", inputId: "escape-sequences-with-zwj", fixedBy: 356),
149152

150153
//
151154
// Bug 4 (#355): `BPETokenizer.bpe(token:)` switches from grapheme-cluster
@@ -185,12 +188,6 @@ private let expectedDivergences: Set<ExpectedDivergence> = [
185188
.init(modelId: "TinyLlama/TinyLlama-1.1B-Chat-v1.0", inputId: "whitespace-runs", fixedBy: 0, note: "Metaspace leading-whitespace runs"),
186189
.init(modelId: "TinyLlama/TinyLlama-1.1B-Chat-v1.0", inputId: "whitespace-trailing-tabs", fixedBy: 0, note: "Metaspace leading-whitespace runs"),
187190

188-
// T5 Unigram divergences not addressed by #356. Both surface different
189-
// segmentation around ZWJ + VS-16 + combining-keycap shapes. Likely a
190-
// second-order interaction in the Unigram lattice's UNK fusing.
191-
.init(modelId: "google-t5/t5-small", inputId: "multiscript-legacy-symbols", fixedBy: 0, note: "Unigram TM trademark / VS-16 segmentation"),
192-
.init(modelId: "google-t5/t5-small", inputId: "escape-sequences-with-zwj", fixedBy: 0, note: "Unigram ZWJ-after-text edge"),
193-
194191
// Qwen2.5 byte-level BPE picks a different merge ordering on Thai
195192
// (and Thai-inside-multiscript) than HF Python. Byte-level encoding
196193
// means there are no combining-mark traps; this is a merge-priority

0 commit comments

Comments
 (0)