Commit 2777d68
expectedDivergences: reclassify 2 T5 entries from pending-investigation to huggingface#356
Verified locally on a combined branch (huggingface#354 ⊕ huggingface#355 ⊕ huggingface#356 ⊕ this PR): all
three T5 divergences are fixed by huggingface#356's scalar-iteration switch. Two
were originally listed as `fixedBy: 0` ("Unigram TM trademark / VS-16
segmentation" and "Unigram ZWJ-after-text edge"); the combined-branch run
fired cleanup hints for both, alongside the emoji-keycap-and-flags hint.
Root cause is the same as the keycap case: a vocab-relevant scalar
(TM glyph U+2122, ZWJ U+200D) is hidden inside a grapheme cluster that
the old `Character`-based Unigram lattice never decomposed. Moving the
iteration unit to `Unicode.Scalar` exposes it.
`expectedDivergences` count drops from 10 pending-investigation entries
to 8 (5 TinyLlama Metaspace whitespace + 3 Qwen2.5 BPE merge-ordering).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 00cb4cf commit 2777d68
1 file changed
Lines changed: 6 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
145 | | - | |
146 | | - | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
147 | 148 | | |
148 | | - | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
149 | 152 | | |
150 | 153 | | |
151 | 154 | | |
| |||
185 | 188 | | |
186 | 189 | | |
187 | 190 | | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | 191 | | |
195 | 192 | | |
196 | 193 | | |
| |||
0 commit comments