Commit d834f01
Put the segmentation fork under a drawn lexicon at all
script_segment's multi-match fork -- two vocabulary entries match the
same token, longest-first chooses, and the stage reports the reading it
passed over -- was fuzzed by nothing. The cause is structural rather
than statistical: the pool's only hangul entries were 김 and 남궁, and
no entry was a proper PREFIX of another, so `matches` could never hold
more than one length whatever the draw did.
남 is a shipped Korean surname and a prefix of 남궁, so a lexicon drawn
with both makes 남궁민준 match twice and sends the parse down the
ambiguity-emission-plus-index-remap path.
Instrumented rather than assumed, because reachable is not the same as
reached: both entries have to land in the same drawn `surnames` (0.8%
of draws), so the fork fires about once in 900 examples -- 42 over
36000 measured -- and the committed derandomize=True seed does not
reach it. A randomized run is what sees it. Said in the note so the
next reader does not conclude from one green run that the entry is
doing nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>1 parent 2af1340 commit d834f01
1 file changed
Lines changed: 11 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
193 | | - | |
| 193 | + | |
194 | 194 | | |
195 | 195 | | |
196 | | - | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
197 | 205 | | |
198 | 206 | | |
199 | 207 | | |
| |||
204 | 212 | | |
205 | 213 | | |
206 | 214 | | |
207 | | - | |
| 215 | + | |
208 | 216 | | |
209 | 217 | | |
210 | 218 | | |
| |||
0 commit comments