Skip to content

Commit d834f01

Browse files
derek73claude
andcommitted
Put the segmentation fork under a drawn lexicon at all
script_segment's multi-match fork -- two vocabulary entries match the same token, longest-first chooses, and the stage reports the reading it passed over -- was fuzzed by nothing. The cause is structural rather than statistical: the pool's only hangul entries were 김 and 남궁, and no entry was a proper PREFIX of another, so `matches` could never hold more than one length whatever the draw did. 남 is a shipped Korean surname and a prefix of 남궁, so a lexicon drawn with both makes 남궁민준 match twice and sends the parse down the ambiguity-emission-plus-index-remap path. Instrumented rather than assumed, because reachable is not the same as reached: both entries have to land in the same drawn `surnames` (0.8% of draws), so the fork fires about once in 900 examples -- 42 over 36000 measured -- and the committed derandomize=True seed does not reach it. A randomized run is what sees it. Said in the note so the next reader does not conclude from one green run that the entry is doing nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 2af1340 commit d834f01

1 file changed

Lines changed: 11 additions & 3 deletions

File tree

tests/v2/test_properties.py

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -190,10 +190,18 @@ def test_particle_fork_is_never_double_reported(text: str) -> None:
190190
# would be a dead entry that trips Lexicon's multi-word warning.
191191
# The CJK tail (#271) is what lets a drawn `surnames` set activate
192192
# script_segment at all: hangul is the script segmented by default, so
193-
# "김"/"남궁" are what make the stage fire (see _names_using, which
193+
# "김"/"남궁"/"남" are what make the stage fire (see _names_using, which
194194
# supplies the unspaced token to fire it ON -- and, since the drawn
195195
# policy picks its own segment_scripts, only when that policy
196-
# activates hangul too). The Han rows ride along for script_orders,
196+
# activates hangul too). "남" is there for the stage's multi-match
197+
# FORK, which nothing else in this pool can reach: it is a proper
198+
# PREFIX of "남궁", so a lexicon drawn with both makes "남궁민준" match
199+
# twice, and longest-first then has to choose and report the reading
200+
# it passed over. Reachable is all it is -- both entries have to land
201+
# in the same drawn `surnames`, 0.8% of draws, so the fork fires
202+
# roughly once in 900 examples (42 over 36000 measured) and the
203+
# committed 250-example seed does not reach it at all; a randomized
204+
# run is what sees it. The Han rows ride along for script_orders,
197205
# and are not inert here either -- `_policies` draws segment_scripts
198206
# freely, so HAN is activated in 37.7% of drawn policies (measured
199207
# over 20000 draws), and an activated Han token STANDING EARLIER takes
@@ -204,7 +212,7 @@ def test_particle_fork_is_never_double_reported(text: str) -> None:
204212
"van", "de", "la", "bin", "abdul", "abu", "dr", "sir", "prof",
205213
"md", "jr", "iii", "esq", "ma", "do", "and", "y", "née", "geb",
206214
"a", "b", "ph.d", "عبد", "фон", "μεγα",
207-
"김", "남궁", "毛", "欧阳",
215+
"김", "남궁", "남", "毛", "欧阳",
208216
])
209217

210218
# given_name_titles is the one field matched as a space-joined run, so

0 commit comments

Comments
 (0)