Pin the interpunct behavior: case rows, corpus, differential guards (#298)

derek73 · claude · derek73 · commit 96832ce8a69c · 2026-07-30T10:30:25.000-07:00
Five case rows land in tests/v2/cases.py, in the table's canonical given/family/middle spelling: - zh_interpunct_transcription_source_order: 威廉·莎士比亚 divides on the B7 and keeps source order (fix(#298)). - zh_interpunct_nakaguro_typed_stays_roster: the SAME transcription typed with the Japanese nakaguro reads as a 姓・名 roster pair, family-first (fix(#272)). A limitation row, chosen not accidental: the design is codepoint-scoped (spec 2026-07-30 decision 5), so cross-convention input reads by the convention of the codepoint it was typed with -- only the Chinese B7 marks a transcription. - ja_interpunct_b7_katakana: sloppy-IME B7 between katakana divides like the nakaguro; order was already positional (fix(#298)). - latin_punt_volat_is_name_interior: Gal·la Marcet, the flank guard's reason -- Latin-flanked B7 is interior, never a divider (parity). - zh_interpunct_suppresses_segmentation: 马丁·路德·金 under zh -- the dot gates segmentation off, 马丁 stays whole (fix(#298)). The differential CJK rule's character class gains U+00B7, the second sanctioned extra beside U+FF65: both dots change parses without being classified by any Script (B7 doubles as Catalan's punt volat). test_differential_cjk_rule_matches_the_script_ranges pins the pair as sanctioned_extras; the compound rule's lookahead copies classified spans only and is untouched. corpus_cjk.jsonl regenerates 31 -> 35: the four CJK-bearing rows enter (Gal·la Marcet correctly does not -- no classified codepoint). Harness hand-run: 689 names, 48 intentional diffs, zero unexplained; the CJK rule classifies 27 including all the B7 names. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
diff --git a/tests/v2/cases.py b/tests/v2/cases.py
@@ -651,4 +651,41 @@ def __post_init__(self) -> None:
          notes="delimited content tokenizes under the same separator "
                "rules: the dot renders back as a space in the nickname "
                "join -- a decision, not an accident"),
+    Case("zh_interpunct_transcription_source_order", "威廉·莎士比亚",
+         {"given": "威廉", "family": "莎士比亚"},
+         classification="fix(#298)",
+         notes="间隔号-divided Han is a transcribed foreign name and "
+               "keeps source order -- the B7 is the transcription "
+               "marker, playing the role pure katakana plays in the "
+               "kana license; it divides only between classified "
+               "characters"),
+    Case("zh_interpunct_nakaguro_typed_stays_roster", "威廉・莎士比亚",
+         {"given": "莎士比亚", "family": "威廉"},
+         classification="fix(#272)",
+         notes="the SAME transcription typed with the Japanese "
+               "nakaguro reads as the dot's own typography says -- a "
+               "姓・名 roster pair, family-first (#272) -- because the "
+               "nakaguro records nothing (spec 2026-07-30 decision 5: "
+               "codepoint-scoped; only the Chinese B7 marks a "
+               "transcription). A limitation row: chosen, not "
+               "accidental -- cross-convention input reads by the "
+               "convention of the codepoint it was typed with"),
+    Case("ja_interpunct_b7_katakana", "マイケル·ジャクソン",
+         {"given": "マイケル", "family": "ジャクソン"},
+         classification="fix(#298)",
+         notes="sloppy-IME B7 between katakana divides like the "
+               "nakaguro; katakana was never licensed, so order was "
+               "already positional -- the split is the fix"),
+    Case("latin_punt_volat_is_name_interior", "Gal·la Marcet",
+         {"given": "Gal·la", "family": "Marcet"},
+         notes="the Catalan punt volat: U+00B7 with Latin neighbors "
+               "is interior to the name, never a divider -- the flank "
+               "guard's whole reason"),
+    Case("zh_interpunct_suppresses_segmentation", "马丁·路德·金",
+         {"given": "马丁", "middle": "路德", "family": "金"},
+         locale="zh",
+         classification="fix(#298)",
+         notes="马 is a listed zh surname, but a 间隔号-divided name "
+               "is a transcription: the dot gates segmentation off, "
+               "so 马丁 stays whole and the positional read stands"),
 )
diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py
@@ -192,16 +192,20 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None:
     it, see the comment there -- so the comparison runs over the BMP
     spans only.
 
-    The rule is also WIDER than the table by exactly one span, which
+    The rule is also WIDER than the table by exactly two spans, which
     the equality has to know about or it would just fail forever. The
     halfwidth middle dot U+FF65 changes parses without being
     classified as anything: tokenize separates on it, so a halfwidth
     transcription splits where 1.4 kept one token, while halfwidth
-    kana stays out of _SCRIPT_RANGES on purpose. Naming that span here
-    rather than relaxing the comparison to a subset check is what
-    keeps the pin honest in both directions: a THIRD source of
-    divergence still fails, and the one sanctioned difference has to
-    be written down to exist.
+    kana stays out of _SCRIPT_RANGES on purpose. U+00B7 -- the
+    context-sensitive 间隔号 (#298) -- is the same situation: it
+    divides between classified characters and pins source order,
+    changing parses, yet doubles as Catalan's punt volat and so can
+    never itself be classified. Naming those spans here rather than
+    relaxing the comparison to a subset check is what keeps the pin
+    honest in both directions: an unsanctioned source of divergence
+    still fails, and each sanctioned difference has to be written
+    down to exist.
     """
     toml_path = (Path(__file__).parents[2] / "tools" / "differential"
                  / "expected_changes.toml")
@@ -222,16 +226,18 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None:
         (int(lo, 16), int(hi, 16))
         for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})",
                                  matched[0]["name_regex"])}
-    # the one span the rule carries that no Script claims (see above)
-    halfwidth_dot = (0xFF65, 0xFF65)
-    assert not any(lo <= halfwidth_dot[0] <= hi
-                   for spans in _policy._SCRIPT_RANGES.values()
-                   for lo, hi in spans), (
-        "U+FF65 is classified now; drop it from the sanctioned extras")
+    # the two spans the rule carries that no Script claims (see above)
+    sanctioned_extras = {(0xFF65, 0xFF65), (0xB7, 0xB7)}
+    for extra, _ in sorted(sanctioned_extras):
+        assert not any(lo <= extra <= hi
+                       for spans in _policy._SCRIPT_RANGES.values()
+                       for lo, hi in spans), (
+            f"U+{extra:04X} is classified now; drop it from the "
+            "sanctioned extras")
     expected = {span
                 for spans in _policy._SCRIPT_RANGES.values()
                 for span in spans
-                if span[1] <= 0xFFFF} | {halfwidth_dot}
+                if span[1] <= 0xFFFF} | sanctioned_extras
     assert declared == expected, (
         f"{toml_path.name}'s CJK name_regex declares {sorted(declared)}; "
         f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}")
diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl
@@ -6,10 +6,13 @@
 "みなみ"
 "マイケル"
 "マイケル ジャクソン"
+"マイケル·ジャクソン"
 "マイケル・ジャクソン"
 "佐々木 太郎"
 "司马相如"
 "夏侯惇"
+"威廉·莎士比亚"
+"威廉・莎士比亚"
 "山田 エミ"
 "山田 太郎 (マイケル・ジャクソン)"
 "山田「タロ」太郎"
@@ -20,6 +23,7 @@
 "田中『ハナ』花子"
 "諸葛亮"
 "阿明"
+"马丁·路德·金"
 "高橋 みなみ"
 "高橋みなみ"
 "高橋・一郎"
diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml
@@ -32,17 +32,22 @@ issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentati
 # supplementary-plane kana is hentaigana and other archaic forms no
 # modern name uses, so _SCRIPT_RANGES does not list it either.
 #
-# U+FF65 is the one span here that _SCRIPT_RANGES does NOT have, and
-# it is deliberate: the halfwidth middle dot separates tokens like its
-# fullwidth twin, so 'ﾏｲｹﾙ･ｼﾞｬｸｿﾝ' splits where 1.4 left one token,
-# but halfwidth kana is excluded from CLASSIFICATION on purpose (a
-# separate normalization problem). The rest of the halfwidth block
-# (U+FF66-U+FF9F) is left out for the usual tightness reason, and it
-# was measured rather than assumed: a dotless halfwidth name such as
-# 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on both sides, so covering the
-# block would pre-excuse a future regression on a shape that is
-# parity today. The dot is the whole mechanism; the dot is the whole
-# span.
+# U+FF65 and U+00B7 are the two spans here that _SCRIPT_RANGES does
+# NOT have, and both are deliberate. The halfwidth middle dot
+# separates tokens like its fullwidth twin, so 'ﾏｲｹﾙ･ｼﾞｬｸｿﾝ' splits
+# where 1.4 left one token, but halfwidth kana is excluded from
+# CLASSIFICATION on purpose (a separate normalization problem). The
+# rest of the halfwidth block (U+FF66-U+FF9F) is left out for the
+# usual tightness reason, and it was measured rather than assumed: a
+# dotless halfwidth name such as 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on
+# both sides, so covering the block would pre-excuse a future
+# regression on a shape that is parity today. U+00B7, the
+# context-sensitive 间隔号 (#298), is FF65's situation again: it
+# divides only between classified-script characters and pins source
+# order, so '威廉·莎士比亚' splits where 1.4 kept one token -- but a
+# codepoint that doubles as Catalan's punt volat can never itself be
+# classified as a CJK script. The dots are the whole mechanism; the
+# dots are the whole spans.
 #
 # corpus_cjk.jsonl (#295) exists so this rule fires in every real
 # run: build_cjk_corpus.py harvests every CJK-bearing text in
@@ -52,7 +57,7 @@ issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentati
 # two corpora have by construction: v1's banks had no reason to test
 # CJK, and build_issues_corpus.py requires an internal space, which
 # unspaced names never have.
-name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65]"
+name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65\\u00B7-\\u00B7]"
 fields = ["first", "middle", "last"]
 
 [[change]]
