Classify the #272 kana fixes in the differential harness

The CJK rule's character class gains the two kana blocks, and its
label names both issues: #272's kana license and nakaguro separator
move name pieces between the same first/middle/last fields on the same
native-script input, so this is one diff shape rather than two, and
splitting the rule by issue would buy no tightness.

The sync pin widens with it, and stops scoping itself by issue. That
scoping rested on a premise #272 falsified -- the kana members were
classification-only, keyed by no default -- and comparing against the
whole _SCRIPT_RANGES table is the stronger promise anyway: a new
script now fails this test until someone decides in writing whether
the rule should cover it, which the added set assertion says out loud.

Both corpora still contain no CJK name at all, so the rule classifies
nothing in a real run; the harness passes 654 names with 18 classified
diffs and none unexplained. That blind spot is #295's subject, cited
in the rule's comment, and it is why the behavioral guarantee lives in
the case table rather than here. corpus_issues.jsonl is deliberately
not regenerated: a dry run adds three Latin strings from #296 and, as
#295 predicts, no CJK -- unrelated tracker content that belongs in its
own reviewable act.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: derek73

tests/v2/test_regex_sync.py

@@ -164,49 +164,56 @@ def test_comma_char_matches_the_pipeline_comma_set() -> None:
 
 
 def test_differential_cjk_rule_matches_the_script_ranges() -> None:
-    """The #271 rule in tools/differential/expected_changes.toml hand-
-    copies the CJK spans from _vocab._SCRIPT_RANGES into a character
+    """The CJK rule in tools/differential/expected_changes.toml hand-
+    copies the script spans from _vocab._SCRIPT_RANGES into a character
     class. A TOML file cannot import the constant, so this is the one
     copy with no possible alternative -- and the one whose divergence
     is quietest, because the harness is run by hand rather than in CI.
 
     Both failure directions matter, which is why this compares sets
     rather than checking coverage. A span MISSING from the class turns
-    an intended #271 change into an UNEXPLAINED diff (a release
+    an intended #271/#272 change into an UNEXPLAINED diff (a release
     blocker for the wrong reason); a span that should not be there
     silently classifies a real regression as intended, which is the
     failure the whole harness exists to prevent.
 
-    Han's astral block is out of scope on both sides. The rule omits
-    it deliberately -- no corpus name reaches it, see the comment
-    there -- so the comparison runs over the BMP spans only, and a new
-    BMP script added to _SCRIPT_RANGES for the SAME #271 behavior
-    (family-first order / hangul segmentation) still fails here until
-    the rule covers it.
-
-    #272 added HIRAGANA/KATAKANA to _SCRIPT_RANGES for an unrelated
-    reason (kana classification for effective_script's license, not
-    the order-flip or segmentation #271's diff rule explains -- no
-    default keys on either kana member), so the comparison is scoped
-    to the #271 scripts by name rather than every table entry; a
-    future script added for #271's own reason still must extend both
-    sides, same as before.
+    Every table entry is in scope. The rule covered HAN and HANGUL
+    alone while the kana members existed only for classification, but
+    #272 gave HIRAGANA a default order entry and made the kana blocks
+    part of the same first/middle/last diff shape the rule explains,
+    so scoping it by issue no longer draws a real line. Comparing
+    against the whole table is also the stronger promise: a script
+    added to _SCRIPT_RANGES for ANY reason fails here until someone
+    decides, in writing, whether the rule should cover it.
+
+    Han's astral block is the single exception, out of scope on both
+    sides. The rule omits it deliberately -- no corpus name reaches
+    it, see the comment there -- so the comparison runs over the BMP
+    spans only.
     """
     toml_path = (Path(__file__).parents[2] / "tools" / "differential"
                  / "expected_changes.toml")
     rules = tomllib.loads(toml_path.read_text())["change"]
-    matched = [r for r in rules if "#271" in r["issue"]]
+    matched = [r for r in rules
+               if "#271" in r["issue"] or "#272" in r["issue"]]
     assert len(matched) == 1, (
-        f"expected exactly one #271 rule in {toml_path.name}, "
+        f"expected exactly one CJK rule in {toml_path.name}, "
         f"found {len(matched)}")
+    # A new table entry must force an explicit decision rather than
+    # quietly widening (or failing to widen) the rule above.
+    assert set(_vocab._SCRIPT_RANGES) == {
+        Script.HAN, Script.HANGUL, Script.HIRAGANA, Script.KATAKANA}, (
+        "a Script joined _SCRIPT_RANGES: decide whether the "
+        f"differential rule in {toml_path.name} should cover it, then "
+        "update this assertion")
     declared = {
         (int(lo, 16), int(hi, 16))
         for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})",
                                  matched[0]["name_regex"])}
     expected = {span
-                for script in (Script.HAN, Script.HANGUL)
-                for span in _vocab._SCRIPT_RANGES[script]
+                for spans in _vocab._SCRIPT_RANGES.values()
+                for span in spans
                 if span[1] <= 0xFFFF}
     assert declared == expected, (
-        f"{toml_path.name}'s #271 name_regex declares {sorted(declared)}; "
+        f"{toml_path.name}'s CJK name_regex declares {sorted(declared)}; "
         f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}")

tools/differential/expected_changes.toml

@@ -7,25 +7,40 @@
 # ahead of `fields`-only ones before matching.
 
 [[change]]
-issue = "fix(#271) native-script CJK: family-first order + hangul segmentation"
+issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentation, the kana license and the nakaguro"
 # '毛 泽东', '김민준': script_orders flips first/last for a name written
 # wholly in Han or Hangul, and the Korean surnames that now ship as
 # default vocabulary additionally split an unspaced hangul token into
-# last + first. Seeded from the fix(#271) rows in tests/v2/cases.py.
+# last + first. '高橋 みなみ', 'マイケル・ジャクソン': #272 extends the
+# same order table to kana-bearing Japanese names and makes the
+# katakana middle dot a token separator, which moves pieces between the
+# same three fields. Seeded from the fix(#271) and fix(#272) rows in
+# tests/v2/cases.py.
+#
+# One rule, not two, because it is one diff shape: both issues move
+# name pieces between first/middle/last on native-script CJK input, and
+# splitting the class by issue would need a rule per script with no
+# gain in tightness (the fields list is already the narrow half).
 #
 # Scoped by script, which is exactly the scope of the behavior --
-# neither change can touch a name with no CJK character in it. Han's
+# nothing here can touch a name with no CJK character in it. Han's
 # astral block (U+20000-U+323AF) is deliberately left out of the class:
 # no name in either corpus reaches it, and a rule should be no wider
 # than the diffs it has to explain. Extend it -- and the sync pin in
 # tests/v2/test_regex_sync.py, which fails if the two disagree -- if
-# one ever does.
+# one ever does. The kana spans are BMP-only for a different reason:
+# supplementary-plane kana is hentaigana and other archaic forms no
+# modern name uses, so _SCRIPT_RANGES does not list it either.
 #
 # Expected to match nothing against the current corpora, which contain
-# no CJK at all: build_issues_corpus.py requires an internal space, and
-# unspaced names are the shape this classifies. Kept for the reason the
-# suffix-delimiter rule below is kept -- ready the moment a name lands.
-name_regex = "[\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3]"
+# no CJK at all -- build_issues_corpus.py requires an internal space,
+# and unspaced names are the shape this classifies. That blind spot is
+# tracked as #272's and #271's shared gap in issue #295, which also
+# records the provenance decision a fix needs; until it is closed the
+# behavioral guarantee lives in the fix(#271)/fix(#272) rows of
+# tests/v2/cases.py, and this rule is kept ready for the moment a CJK
+# name lands in a corpus.
+name_regex = "[\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3]"
 fields = ["first", "middle", "last"]
 
 [[change]]