Simplify: single-source the strip set, auto-discover span pins, trim accretion

The strip-policy set is now defined ONCE (_stripped, shared by
_ignorable and _flank), making the flank-transparency invariant
structural: a new strip class added to the policy stays transparent
to the interpunct guard by construction instead of silently
re-opening the RTL-quote bug. _flank takes a range and the policy
(three params instead of five; the bound arithmetic lives at the
call sites where the region is in scope), and the right flank is now
walked only after the left one classifies -- half the guard work on
every Catalan punt-volat name. The dead 'not is_separator' conjunct
on the interpunct test is gone (unfalsifiable: B7 is no comma, no
space, and _ignorable cannot claim it).

The differential span pins single-source through _SANCTIONED_EXTRAS,
_declared_spans and _expected_bmp_spans, and the compound-rule pin
auto-discovers every span-bearing rule that intersects the table
instead of reading a hand-maintained slug roster whose own docstring
admitted its failure mode ('or its span copy goes unpinned') -- a
third compound rule is now pinned by existing.

Review-round comment accretion trimmed to single homes: the nakaguro
roster doctrine lives at _NAME_DOT_SEPARATORS (with the 高橋・一郎
example), the placement rationale at _INTERPUNCT, the suppression
rationale in _effective_order's docstring. Two duplicate tests
deleted: test_undotted_han_still_reads_family_first was byte-identical
to test_script_order_applies_when_every_piece_is_one_script, and the
assign-level nakaguro roster test restated the pre-existing
ja_nakaguro_han_takes_the_han_order case row plus the tokenize-level
non-recording pin.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: derek73

nameparser/_pipeline/_assign.py

@@ -89,7 +89,9 @@ def _effective_order(policy: Policy,
     order governs the positional read; anything else -- Latin, mixed
     scripts, no entry -- falls back to name_order. A 间隔号-divided
     name (`dot_divided`, #298) suppresses the whole lookup first: the
-    dot marks a transcription, which keeps name_order. Piece-level, after
+    dot marks a transcription -- playing the role pure katakana plays
+    in the kana license, orthography naming the convention -- so the
+    license yields to name_order. Piece-level, after
     title/suffix peeling: 'Dr. 毛泽东' is a wholly-Han NAME under a
     Latin title. Kana-licensed tokens (高橋みなみ, #272) resolve to
     HIRAGANA the same way a wholly-Han or wholly-Hangul token resolves
@@ -107,13 +109,8 @@ def _effective_order(policy: Policy,
     resolves the SCRIPT for a single token. This function calls that
     one per token below.
     """
-    # #298: a 间隔号-divided name is a transcription -- the B7 plays
-    # for Han exactly the role pure katakana plays in the kana license
-    # (orthography naming the convention), so the family-first license
-    # yields to the positional read. Checked before the script_orders
-    # lookup: the suppression outranks the license. Codepoint-scoped:
-    # only U+00B7 records (spec 2026-07-30 decision 5) -- the nakaguro
-    # is roster typography whose pieces the license reads correctly.
+    # #298 transcription marker -- see the docstring; codepoint-scoped
+    # (only U+00B7 records, spec 2026-07-30 decision 5)
     if dot_divided:
         return policy.name_order
     if not policy.script_orders:

nameparser/_pipeline/_tokenize.py

@@ -13,10 +13,8 @@
 separators and never enter a token, so spans always index the
 original exactly as given. Whitespace and the name-dot are
 unconditional; emoji/bidi stripping alone is policy-gated
-(Policy.strip_emoji/strip_bidi). The Chinese interpunct U+00B7 is the
-one context-sensitive separator: it divides (and records) only
-between classified-script neighbors, so its rule lives in
-_tokenize_region, which has the index _ignorable lacks.
+(Policy.strip_emoji/strip_bidi). The Chinese interpunct U+00B7 is
+context-sensitive -- see _INTERPUNCT below.
 
 v1's squash_emoji/squash_bidi REMOVED the char and joined neighbors
 ('A\U0001f600B' -> 'AB'); here an ignorable char is a SEPARATOR
@@ -32,7 +30,7 @@
 from nameparser._pipeline._state import (
     COMMA_CHARS, ParseState, WorkToken,
 )
-from nameparser._policy import _SCRIPT_RANGES, _script_matcher
+from nameparser._policy import Policy, _SCRIPT_RANGES, _script_matcher
 from nameparser._types import Role, Span
 
 # Ported from v1 (nameparser/config/regexes.py, "emoji" and "bidi") --
@@ -49,9 +47,11 @@
 # The katakana middle dot and its halfwidth twin divide the parts of
 # a foreign name transcribed into katakana (マイケル・ジャクソン) --
 # native names never contain them, so they separate unconditionally,
-# like whitespace (amendment 2026-07-29 section 1b). The Chinese
-# interpunct U+00B7 is context-sensitive instead -- see _INTERPUNCT
-# below.
+# like whitespace (amendment 2026-07-29 section 1b). They record no
+# offset: the nakaguro also divides kanji roster pairs (高橋・一郎,
+# 姓・名 -- read family-first by the script license, #272), so it is
+# not a transcription marker. Only the Chinese 间隔号 is (#298), and
+# it is context-sensitive -- see _INTERPUNCT below.
 _NAME_DOT_SEPARATORS = frozenset({"\u30FB", "\uFF65"})
 
 _INTERPUNCT = "\u00B7"
@@ -70,21 +70,28 @@ def _is_emoji(ch: str) -> bool:
     return any(lo <= cp <= hi for lo, hi in _EMOJI_RANGES)
 
 
-def _flank(text: str, i: int, step: int, limit: int,
-           state: ParseState) -> str | None:
+def _stripped(ch: str, policy: Policy) -> bool:
+    """True when the strip policy removes `ch` from the token stream.
+    The ONE definition of that set, shared by _ignorable and _flank: a
+    character that vanishes from tokens must not occupy a flank
+    position either, so a new strip class added here stays transparent
+    to the interpunct guard by construction."""
+    if policy.strip_bidi and _BIDI.match(ch):
+        return True
+    return bool(policy.strip_emoji and _is_emoji(ch))
+
+
+def _flank(text: str, indices: range, policy: Policy) -> str | None:
     """The nearest flank character the strip policy would keep, or
-    None at the region bound. Stripped invisibles (bidi, emoji) are
-    TRANSPARENT here: they vanish from the token stream, so they must
-    not occupy a flank position either -- an RTL document quoting a
-    transcription puts U+200F beside the dot in visually identical
-    text. Whitespace and the other separators stay guard-defeating:
-    a dot beside a space is not between characters."""
-    while i != limit:
+    None if the range exhausts. Stripped invisibles are TRANSPARENT
+    here: an RTL document quoting a transcription puts U+200F beside
+    the dot in visually identical text. Whitespace and the other
+    separators stay guard-defeating: a dot beside a space is not
+    between characters."""
+    for i in indices:
         ch = text[i]
-        if not (state.policy.strip_bidi and _BIDI.match(ch)) \
-                and not (state.policy.strip_emoji and _is_emoji(ch)):
+        if not _stripped(ch, policy):
             return ch
-        i += step
     return None
 
 
@@ -97,24 +104,14 @@ def _ignorable(ch: str, state: ParseState) -> bool:
         # skip the checks below for every ASCII letter
         return False
     # unconditional, like whitespace -- not policy-gated, so this sits
-    # ahead of the policy checks rather than in _tokenize_region beside
-    # COMMA_CHARS (commas RECORD an offset; these dots must not: the
-    # nakaguro is Japanese roster/divider typography whose pieces the
-    # script license already reads correctly -- #272, 高橋・一郎
-    # family-first, マイケル・ジャクソン positional -- while only the
-    # Chinese 间隔号 carries the transcription meaning and records,
-    # #298, handled in _tokenize_region). The only load-bearing
-    # constraint is "after the isascii fast path" (both dots are
-    # non-ASCII); being ahead of the bidi/emoji checks below is NOT
-    # load-bearing -- the three sets are disjoint, so a future edit is
-    # free to reorder them.
+    # ahead of the policy check rather than in _tokenize_region beside
+    # COMMA_CHARS (commas RECORD an offset; these dots must not --
+    # only the 间隔号 marks a transcription and records, #298, see
+    # _NAME_DOT_SEPARATORS above). The only load-bearing constraint is
+    # "after the isascii fast path" (both dots are non-ASCII).
     if ch in _NAME_DOT_SEPARATORS:
         return True
-    if state.policy.strip_bidi and _BIDI.match(ch):
-        return True
-    if state.policy.strip_emoji:
-        return _is_emoji(ch)
-    return False
+    return _stripped(ch, state.policy)
 
 
 def _tokenize_region(state: ParseState, start: int, end: int,
@@ -126,24 +123,20 @@ def _tokenize_region(state: ParseState, start: int, end: int,
     for i in range(start, end):
         ch = text[i]
         is_separator = ch in COMMA_CHARS or _ignorable(ch, state)
-        if not is_separator and ch == _INTERPUNCT:
-            # flanks are region-local (the _flank walks stop at the
-            # region bounds): a B7 at a region edge has a masked or
-            # absent neighbor and stays token text. Region-local on
-            # purpose and defensively: under the default delimiters a
-            # masked span's edge character is never classified, so
-            # the two bounds are indistinguishable -- but a custom
-            # delimiter ending in a classified character would
-            # otherwise let a B7 split and record across a mask seam.
-            # The flank scan reads raw text like segmentation matching
-            # does (#272's stance): NFD hangul degrades to no-split,
-            # never wrong-split -- classification NFC-normalizes but
-            # the guard does not.
-            left = _flank(text, i - 1, -1, start - 1, state)
-            right = _flank(text, i + 1, +1, end, state)
-            is_separator = (left is not None and right is not None
-                            and _classified_char(left)
-                            and _classified_char(right))
+        if ch == _INTERPUNCT:
+            # Region-local flanks: a B7 at a region edge stays token
+            # text, and the bound also stops a custom delimiter's
+            # classified edge character from acting as a flank across
+            # a mask seam. The scan reads raw text like segmentation
+            # matching does (#272's stance): NFD hangul degrades to
+            # no-split, never wrong-split -- classification
+            # NFC-normalizes but the guard does not.
+            left = _flank(text, range(i - 1, start - 1, -1),
+                          state.policy)
+            if left is not None and _classified_char(left):
+                right = _flank(text, range(i + 1, end), state.policy)
+                is_separator = (right is not None
+                                and _classified_char(right))
         if is_separator:
             if tok_start is not None:
                 tokens.append(WorkToken(text[tok_start:i],

tests/v2/pipeline/test_assign.py

@@ -228,21 +228,6 @@ def test_interpunct_suppression_yields_to_explicit_name_order() -> None:
     assert _by_role(out, Role.GIVEN) == "莎士比亚"
 
 
-def test_nakaguro_divided_kanji_keeps_the_roster_reading() -> None:
-    # the Japanese dot is roster formatting (姓・名), NOT the
-    # transcription marker -- #272's family-first reading stands
-    # (codepoint scope, spec decision 5)
-    out = _assigned("高橋・一郎")
-    assert _by_role(out, Role.FAMILY) == "高橋"
-    assert _by_role(out, Role.GIVEN) == "一郎"
-
-
-def test_undotted_han_still_reads_family_first() -> None:
-    out = _assigned("毛 泽东")
-    assert _by_role(out, Role.FAMILY) == "毛"
-    assert _by_role(out, Role.GIVEN) == "泽东"
-
-
 def test_script_with_no_table_entry_falls_back() -> None:
     # A single, well-defined script the table simply does not list:
     # resolution must fall through to name_order rather than pick an

tests/v2/test_regex_sync.py

@@ -164,6 +164,33 @@ def test_comma_char_matches_the_pipeline_comma_set() -> None:
     assert set(_render._COMMA_CHAR.pattern.strip("[]")) == set(COMMA_CHARS)
 
 
+# The one sanctioned divergence between the differential rules'
+# character classes and _SCRIPT_RANGES: the halfwidth middle dot
+# separates tokens without being classified (halfwidth kana stays out
+# of the table on purpose). U+00B7 is deliberately NOT here -- its
+# flank guard means every name it can change matches through a
+# classified flanking character already. Single-sourced: both span
+# pins below read this set.
+_SANCTIONED_EXTRAS = frozenset({(0xFF65, 0xFF65)})
+
+
+def _declared_spans(name_regex: str) -> set[tuple[int, int]]:
+    """The \\uXXXX-\\uXXXX span pairs a rule's character class declares."""
+    return {
+        (int(lo, 16), int(hi, 16))
+        for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})",
+                                 name_regex)}
+
+
+def _expected_bmp_spans() -> set[tuple[int, int]]:
+    """What a full CJK character class in the toml must declare: the
+    table's BMP spans plus the sanctioned extras."""
+    return {span
+            for spans in _policy._SCRIPT_RANGES.values()
+            for span in spans
+            if span[1] <= 0xFFFF} | set(_SANCTIONED_EXTRAS)
+
+
 def test_differential_cjk_rule_matches_the_script_ranges() -> None:
     """The CJK rule in tools/differential/expected_changes.toml hand-
     copies the script spans from _policy._SCRIPT_RANGES into a character
@@ -224,67 +251,64 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None:
         "a Script joined _SCRIPT_RANGES: decide whether the "
         f"differential rule in {toml_path.name} should cover it, then "
         "update this assertion")
-    declared = {
-        (int(lo, 16), int(hi, 16))
-        for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})",
-                                 matched[0]["name_regex"])}
-    # the two spans the rule carries that no Script claims (see above)
-    sanctioned_extras = {(0xFF65, 0xFF65)}
-    for xlo, xhi in sanctioned_extras:
+    declared = _declared_spans(matched[0]["name_regex"])
+    # the extras must stay UNclassified, or they belong in the table
+    for xlo, xhi in _SANCTIONED_EXTRAS:
         assert not any(lo <= xhi and xlo <= hi
                        for spans in _policy._SCRIPT_RANGES.values()
                        for lo, hi in spans), (
             f"U+{xlo:04X}-U+{xhi:04X} is classified now; drop it "
-            "from the sanctioned extras")
-    expected = {span
-                for spans in _policy._SCRIPT_RANGES.values()
-                for span in spans
-                if span[1] <= 0xFFFF} | sanctioned_extras
+            "from _SANCTIONED_EXTRAS")
+    expected = _expected_bmp_spans()
     assert declared == expected, (
         f"{toml_path.name}'s CJK name_regex declares {sorted(declared)}; "
         f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}")
 
 
-@pytest.mark.parametrize("slug", ["cjk-delimited-nickname",
-                                  "cjk-comma-compound"])
-def test_differential_compound_rules_match_the_script_ranges(
-        slug: str) -> None:
-    """The compound rules carry FURTHER hand copies of the script
-    spans in the toml: each one's require-a-classified-codepoint
-    lookahead exists so its trigger set alone (delimiters; a comma)
-    cannot claim a Latin name's regression ('John 「Jack」 Kennedy',
-    'Smith, Jr.' -- neither contains a classified character). Pin each
-    copy to the table exactly as the canonical CJK rule's is, plus the
-    nickname rule's delimiter set itself, so widening either is an
-    explicit decision here rather than a silent absorption change.
-
-    Selection note for future rule authors: the canonical-CJK-rule pin
+def test_every_span_bearing_rule_matches_the_script_ranges() -> None:
+    """Auto-discovered pin for every FURTHER hand copy of the script
+    spans in the toml: any rule whose character class declares spans
+    intersecting _SCRIPT_RANGES must declare the whole expected class
+    (table BMP spans + sanctioned extras). The compound rules'
+    require-a-classified-codepoint lookaheads exist so their trigger
+    sets alone (delimiters; a comma) cannot claim a Latin name's
+    regression -- and each such lookahead is a copy nothing else
+    checks. Discovery replaces a hand-maintained slug roster: a new
+    compound rule's copy is pinned by existing here, not by an author
+    remembering to enroll it. Rules whose spans touch OTHER scripts
+    (Cyrillic, say) are out of scope and skipped by the intersection
+    test.
+
+    Selection note for future rule authors: the canonical-rule pin
     above selects by the literal '#271'/'#272' substrings and asserts
-    uniqueness -- the compound slugs avoid them on purpose, and any
-    new rule's must too (and its slug joins this parametrize list, or
-    its span copy goes unpinned).
+    uniqueness -- compound slugs must avoid them.
     """
     toml_path = (Path(__file__).parents[2] / "tools" / "differential"
                  / "expected_changes.toml")
     rules = tomllib.loads(toml_path.read_text())["change"]
-    matched = [r for r in rules if slug in r["issue"]]
-    assert len(matched) == 1
-    regex = matched[0]["name_regex"]
-    if slug == "cjk-delimited-nickname":
-        assert "[「」『』・･]" in regex, (
-            "the compound rule's delimiter set changed; decide "
-            "deliberately")
-    declared = {
-        (int(lo, 16), int(hi, 16))
-        for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})",
-                                 regex)}
-    expected = {span
-                for spans in _policy._SCRIPT_RANGES.values()
-                for span in spans
-                if span[1] <= 0xFFFF} | {(0xFF65, 0xFF65)}
-    assert declared == expected, (
-        f"{slug}'s codepoint lookahead declares {sorted(declared)}; "
-        f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}")
+    table_spans = _expected_bmp_spans()
+    checked = []
+    for rule in rules:
+        regex = rule.get("name_regex")
+        if not isinstance(regex, str):
+            continue
+        declared = _declared_spans(regex)
+        if not declared & table_spans:
+            continue
+        checked.append(rule["issue"])
+        assert declared == table_spans, (
+            f"{rule['issue']!r} declares {sorted(declared)}; expected "
+            f"{sorted(table_spans)}")
+    # the canonical rule plus both compound rules, today -- if this
+    # count drops, a hand copy fell out of discovery's reach
+    assert len(checked) >= 3, checked
+    # the delimiter compound's trigger set is its own decision surface
+    nickname = [r for r in rules
+                if "cjk-delimited-nickname" in r["issue"]]
+    assert len(nickname) == 1
+    assert "[\u300C\u300D\u300E\u300F\u30FB\uFF65]" in nickname[0][
+        "name_regex"] or "[「」『』・･]" in nickname[0]["name_regex"], (
+        "the compound rule's delimiter set changed; decide deliberately")
 
 
 def test_cjk_corpus_matches_the_case_table() -> None: