Classify 〆 (U+3006) as Han

The one behavior change of this branch: the HAN entry's opening span
widens from the 々 singleton (0x3005, 0x3005) to (0x3005, 0x3006),
taking in 〆, the shime mark. Unlike 々 -- Script=Han under UAX #24,
merely outside every CJK ideograph block -- 〆 carries Script=Common,
so this is the table deliberately reaching PAST the Script property,
not around a block boundary. It earns the reach: 〆 functions solely
as a component of Japanese surnames (〆木 Shimeki, 〆谷 Shimetani,
〆野) and appears in no other script's names. Before the change 〆木
was a mixed-script token: unlicensed, positional, given-first -- the
name reversed and never gated into segmentation.

The widening also makes 〆木太郎 wholly Japanese to the ja pack's
adapter, which would have handed it to namedivider -- whose 0.4.x
rule path cuts every attested 〆 surname at offset 1 (〆 | 木太郎,
score 1.0, algorithm='rule'; measured): a wrong family arriving with
NO SEGMENTATION report, since 1.0 clears the floor. The adapter now
declines 〆-bearing tokens before the divider is consulted (pinned
stub-based in tests/v2/test_locales.py, no extra needed): the
family-first ORDER fix stands regardless, and division can return
when a divider knows the mark.

The differential toml's character class is a hand copy of the table,
and its sync guard tripped in between exactly as designed:
test_differential_cjk_rule_matches_the_script_ranges failed with
"expected_changes.toml's CJK name_regex declares [(12293, 12293),
...]; _SCRIPT_RANGES' BMP spans are [(12293, 12294), ...]" until the
CJK rule's 々-々 was widened to 々-〆 to match.
Differential harness re-run by hand: 654 corpus names, 18 intentional
diffs, 0 unexplained, exit 0 -- neither corpus contains a 〆 name, so
totals are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: derek73

docs/release_log.rst

@@ -22,6 +22,7 @@ Release Log
     - Fix names written wholly in Han or Hangul parsing given-first: native-script CJK now reads family-first by default, through the new ``Policy.script_orders`` table, so ``"毛 泽东"`` gives family ``毛`` where 1.x gave family ``泽东``. A name that is a single unspaced token moves the same way — ``"毛泽东"`` and ``"山田太郎"`` now land in ``family``/``last`` where 1.x put them in ``given``/``first``, with the string itself untouched, which makes it the easiest form of this change to miss — a lone token renders the same whichever field holds it, whereas the spaced form does show up in the output (``str(HumanName("毛 泽东"))`` is now ``"泽东 毛"``). No language detection is involved: Chinese and Japanese both write the family name first in native script, so the script settles the order without anyone having to know which language it is. Latin-script and mixed-script names are never affected, and an explicit comma still wins. **Default-on: changes parse output for wholly-CJK names**, through ``HumanName`` as well as the 2.0 API (closes #271)
     - Fix unspaced Korean names not splitting: the census surname list now ships as default vocabulary (``Lexicon.surnames``) with hangul segmentation on by default (``Policy.segment_scripts``), so ``"김민준"`` parses family ``김``, given ``민준`` where 1.x returned the whole string as ``first``. Rendering follows the split, so ``str(HumanName("김민준"))`` is now ``"민준 김"`` where 1.x echoed the input back unchanged. Nothing but Korean is written in hangul and its surnames are a closed census set, which is what makes the split safe as a default rather than a pack. **Default-on**, and it reaches ``HumanName`` too. ``Policy(segment_scripts=())`` turns the split off; ``Policy(script_orders={})`` separately restores the positional reading; clearing both restores 2.0 behavior exactly (#271)
     - Fix Japanese names carrying kana parsing given-first: the family-first rule above extends to any name whose characters stay within kanji and kana while carrying at least one kana character, so ``"高橋 みなみ"`` gives family ``高橋`` and ``"山田 エミ"`` family ``山田`` where 1.x read both the other way round, and a lone such token (``"高橋みなみ"``, ``"みなみ"``) lands in ``family``/``last`` where 1.x put it in ``given``/``first``. The reasoning is the one hangul already uses: hiragana never transcribes a foreign name, and a transcription is kana ALONE, so kanji-plus-kana is a Japanese person's name written in Japanese order. A name written **wholly in katakana** is deliberately excluded and stays positional — it is predominantly a transcribed foreign name (``"マイケル ジャクソン"``) already in given-first order. **Default-on: changes parse output for kana-bearing Japanese names**, through ``HumanName`` as well as the 2.0 API, and the spaced forms change what the name renders as (``str(HumanName("高橋 みなみ"))`` is now ``"みなみ 高橋"``). ``Policy(script_orders={})`` clears this entry along with the Han and Hangul ones (#272)
+    - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names
     - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is deliberately NOT included: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``) (#272)
     - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
 

nameparser/_policy.py

@@ -66,7 +66,8 @@ class Script(StrEnum):
 # so the packs' predicates build on it too, through _script_matcher
 # below (the hand-copies this replaced dated from when the table
 # lived in the pipeline, which packs must not import).
-# HAN: the ideographic iteration mark U+3005, the URO plus Extension
+# HAN: the ideographic iteration mark U+3005 and the shime mark
+# U+3006, the URO plus Extension
 # A, the compatibility block, and the supplementary-plane block
 # (Ext B-I + CJK Compat Ideographs Supplement, 0x20000-0x323AF) --
 # rare surnames are the biggest real source of supplementary-plane
@@ -76,13 +77,21 @@ class Script(StrEnum):
 # U+3005 々 is the block-vs-Script case, running the OPPOSITE way to
 # U+30FB below: 々 already IS Script=Han under UAX #24 (Scripts.txt
 # reads `3005 ; Han`), but it sits in CJK Symbols and Punctuation,
-# outside every CJK ideograph block this table spans -- so the
-# singleton entry is what a BLOCK table needs to reach a character the
-# Script property would have classified correctly for free. It earns
-# the reach: 々 repeats the preceding kanji and appears only inside
-# Han-written names -- 佐々木 (Sasaki, a top-20 Japanese surname),
-# 野々村, 奈々. Omitting it made 佐々木 a mixed-script token: the name
-# reversed and never gated into segmentation.
+# outside every CJK ideograph block this table spans -- so a
+# singleton entry was what a BLOCK table needed to reach a character
+# the Script property would have classified correctly for free. It
+# earns the reach: 々 repeats the preceding kanji and appears only
+# inside Han-written names -- 佐々木 (Sasaki, a top-20 Japanese
+# surname), 野々村, 奈々. Omitting it made 佐々木 a mixed-script token:
+# the name reversed and never gated into segmentation.
+# U+3006 〆 (the shime mark) extends that singleton to a two-codepoint
+# span on a DIFFERENT justification: unlike 々, 〆 is Script=Common
+# under UAX #24, so this is the table deliberately reaching PAST the
+# Script property, not around a block boundary -- justified because 〆
+# functions solely as a component of Japanese surnames (〆木 Shimeki,
+# 〆谷 Shimetani, 〆野) and appears in no other script's names.
+# Leaving it out made 〆木 a mixed-script token: the name reversed and
+# never gated into segmentation.
 # HANGUL: precomposed syllables only -- modern Korean
 # text never writes names as bare jamo.
 # HIRAGANA/KATAKANA (#272): the two kana blocks, each in full. There
@@ -120,7 +129,7 @@ class Script(StrEnum):
 # future script would make the result order-dependent instead of
 # well-defined.
 _SCRIPT_RANGES: dict[Script, tuple[tuple[int, int], ...]] = {
-    Script.HAN: ((0x3005, 0x3005), (0x3400, 0x4DBF), (0x4E00, 0x9FFF),
+    Script.HAN: ((0x3005, 0x3006), (0x3400, 0x4DBF), (0x4E00, 0x9FFF),
                  (0xF900, 0xFAFF), (0x20000, 0x323AF)),
     Script.HANGUL: ((0xAC00, 0xD7A3),),
     Script.HIRAGANA: ((0x3040, 0x309F),),

nameparser/locales/ja.py

@@ -127,8 +127,10 @@ def ja_segmenter(*, gbdt: bool = False) -> Segmenter:
     matters for offline, sandboxed or air-gapped deployments.
 
     The adapter declines -- returns None, leaving the token whole --
-    for text outside the Japanese repertoire, for text too short to
-    divide, for any answer that fails to reconstruct its input, and for
+    for text outside the Japanese repertoire, for text bearing the
+    shime mark 〆 (namedivider 0.4.x's rule path cuts it in the wrong
+    place), for text too short to divide, for any answer that fails to
+    reconstruct its input, and for
     any score outside [0, 1] by more than float noise (a divider
     scoring 87.0 is broken, and its division is worth no more than its
     score). An
@@ -168,6 +170,15 @@ def segment(text: str) -> Segmentation | None:
         # answer for it regardless -- the same union DEVIATES declares.
         if not _wholly_japanese(text):
             return None
+        # namedivider 0.4.x has no knowledge of the shime mark: its
+        # rule path cuts 〆木太郎 to 〆 | 木太郎 at confidence 1.0
+        # (measured -- wrong for every attested 〆 surname: 〆木, 〆谷,
+        # 〆野), so a wrong family would arrive with no SEGMENTATION
+        # report. Decline instead: the family-first ORDER fix for 〆
+        # stands regardless, and division can return when a divider
+        # knows the mark.
+        if "〆" in text:
+            return None
         # namedivider RAISES below two characters ("Name length needs
         # at least 2 chars"), and a segmenter's exceptions propagate by
         # contract, so a one-character token -- routine in spaced input

tests/v2/cases.py

@@ -619,10 +619,25 @@ def __post_init__(self) -> None:
          notes="々 (U+3005, the ideographic iteration mark) repeats "
                "the preceding kanji and is Script=Han under UAX #24, "
                "but sits outside every CJK ideograph BLOCK -- and the "
-               "classifier is a block table, so it needs a singleton "
+               "classifier is a block table, so it needs its own "
                "entry to count as Han; without one 佐々木 -- a top-20 "
                "Japanese surname -- would be a mixed-script token and "
                "reverse"),
+    Case("ja_shime_mark_is_han", "〆木 太郎",
+         {"family": "〆木", "given": "太郎"},
+         classification="fix(shime-mark)",
+         notes="〆 (U+3006, the shime mark) opens real Japanese "
+               "surnames -- 〆木 Shimeki, 〆谷 Shimetani, 〆野 -- but "
+               "carries Script=Common under UAX #24; the table counts "
+               "it as Han anyway (a deliberate step PAST the Script "
+               "property, unlike 々's), else 〆木 is a mixed-script "
+               "token and the name reverses"),
+    Case("ja_shime_lone_token_takes_family", "〆木太郎",
+         {"family": "〆木太郎"},
+         classification="fix(shime-mark)",
+         notes="wholly-Han lone token under the family-first entry: "
+               "the whole token lands in family, unsegmented by "
+               "default"),
     Case("ja_nakaguro_inside_a_nickname",
          "山田 太郎 (マイケル・ジャクソン)",
          {"family": "山田", "given": "太郎",

tests/v2/pipeline/test_vocab.py

@@ -183,7 +183,7 @@ def test_script_ranges_are_pairwise_disjoint() -> None:
     # every entry of every script rather than script by script,
     # because the adjacent blocks are exactly where a future edit
     # would go wrong -- HIRAGANA ends at U+309F and KATAKANA opens at
-    # U+30A0 with no gap at all, and HAN's singleton U+3005 sits just
+    # U+30A0 with no gap at all, and HAN's U+3005-U+3006 span sits just
     # below both.
     spans = [(lo, hi, script)
              for script, ranges in _SCRIPT_RANGES.items()

tests/v2/test_locales.py

@@ -269,6 +269,26 @@ def test_ja_adapter_guard_stack_against_a_stub(
     assert seg("林") is None                # namedivider raises below 2
 
 
+def test_ja_adapter_declines_shime_tokens(
+        monkeypatch: pytest.MonkeyPatch) -> None:
+    # Since U+3006 classifies as Han, 〆木太郎 is wholly Japanese and
+    # reaches the adapter -- but namedivider 0.4.x has no knowledge of
+    # the shime mark: measured, its rule path cuts every attested 〆
+    # surname at offset 1 (〆木太郎 -> 〆 | 木太郎, score 1.0,
+    # algorithm='rule'), a wrong family arriving with NO SEGMENTATION
+    # report because 1.0 clears the floor. The adapter declines these
+    # tokens before the divider is ever consulted.
+    calls: list[str] = []
+
+    def divide(text: str) -> _FakeDivided:
+        calls.append(text)
+        return _FakeDivided(text[:1], text[1:], 1.0)
+
+    _fake_namedivider(monkeypatch, divide)
+    assert locales.ja_segmenter()("〆木太郎") is None
+    assert calls == []          # the decline sits before divide_name
+
+
 def test_ja_adapter_defensive_branches_against_a_stub(
         monkeypatch: pytest.MonkeyPatch) -> None:
     # reconstruction failure -> decline; an empty side -> the stated

tools/differential/expected_changes.toml

@@ -50,9 +50,10 @@ issue = "fix(#271/#272) native-script CJK: family-first order, hangul segmentati
 # tracked as #272's and #271's shared gap in issue #295, which also
 # records the provenance decision a fix needs; until it is closed the
 # behavioral guarantee lives in the fix(#271)/fix(#272) rows of
-# tests/v2/cases.py, and this rule is kept ready for the moment a CJK
-# name lands in a corpus.
-name_regex = "[\\u3005-\\u3005\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65]"
+# tests/v2/cases.py and the fix(shime-mark) rows, whose 〆 span rides
+# in on the same HAN entry, and this rule is kept ready for the moment
+# a CJK name lands in a corpus.
+name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65]"
 fields = ["first", "middle", "last"]
 
 [[change]]