Classify the ideographic iteration mark as Han (#272)

々 (U+3005) repeats the preceding kanji and carries Script=Common
under UAX #24, so it fell outside every range in _SCRIPT_RANGES. That
made 佐々木 a mixed-script token: "佐々木 太郎" REVERSED, giving given
佐々木 and family 太郎, and the token never gated into segmentation.
佐々木 (Sasaki) is a top-20 Japanese surname, and 野々村 and 奈々 are
ordinary names, so this was not an exotic corner.

The mark joins the HAN ranges on the same block-style reasoning U+30FB
already gets -- it can only appear inside a Han-written name, being a
repeat of the character before it -- along with all three hand-copies
the sync pins police. Verified with the extra rather than assumed:
namedivider reads 佐々木健 as 佐々木 + 健, so the integration
assertion pins the real answer. 佐 is absent from the zh surname list,
so the zh pack's reading of these names does not change.

Also from the branch review: the differential rule gains U+FF65, whose
halfwidth transcriptions do split ('ﾏｲｹﾙ･ｼﾞｬｸｿﾝ') even though
halfwidth kana is deliberately unclassified -- the sync pin names that
one sanctioned extra rather than relaxing to a subset check, so a
second source of divergence still fails. The rest of the halfwidth
block stays out: a dotless 'ﾏｲｹﾙ ｼﾞｬｸｿﾝ' is byte-identical on both
sides, measured, and covering it would pre-excuse a future regression.
Plus four documentation corrections: the nakaguro's exemption from
both policy opt-outs stated in AGENTS.md where the doctrine lives,
usage.rst's forward pointer widened to kana, _tokenize.py's dangling
promise of a follow-up issue replaced with the standing reason, and
the available() tuple moved off the #271 bullet that predates ja.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: derek73

AGENTS.md

@@ -72,7 +72,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG)
 
 The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic).
 
-**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Four defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes.
+**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Four defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) would need a guard. Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes.
 
 ### Configuration layer (`nameparser/config/`)
 

docs/release_log.rst

@@ -4,10 +4,10 @@ Release Log
 
     **East Asian name support**
 
-    - Add the Chinese locale pack ``locales.ZH`` — opt-in Han segmentation for unspaced names like ``毛泽东``, with the surname vocabulary it needs. A pack rather than a default because a Chinese surname list corrupts the Japanese names written in the same characters (``高橋一郎`` would split ``高`` + ``橋一郎``); Japanese data goes through ``locales.JA`` and its segmenter instead. It sets no name order, since native-script Han already reads family-first without it. ``locales.available()`` is now ``('ja', 'ru', 'tr_az', 'zh')`` (#271)
+    - Add the Chinese locale pack ``locales.ZH`` — opt-in Han segmentation for unspaced names like ``毛泽东``, with the surname vocabulary it needs. A pack rather than a default because a Chinese surname list corrupts the Japanese names written in the same characters (``高橋一郎`` would split ``高`` + ``橋一郎``); Japanese data goes through ``locales.JA`` and its segmenter instead. It sets no name order, since native-script Han already reads family-first without it (#271)
     - Add ``Lexicon.surnames``, ``Policy.script_orders``, ``Policy.segment_scripts``, the ``Script`` enum and the ``DEFAULT_SCRIPT_ORDERS`` constant to the public API. This is the first behavior nameparser keys on the script a name is written in; it is allowed only where the script itself settles a convention, never as a proxy for guessing the language (#271)
     - Add ``AmbiguityKind.SEGMENTATION``, reported when a surname split had a vocabulary-supported alternative: ``"남궁민수"`` is 남궁 + 민수 by the compound surname but 남 + 궁민수 by the single-syllable one, and longest-match had to pick. A name with only one possible split decided nothing and reports nothing (#271)
-    - Add the Japanese locale pack ``locales.JA`` and the segmenter factory ``locales.ja_segmenter()``, which together divide an unspaced Japanese name: ``parser_for(locales.JA, segmenter=locales.ja_segmenter())`` reads ``山田太郎`` as family ``山田``, given ``太郎``. The two halves are separate because no surname list can do this job — family and given names draw on the same kanji and the reading, not the spelling, decides most divisions — so the pack activates the stage and a third-party divider performs it. ``ja_segmenter()`` wraps `namedivider-python <https://pypi.org/project/namedivider-python/>`_, installed with the new ``nameparser[ja]`` extra; the core stays dependency-free, and ``ja_segmenter(gbdt=True)`` selects namedivider's more accurate gradient-boosted model, which downloads its data on first use (closes #272)
+    - Add the Japanese locale pack ``locales.JA`` and the segmenter factory ``locales.ja_segmenter()``, which together divide an unspaced Japanese name: ``parser_for(locales.JA, segmenter=locales.ja_segmenter())`` reads ``山田太郎`` as family ``山田``, given ``太郎``. The two halves are separate because no surname list can do this job — family and given names draw on the same kanji and the reading, not the spelling, decides most divisions — so the pack activates the stage and a third-party divider performs it. ``ja_segmenter()`` wraps `namedivider-python <https://pypi.org/project/namedivider-python/>`_, installed with the new ``nameparser[ja]`` extra; the core stays dependency-free, and ``ja_segmenter(gbdt=True)`` selects namedivider's more accurate gradient-boosted model, which downloads its data on first use. With both packs registered, ``locales.available()`` is now ``('ja', 'ru', 'tr_az', 'zh')`` (closes #272)
     - Add ``Segmentation`` and the ``Segmenter`` type alias to the public API, plus the keyword-only ``Parser(segmenter=...)`` hook they describe: any callable from a token's text to a ``Segmentation`` (the interior offsets to cut at, and a confidence) or ``None`` to decline. It is consulted only for scripts listed in ``Policy.segment_scripts``, and only where the surname vocabulary declined first, so ``parser_for(locales.ZH, locales.JA, segmenter=...)`` composes — a listed Chinese surname wins, the segmenter takes the rest (#272)
     - Add the ``Script`` members ``HIRAGANA`` and ``KATAKANA``. Two members rather than one ``KANA`` because the parser treats them differently: hiragana never transcribes a foreign name, while a wholly-katakana name usually is one (#272)
 

docs/usage.rst

@@ -52,9 +52,9 @@ given-then-family:
     'Doe'
 
 For family-first input *without* a comma — common outside Europe — set
-``name_order``; see :doc:`customize`. Names written in Han or Hangul
-are the exception that needs no setting at all: see `East Asian
-names`_ below.
+``name_order``; see :doc:`customize`. Names written in Han or Hangul,
+and Japanese names written in kanji and kana, are the exception that
+needs no setting at all: see `East Asian names`_ below.
 
 Words that attach to their neighbors
 --------------------------------------

nameparser/_pipeline/_tokenize.py

@@ -44,9 +44,13 @@
 # The katakana middle dot and its halfwidth twin divide the parts of
 # a foreign name transcribed into katakana (マイケル・ジャクソン) --
 # native names never contain them, so they separate unconditionally,
-# like whitespace (amendment 2026-07-29 section 1b). U+00B7 is
-# deliberately NOT here: it is also the Catalan punt volat, interior
-# to real names (Gal·la) -- a follow-up issue owns that decision.
+# like whitespace (amendment 2026-07-29 section 1b). The Chinese
+# interpunct U+00B7 (威廉·莎士比亚) is deliberately NOT here: it is
+# also the Catalan punt volat, which sits INSIDE a single name piece
+# (Gal·la), so separating on it unconditionally would break names it
+# has no business touching. Admitting it would need a flanked-by-CJK
+# guard, which is a different rule from this set's "these codepoints
+# are always separators".
 _NAME_DOT_SEPARATORS = frozenset({"\u30FB", "\uFF65"})
 
 

nameparser/_pipeline/_vocab.py

@@ -37,14 +37,21 @@
 # per-char on the _EMOJI_RANGES precedent in _tokenize.py, on the
 # theory that a range test needs no regex; measured at token scale the
 # compiled regex wins by 3-9x, and by 89x on long tokens.)
-# HAN: the URO plus Extension A, the compatibility block,
-# and the supplementary-plane block (Ext B-I + CJK Compat Ideographs
-# Supplement, 0x20000-0x323AF) -- rare surnames are the biggest real
-# source of supplementary-plane hanzi in personal names (e.g. 𠮷田's
-# 𠮷, U+20BB7), so leaving them out silently mis-orders those names;
-# unassigned gaps inside the span are harmless, since no real name
-# contains an unassigned codepoint. HANGUL: precomposed syllables
-# only -- modern Korean text never writes names as bare jamo.
+# HAN: the ideographic iteration mark U+3005, the URO plus Extension
+# A, the compatibility block, and the supplementary-plane block
+# (Ext B-I + CJK Compat Ideographs Supplement, 0x20000-0x323AF) --
+# rare surnames are the biggest real source of supplementary-plane
+# hanzi in personal names (e.g. 𠮷田's 𠮷, U+20BB7), so leaving them
+# out silently mis-orders those names; unassigned gaps inside the span
+# are harmless, since no real name contains an unassigned codepoint.
+# U+3005 々 is the block-vs-Script case again (it is Script=Common
+# under UAX #24, like U+30FB below): it REPEATS the preceding kanji
+# and appears only inside Han-written names -- 佐々木 (Sasaki, a
+# top-20 Japanese surname), 野々村, 奈々 -- so a token carrying it is
+# as Han as the character it iterates. Omitting it made 佐々木 a
+# mixed-script token: the name reversed and never gated into
+# segmentation. HANGUL: precomposed syllables only -- modern Korean
+# text never writes names as bare jamo.
 # HIRAGANA/KATAKANA (#272): the two kana blocks, each in full. There
 # IS a supplementary-plane kana repertoire (Kana Supplement, Kana
 # Extended-A/B, Small Kana Extension, U+1AFF0-U+1B16F, 311 assigned
@@ -75,8 +82,8 @@
 # (dict iteration order), so an overlapping future script would make
 # the result order-dependent instead of well-defined.
 _SCRIPT_RANGES: dict[Script, tuple[tuple[int, int], ...]] = {
-    Script.HAN: ((0x3400, 0x4DBF), (0x4E00, 0x9FFF), (0xF900, 0xFAFF),
-                 (0x20000, 0x323AF)),
+    Script.HAN: ((0x3005, 0x3005), (0x3400, 0x4DBF), (0x4E00, 0x9FFF),
+                 (0xF900, 0xFAFF), (0x20000, 0x323AF)),
     Script.HANGUL: ((0xAC00, 0xD7A3),),
     Script.HIRAGANA: ((0x3040, 0x309F),),
     Script.KATAKANA: ((0x30A0, 0x30FF),),

nameparser/locales/ja.py

@@ -81,7 +81,7 @@
 # kana-licensed composite the pack acts on (山田エミ) -- DEVIATES must
 # declare it, and the adapter must not decline it. A pure-katakana
 # token never reaches either: the stage's own gate stops it.
-_JA_RANGES = ((0x3040, 0x309F), (0x30A0, 0x30FF),
+_JA_RANGES = ((0x3005, 0x3005), (0x3040, 0x309F), (0x30A0, 0x30FF),
               (0x3400, 0x4DBF), (0x4E00, 0x9FFF), (0xF900, 0xFAFF),
               (0x20000, 0x323AF))
 

nameparser/locales/zh.py

@@ -101,8 +101,8 @@
 # _pipeline/_vocab.py's _SCRIPT_RANGES[Script.HAN] (layering forbids
 # a pack importing the pipeline; the sync test in
 # tests/v2/test_locales.py pins the equality).
-_HAN_RANGES = ((0x3400, 0x4DBF), (0x4E00, 0x9FFF), (0xF900, 0xFAFF),
-               (0x20000, 0x323AF))
+_HAN_RANGES = ((0x3005, 0x3005), (0x3400, 0x4DBF), (0x4E00, 0x9FFF),
+               (0xF900, 0xFAFF), (0x20000, 0x323AF))
 
 
 def DEVIATES(name: str) -> bool:

tests/v2/cases.py

@@ -613,6 +613,14 @@ def __post_init__(self) -> None:
          notes="parity: katakana deliberately has no entry, so the "
                "positional default holds -- transcribed foreign names "
                "keep source order"),
+    Case("ja_iteration_mark_is_han", "佐々木 太郎",
+         {"family": "佐々木", "given": "太郎"},
+         classification="fix(#272)",
+         notes="々 (U+3005, the ideographic iteration mark) repeats "
+               "the preceding kanji and is Script=Common under UAX "
+               "#24; it counts as Han here, or 佐々木 -- a top-20 "
+               "Japanese surname -- would be a mixed-script token and "
+               "reverse"),
     Case("ja_nakaguro_inside_a_nickname",
          "山田 太郎 (マイケル・ジャクソン)",
          {"family": "山田", "given": "太郎",

tests/v2/test_locales.py

@@ -246,6 +246,12 @@ def test_ja_end_to_end() -> None:
     # kana-licensed composite: gated in through the HIRAGANA entry
     n = p.parse("高橋みなみ")
     assert (n.family, n.given) == ("高橋", "みなみ")
+    # the iteration mark 々 gates in as Han and namedivider reads the
+    # repeated character correctly -- verified against 0.4.1 rather
+    # than assumed, since 々 is Script=Common and a divider could
+    # plausibly have choked on it or cut beside it
+    n = p.parse("佐々木健")
+    assert (n.family, n.given) == ("佐々木", "健")
 
 
 @_needs_ja
@@ -576,6 +582,7 @@ def _default_corpus() -> list[str]:
     "高橋みなみ",    # kanji + hiragana: the kana license's carrier case
     "山田エミ",      # kanji + katakana: licensed as a MIXED token, where
                      # pure katakana (マイケル) is deliberately not
+    "佐々木健",      # the iteration mark 々 inside a top-20 surname
 ]
 
 

tests/v2/test_parser.py

@@ -483,6 +483,20 @@ def test_kana_licensed_names_read_family_first_by_default() -> None:
     assert parse("高橋みなみ").family == "高橋みなみ"
 
 
+def test_iteration_mark_counts_as_han() -> None:
+    # 々 (U+3005) repeats the preceding kanji and is Script=Common
+    # under UAX #24, so classifying by the Script property would put a
+    # 佐々木 token in no script at all: the name reversed, and the
+    # token never reached the segmentation gate. It is in the HAN
+    # ranges by BLOCK-style reasoning, the same call U+30FB gets.
+    n = parse("佐々木 太郎")
+    assert (n.family, n.given) == ("佐々木", "太郎")
+    n = parse("野々村 真")
+    assert (n.family, n.given) == ("野々村", "真")
+    # and a lone one takes the family role like any other Han token
+    assert parse("奈々").family == "奈々"
+
+
 def test_pure_katakana_stays_positional() -> None:
     # transcribed foreign names keep their original (usually Western)
     # order; katakana alone licenses nothing