Merge pull request #305 from derek73/v2.1/interpunct

间隔号-divided transcriptions keep source order

Author: derek73

AGENTS.md

@@ -75,7 +75,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG)
 
 The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic).
 
-**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Four defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) would need a guard. Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes.
+**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it. Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps its source order and never segments — the orthography names the convention, exactly as pure katakana does, with the divider carrying the signal instead of the script. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes.
 
 ### Configuration layer (`nameparser/config/`)
 

docs/concepts.rst

@@ -58,7 +58,9 @@ layer, which assigns purely by where a word sits: the first unclaimed
 word is the given name, the last is the family name, and anything
 between them is the middle name. ``name_order``, an explicit comma,
 and — for a name written wholly in one East Asian script —
-``script_orders`` change what "first" and "last" mean here; nothing
+``script_orders`` change what "first" and "last" mean here; the
+Chinese interpunct ``·`` dividing such a name walks the last of those
+back, marking a transcription that keeps its source order; nothing
 else does.
 
 This is the whole parser in two sentences, and it explains its

docs/migrate.rst

@@ -429,7 +429,11 @@ now.
 surface is frozen for 2.x — so the way out is the 2.0 API:
 ``Parser(policy=Policy(script_orders={}, segment_scripts=()))``
 restores 1.4's reading of every shape above that turns on order or
-splitting. The middle dot is the one exception: it is decided in
-tokenization rather than by policy, so a name written with one still
-divides at the dot, and still renders with a space, whatever those two
-fields are set to.
+splitting. The middle dots are the exception: both the katakana dot
+and the Chinese interpunct ``·`` (U+00B7, dividing a transcription
+like ``威廉·莎士比亚``, and a separator only between classified-script
+characters — each side judged on its own — where the katakana dot is
+unconditional) are decided
+in tokenization rather than by policy, so a name written with either
+still divides at the dot, and still renders with a space, whatever
+those two fields are set to.

docs/release_log.rst

@@ -23,7 +23,8 @@ Release Log
     - Fix unspaced Korean names not splitting: the census surname list now ships as default vocabulary (``Lexicon.surnames``) with hangul segmentation on by default (``Policy.segment_scripts``), so ``"김민준"`` parses family ``김``, given ``민준`` where 1.x returned the whole string as ``first``. Rendering follows the split, so ``str(HumanName("김민준"))`` is now ``"민준 김"`` where 1.x echoed the input back unchanged. Nothing but Korean is written in hangul and its surnames are a closed census set, which is what makes the split safe as a default rather than a pack. **Default-on**, and it reaches ``HumanName`` too. ``Policy(segment_scripts=())`` turns the split off; ``Policy(script_orders={})`` separately restores the positional reading; clearing both restores 2.0 behavior exactly (#271)
     - Fix Japanese names carrying kana parsing given-first: the family-first rule above extends to any name whose characters stay within kanji and kana while carrying at least one kana character, so ``"高橋 みなみ"`` gives family ``高橋`` and ``"山田 エミ"`` family ``山田`` where 1.x read both the other way round, and a lone such token (``"高橋みなみ"``, ``"みなみ"``) lands in ``family``/``last`` where 1.x put it in ``given``/``first``. The reasoning is the one hangul already uses: hiragana never transcribes a foreign name, and a transcription is kana ALONE, so kanji-plus-kana is a Japanese person's name written in Japanese order. A name written **wholly in katakana** is deliberately excluded and stays positional — it is predominantly a transcribed foreign name (``"マイケル ジャクソン"``) already in given-first order. **Default-on: changes parse output for kana-bearing Japanese names**, through ``HumanName`` as well as the 2.0 API, and the spaced forms change what the name renders as (``str(HumanName("高橋 みなみ"))`` is now ``"みなみ 高橋"``). ``Policy(script_orders={})`` clears this entry along with the Han and Hangul ones (#272)
     - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303)
-    - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is deliberately NOT included: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``) (#272)
+    - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272)
+    - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298)
     - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
 
 * 2.0.0 - July 27, 2026

docs/usage.rst

@@ -188,6 +188,24 @@ language is up to you. When you know the data is Chinese, apply the
     >>> parser_for(locales.ZH).parse("毛泽东").family
     '毛'
 
+Chinese also has a transcription convention, and it is written in the
+punctuation: a foreign name transcribed into Han characters keeps its
+source order and divides its parts with the 间隔号, the interpunct
+``·`` (U+00B7) — 威廉·莎士比亚 is William Shakespeare, given name
+first. The dot itself is the marker, so nameparser reads U+00B7 as a
+token separator when it sits between classified-script characters
+(each side judged on its own — a hangul character beside a Han one
+qualifies), and a dot anywhere in the name reads the whole name as a
+transcription listing: it keeps the order it was written in and is
+never segmented — the role pure katakana plays for Japanese
+transcriptions, played here by the divider instead of the script.
+Because only classified characters on both sides make it a divider,
+the same codepoint interior to a Latin-script name — the Catalan punt
+volat in ``Gal·la`` — is untouched, and a dot with a classified
+character on just one side (``王·Smith``) stays part of the word
+undivided. The Japanese middle dot ・ (covered next) is a different
+mark carrying a different convention, and keeps its own reading.
+
 Japanese
 ~~~~~~~~~
 
@@ -266,8 +284,15 @@ positional rules — order genuinely varies in romanized data, so
 nothing script-based applies. A name written wholly in katakana stays
 positional for the reason given above, pack or no pack: it is
 predominantly a transcription, and a transcription is already in the
-order it should be read in. And a comma disables the script behaviors
-entirely, on the reasoning ``name_order`` already follows: whoever
+order it should be read in. A Han transcription written with a space
+instead of the 间隔号 (威廉 莎士比亚) carries nothing to distinguish it
+from a native two-token name and keeps the family-first reading — the
+dot is the marker, and without it there is no signal. The same holds
+for a transcription typed with the Japanese middle dot (威廉・莎士比亚):
+each dot carries its own script's convention, the nakaguro's is
+Japanese roster formatting rather than transcription, so only the
+Chinese dot rescues the source order. And a comma disables the script
+behaviors entirely, on the reasoning ``name_order`` already follows: whoever
 wrote the comma has already said where the family name ends.
 
 A division the parser had to choose is reported rather than hidden.