Merge pull request #294 from derek73/v2.1/cjk-defaults

Parse native-script CJK names by default and add the ZH pack (#271)

Author: derek73

AGENTS.md

@@ -56,8 +56,10 @@ title; particles join forward, so ``de la`` attaches to ``Vega``.
 Whatever the vocabulary layer has not claimed is left to a positional
 layer, which assigns purely by where a word sits: the first unclaimed
 word is the given name, the last is the family name, and anything
-between them is the middle name. ``name_order`` and an explicit comma
-change what "first" and "last" mean here; nothing else does.
+between them is the middle name. ``name_order``, an explicit comma,
+and — for a name written wholly in one East Asian script —
+``script_orders`` change what "first" and "last" mean here; nothing
+else does.
 
 This is the whole parser in two sentences, and it explains its
 character. A word nameparser has never seen still gets a sensible role,

docs/concepts.rst

@@ -303,6 +303,58 @@ family-then-given regardless of the configured order:
     >>> family_first.parse("Thomas, John").family
     'Thomas'
 
+East Asian defaults, and turning them off
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Two defaults key on the *script* a name is written in rather than on
+anything you set: a name written wholly in Han or Hangul is assigned
+family-first (``script_orders``), and an unspaced hangul name is split
+into surname and given name against the shipped Korean census list
+(``segment_scripts``). :ref:`east-asian-names` explains the naming
+conventions both rest on — this section is how to switch them off,
+which you can do separately:
+
+.. doctest::
+
+    >>> parse("김민준").family                    # both defaults on
+    '김'
+    >>> positional = Parser(policy=Policy(script_orders={}))
+    >>> positional.parse("김민준").family         # still split
+    '민준'
+    >>> unsplit = Parser(policy=Policy(segment_scripts=()))
+    >>> unsplit.parse("김민준").family            # one token, not split
+    '김민준'
+
+The two switches interact, and clearing only ``script_orders``
+produces a third behavior rather than the old one: the split still
+runs, so ``김민준`` still becomes two tokens, and the positional
+default then assigns them given-first — the surname lands in
+``given``. To restore nameparser 2.0's reading exactly, clear both
+fields.
+
+To teach the splitter a surname it doesn't ship with, add it to the
+``surnames`` vocabulary like any other word:
+
+.. doctest::
+
+    >>> lex = Lexicon.default().add(surnames={"김민"})
+    >>> Parser(lexicon=lex).parse("김민준").family
+    '김민'
+
+Chinese surnames are deliberately absent from that default set,
+because splitting Han text requires knowing Chinese from Japanese;
+:doc:`locales` covers the opt-in ``zh`` pack that supplies them.
+
+.. note::
+
+   Both fields are annotated with their canonical *storage* type
+   rather than with everything the constructor accepts — the same as
+   ``capitalization_exceptions``. Under mypy the readable spellings
+   above (``script_orders={...}``, ``segment_scripts=(...)``) need a
+   ``# type: ignore[arg-type]``; ``script_orders=()`` and
+   ``segment_scripts=frozenset(...)`` check clean and mean the same
+   thing.
+
 Nicknames, maiden names, and brackets
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/customize.rst

@@ -201,12 +201,21 @@ parser_for(locales.RU).parse("Ivanova Anna Sergeevna").given   # "Anna"
 
 A locale pack is pure data: a vocabulary fragment plus a partial policy
 patch, folded in at parser construction. 2.0.0 ships `RU` and `TR_AZ`
-(formalizing the patronymic support added in 1.3.0); Chinese/Korean
-(bundled surname lists + segmentation of unspaced input), Vietnamese,
-and Japanese (via a `nameparser[ja]` extra with a pluggable segmenter)
-are designed and staged for 2.x minors. Vocabulary that cannot misfire
-on other languages' names (e.g. Cyrillic honorifics) goes in the
-default lexicon instead of packs.
+(formalizing the patronymic support added in 1.3.0); Vietnamese and
+Japanese (via a `nameparser[ja]` extra with a pluggable segmenter) are
+designed and staged for later 2.x minors. Vocabulary that cannot
+misfire on other languages' names (e.g. Cyrillic honorifics) goes in
+the default lexicon instead of packs.
+
+*Amended for 2.1 (#271):* the Chinese/Korean work landed, and split
+along that last line rather than shipping as one pack. Korean surnames
+and family-first order for native-script CJK cannot misfire on other
+languages' names, so both became defaults; only the Chinese surname
+list needed a pack (`ZH`), because it would corrupt the Japanese names
+written in the same characters. The never-auto-detect rule holds as
+written for a name's *language*; its **script** is a different signal,
+and where the script alone settles a convention, behavior may key on
+it.
 
 ## Migration: what actually happens to existing code
 

docs/design/nameparser-2.0-rfc.md

@@ -55,6 +55,24 @@ precedes the *given* name — as Arabic ones do — so the word after it
 isn't read as a family name. Listing it in ``given_name_titles`` alone
 raises ``ValueError`` rather than quietly doing nothing.
 
+Two East Asian behaviors are on by default for the same reason, except
+that what selects them is the *script* rather than the word. The
+background (covered fully under :ref:`east-asian-names` in
+:doc:`usage`): Chinese, Japanese, and Korean names put the family name
+first in native script and are usually written with no space between
+the parts. Both defaults follow from facts the script alone
+establishes. A name written wholly in Han or Hangul is assigned
+family-first, because every language written in those scripts orders
+names that way — no language guess is involved. An unspaced hangul
+name is additionally split into surname and given name, because hangul
+is written by nothing but Korean and Korean surnames are a closed
+census set that ships as default vocabulary. Splitting an unspaced
+*Han* name is the one behavior the script cannot license — the same
+characters could be a Chinese or a Japanese name, and a Chinese
+surname list splits Japanese names in the wrong place — so it is not a
+default: the ``zh`` pack below turns it on when you can declare the
+data Chinese.
+
 A pack is for something different: a *structural* rule, like reordering
 a patronymic, that vocabulary alone can't express.
 
@@ -91,7 +109,7 @@ takes:
 .. doctest::
 
     >>> locales.available()
-    ('ru', 'tr_az')
+    ('ru', 'tr_az', 'zh')
     >>> locales.get("ru") is locales.RU
     True
 
@@ -114,10 +132,19 @@ parsing, equivalent to ``parser_for(locales.get("ru"))``.
        (``oglu``, ``qizi``, ``uulu``, and their Latin- and
        Cyrillic-script variants) and reads the name around it as
        given/middle/family.
-
-Both shipped packs are policy-only — they carry no vocabulary of their
-own; see :doc:`concepts` for why that split (language vocabulary vs.
-behavior) is drawn where it is.
+   * - ``zh``
+     - Chinese surname segmentation — splits an unspaced Han name into
+       surname and given name (``毛泽东`` → family ``毛``, given
+       ``泽东``) against the surname list the pack ships. It sets no
+       name order: native-script Han already reads family-first
+       without a pack.
+
+``ru`` and ``tr_az`` are policy-only — they carry no vocabulary of
+their own. ``zh`` is both halves at once: a surname list, plus the one
+policy field that turns segmentation on for the script it covers. See
+:doc:`concepts` for how that split (language vocabulary vs. behavior)
+is drawn, and `Contributing a pack to nameparser`_ for which half a
+new naming rule belongs in.
 
 .. warning::
 
@@ -158,7 +185,9 @@ right and is validated on its own, before it is unioned onto the base
 — so a fragment that marks a word must also carry the word it marks.
 To make an existing base title precede the given name, restate the
 title in the fragment rather than listing it in ``given_name_titles``
-alone. Both shipped packs are policy-only, so neither hits this.
+alone. ``zh`` is the shipped worked example: its
+``Lexicon(surnames=...)`` has to satisfy every ``Lexicon`` rule
+standing alone, before anything unions it onto the base.
 
 .. doctest::
 
@@ -202,24 +231,41 @@ by ``tests/v2/test_locales.py``:
    string, return whether *this pack alone* might parse it differently
    from the default parser. Over-declaring is safe; under-declaring is
    not — when in doubt, ``DEVIATES`` should say yes.
-#. Add a rotator list to ``tests/v2/test_locales.py`` with at least one
-   name exercising every alternation branch of every marker regex the
-   pack defines — ``test_rotators_cover_every_marker_branch`` fails
-   until each branch is hit.
+#. Add a rotator list to ``tests/v2/test_locales.py``. Every pack needs
+   one, but what it has to contain follows from how the pack declares
+   its scope. A pack declaring by *marker regex* (``ru``, ``tr_az``)
+   needs at least one name exercising every alternation branch of every
+   regex it defines — ``test_rotators_cover_every_marker_branch`` fails
+   until each branch is hit. A pack declaring by *codepoint range*
+   (``zh``) has no branches to sweep and drops out of that test, so its
+   rotators have to carry the same weight by hand: the unspaced names
+   the pack must split, one per shape of the vocabulary it ships —
+   single surname, compound surname, and any spelling variant it means
+   to cover.
 #. Keep the non-interference gate green over the shared corpus plus
    your rotators: every name the packed parser parses differently from
    the default must be one your ``DEVIATES`` predicate flags — no
    silent, undeclared side effects on names outside the pack's stated
    scope.
-#. Keep the pack policy-only in 2.0 — ``ru`` and ``tr_az`` both ship
-   an empty :class:`~nameparser.Lexicon`; a pack that wants to carry
-   its own vocabulary is a later conversation.
+#. Decide which layer the vocabulary belongs in, if the pack carries
+   any. Vocabulary that is *self-selecting* — able to match only text of
+   the tradition it came from, the way a hangul surname can only ever
+   match hangul — is default-safe, and belongs in the default lexicon
+   (``nameparser/config/``) rather than in a pack: a pack nobody knows
+   to ask for is vocabulary nobody gets. Vocabulary that *declares a
+   language its script does not* — a Chinese surname list, which
+   silently mangles the Japanese names written in the same characters
+   — belongs in the pack, where asking for it is the declaration.
+   ``ru`` and ``tr_az`` need no vocabulary at all and ship an empty
+   :class:`~nameparser.Lexicon`; ``nameparser/locales/zh.py`` is the
+   template for one that does.
 #. Curate vocabulary conservatively, the same rule as
    :doc:`customize`: when you're unsure whether a word or a marker
    belongs, leave it out.
 
-``nameparser/locales/ru.py`` is the reference implementation to copy
-from. Staged packs in progress are tracked in issues `#271
-<https://github.com/derek73/python-nameparser/issues/271>`_, `#272
-<https://github.com/derek73/python-nameparser/issues/272>`_, and `#146
-<https://github.com/derek73/python-nameparser/issues/146>`_.
+``nameparser/locales/ru.py`` is the reference implementation for a
+policy-only pack, ``nameparser/locales/zh.py`` for one that carries
+vocabulary. Packs still in progress are tracked in issues `#272
+<https://github.com/derek73/python-nameparser/issues/272>`_ (Japanese)
+and `#146 <https://github.com/derek73/python-nameparser/issues/146>`_
+(Vietnamese).

docs/locales.rst

@@ -339,3 +339,45 @@ into ``middle``/``last`` (``"Jane Smith née Jones"``), and with a
 custom suffix delimiter configured, a no-space delimiter group renders
 whole (``"RN/CRNA"``) where 1.x split it (``"RN, CRNA"``) — the role
 assignment is identical, only the rendered string differs.
+
+2.1 adds two more, and unlike most of the 2.0 API these do reach
+``HumanName``: a name written wholly in Han or Hangul is read
+family-first, and an unspaced Korean name is split into surname and
+given name.
+
+.. doctest::
+
+    >>> HumanName("毛 泽东").last
+    '毛'
+    >>> HumanName("김민준").last, HumanName("김민준").first
+    ('김', '민준')
+
+1.4 read the first as ``first="毛"``/``last="泽东"``, and left the
+second whole in ``first``. The Korean one also changes what the name
+*renders* as — ``str(HumanName("김민준"))`` was ``"김민준"`` and is now
+``"민준 김"``, because the split inserts a token boundary that the
+default format then writes given-name-first.
+
+A third shape changes even though nothing splits it. 1.4 routed a lone
+token to ``first`` whatever it was, so an unspaced Chinese or Japanese
+name landed there entire; 2.1 reads it as native-script CJK and puts it
+in ``last`` instead. Nothing is segmented — Han splitting is opt-in
+through a locale pack and ``HumanName`` cannot apply one — but the
+field the whole string arrives in is different:
+
+.. doctest::
+
+    >>> HumanName("毛泽东").last, HumanName("毛泽东").first
+    ('毛泽东', '')
+    >>> HumanName("山田太郎").last, HumanName("山田太郎").first
+    ('山田太郎', '')
+
+Both were ``first`` under 1.4. If you feed unspaced CJK through
+``HumanName`` and read ``first``, that is the change most likely to
+reach you, and it is silent — the string is intact, just in the other
+field.
+
+``Constants`` has no switch for any of this — the v1 configuration
+surface is frozen for 2.x — so the way out is the 2.0 API:
+``Parser(policy=Policy(script_orders={}, segment_scripts=()))``
+restores 1.4's reading of all three shapes.

docs/migrate.rst

@@ -73,6 +73,9 @@ Configuration
 .. autoclass:: nameparser.PatronymicRule
    :members:
 
+.. autoclass:: nameparser.Script
+   :members:
+
 .. _name-order-constants:
 
 Name-order constants
@@ -102,6 +105,21 @@ because only these three orders have defined assignment semantics.
    Family name first, given name *last*, the words between middle —
    e.g. Vietnamese full-name order.
 
+.. py:data:: nameparser.DEFAULT_SCRIPT_ORDERS
+   :value: ((Script.HAN, FAMILY_FIRST), (Script.HANGUL, FAMILY_FIRST))
+
+   The default :attr:`~nameparser.Policy.script_orders` table: a name
+   written wholly in Han or Hangul reads family-first. A matching
+   entry in this table takes precedence over ``name_order``, including
+   a ``name_order`` you set explicitly — ``name_order`` governs only
+   the names no entry matches. The values are drawn from the same
+   three constants above, and the same restriction applies. Build on it for
+   additive customization —
+   ``script_orders=dict(DEFAULT_SCRIPT_ORDERS) | {Script.HAN:
+   GIVEN_FIRST}`` — and pass ``script_orders={}`` to opt out entirely
+   and get the purely positional read back. Latin-script and
+   mixed-script names are never affected either way.
+
 Delimiter defaults
 ^^^^^^^^^^^^^^^^^^
 
@@ -171,6 +189,8 @@ HumanName.config Defaults
    :members:
 .. automodule:: nameparser.config.maiden_markers
    :members:
+.. automodule:: nameparser.config.surnames
+   :members:
 .. automodule:: nameparser.config.capitalization
    :members:
 .. automodule:: nameparser.config.regexes

docs/modules.rst

@@ -1,5 +1,22 @@
 Release Log
 ===========
+* 2.1.0 - Unreleased
+
+    **East Asian name support**
+
+    - Add the Chinese locale pack ``locales.ZH`` — opt-in Han segmentation for unspaced names like ``毛泽东``, with the surname vocabulary it needs. A pack rather than a default because a Chinese surname list corrupts the Japanese names written in the same characters (``高橋一郎`` would split ``高`` + ``橋一郎``); Japanese needs the pluggable segmenter staged in #272. It sets no name order, since native-script Han already reads family-first without it. ``locales.available()`` is now ``('ru', 'tr_az', 'zh')`` (#271)
+    - Add ``Lexicon.surnames``, ``Policy.script_orders``, ``Policy.segment_scripts``, the ``Script`` enum and the ``DEFAULT_SCRIPT_ORDERS`` constant to the public API. This is the first behavior nameparser keys on the script a name is written in; it is allowed only where the script itself settles a convention, never as a proxy for guessing the language (#271)
+    - Add ``AmbiguityKind.SEGMENTATION``, reported when a surname split had a vocabulary-supported alternative: ``"남궁민수"`` is 남궁 + 민수 by the compound surname but 남 + 궁민수 by the single-syllable one, and longest-match had to pick. A name with only one possible split decided nothing and reports nothing (#271)
+
+    **Breaking Changes**
+
+    - Change the pickle compatibility of the 2.0 API's ``Policy`` and ``Lexicon``: the new fields change the field layout their guarded ``__setstate__`` checks, so a pickle written by 2.0.0 raises ``ValueError`` naming the missing fields instead of loading. Re-pickle after upgrading. ``HumanName`` pickles are unaffected — the facade pickles v1-shaped component state, not these objects
+
+    **Behavior Changes**
+
+    - Fix names written wholly in Han or Hangul parsing given-first: native-script CJK now reads family-first by default, through the new ``Policy.script_orders`` table, so ``"毛 泽东"`` gives family ``毛`` where 1.x gave family ``泽东``. A name that is a single unspaced token moves the same way — ``"毛泽东"`` and ``"山田太郎"`` now land in ``family``/``last`` where 1.x put them in ``given``/``first``, with the string itself untouched, which makes it the easiest form of this change to miss — a lone token renders the same whichever field holds it, whereas the spaced form does show up in the output (``str(HumanName("毛 泽东"))`` is now ``"泽东 毛"``). No language detection is involved: Chinese and Japanese both write the family name first in native script, so the script settles the order without anyone having to know which language it is. Latin-script and mixed-script names are never affected, and an explicit comma still wins. **Default-on: changes parse output for wholly-CJK names**, through ``HumanName`` as well as the 2.0 API (closes #271)
+    - Fix unspaced Korean names not splitting: the census surname list now ships as default vocabulary (``Lexicon.surnames``) with hangul segmentation on by default (``Policy.segment_scripts``), so ``"김민준"`` parses family ``김``, given ``민준`` where 1.x returned the whole string as ``first``. Rendering follows the split, so ``str(HumanName("김민준"))`` is now ``"민준 김"`` where 1.x echoed the input back unchanged. Nothing but Korean is written in hangul and its surnames are a closed census set, which is what makes the split safe as a default rather than a pack. **Default-on**, and it reaches ``HumanName`` too. ``Policy(segment_scripts=())`` turns the split off; ``Policy(script_orders={})`` separately restores the positional reading; clearing both restores 2.0 behavior exactly (#271)
+
 * 2.0.0 - July 27, 2026
 
     Two release candidates preceded this release (rc1 on 2026-07-23,