Merge pull request #297 from derek73/v2.1/ja-segmenter

Japanese names: the kana license by default and a pluggable segmenter (#272)

Author: derek73

.github/workflows/python-package.yml

@@ -61,3 +61,40 @@ jobs:
         token: ${{ secrets.CODECOV_TOKEN }}
         files: ./coverage.xml
         fail_ci_if_error: false
+
+  # The optional nameparser[ja] segmenter (#272). Its own job, not a
+  # matrix entry: the matrix above must keep proving that the package
+  # installs and passes with NO third-party dependency at all, which is
+  # the promise the extra is carved out of. Everything the extra adds
+  # is under tests/v2/ (the ja pack's integration tests, which skip
+  # without it).
+  ja-extra:
+    runs-on: ubuntu-latest
+    # 3.14, not the matrix's newest: namedivider-python pulls lightgbm,
+    # which pulls scipy/numpy, and those have no 3.15 wheels yet (the
+    # sdist build fails). 3.15 is pre-release and this matrix is ahead
+    # of the ecosystem; revisit when it goes final.
+    env:
+      UV_PYTHON: "3.14"
+    steps:
+    - uses: actions/checkout@v7
+    - name: Set up Python 3.14
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.14"
+    - name: Install uv
+      uses: astral-sh/setup-uv@v6
+      with:
+        enable-cache: true
+    - name: Install dependencies
+      run: uv sync --group dev --extra ja
+    - name: Run Tests
+      # The import check first: every test the extra adds is guarded by
+      # a find_spec("namedivider") skipif, so a job where the extra
+      # silently failed to install is a job where all of them SKIP and
+      # the step still passes green. This line is what makes that
+      # failure loud -- without it the job proves nothing on the day it
+      # matters.
+      run: |
+        uv run python -c "import namedivider"
+        uv run pytest tests/v2/ -q

AGENTS.md

@@ -146,7 +146,11 @@ Honest ambiguity
 Parsing never raises. Pass in a string that doesn't look like a name
 at all, and you get back a :class:`~nameparser.ParsedName` with empty
 fields, not an exception. The parser's job is to make a reasonable
-call on real-world text, not to reject it.
+call on real-world text, not to reject it. The single exception is
+code you supplied yourself: a :data:`~nameparser.Segmenter` passed to
+``Parser(segmenter=...)`` runs inside the parse, and its own
+exceptions propagate rather than being swallowed — a failure there is
+a bug in your callable, not a fact about the name.
 
 Some calls are irreducibly ambiguous — both readings are legitimate,
 and no amount of rule-tuning resolves them without breaking some other

docs/concepts.rst

@@ -307,12 +307,12 @@ East Asian defaults, and turning them off
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Two defaults key on the *script* a name is written in rather than on
-anything you set: a name written wholly in Han or Hangul is assigned
-family-first (``script_orders``), and an unspaced hangul name is split
-into surname and given name against the shipped Korean census list
-(``segment_scripts``). :ref:`east-asian-names` explains the naming
-conventions both rest on — this section is how to switch them off,
-which you can do separately:
+anything you set: a name written wholly in Han or Hangul — or one
+mixing kanji with kana — is assigned family-first (``script_orders``),
+and an unspaced hangul name is split into surname and given name
+against the shipped Korean census list (``segment_scripts``).
+:ref:`east-asian-names` explains the naming conventions both rest on —
+this section is how to switch them off, which you can do separately:
 
 .. doctest::
 
@@ -345,6 +345,18 @@ Chinese surnames are deliberately absent from that default set,
 because splitting Han text requires knowing Chinese from Japanese;
 :doc:`locales` covers the opt-in ``zh`` pack that supplies them.
 
+The Japanese behaviors ride these same two fields, so they need no
+switches of their own: ``script_orders={}`` clears the kana-licensed
+entry along with the Han and Hangul ones, and ``segment_scripts=()``
+deactivates every script at once, which also stops a parser consulting
+whatever segmenter it was given. The segmenter has an off-switch as
+well — ``Parser(segmenter=None)``, which is the default; see
+:ref:`segmenter-contract` for what one is expected to do with text it
+does not handle. One Japanese behavior is not a policy field at all:
+the katakana middle dot ・ separates tokens the way a space does,
+decided in tokenization, so it applies however these two fields are
+set.
+
 .. note::
 
    Both fields are annotated with their canonical *storage* type

docs/customize.rst

@@ -61,17 +61,18 @@ background (covered fully under :ref:`east-asian-names` in
 :doc:`usage`): Chinese, Japanese, and Korean names put the family name
 first in native script and are usually written with no space between
 the parts. Both defaults follow from facts the script alone
-establishes. A name written wholly in Han or Hangul is assigned
+establishes. A name written wholly in Han or Hangul — or one mixing
+kanji with kana, a combination only Japanese produces — is assigned
 family-first, because every language written in those scripts orders
-names that way — no language guess is involved. An unspaced hangul
+names that way; no language guess is involved. An unspaced hangul
 name is additionally split into surname and given name, because hangul
 is written by nothing but Korean and Korean surnames are a closed
 census set that ships as default vocabulary. Splitting an unspaced
 *Han* name is the one behavior the script cannot license — the same
 characters could be a Chinese or a Japanese name, and a Chinese
 surname list splits Japanese names in the wrong place — so it is not a
-default: the ``zh`` pack below turns it on when you can declare the
-data Chinese.
+default: the ``zh`` and ``ja`` packs below turn it on for data whose
+language you can declare.
 
 A pack is for something different: a *structural* rule, like reordering
 a patronymic, that vocabulary alone can't express.
@@ -109,7 +110,7 @@ takes:
 .. doctest::
 
     >>> locales.available()
-    ('ru', 'tr_az', 'zh')
+    ('ja', 'ru', 'tr_az', 'zh')
     >>> locales.get("ru") is locales.RU
     True
 
@@ -123,6 +124,14 @@ parsing, equivalent to ``parser_for(locales.get("ru"))``.
 
    * - Code
      - Turns on
+   * - ``ja``
+     - Japanese segmentation — activates division for unspaced
+       Japanese names, which needs a segmenter to act: install
+       ``nameparser[ja]`` and pass
+       ``parser_for(locales.JA, segmenter=locales.ja_segmenter())``
+       (``山田太郎`` → family ``山田``, given ``太郎``). The pack ships
+       no surname list, because no list divides a kanji name, and no
+       order: a Japanese name already reads family-first by default.
    * - ``ru``
      - East Slavic patronymic order — detects a formal
        given/patronymic/family shape (Cyrillic and transliterated
@@ -139,9 +148,9 @@ parsing, equivalent to ``parser_for(locales.get("ru"))``.
        name order: native-script Han already reads family-first
        without a pack.
 
-``ru`` and ``tr_az`` are policy-only — they carry no vocabulary of
-their own. ``zh`` is both halves at once: a surname list, plus the one
-policy field that turns segmentation on for the script it covers. See
+``ja``, ``ru`` and ``tr_az`` are policy-only — they carry no vocabulary
+of their own. ``zh`` is both halves at once: a surname list, plus the
+one policy field that turns segmentation on for the script it covers. See
 :doc:`concepts` for how that split (language vocabulary vs. behavior)
 is drawn, and `Contributing a pack to nameparser`_ for which half a
 new naming rule belongs in.
@@ -167,6 +176,47 @@ new naming rule belongs in.
    mixes traditions, parse the subsets separately with different
    parsers rather than enabling a pack over all of it.
 
+.. _segmenter-contract:
+
+Segmenters
+-----------
+
+``ja`` is policy-only in a second sense: it turns division on for
+Japanese text without supplying anything to divide with. That job goes
+to a **segmenter**, which is passed to
+:func:`~nameparser.parser_for` rather than carried by the pack — a
+:class:`~nameparser.Locale` is pure data, and a third-party callable is
+neither pure nor data. :func:`~nameparser.locales.ja_segmenter` is the
+shipped one; writing your own is worth it for any script whose
+divisions you know better than a surname list does.
+
+A :data:`~nameparser.Segmenter` is any callable taking a token's text
+and returning a :class:`~nameparser.Segmentation` — the interior
+offsets to cut at, plus how confident you are — or ``None`` to decline,
+leaving the token whole. Declining is the load-bearing half of the
+contract, because ``segment_scripts`` unions across packs: your
+segmenter is offered every token of every activated script, not only
+the ones its own pack turned on. Recognize the text you can actually
+read and return ``None`` for the rest, rather than answering for a
+script you never meant to handle. Exceptions are the one thing that
+does not stay inside the parse — a segmenter is your code, so its
+errors propagate out of ``parse()`` instead of being absorbed as
+content errors.
+
+The pack half of that arrangement carries no words at all, which makes
+it the shortest kind of pack there is:
+
+.. doctest::
+
+    >>> from nameparser import Lexicon, Locale, PolicyPatch, Script
+    >>> mine = Locale(code="myscript", lexicon=Lexicon.empty(),
+    ...               policy=PolicyPatch(
+    ...                   segment_scripts=frozenset({Script.HAN})))
+
+``nameparser/locales/ja.py`` is the shipped example of exactly that
+shape: activation is the pack's entire contribution, and a pack
+applied without a segmenter simply divides nothing.
+
 Creating your own Locale
 -------------------------
 
@@ -237,11 +287,14 @@ by ``tests/v2/test_locales.py``:
    needs at least one name exercising every alternation branch of every
    regex it defines — ``test_rotators_cover_every_marker_branch`` fails
    until each branch is hit. A pack declaring by *codepoint range*
-   (``zh``) has no branches to sweep and drops out of that test, so its
-   rotators have to carry the same weight by hand: the unspaced names
-   the pack must split, one per shape of the vocabulary it ships —
-   single surname, compound surname, and any spelling variant it means
-   to cover.
+   (``zh``, ``ja``) has no branches to sweep and drops out of that
+   test, so its rotators have to carry the same weight by hand: the
+   unspaced names the pack must split, one per shape of the vocabulary
+   it ships — single surname, compound surname, and any spelling
+   variant it means to cover. A pack that ships no vocabulary lists
+   the shapes its *segmenter* must divide instead, and marks the
+   rotator tests to skip when the optional dependency is absent, so
+   the contract tests still run everywhere.
 #. Keep the non-interference gate green over the shared corpus plus
    your rotators: every name the packed parser parses differently from
    the default must be one your ``DEVIATES`` predicate flags — no
@@ -256,16 +309,17 @@ by ``tests/v2/test_locales.py``:
    language its script does not* — a Chinese surname list, which
    silently mangles the Japanese names written in the same characters
    — belongs in the pack, where asking for it is the declaration.
-   ``ru`` and ``tr_az`` need no vocabulary at all and ship an empty
-   :class:`~nameparser.Lexicon`; ``nameparser/locales/zh.py`` is the
-   template for one that does.
+   ``ja``, ``ru`` and ``tr_az`` need no vocabulary at all and ship an
+   empty :class:`~nameparser.Lexicon`; ``nameparser/locales/zh.py`` is
+   the template for one that does.
 #. Curate vocabulary conservatively, the same rule as
    :doc:`customize`: when you're unsure whether a word or a marker
    belongs, leave it out.
 
 ``nameparser/locales/ru.py`` is the reference implementation for a
 policy-only pack, ``nameparser/locales/zh.py`` for one that carries
-vocabulary. Packs still in progress are tracked in issues `#272
-<https://github.com/derek73/python-nameparser/issues/272>`_ (Japanese)
-and `#146 <https://github.com/derek73/python-nameparser/issues/146>`_
+vocabulary, and ``nameparser/locales/ja.py`` for one whose whole
+contribution is turning a stage on. Packs still in progress are
+tracked in issue `#146
+<https://github.com/derek73/python-nameparser/issues/146>`_
 (Vietnamese).

docs/locales.rst

@@ -340,10 +340,11 @@ custom suffix delimiter configured, a no-space delimiter group renders
 whole (``"RN/CRNA"``) where 1.x split it (``"RN, CRNA"``) — the role
 assignment is identical, only the rendered string differs.
 
-2.1 adds two more, and unlike most of the 2.0 API these do reach
-``HumanName``: a name written wholly in Han or Hangul is read
-family-first, and an unspaced Korean name is split into surname and
-given name.
+2.1 adds three more, and unlike most of the 2.0 API these do reach
+``HumanName``: a name written in East Asian script is read
+family-first, an unspaced Korean name is split into surname and given
+name, and the katakana middle dot separates tokens the way a space
+does.
 
 .. doctest::
 
@@ -377,7 +378,58 @@ Both were ``first`` under 1.4. If you feed unspaced CJK through
 reach you, and it is silent — the string is intact, just in the other
 field.
 
+Japanese kana carries the same order rule, which widens both shapes
+past the wholly-Han text described above. A name that mixes kanji with
+hiragana or katakana is read family-first, and a lone kana-bearing
+token moves from ``first`` to ``last`` exactly as ``毛泽东`` does:
+
+.. doctest::
+
+    >>> HumanName("高橋 みなみ").last, HumanName("高橋 みなみ").first
+    ('高橋', 'みなみ')
+    >>> HumanName("山田 エミ").last
+    '山田'
+    >>> HumanName("高橋みなみ").last, HumanName("高橋みなみ").first
+    ('高橋みなみ', '')
+
+1.4 read the spaced ones as ``first="高橋"``/``last="みなみ"`` and
+``first="山田"``/``last="エミ"``, and put the unspaced one whole in
+``first``. The spaced shapes change what the name renders as too:
+``str(HumanName("高橋 みなみ"))`` was ``"高橋 みなみ"`` and is now
+``"みなみ 高橋"``. A name written *wholly* in katakana is deliberately
+left alone — it is usually a transcribed foreign name already in
+given-first order — so ``HumanName("マイケル ジャクソン")`` reads
+``first="マイケル"``/``last="ジャクソン"`` on both versions.
+
+One more shape changes for a different reason: the katakana middle dot
+``・``, which divides the parts of such a transcription, is now a token
+separator rather than an ordinary character. 1.4 saw one token and put
+it in ``first``; 2.1 sees two:
+
+.. doctest::
+
+    >>> HumanName("マイケル・ジャクソン").first
+    'マイケル'
+    >>> HumanName("マイケル・ジャクソン").last
+    'ジャクソン'
+    >>> HumanName("高橋・一郎").last, HumanName("高橋・一郎").first
+    ('高橋', '一郎')
+
+The two divide the same way and land in opposite fields, because the
+katakana pair keeps its source order while the kanji pair takes the
+family-first rule. Separating also changes the rendered string, the
+same way the Korean split does: the dot comes back as a space, so
+``str(HumanName("マイケル・ジャクソン"))`` was ``"マイケル・ジャクソン"``
+and is now ``"マイケル ジャクソン"``. That reaches delimited content
+too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` was
+``"マイケル・ジャクソン"`` under 1.4 and is ``"マイケル ジャクソン"``
+now.
+
 ``Constants`` has no switch for any of this — the v1 configuration
 surface is frozen for 2.x — so the way out is the 2.0 API:
 ``Parser(policy=Policy(script_orders={}, segment_scripts=()))``
-restores 1.4's reading of all three shapes.
+restores 1.4's reading of every shape above that turns on order or
+splitting. The middle dot is the one exception: it is decided in
+tokenization rather than by policy, so a name written with one still
+divides at the dot, and still renders with a space, whatever those two
+fields are set to.

docs/migrate.rst

@@ -14,6 +14,25 @@ Parsing
 
 .. autofunction:: nameparser.parser_for
 
+.. autoclass:: nameparser.Segmentation
+   :members:
+
+.. py:data:: nameparser.Segmenter
+   :value: Callable[[str], Segmentation | None]
+
+   The type of the optional ``Parser(segmenter=...)`` hook: a callable
+   given one token's text, returning a
+   :class:`~nameparser.Segmentation` that divides it, or ``None`` to
+   decline and leave it whole. An alias, not a class — any callable of
+   that shape qualifies, and nothing needs to be subclassed or
+   registered. It is consulted only for tokens whose script is listed
+   in :attr:`Policy.segment_scripts
+   <nameparser.Policy.segment_scripts>`, and only where the surname
+   vocabulary declined first; see :ref:`segmenter-contract` for what a
+   segmenter owes its caller.
+   :func:`~nameparser.locales.ja_segmenter` is the shipped
+   implementation.
+
 Results
 ~~~~~~~
 
@@ -106,10 +125,15 @@ because only these three orders have defined assignment semantics.
    e.g. Vietnamese full-name order.
 
 .. py:data:: nameparser.DEFAULT_SCRIPT_ORDERS
-   :value: ((Script.HAN, FAMILY_FIRST), (Script.HANGUL, FAMILY_FIRST))
+   :value: ((Script.HAN, FAMILY_FIRST), (Script.HANGUL, FAMILY_FIRST), (Script.HIRAGANA, FAMILY_FIRST))
 
    The default :attr:`~nameparser.Policy.script_orders` table: a name
-   written wholly in Han or Hangul reads family-first. A matching
+   written wholly in Han or Hangul reads family-first, and so does a
+   Japanese name mixing kanji with kana, which resolves to the
+   ``HIRAGANA`` entry whichever of the two syllabaries it actually
+   uses — that member is the license's carrier key, not a claim about
+   the characters present. A name written wholly in katakana has no
+   entry on purpose and stays positional. A matching
    entry in this table takes precedence over ``name_order``, including
    a ``name_order`` you set explicitly — ``name_order`` governs only
    the names no entry matches. The values are drawn from the same
@@ -147,6 +171,8 @@ Locales
 .. automodule:: nameparser.locales
    :members: get, available
 
+.. autofunction:: nameparser.locales.ja_segmenter
+
 1.x compatibility layer
 ------------------------