Add the kana Script members and the effective_script kana license (#272)

Task 1 of the #272 Japanese-support plan: classification only, no
order/segmentation defaults change. Script gains HIRAGANA (U+3040-
U+309F) and KATAKANA (U+30A0-U+30FF, which includes the prolonged
mark and the U+30FB middle dot). _vocab.effective_script extends
single_script with the kana license from the 2026-07-29 amendment: a
mixed Han/hiragana/katakana token is Japanese and resolves to the
HIRAGANA carrier entry; pure-katakana keeps single_script's own
answer since nothing defaults on it.

Updated two pre-existing pins that this table change broke: the
single_script test asserting kana was unclassified (now KATAKANA),
and test_regex_sync's #271 differential-rule sync check, which now
scopes its expected spans to the HAN/HANGUL scripts #271 actually
governs rather than every _SCRIPT_RANGES entry, since #272 grows the
same table for an unrelated reason.

Quality-review fix-first pass, folded into this same commit: both
classifiers now run on an NFC-normalized copy of the text, never the
raw input. NFD decomposes precomposed katakana onto a base character
plus a combining voiced/semi-voiced sound mark (U+3099/U+309A), which
sit in the HIRAGANA block rather than katakana's -- classifying raw
NFD text could therefore see a pure-katakana token as Han-free but
kana-mixed and wrongly hand it the kana license (HIRAGANA) instead of
declining it (KATAKANA). Separately, NFD decomposes Hangul syllables
onto bare jamo (U+1100-U+11FF), entirely outside the HANGUL range, so
raw NFD Korean input missed the shipped family-first order rule
altogether and silently fell back to the positional default -- a live
gap in #294's shipped behavior, not merely a #272 nicety. Both
directions are demonstrated in tests/v2/pipeline/test_vocab.py by
simulating the pre-fix classifier against NFD input built with
unicodedata.normalize() (never pasted as decomposed literals): the
pre-fix classifier returns None for both an NFD katakana and an NFD
hangul string, and would have returned HIRAGANA (wrongly) for the raw
NFD katakana case under a naive license check. Matching (is_initial,
suffix lookups, etc.) deliberately stays on raw text elsewhere in the
module -- NFD only ever costs a match there, never a wrong one, so
that asymmetry is safe and unchanged. The normalized copy is used for
classification only; token text and rendered spans are exactly what
the caller wrote, still NFD if that's what came in (see the new
test_nfd_korean_input_still_reads_family_first in test_parser.py,
which compares NFC-normalized field values since the rendered family/
given text itself stays NFD).

Also folded in from the same review: reworded the "no supplementary-
plane kana to chase" claim (false -- 311 assigned codepoints exist;
none are worth chasing, since they're archaic/phonetic-extension
forms no modern name uses); added the halfwidth-kana exclusion
rationale and the block-vs-UAX#24 clause to the range-table comment;
corrected "kana-only" to "katakana-only" in effective_script's
docstring (さくらエミ is kana-only and licensed); added a
forward-pointer from single_script to effective_script; moved
_JA_SCRIPTS/_JA_PATTERN up beside _SCRIPT_PATTERNS and changed
_JA_SCRIPTS from a sorted frozenset to a plain table-ordered tuple
(nothing uses membership); made HIRAGANA's Policy docstring
self-contained; folded the temporal-named Script-enum test into
test_script_values_are_the_public_names; and trimmed the
Han/Hangul-duplicate assertions out of the kana single_script test
(renamed test_kana_singles_classify).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: derek73

nameparser/_pipeline/_vocab.py

@@ -9,6 +9,7 @@
 from __future__ import annotations
 
 import re
+import unicodedata
 
 from nameparser._lexicon import Lexicon, _normalize
 from nameparser._policy import Script
@@ -42,17 +43,37 @@
 # 𠮷, U+20BB7), so leaving them out silently mis-orders those names;
 # unassigned gaps inside the span are harmless, since no real name
 # contains an unassigned codepoint. HANGUL: precomposed syllables
-# only -- modern Korean text never writes names as bare jamo. Kana is
-# DELIBERATELY absent: a kana token identifies Japanese, whose
-# conventions are #272's segmenter, not this table's. The ranges
-# below must stay mutually disjoint: single_script returns the FIRST
-# covering entry (dict iteration order), so an overlapping future
-# script (e.g. a ja entry that also covers Han) would make the result
-# order-dependent instead of well-defined.
+# only -- modern Korean text never writes names as bare jamo.
+# HIRAGANA/KATAKANA (#272): the two kana blocks, each in full. There
+# IS a supplementary-plane kana repertoire (Kana Supplement, Kana
+# Extended-A/B, Small Kana Extension, U+1AFF0-U+1B16F, 311 assigned
+# codepoints) but none of it is WORTH chasing the way Han's astral
+# block is: those codepoints are hentaigana and other archaic/
+# phonetic-extension forms no modern Japanese name uses, unlike
+# supplementary Han, which real surnames genuinely need. Halfwidth
+# kana (U+FF65-U+FF9D) is likewise deliberately excluded -- legacy
+# bank/CSV data uses it, but it is a separate normalization problem;
+# Task 2b's separator handling only touches the halfwidth DOT
+# (U+FF65), not the rest of that block. This table classifies by
+# Unicode BLOCK, not the UAX #24 Script property: U+30A0, U+30FB
+# (the middle dot), and U+30FC (the prolonged sound mark) all carry
+# Script=Common under UAX #24, and the combining kana voicing marks
+# U+3099-U+309C are Common/Inherited -- yet every one of them is
+# needed here, and block membership, not the Script property, is
+# what puts them in range. The katakana block's upper end (U+30FF)
+# including the middle dot U+30FB is load-bearing for
+# effective_script's kana license below (see its docstring) -- a
+# later task turns U+30FB into a tokenize-level separator, but until
+# then it classifies as ordinary katakana. The ranges below must stay
+# mutually disjoint: single_script returns the FIRST covering entry
+# (dict iteration order), so an overlapping future script would make
+# the result order-dependent instead of well-defined.
 _SCRIPT_RANGES: dict[Script, tuple[tuple[int, int], ...]] = {
     Script.HAN: ((0x3400, 0x4DBF), (0x4E00, 0x9FFF), (0xF900, 0xFAFF),
                  (0x20000, 0x323AF)),
     Script.HANGUL: ((0xAC00, 0xD7A3),),
+    Script.HIRAGANA: ((0x3040, 0x309F),),
+    Script.KATAKANA: ((0x30A0, 0x30FF),),
 }
 
 # Derived, never hand-written: one character class per script, in the
@@ -65,6 +86,21 @@
     for script, ranges in _SCRIPT_RANGES.items()
 }
 
+#: The Japanese repertoire: the union effective_script's kana license
+#: quantifies over. A tuple in the table's own order (HAN, then
+#: HIRAGANA, then KATAKANA -- HANGUL simply omitted, the relative
+#: order of the rest is unchanged), not a frozenset: nothing consults
+#: membership today, only iterates to build the pattern below, so
+#: there is nothing set-ness would buy; a later task that needs
+#: membership can convert it then.
+_JA_SCRIPTS = (Script.HAN, Script.HIRAGANA, Script.KATAKANA)
+_JA_PATTERN = re.compile(
+    "["
+    + "".join(f"\\U{lo:08x}-\\U{hi:08x}"
+              for s in _JA_SCRIPTS
+              for lo, hi in _SCRIPT_RANGES[s])
+    + "]+")
+
 
 def is_initial(text: str) -> bool:
     """'A.' / 'j.' / bare capital -- v1's is_an_initial."""
@@ -162,18 +198,72 @@ def period_joined_vocab(text: str, lexicon: Lexicon) -> str | None:
     return None
 
 
+def _normalized_for_script(text: str) -> str | None:
+    """The guard AND the NFC normalization single_script and
+    effective_script's license path both need, single-sourced so they
+    cannot drift: None for the two shapes neither ever classifies
+    (empty, and the common all-ASCII Latin token -- skipped before
+    normalizing, since ASCII is already NFC and every _SCRIPT_RANGES
+    entry is non-ASCII regardless), else an NFC-normalized copy.
+
+    NFC, not raw: NFD input decomposes precomposed katakana onto a
+    base character plus a COMBINING mark (U+3099/U+309A, which sit in
+    the HIRAGANA block, not katakana's), so classifying raw NFD text
+    can hand a pure-katakana token the kana license by accident; NFD
+    also decomposes Hangul syllables onto bare jamo (U+1100-U+11FF),
+    entirely outside the HANGUL range, so raw NFD Korean input misses
+    the shipped family-first order rule rather than merely misfiring.
+    Normalizing first fixes both. This is classification-only and
+    read-only: the returned copy is never what gets tokenized, so
+    token text and spans stay exactly what the caller wrote.
+
+    MATCHING (is_initial, suffix lookups, etc.) deliberately stays on
+    raw text elsewhere in this module -- unlike script classification,
+    NFD only ever costs a match there (a suffix word written NFD fails
+    to match its NFC vocabulary entry), never wrong-matches, so the
+    asymmetry is safe: one direction needs a fix, the other doesn't.
+    """
+    if not text or text.isascii():
+        return None
+    return unicodedata.normalize("NFC", text)
+
+
 def single_script(text: str) -> Script | None:
     """The one Script whose ranges cover EVERY char of `text`, else
     None (mixed-script text has no well-defined convention to apply;
-    the caller falls back to the positional default)."""
-    if not text:
-        return None  # the + below needs one char; "" belongs to no script
-    if text.isascii():
-        # every _SCRIPT_RANGES entry is non-ASCII (lowest today is
-        # U+3400): skip the patterns for the overwhelmingly common
-        # Latin token (the _tokenize._ignorable ASCII-floor precedent)
+    the caller falls back to the positional default). Classifies an
+    NFC-normalized copy of `text` -- see _normalized_for_script.
+    Callers wanting the kana-mixed license (a kanji+kana composite
+    resolving to HIRAGANA) want effective_script, not this function."""
+    normalized = _normalized_for_script(text)
+    if normalized is None:
         return None
     for script, pattern in _SCRIPT_PATTERNS.items():
-        if pattern.fullmatch(text):
+        if pattern.fullmatch(normalized):
             return script
     return None
+
+
+def effective_script(text: str) -> Script | None:
+    """single_script, extended by the kana license (#272 amendment):
+    a MIXED token wholly within Han∪hiragana∪katakana is Japanese --
+    it necessarily contains kana (pure Han is not mixed), cannot be
+    Chinese, and is not a foreign transcription (those are
+    katakana-only: マイケル has no kanji, but さくらエミ -- hiragana
+    plus katakana -- is kana-only AND licensed) -- and resolves to the
+    HIRAGANA carrier entry. Pure-katakana stays KATAKANA
+    (single_script's answer): a lone katakana token is predominantly a
+    transcribed foreign name, so nothing defaults on it."""
+    script = single_script(text)
+    if script is not None:
+        return script
+    # normalized is None for both shapes _JA_PATTERN could never match
+    # anyway (empty text, or the all-ASCII text single_script's fast
+    # path already ruled out) -- real work, not a leftover "if text"
+    # guard: unlike the pre-NFC version, None here also covers the
+    # ASCII case, which single_script's own empty check alone would
+    # not.
+    normalized = _normalized_for_script(text)
+    if normalized is not None and _JA_PATTERN.fullmatch(normalized):
+        return Script.HIRAGANA
+    return None

nameparser/_policy.py

@@ -45,6 +45,15 @@ class Script(StrEnum):
     HAN = "han"
     #: Korean Hangul (precomposed syllables). Unambiguously Korean.
     HANGUL = "hangul"
+    #: Japanese hiragana. Never transcribes foreign names, so a mixed
+    #: kanji+kana token (高橋みなみ) is Japanese and resolves HERE --
+    #: this member is the carrier key in script_orders/segment_scripts.
+    HIRAGANA = "hiragana"
+    #: Japanese katakana. A PURE-katakana token is predominantly a
+    #: transcribed foreign name in its original order (マイケル), so
+    #: no default behavior keys on this member; it exists so the
+    #: classifier can name what it deliberately declines.
+    KATAKANA = "katakana"
 
 
 # Order-spec constants (#270). Each reads as its contents because roles

tests/v2/pipeline/test_vocab.py

@@ -1,7 +1,9 @@
+import unicodedata
+
 from nameparser._lexicon import Lexicon
 from nameparser._pipeline._vocab import (
-    _SCRIPT_RANGES, is_initial, is_suffix_lenient, is_suffix_strict,
-    single_script,
+    _SCRIPT_RANGES, effective_script, is_initial, is_suffix_lenient,
+    is_suffix_strict, single_script,
 )
 from nameparser._policy import Script
 
@@ -51,7 +53,10 @@ def test_single_script_requires_every_char_in_one_script() -> None:
     assert single_script("Smith") is None
     assert single_script("毛zedong") is None            # mixed
     assert single_script("毛김") is None                 # mixed CJK
-    assert single_script("イチロー") is None             # kana: not HAN
+    # kana classifies as its own script (#272), not HAN -- was None
+    # before kana had a table entry; single_script's job is telling
+    # kana apart from Han, not lumping the two together
+    assert single_script("イチロー") is Script.KATAKANA
 
 
 def test_single_script_range_edges() -> None:
@@ -82,3 +87,62 @@ def test_no_script_range_reaches_ascii() -> None:
     # it fails here rather than silently going unclassified.
     assert all(lo >= 0x80
                for ranges in _SCRIPT_RANGES.values() for lo, _ in ranges)
+
+
+def test_kana_singles_classify() -> None:
+    assert single_script("みなみ") is Script.HIRAGANA
+    assert single_script("エミ") is Script.KATAKANA
+    assert single_script("ー") is Script.KATAKANA   # prolonged mark, in-block
+
+
+def test_effective_script_kana_license() -> None:
+    # pure single-script tokens pass through unchanged
+    assert effective_script("山田") is Script.HAN
+    assert effective_script("みなみ") is Script.HIRAGANA
+    assert effective_script("マイケル") is Script.KATAKANA
+    # the license: a MIXED token wholly in Han∪kana is Japanese and
+    # resolves to the HIRAGANA carrier entry
+    assert effective_script("高橋みなみ") is Script.HIRAGANA  # kanji+hira
+    assert effective_script("山田エミ") is Script.HIRAGANA    # kanji+kata
+    assert effective_script("さくらエミ") is Script.HIRAGANA  # hira+kata
+    # outside the license: anything beyond the JA repertoire
+    assert effective_script("毛김") is None
+    assert effective_script("山田x") is None
+    # NOT None: U+30FB sits INSIDE the katakana block (verified by
+    # codepoint), so this stays a PURE-katakana token, same as
+    # "マイケル" above -- the license only ever fires on a MIXED
+    # token, and this one isn't mixed. It has no order-default entry
+    # (DEFAULT_SCRIPT_ORDERS carries no KATAKANA key), which is where
+    # "declines the license" actually shows up. A later task turns
+    # U+30FB into a tokenize-level separator, so this string arrives
+    # at effective_script as two tokens instead of one.
+    assert effective_script("マイケル・ジャクソン") is Script.KATAKANA
+    assert effective_script("") is None
+
+
+def test_nfd_katakana_still_classifies_and_declines_the_license() -> None:
+    # Built via normalize(), never pasted as decomposed literals --
+    # NFD "ガガ" is base katakana カ + COMBINING VOICED SOUND MARK
+    # (U+3099) twice; U+3099 sits in the HIRAGANA block, not
+    # katakana's, so classifying raw NFD text would see one char from
+    # each block and either call it mixed (single_script: None) or
+    # wrongly grant the kana license (effective_script: HIRAGANA) for
+    # what is really one pure-katakana token. NFC-normalizing first
+    # (the #272 amendment's NFC decision) recomposes it back to ガガ,
+    # which reads as ordinary katakana either way -- the license
+    # still correctly declines, because this token isn't mixed.
+    nfd = unicodedata.normalize("NFD", "ガガ")
+    assert nfd != "ガガ"  # sanity: confirms the decomposition actually ran
+    assert single_script(nfd) is Script.KATAKANA
+    assert effective_script(nfd) is Script.KATAKANA
+
+
+def test_nfd_hangul_still_classifies() -> None:
+    # NFD decomposes each precomposed syllable onto 2-3 jamo
+    # (U+1100-U+11FF), entirely outside the HANGUL range -- raw NFD
+    # input would silently miss the shipped family-first order rule
+    # rather than merely misclassify. Built via normalize(), not
+    # pasted decomposed literals, same reason as above.
+    nfd = unicodedata.normalize("NFD", "김민준")
+    assert nfd != "김민준"  # sanity: confirms the decomposition actually ran
+    assert single_script(nfd) is Script.HANGUL

tests/v2/test_parser.py

@@ -1,5 +1,6 @@
 import dataclasses
 import pickle
+import unicodedata
 
 import pytest
 
@@ -451,6 +452,21 @@ def test_wholly_cjk_names_read_family_first_by_default() -> None:
     assert parse("毛泽东").family == "毛泽东"
 
 
+def test_nfd_korean_input_still_reads_family_first() -> None:
+    # fix(#271) classification, landed via the #272 NFC-classification
+    # amendment: NFD decomposes each Hangul syllable onto bare jamo,
+    # entirely outside the HANGUL range, so raw NFD input used to miss
+    # script_orders' family-first rule and fall back to the positional
+    # default -- a live gap in #294's shipped behavior until
+    # single_script started classifying an NFC-normalized copy.
+    n = parse(unicodedata.normalize("NFD", "김 민준"))
+    # classification-only: the rendered text is exactly what was
+    # typed (still NFD, spans untouched), so compare NFC-normalized --
+    # the point under test is the ORDER (family first), not encoding
+    assert (unicodedata.normalize("NFC", n.family),
+            unicodedata.normalize("NFC", n.given)) == ("김", "민준")
+
+
 def test_latin_names_are_untouched_by_script_orders() -> None:
     n = parse("John Smith")
     assert (n.given, n.family) == ("John", "Smith")

tests/v2/test_policy.py

@@ -63,6 +63,7 @@ def test_name_order_rejects_plain_string_tuples() -> None:
 
 def test_script_values_are_the_public_names() -> None:
     assert Script.HAN == "han" and Script.HANGUL == "hangul"
+    assert Script.HIRAGANA == "hiragana" and Script.KATAKANA == "katakana"
 
 
 def test_patronymic_rules_coerce_and_reject() -> None:

tests/v2/test_regex_sync.py

@@ -23,6 +23,7 @@
 
 from nameparser.config import regexes as _config
 from nameparser._pipeline import _assign, _post_rules, _tokenize, _vocab
+from nameparser._policy import Script
 from nameparser import _render
 
 
@@ -114,6 +115,9 @@ def test_initial_copies_agree_with_each_other_and_config() -> None:
     ("_tokenize", "_BIDI"): None,       # re_bidi, not a REGEXES key
     # Mirrors _pipeline._state.COMMA_CHARS, not nameparser.config
     ("_render", "_COMMA_CHAR"): None,
+    # #272: derived from _SCRIPT_RANGES itself (like _SCRIPT_PATTERNS),
+    # not hand-copied from anywhere -- no config counterpart to pin.
+    ("_vocab", "_JA_PATTERN"): None,
 }
 
 _MODULES = {"_assign": _assign, "_post_rules": _post_rules,
@@ -176,8 +180,17 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None:
     Han's astral block is out of scope on both sides. The rule omits
     it deliberately -- no corpus name reaches it, see the comment
     there -- so the comparison runs over the BMP spans only, and a new
-    BMP script added to _SCRIPT_RANGES still fails here until the rule
-    covers it.
+    BMP script added to _SCRIPT_RANGES for the SAME #271 behavior
+    (family-first order / hangul segmentation) still fails here until
+    the rule covers it.
+
+    #272 added HIRAGANA/KATAKANA to _SCRIPT_RANGES for an unrelated
+    reason (kana classification for effective_script's license, not
+    the order-flip or segmentation #271's diff rule explains -- no
+    default keys on either kana member), so the comparison is scoped
+    to the #271 scripts by name rather than every table entry; a
+    future script added for #271's own reason still must extend both
+    sides, same as before.
     """
     toml_path = (Path(__file__).parents[2] / "tools" / "differential"
                  / "expected_changes.toml")
@@ -191,8 +204,9 @@ def test_differential_cjk_rule_matches_the_script_ranges() -> None:
         for lo, hi in re.findall(r"\\u([0-9A-Fa-f]{4})-\\u([0-9A-Fa-f]{4})",
                                  matched[0]["name_regex"])}
     expected = {span
-                for spans in _vocab._SCRIPT_RANGES.values()
-                for span in spans if span[1] <= 0xFFFF}
+                for script in (Script.HAN, Script.HANGUL)
+                for span in _vocab._SCRIPT_RANGES[script]
+                if span[1] <= 0xFFFF}
     assert declared == expected, (
         f"{toml_path.name}'s #271 name_regex declares {sorted(declared)}; "
         f"_SCRIPT_RANGES' BMP spans are {sorted(expected)}")