Merge pull request #310 from derek73/v2.1/cjk-honorifics

Recognize spaced CJK postnominal honorifics as suffixes

Author: derek73

docs/release_log.rst

@@ -25,6 +25,7 @@ Release Log
     - Fix names containing 〆 (U+3006, the shime mark that opens Japanese surnames like 〆木 and 〆谷) parsing given-first: the script classifier now counts 〆 as Han, extending the 々 entry the table already carries, and going a step further than it — 々 is Script=Han and merely outside the ideograph blocks, while 〆 is Script=Common — so these names take the East Asian family-first reading like any other wholly-Han name. ``Policy(script_orders={})`` restores the positional reading for these names exactly as for other wholly-Han names (#303)
     - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272)
     - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298)
+    - Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese, with 先生/博士/教授 shared with Japanese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching — which also reaches a glued surname+honorific token, since segmentation splits off the surname first (``김씨`` reads family 김, suffix 씨; ``王先生`` the same under ``locales.ZH``); a glued honorific after a given name (``김민준씨``) and glued kana (``山田太郎様``) stay out of reach, tracked as #308 (closes #307)
     - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
     - Fix the Ukrainian conjunction ``й`` not joining the pieces around it: it is the euphonic alternate of ``і``, the two chosen by the surrounding vowel and consonant rather than by meaning (``"Олесь і Олена"`` but ``"Марія й Петро"``), so real Ukrainian data carries both spellings and shipping only ``і`` recognized just one of them. ``"Олесь й Олена Коваленки"`` now gives given ``"Олесь й Олена"`` where the ``й`` previously landed in ``middle``. Same treatment as the ``и``/``і`` entries added in 2.0.0, single-letter carve-out included: the conjunction joins only once the name has enough pieces, and a punctuated initial still wins, so ``"Й. Сліпий"`` is unaffected. Raised in a comment on #267
 

docs/usage.rst

@@ -295,6 +295,16 @@ Chinese dot rescues the source order. And a comma disables the script
 behaviors entirely, on the reasoning ``name_order`` already follows: whoever
 wrote the comma has already said where the family name ends.
 
+Honorifics and degrees follow a CJK name, and the spaced forms are
+recognized as suffixes — ``王小明 先生`` reads family 王小明 with 先生
+in ``suffix``, and Korean's standardly-spaced 씨 routes the same way.
+A glued honorific is reached in exactly one shape — surname +
+honorific, where segmentation splits off the surname and the
+honorific is what remains: 김씨 reads family 김 with suffix 씨, and
+王先生 the same way under the zh pack. A glued honorific after a
+given name (김민준씨, 王小明先生) and glued kana (山田太郎様, 田中さん)
+stay part of the name.
+
 A division the parser had to choose is reported rather than hidden.
 When an unspaced name has more than one vocabulary-supported split —
 ``남궁민수`` is 남궁 + 민수 by the two-syllable surname but 남 + 궁민수

nameparser/config/suffixes.py

@@ -36,6 +36,45 @@
     'ז״ל',      # same, U+05F4 gershayim
     'שליט"א',   # honorific for a living rabbi, ASCII quote
     'שליט״א',   # same, U+05F4 gershayim
+
+    # #307: CJK postnominal honorifics and degrees, whole-token
+    # matched. That reaches the spaced forms directly, and glued
+    # forms in exactly ONE shape: surname+honorific (김씨; 王先生
+    # under the zh pack), where segmentation splits off the surname
+    # before suffix classification and the honorific is what remains.
+    # A glued honorific after a GIVEN name (김민준씨, 王小明先生) and
+    # glued kana (山田太郎様; 田中さん additionally ships no さん
+    # entry at all) stay out of reach, tracked as #308.
+    # Self-selecting like the Korean surnames: a Han or hangul entry
+    # can only ever match CJK text. Vetting per the мл/ст standard
+    # above -- three entries worth naming:
+    # - 氏: a rare Japanese surname reading exists, but a bare
+    #   trailing 氏 after a name is the news-style honorific, and a
+    #   氏-surnamed person writes it FIRST.
+    # - 양/군: 양 is also a top-tier surname (Yang) -- but a surname
+    #   LEADS, so the trailing-only suffix gate never sees it there
+    #   (양 미선 keeps family 양); as single-syllable GIVEN names in
+    #   final position both are vanishingly rare in practice.
+    # - 博士: an attested Japanese given name (ひろし, Hiroshi) --
+    #   the doctorate reading vastly dominates the spaced trailing
+    #   position this set matches, and a glued 田中博士 is untouched.
+    # Bare hangul 선생/교수 are deliberately absent (only the -님
+    # honorific forms ship): the bare forms read as common nouns as
+    # readily as address terms. Further candidates (여사, 太太, 殿)
+    # wait on the same case-by-case argument.
+    '씨',        # ko Mr./Ms. -- standardly spaced in Korean orthography
+    '박사',      # ko doctorate holder ("Dr.")
+    '선생님',    # ko teacher/respected elder
+    '교수님',    # ko professor (honorific form)
+    '군',        # ko young man ("Master")
+    '양',        # ko young woman ("Miss")
+    '先生',      # zh Mr. / ja teacher-master -- honorific in both
+    '女士',      # zh Ms./Madam
+    '小姐',      # zh Miss
+    '博士',      # zh+ja doctorate holder, shared Han
+    '教授',      # zh+ja professor, shared Han
+    '様',        # ja formal Mr./Ms. (the mail-addressing honorific)
+    '氏',        # ja news-style Mr. (田中氏)
 }
 """
 

tests/v2/cases.py

@@ -707,4 +707,80 @@ def __post_init__(self) -> None:
          notes="one classified neighbor is not enough: the guard "
                "requires both, so the undivided dot remains part of "
                "the word -- declining, not deciding"),
+    Case("zh_honorific_suffix_spaced", "王小明 先生",
+         {"family": "王小明", "suffix": "先生"},
+         classification="fix(#307)",
+         notes="CJK honorifics FOLLOW the name; a spaced 先生 (Mr.) is "
+               "a suffix, and recognizing it must come before the "
+               "family-first order hands it a role -- unrecognized it "
+               "read as the GIVEN name under the 2.1 defaults"),
+    Case("ko_honorific_ssi", "김민준 씨",
+         {"family": "김", "given": "민준", "suffix": "씨"},
+         classification="fix(#307)",
+         notes="Korean orthography standardly SPACES 씨, so the "
+               "whole-token suffix machinery reaches it; the name "
+               "still segments (suffix classification runs after the "
+               "script_segment stage, which only ever saw 김민준)"),
+    Case("ko_degree_baksa", "김민준 박사",
+         {"family": "김", "given": "민준", "suffix": "박사"},
+         classification="fix(#307)",
+         notes="박사 (doctorate) is the ko analogue of a trailing "
+               "PhD: fix(suffix-routing)'s two-token shape, one "
+               "script over"),
+    Case("ja_sama_spaced", "田中 太郎 様",
+         {"family": "田中", "given": "太郎", "suffix": "様"},
+         classification="fix(#307)",
+         notes="the spaced 様 of forms and databases; the glued "
+               "mail-addressing form 山田太郎様 is #308's mechanism, "
+               "out of reach of whole-token matching"),
+    Case("ko_honorific_glued_via_segmentation", "김씨",
+         {"family": "김", "suffix": "씨"},
+         classification="fix(#307)",
+         notes="glued hangul is reached ONLY in the surname+honorific "
+               "shape: default segmentation splits off the surname, "
+               "and the honorific is what remains -- a partial "
+               "delivery of #308 that falls out of stage order. A "
+               "glued honorific after a GIVEN name (김민준씨, the row "
+               "below) stays out of reach"),
+    Case("ko_honorific_after_comma", "김민준, 씨",
+         {"family": "김민준", "suffix": "씨"},
+         classification="fix(#307)",
+         notes="the post-comma lenient gate admits the honorific too; "
+               "the comma disables segmentation per the comma "
+               "doctrine, so 김민준 stays whole"),    Case("ko_honorific_glued_given_stays", "김민준씨",
+         {"family": "김", "given": "민준씨"},
+         notes="the boundary of the glued reach: segmentation splits "
+               "off 김 and the REMAINDER is 민준씨, not a listed "
+               "honorific token -- the common full-name glued shape "
+               "is #308's mechanism, pinned here so the docs' scoped "
+               "claim stays true"),
+    Case("zh_honorific_glued_surname", "王先生",
+         {"family": "王", "suffix": "先生"},
+         locale="zh",
+         classification="fix(#307)",
+         notes="the Han twin of 김씨: the zh pack's segmentation "
+               "splits off the surname and the remaining 先生 is the "
+               "honorific token"),
+    Case("zh_honorific_glued_given_stays", "王小明先生",
+         {"family": "王", "given": "小明先生"},
+         locale="zh",
+         notes="the Han boundary twin: 小明先生 is the remainder, not "
+               "an honorific token -- glued full names stay #308"),
+    Case("ko_suffix_matching_is_whole_token", "김지양",
+         {"family": "김", "given": "지양"},
+         notes="지양 ENDS with the honorific 양 but is a given name: "
+               "suffix matching is whole-token, never endswith -- the "
+               "pin the differential rule's anchor mirrors at the "
+               "name-string level"),
+    Case("ko_surname_yang_leads", "양 미선",
+         {"family": "양", "given": "미선"},
+         notes="양 is both a top-tier surname (Yang) and a shipped "
+               "honorific: position decides, and a surname LEADS -- "
+               "the trailing-only suffix gate never sees it here"),
+    Case("ko_honorific_stack", "김민준 박사 씨",
+         {"family": "김", "given": "민준", "suffix": "박사, 씨"},
+         classification="fix(#307)",
+         notes="a trailing RUN of honorifics peels whole, like "
+               "'Smith PhD MD' -- the multi-suffix loop the peel "
+               "shares with Latin suffixes"),
 )

tests/v2/test_regex_sync.py

@@ -334,3 +334,33 @@ def test_cjk_corpus_matches_the_case_table() -> None:
     assert checked_in == module.selected_names(), (
         "corpus_cjk.jsonl is stale: regenerate with "
         "`uv run python tools/differential/build_cjk_corpus.py`")
+
+
+def test_differential_honorific_rule_matches_the_suffix_vocabulary() -> None:
+    """The fix(cjk-honorific-suffix) rule's alternation is a hand copy
+    of SUFFIX_NOT_ACRONYMS' CJK entries (#307) -- the toml cannot
+    import them. The expected set is DERIVED from the config by script
+    membership (a classified codepoint anywhere in the entry), so
+    adding a CJK honorific without widening the rule -- or widening
+    the rule with something the vocabulary does not ship -- fails
+    here. The span-bearing pins above skip this rule on purpose: its
+    trigger is the alternation, not a character class.
+    """
+    from nameparser.config.suffixes import SUFFIX_NOT_ACRONYMS
+
+    toml_path = (Path(__file__).parents[2] / "tools" / "differential"
+                 / "expected_changes.toml")
+    rules = tomllib.loads(toml_path.read_text())["change"]
+    matched = [r for r in rules if "cjk-honorific-suffix" in r["issue"]]
+    assert len(matched) == 1
+    regex = matched[0]["name_regex"]
+    prefix = "(?:^| )(?:"
+    assert regex.startswith(prefix) and regex.endswith(")$"), (
+        "the honorific rule's shape changed; update this parser")
+    declared = set(regex[len(prefix):-2].split("|"))
+    has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES)
+    expected = {entry for entry in SUFFIX_NOT_ACRONYMS
+                if has_classified(entry)}
+    assert declared == expected, (
+        f"rule declares {sorted(declared)}; the config's CJK suffix "
+        f"entries are {sorted(expected)}")

tools/differential/corpus_cjk.jsonl

@@ -22,6 +22,10 @@
 "毛 김"
 "毛泽东"
 "王·Smith"
+"王先生"
+"王小明 先生"
+"王小明先生"
+"田中 太郎 様"
 "田中『ハナ』花子"
 "諸葛亮"
 "阿明"
@@ -32,7 +36,15 @@
 "高橋一郎"
 "김 민준"
 "김민준"
+"김민준 박사"
+"김민준 박사 씨"
+"김민준 씨"
+"김민준, 씨"
+"김민준씨"
+"김씨"
+"김지양"
 "남궁"
 "남궁민수"
 "남궁민수, 지훈"
 "마이클·잭슨"
+"양 미선"

tools/differential/expected_changes.toml

@@ -177,6 +177,35 @@ issue = "fix(cjk-comma-compound) comma routing compounds with the CJK order flip
 name_regex = "(?s)(?=.*,)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])"
 fields = ["first", "middle", "last", "title", "suffix"]
 
+[[change]]
+issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip"
+# '王小明 先生', '김민준 씨', '田中 太郎 様': #307 ships the spaced CJK
+# honorifics as suffix vocabulary, so a trailing 先生/씨/様 moves to
+# `suffix` where 1.4 read it as a name part -- and the remaining
+# wholly-CJK name simultaneously takes the family-first flip, so a
+# three-token name moves `middle` too, past the fields-only
+# suffix-routing rule. Keyed on the honorific alternation itself
+# (the maiden-marker rule's pattern): the trigger tokens are wholly
+# CJK characters, so the rule matches only a name whose TRAILING
+# token is a listed honorific -- a mostly-Latin name with one
+# ('Wang Xiaoming 先生') is inside its shadow, accepted because the
+# recognized honorific is the diff's cause there too, and the
+# alternation is a hand copy of SUFFIX_NOT_ACRONYMS' CJK entries --
+# pinned by tests/v2/test_regex_sync.py, which derives the expected
+# set from the config by script membership. Anchored to a WHOLE
+# trailing token ((?:^| )...$), judged on the NAME STRING: without
+# it, any name string merely ENDING in 양/군 -- the glued given name
+# 김지양, the spaced 김 지양 -- would match and could have a real
+# regression absorbed as intentional (the case table pins the
+# parser-side twin: suffix matching is whole-token, never endswith).
+# Glued surname+honorific diffs (김씨 -- reached via segmentation)
+# fall to the fields-only suffix-routing rule by their
+# {first, last, suffix} shape (v1 held the whole token in first, so
+# last moves too), which is field-honest even though that rule's
+# prose describes the two-token Latin case.
+name_regex = "(?:^| )(?:씨|박사|선생님|교수님|군|양|先生|女士|小姐|博士|教授|様|氏)$"
+fields = ["first", "middle", "last", "suffix"]
+
 [[change]]
 issue = "feat(#269) non-Latin titles/conjunctions recognized"
 # 'г-н Иван Петров' (Cyrillic title), 'Хосе И Мария Сантос' (Cyrillic