Skip to content

Commit 50f722c

Browse files
derek73claude
andcommitted
Fix the fork example and document the hangul render change (#271)
The SEGMENTATION fork example was false for the shipped vocabulary. 欧 is not in the zh surname list, so 欧阳明 has exactly one possible split and emits nothing -- the stage tests only show otherwise because their synthetic lexicon adds 欧 on purpose. Both prose copies now use 夏侯惇, where 夏侯 wins and 夏 is genuinely listed too, so the example describes a fork the shipped data can actually produce (verified: it emits, and test_locales already pins that reading). The stage tests keep 欧阳明 -- their lexicon is synthetic by design and says so. Also documents the rendered-string change for Korean names, which no doc mentioned: str(HumanName("김민준")) was "김민준" in 1.4 and is "민준 김" now, because segmentation inserts a boundary the default format then writes given-first. Writing that turned up an error in the release log entry from the last commit, fixed here: the claim that the Han case is invisible in the output string holds only for a LONE token. The spaced form reorders, so str(HumanName("毛 泽东")) is "泽东 毛" where 1.4 echoed the input. Both bullets now say which shape they mean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 5ff1131 commit 50f722c

4 files changed

Lines changed: 8 additions & 5 deletions

File tree

docs/migrate.rst

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -353,7 +353,10 @@ given name.
353353
('김', '민준')
354354

355355
1.4 read the first as ``first="毛"``/``last="泽东"``, and left the
356-
second whole in ``first``.
356+
second whole in ``first``. The Korean one also changes what the name
357+
*renders* as — ``str(HumanName("김민준"))`` was ``"김민준"`` and is now
358+
``"민준 김"``, because the split inserts a token boundary that the
359+
default format then writes given-name-first.
357360

358361
A third shape changes even though nothing splits it. 1.4 routed a lone
359362
token to ``first`` whatever it was, so an unspaced Chinese or Japanese

docs/release_log.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ Release Log
1414

1515
**Behavior Changes**
1616

17-
- Fix names written wholly in Han or Hangul parsing given-first: native-script CJK now reads family-first by default, through the new ``Policy.script_orders`` table, so ``"毛 泽东"`` gives family ```` where 1.x gave family ``泽东``. A name that is a single unspaced token moves the same way — ``"毛泽东"`` and ``"山田太郎"`` now land in ``family``/``last`` where 1.x put them in ``given``/``first``, with the string itself untouched, which makes it the easiest form of this change to miss. No language detection is involved: Chinese and Japanese both write the family name first in native script, so the script settles the order without anyone having to know which language it is. Latin-script and mixed-script names are never affected, and an explicit comma still wins. **Default-on: changes parse output for wholly-CJK names**, through ``HumanName`` as well as the 2.0 API (closes #271)
18-
- Fix unspaced Korean names not splitting: the census surname list now ships as default vocabulary (``Lexicon.surnames``) with hangul segmentation on by default (``Policy.segment_scripts``), so ``"김민준"`` parses family ````, given ``민준`` where 1.x returned the whole string as ``first``. Nothing but Korean is written in hangul and its surnames are a closed census set, which is what makes the split safe as a default rather than a pack. **Default-on**, and it reaches ``HumanName`` too. ``Policy(segment_scripts=())`` turns the split off; ``Policy(script_orders={})`` separately restores the positional reading; clearing both restores 2.0 behavior exactly (#271)
17+
- Fix names written wholly in Han or Hangul parsing given-first: native-script CJK now reads family-first by default, through the new ``Policy.script_orders`` table, so ``"毛 泽东"`` gives family ``毛`` where 1.x gave family ``泽东``. A name that is a single unspaced token moves the same way — ``"毛泽东"`` and ``"山田太郎"`` now land in ``family``/``last`` where 1.x put them in ``given``/``first``, with the string itself untouched, which makes it the easiest form of this change to miss — a lone token renders the same whichever field holds it, whereas the spaced form does show up in the output (``str(HumanName("毛 泽东"))`` is now ``"泽东 毛"``). No language detection is involved: Chinese and Japanese both write the family name first in native script, so the script settles the order without anyone having to know which language it is. Latin-script and mixed-script names are never affected, and an explicit comma still wins. **Default-on: changes parse output for wholly-CJK names**, through ``HumanName`` as well as the 2.0 API (closes #271)
18+
- Fix unspaced Korean names not splitting: the census surname list now ships as default vocabulary (``Lexicon.surnames``) with hangul segmentation on by default (``Policy.segment_scripts``), so ``"김민준"`` parses family ````, given ``민준`` where 1.x returned the whole string as ``first``. Rendering follows the split, so ``str(HumanName("김민준"))`` is now ``"민준 김"`` where 1.x echoed the input back unchanged. Nothing but Korean is written in hangul and its surnames are a closed census set, which is what makes the split safe as a default rather than a pack. **Default-on**, and it reaches ``HumanName`` too. ``Policy(segment_scripts=())`` turns the split off; ``Policy(script_orders={})`` separately restores the positional reading; clearing both restores 2.0 behavior exactly (#271)
1919

2020
* 2.0.0 - July 27, 2026
2121

nameparser/_pipeline/_script_segment.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
inserts the missing token boundary by vocabulary: the first token
1313
written wholly in an activated script is matched longest-first against
1414
Lexicon.surnames, and a hit splits it in two. Compound-before-single
15-
("欧阳明" is 欧阳 + , though is itself a surname) falls out of
15+
("夏侯惇" is 夏侯 + , though is itself a surname) falls out of
1616
longest-first. The split makes two sub-slices of the one token,
1717
rewriting nothing -- spans still index the original exactly, so the
1818
anti-#100 invariant holds by construction.

nameparser/_types.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -281,7 +281,7 @@ class AmbiguityKind(StrEnum):
281281
#: the parse is best-effort over the extra segments.
282282
COMMA_STRUCTURE = "comma-structure"
283283
#: An unspaced CJK token had more than one vocabulary-supported
284-
#: surname split ("欧阳明": 欧阳 + was taken, + 阳明 also
284+
#: surname split ("夏侯惇": 夏侯 + was taken, + 侯惇 also
285285
#: matched); the longest surname won, and ``detail`` names both
286286
#: readings. Points at the two tokens the split produced. (#271)
287287
SEGMENTATION = "segmentation"

0 commit comments

Comments
 (0)