Say what the peel actually does, in prose (#308)

Two overclaims in the user docs and four stale notes in the tests.

usage.rst said the honorific is split off "before anything else reads
the name". Comma structure and interpunct detection both run first and
both gate the peel: "田中さん, 太郎" keeps the whole 田中さん as the
family name, pinned by test_family_comma_skips_the_peel and documented
nowhere. Said now, next to the comma rule it belongs to.

Both usage.rst and the release log promised that the spaced spelling
"composes the same way" and warned that a spaced family-name-only
address would start dividing. The previous commit that narrowed the
carve-out makes both false: spaced forms are unaffected, and the
release log's ja-pack paragraph now says which spelling divides and
which does not, and offers the spaced spelling as the opt-out it now
is.

The case-table notes: the lone-씨 row pointed at the length cap when
the peel-site scan is what holds it, ja_sama_spaced still forward-
pointed to work that has landed, the lone-さん row named its twin by
position and got the wrong row, and test_lexicon's rationale credited
an ambiguous subtraction that happens only in the v1 shim.

Author: derek73

docs/release_log.rst

@@ -27,7 +27,7 @@ Release Log
     - Fix the katakana middle dot ``・`` (U+30FB, and its halfwidth twin U+FF65) being read as part of a name rather than as the divider it is: it now separates tokens exactly as a space does. A transcribed foreign name therefore divides into its parts and, being wholly katakana, keeps its source order — ``"マイケル・ジャクソン"`` gives given ``マイケル``, family ``ジャクソン``, where 1.x left the whole string in ``first`` — while a kanji pair written the same way takes the family-first rule (``"高橋・一郎"`` → family ``高橋``). Native Japanese names never contain this character and no other script's names use it, so the separation is unconditional, which also means it is **not** covered by the two policy opt-outs. Rendering follows, the way the Korean split's does: the dot comes back as a space, so ``str(HumanName("マイケル・ジャクソン"))`` is now ``"マイケル ジャクソン"``, and that reaches delimited content too — the nickname in ``"山田 太郎 (マイケル・ジャクソン)"`` renders ``"マイケル ジャクソン"``. The Chinese interpunct ``·`` is not an unconditional separator like these two: U+00B7 is also the Catalan punt volat and appears inside legitimate names (``Gal·la``), so it divides only between classified-script characters — the transcription treatment it marks is #298's (#272)
     - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298)
     - Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese, with 先生/博士/教授 shared with Japanese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching, which also reaches a glued surname+honorific token, since segmentation splits off the surname first (``김씨`` reads family 김, suffix 씨). The glued forms whole-token matching cannot reach are handled by the peel described below (closes #307)
-    - Fix glued CJK honorifics parsing as part of the name: an honorific written against the name — ``田中さん``, ``山田太郎様``, ``김민준씨``, ``김민준님``, ``王小明先生`` — is now split off the end of the last name token and routed to ``suffix``, where it had been swallowed by the name (the whole of ``田中さん`` was the family name; ``김민준씨`` gave given 민준씨). The peeled name then goes through the ordinary machinery, so the Korean split still happens (``김민준씨`` → family 김, given 민준, suffix 씨) and the ``田中さん`` case stops misreading as a kana-licensed composite. Only entries that can never END a name peel — 씨, 님, 박사, 박사님, 선생님, 교수님, さん, さま, くん, ちゃん, 様, 先生, 教授, 女士, 小姐 — while 양, 군, 氏, 博士 and 殿 are recognized in their spaced form only, because 김지양 is a given name, 田中博士 is Tanaka Hiroshi as readily as Doctor Tanaka, and some ninety Japanese surnames end in 殿 (鵜殿, 真殿); 君 is recognized in neither form, since 王君 is a complete Chinese name, though its kana spelling くん peels. Seven entries are new vocabulary in this change (さん, さま, くん, ちゃん, 殿, 님, 박사님), so their spaced forms route to ``suffix`` too (``田中 さん``, ``田中 殿``, ``김민준 님``, ``김민준 박사님``). 박사님 closes a gap in the shipped set rather than opening new ground: 선생님 and 교수님 shipped in 2.1 without it, so ``김민준박사님`` stranded 박사 in the given name and the spaced ``김민준 박사님`` came back as two suffixes for one honorific. Exactly one honorific peels off a token, and every entry is a whole honorific rather than a part of one. A token that is nothing but an honorific is no longer taken apart either — ``선생님`` and ``박사`` now stay whole where hangul segmentation had split them 선 + 생님 and 박 + 사, since 선 and 박 are listed surnames — which is **default-on** in its own right, hangul segmentation being a default. A configured segmenter benefits twice over: it is handed the name without the honorific, and a recognized honorific beside an undivided name no longer counts as a boundary its writer drew, so ``parser_for(locales.JA, segmenter=ja_segmenter())`` reads both ``山田太郎様`` and the spaced ``山田太郎 様`` as family 山田, given 太郎, suffix 様. That last part cuts both ways and is worth knowing before you upgrade: an honorific no longer exempts a name from the pack's division at all, so a family name written alone with one — ``田中 さん``, ``佐藤 氏`` — now divides the way bare ``田中`` already did (family 田, given 中). Declining the pack, or the segmenter, is how a caller opts out of that presumption. **Default-on: changes parse output for glued CJK honorific forms**, through ``HumanName`` as well as the 2.0 API (closes #308)
+    - Fix glued CJK honorifics parsing as part of the name: an honorific written against the name — ``田中さん``, ``山田太郎様``, ``김민준씨``, ``김민준님``, ``王小明先生`` — is now split off the end of the last name token and routed to ``suffix``, where it had been swallowed by the name (the whole of ``田中さん`` was the family name; ``김민준씨`` gave given 민준씨). The peeled name then goes through the ordinary machinery, so the Korean split still happens (``김민준씨`` → family 김, given 민준, suffix 씨) and the ``田中さん`` case stops misreading as a kana-licensed composite. Only entries that can never END a name peel — 씨, 님, 박사, 박사님, 선생님, 교수님, さん, さま, くん, ちゃん, 様, 先生, 教授, 女士, 小姐 — while 양, 군, 氏, 博士 and 殿 are recognized in their spaced form only, because 김지양 is a given name, 田中博士 is Tanaka Hiroshi as readily as Doctor Tanaka, and some ninety Japanese surnames end in 殿 (鵜殿, 真殿); 君 is recognized in neither form, since 王君 is a complete Chinese name, though its kana spelling くん peels. Seven entries are new vocabulary in this change (さん, さま, くん, ちゃん, 殿, 님, 박사님), so their spaced forms route to ``suffix`` too (``田中 さん``, ``田中 殿``, ``김민준 님``, ``김민준 박사님``). 박사님 closes a gap in the shipped set rather than opening new ground: 선생님 and 교수님 shipped in 2.1 without it, so ``김민준박사님`` stranded 박사 in the given name and the spaced ``김민준 박사님`` came back as two suffixes for one honorific. Exactly one honorific peels off a token, and every entry is a whole honorific rather than a part of one. A token that is nothing but an honorific is no longer taken apart either — ``선생님`` and ``박사`` now stay whole where hangul segmentation had split them 선 + 생님 and 박 + 사, since 선 and 박 are listed surnames — which is **default-on** in its own right, hangul segmentation being a default. A configured segmenter benefits twice over: it is handed the name without the honorific, and the honorific the peel just cut off does not then look to it like a boundary the writer drew — so ``parser_for(locales.JA, segmenter=ja_segmenter())`` reads ``山田太郎様`` as family 山田, given 太郎, suffix 様. Worth knowing before you upgrade: that exemption is what makes a GLUED honorific stop protecting a name from division, so a family name written alone with one — ``田中さん`` — now divides the way bare ``田中`` already did (family 田, given 中, suffix さん). It is exactly and only the peeled tail that is exempt. A SPACED honorific is a boundary its writer did draw, and the name beside it is left as written: ``田中 さん`` and ``佐藤 氏`` keep family 田中 and 佐藤 under the pack — their division is untouched by this release, whichever field the honorific itself lands in. Writing the honorific spaced is therefore an opt-out in its own right, alongside declining the pack or the segmenter. **Default-on: changes parse output for glued CJK honorific forms**, through ``HumanName`` as well as the 2.0 API (closes #308)
     - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
     - Fix the Ukrainian conjunction ``й`` not joining the pieces around it: it is the euphonic alternate of ``і``, the two chosen by the surrounding vowel and consonant rather than by meaning (``"Олесь і Олена"`` but ``"Марія й Петро"``), so real Ukrainian data carries both spellings and shipping only ``і`` recognized just one of them. ``"Олесь й Олена Коваленки"`` now gives given ``"Олесь й Олена"`` where the ``й`` previously landed in ``middle``. Same treatment as the ``и``/``і`` entries added in 2.0.0, single-letter carve-out included: the conjunction joins only once the name has enough pieces, and a punctuated initial still wins, so ``"Й. Сліпий"`` is unaffected. Raised in a comment on #267
 

docs/usage.rst

@@ -298,14 +298,25 @@ wrote the comma has already said where the family name ends.
 Honorifics and degrees follow a CJK name, and both the spaced and the
 glued forms are recognized as suffixes: ``王小明 先生`` reads family
 王小明 with 先生 in ``suffix``, and so do glued ``田中さん``,
-``山田太郎様`` and ``김민준씨`` — the honorific is split off the end of
-the last name token before anything else reads the name, so the split
-and the order rules see the name without it. Everything downstream
-then applies as usual, which is why ``김민준씨`` still divides into
-family 김 and given 민준, and why a configured Japanese segmenter is
-handed 山田太郎 rather than 山田太郎様. The spaced spelling composes
-the same way — an honorific is not a boundary its writer drew between
-family and given, so ``山田太郎 様`` divides too.
+``山田太郎様`` and ``김민준씨``. The honorific is split off the end of
+the last name token before the name is split or ordered, so those
+rules see the name without it — which is why ``김민준씨`` still divides
+into family 김 and given 민준, and why a configured Japanese segmenter
+is handed 山田太郎 rather than 山田太郎様. The comma rule above comes
+first, though, and covers this too: in ``田中さん, 太郎`` the writer
+has already said where the family name ends, so nothing is peeled and
+the family name is the whole ``田中さん``. A 间隔号-divided
+transcription opts out for its own reason, the same way.
+
+Splitting a name is where the two spellings part company. A spaced
+honorific stands beside a name its writer chose to write as one unit,
+and that unit is left alone: under the Japanese pack ``佐藤 氏`` keeps
+family 佐藤, where bare ``佐藤`` would have been divided 佐 + 藤. A
+glued honorific says nothing of the kind — its writer drew no boundary
+anywhere — so ``田中さん`` divides the way bare ``田中`` does, into
+family 田 and given 中, with さん in ``suffix``. Writing the honorific
+spaced is therefore also how you ask for a family name to be kept
+whole, short of declining the pack.
 
 The glued reading is deliberately narrower than the spaced one. A
 spaced honorific sits behind a boundary its writer drew; a glued one

tests/v2/cases.py

@@ -730,9 +730,10 @@ def __post_init__(self) -> None:
     Case("ja_sama_spaced", "田中 太郎 様",
          {"family": "田中", "given": "太郎", "suffix": "様"},
          classification="fix(#307)",
-         notes="the spaced 様 of forms and databases; the glued "
-               "mail-addressing form 山田太郎様 is #308's mechanism, "
-               "out of reach of whole-token matching"),
+         notes="the spaced 様 of forms and databases, which whole-token "
+               "matching reaches on its own; the glued "
+               "mail-addressing form is ja_sama_glued below, reached "
+               "by #308's peel instead"),
     Case("ja_san_spaced", "田中 さん",
          {"family": "田中", "suffix": "さん"},
          classification="fix(#308)",
@@ -810,10 +811,12 @@ def __post_init__(self) -> None:
     Case("ko_glued_tail_alone_never_peels", "씨",
          {"family": "씨"},
          classification="fix(#271)",
-         notes="the empty-remainder guard: a token that IS a tail has "
-               "nothing to peel off. Classified to #271 because that "
-               "is what moves the lone token from first to family; "
-               "the row exists for the guard"),
+         notes="a token that IS a tail is not a peel site at all -- "
+               "every tail is a suffix word, so the site scan steps "
+               "past it and finds nothing else. NOT the length cap, "
+               "which is never reached here. Classified to #271 "
+               "because that is what moves the lone token from first "
+               "to family; the row exists for the guard"),
     Case("ko_honorific_token_alone_stays_whole", "선생님",
          {"family": "선생님"},
          classification="fix(#308)",
@@ -825,8 +828,12 @@ def __post_init__(self) -> None:
     Case("ja_glued_tail_alone_never_peels", "さん",
          {"family": "さん"},
          classification="fix(#272)",
-         notes="the kana twin of the row above; the field placement "
-               "is the kana-licensed order default, not the peel"),
+         notes="the kana twin of ko_glued_tail_alone_never_peels, and "
+               "of that row only: a lone さん carries none of "
+               "ko_honorific_token_alone_stays_whole's risk, since no "
+               "surname vocabulary is written in kana. The field "
+               "placement is the kana-licensed order default, not the "
+               "peel"),
     Case("ja_glued_degree_stays", "田中博士",
          {"family": "田中博士"},
          classification="fix(#271)",

tests/v2/test_lexicon.py

@@ -602,7 +602,9 @@ def test_default_lexicon_honorific_tails_are_all_suffix_words() -> None:
     # Not redundant with suffixes.py's GLUED_HONORIFICS <= SUFFIX_NOT_
     # ACRONYMS assert: that one relates the raw constants at import
     # time (and is gone under python -O); this one relates the
-    # normalized, post-ambiguous-subtraction Lexicon fields, and
-    # survives -O. Structural check, not a content pin.
+    # normalized fields of a constructed Lexicon, and survives -O.
+    # (No ambiguous subtraction here -- that lives in the v1 shim's
+    # _build_snapshot, which this never touches.) Structural check,
+    # not a content pin.
     d = Lexicon.default()
     assert d.honorific_tails <= d.suffix_words