Say the peel is not GATED by comma structure, rather than blind to it (#312)

Author: derek73

docs/release_log.rst

@@ -28,7 +28,7 @@ Release Log
     - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298)
     - Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese, with 先生/博士/教授 shared with Japanese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching, which also reaches a glued surname+honorific token, since segmentation splits off the surname first (``김씨`` reads family 김, suffix 씨). The glued forms whole-token matching cannot reach are handled by the peel described below (closes #307)
     - Fix glued CJK honorifics parsing as part of the name: an honorific written against the name — ``田中さん``, ``山田太郎様``, ``김민준씨``, ``김민준님``, ``王小明先生`` — is now split off the end of the last name token and routed to ``suffix``, where it had been swallowed by the name (the whole of ``田中さん`` was the family name; ``김민준씨`` gave given 민준씨). The peeled name then goes through the ordinary machinery, so the Korean split still happens (``김민준씨`` → family 김, given 민준, suffix 씨) and the ``田中さん`` case stops misreading as a kana-licensed composite. Only entries that can never END a name peel — 씨, 님, 박사, 박사님, 선생님, 교수님, さん, さま, くん, ちゃん, 様, 先生, 教授, 女士, 小姐 — while 양, 군, 氏, 博士 and 殿 are recognized in their spaced form only, because 김지양 is a given name, 田中博士 is Tanaka Hiroshi as readily as Doctor Tanaka, and some ninety Japanese surnames end in 殿 (鵜殿, 真殿); 君 is recognized in neither form, since 王君 is a complete Chinese name, though its kana spelling くん peels. Seven entries are new vocabulary in this change (さん, さま, くん, ちゃん, 殿, 님, 박사님), so their spaced forms route to ``suffix`` too (``田中 さん``, ``田中 殿``, ``김민준 님``, ``김민준 박사님``). 박사님 closes a gap in the shipped set rather than opening new ground: 선생님 and 교수님 shipped in 2.1 without it, so ``김민준박사님`` stranded 박사 in the given name and the spaced ``김민준 박사님`` came back as two suffixes for one honorific. Exactly one honorific peels off a token, and every entry is a whole honorific rather than a part of one. A token that is nothing but an honorific is no longer taken apart either — ``선생님`` and ``박사`` now stay whole where hangul segmentation had split them 선 + 생님 and 박 + 사, since 선 and 박 are listed surnames — which is **default-on** in its own right, hangul segmentation being a default. A configured segmenter benefits twice over: it is handed the name without the honorific, and the honorific the peel just cut off does not then look to it like a boundary the writer drew — so ``parser_for(locales.JA, segmenter=ja_segmenter())`` reads ``山田太郎様`` as family 山田, given 太郎, suffix 様. Worth knowing before you upgrade: that exemption is what makes a GLUED honorific stop protecting a name from division, so a family name written alone with one — ``田中さん`` — now divides the way bare ``田中`` already did (family 田, given 中, suffix さん). It is exactly and only the peeled tail that is exempt. A SPACED honorific is a token boundary its writer typed, and anything standing beside a name calls the segmenter off, so the name is left as written: ``田中 さん`` and ``佐藤 氏`` keep family 田中 and 佐藤 under the pack — the division the pack gives them without this change, whichever field the honorific itself lands in. That is the conservative reading rather than a claim about intent: a spaced honorific cannot be told apart from a spaced given name by position, and counting it as one keeps four real surnames whole (``佐藤 氏``, ``田中 様``, ``鈴木 先生``, ``中村 教授``) at the price of the one division it then declines to make (``山田太郎 様``). Writing the honorific spaced is therefore an opt-out in its own right on the SEGMENTER path, alongside declining the pack or the segmenter. It is no lever where the VOCABULARY divides the name, the two spellings agreeing exactly there — ``김민준 씨`` and ``김민준씨`` both give family 김, given 민준, suffix 씨, as do ``王小明 先生`` and ``王小明先生`` under the Chinese pack — and Korean data has no pack to decline either, hangul segmentation being on by default. **Default-on: changes parse output for glued CJK honorific forms**, through ``HumanName`` as well as the 2.0 API (closes #308)
-    - Fix a comma or a 间隔号 stopping the glued-honorific peel: an honorific written against the name is now split off and routed to ``suffix`` whichever way the name is punctuated, so ``김, 민준씨`` gives family 김, given 민준, suffix 씨 — the same as the spaced ``김 민준씨`` — and likewise ``田中, 太郎さん``, ``田中さん, PhD`` and ``威廉·莎士比亚さん``. Previously each of those left the honorific inside the name. The peel now consults no comma structure and no dot: both of those say where a name divides into surname and given, and an honorific is not part of the name in either spelling. The surname split still stands down for both, unchanged — a comma still means the writer said where the family name ends, and the 间隔号 still marks a transcription. ``田中さん, 太郎`` is unaffected, because the honorific there is not at the end of the name: 太郎 is. **Default-on**, and it reaches ``HumanName`` too (closes #312)
+    - Fix a comma or a 间隔号 stopping the glued-honorific peel: an honorific written against the name is now split off and routed to ``suffix`` whichever way the name is punctuated, so ``김, 민준씨`` gives family 김, given 민준, suffix 씨 — the same as the spaced ``김 민준씨`` — and likewise ``田中, 太郎さん``, ``田中さん, PhD`` and ``威廉·莎士比亚さん``. Previously each of those left the honorific inside the name. No comma structure and no dot gates the peel any more — it reads the comma structure only to find which segments the name occupies, an honorific being as often glued to the given name as to the family: both of those marks say where a name divides into surname and given, and an honorific is not part of the name in either spelling. The surname split still stands down for both, unchanged — a comma still means the writer said where the family name ends, and the 间隔号 still marks a transcription. ``田中さん, 太郎`` is unaffected, because the honorific there is not at the end of the name: 太郎 is. **Default-on**, and it reaches ``HumanName`` too (closes #312)
     - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272)
     - Fix the Ukrainian conjunction ``й`` not joining the pieces around it: it is the euphonic alternate of ``і``, the two chosen by the surrounding vowel and consonant rather than by meaning (``"Олесь і Олена"`` but ``"Марія й Петро"``), so real Ukrainian data carries both spellings and shipping only ``і`` recognized just one of them. ``"Олесь й Олена Коваленки"`` now gives given ``"Олесь й Олена"`` where the ``й`` previously landed in ``middle``. Same treatment as the ``и``/``і`` entries added in 2.0.0, single-letter carve-out included: the conjunction joins only once the name has enough pieces, and a punctuated initial still wins, so ``"Й. Сліпий"`` is unaffected. Raised in a comment on #267