You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The common way to write CJK honorifics is glued to the name: 田中さん, 王先生, and the standard mail-addressing form 山田太郎様. Whole-token vocabulary can never match these, so today the honorific is absorbed whole — and a kana honorific even changes classification (田中さん is Han+hiragana → kana-licensed → family=田中さん), and would mislead a configured ja segmenter.
The fix is a mechanism, not constants: a list-driven tail-peel on the last name token, running before classification/order/segmentation — the honorific-stripping analogue of script_segment. Design questions this issue exists to settle:
Safety: 山田さん must peel while 山田さくら must not — tail-matching against a closed honorific list is probably sufficient (no Japanese given name is さん/様/先生), but the boundary needs the same treatment the segmenter contract got (what may peel, what must decline).
Default-on or opt-in: glued 様/さん at token tail looks unambiguous (the doctrine's "orthography settles it" test), but that claim needs adversarial review against real given names.
Interaction order: the peel must precede effective_script, the family-first license, and the segmentation stage, and the peeled honorific needs a field (suffix) and span bookkeeping consistent with the anti-Strange parsing of name w lastname prefix and title before and after #100 invariant (spans index the original — a peel is a token split, not a rewrite).
Depends on the spaced-honorific vocabulary issue for the list itself.
The common way to write CJK honorifics is glued to the name: 田中さん, 王先生, and the standard mail-addressing form 山田太郎様. Whole-token vocabulary can never match these, so today the honorific is absorbed whole — and a kana honorific even changes classification (田中さん is Han+hiragana → kana-licensed → family=田中さん), and would mislead a configured ja segmenter.
The fix is a mechanism, not constants: a list-driven tail-peel on the last name token, running before classification/order/segmentation — the honorific-stripping analogue of
script_segment. Design questions this issue exists to settle:effective_script, the family-first license, and the segmentation stage, and the peeled honorific needs a field (suffix) and span bookkeeping consistent with the anti-Strange parsing of name w lastname prefix and title before and after #100 invariant (spans index the original — a peel is a token split, not a rewrite).Depends on the spaced-honorific vocabulary issue for the list itself.