Pin the peel's segment choice and its two boundaries (#308)

A mutation pass found one live survivor on a fully-covered line:
replacing the peel's `state.segments[0]` with `range(len(state.tokens))`
-- the natural simplification, since under NO_COMMA the two runs are
usually identical -- changes real output and fails nothing. Extracted
delimited content is still in the token stream at this stage with only
its role set, so "김민준씨 (Jimmy)" would hand the scan-back Jimmy as
its site: no post-nominal to step over, no tail to peel, and the peel
lost entirely. The comment above the segment lookup made it worse by
saying extracted content is unreachable and that no input can produce
it -- true of the surname half it was written for, actively misleading
for the peel, where the choice is what keeps Jimmy out. Case row plus
both comments.

Two boundaries a reader could otherwise meet by surprise:

- "田中さん, PhD" keeps the whole 田中さん as the family name while
  "田中さん PhD" gives family 田中 and suffix "さん, PhD". A one-word
  name part before a comma is FAMILY_COMMA, which opts the stage out
  by doctrine -- the one spelling disagreement #308 does not remove,
  where the 김민준씨 Jr. pair got two rows for removing exactly that.
  A case row, so it is a recorded decision.
- The off-switch docs/customize.rst promises was verified and
  untested: removing a tail leaves the glued name unsplit while the
  spaced spelling still routes the honorific. Pinned at parse level,
  against a peel-is-on baseline so it cannot pass vacuously.

And a clause recording why _is_post_nominal is strict rather than
lenient: the initial veto makes "田中さん V." decline where
"田中さん II" peels, which agrees with what classify does with "V."
downstream. Swapping the predicate fails no test; the clause is the
record instead.

Author: derek73

nameparser/_pipeline/_script_segment.py

@@ -254,6 +254,15 @@ def _is_post_nominal(state: ParseState, i: int) -> bool:
     surname site reads a True as an ANSWER and declines, rather than as
     a token to step past.
 
+    STRICT, not lenient: the initial veto applies, so "V." is not a
+    post-nominal here though bare "v" is a suffix word. That agrees
+    with what classify does with the same token downstream -- "V." is
+    a middle initial -- and the difference is reachable under the
+    default lexicon: "田中さん V." stops its scan-back at the initial
+    and does not peel, while "田中さん II" steps over II and does.
+    Swapping in is_suffix_lenient fails no test; this clause is the
+    record instead.
+
     Suffixes only, deliberately. Should a CJK entry ever join titles,
     the surname site would want it excluded while the peel site must
     not: a title never trails, so widening the peel's scan-back would
@@ -285,6 +294,13 @@ def _peel_honorific_tail(state: ParseState) -> ParseState:
     that returns None where the outright index would raise removes an
     unpinned reliance on the FAMILY_COMMA gate landing first.
 
+    WHICH run is scanned is a separate decision, and the one a reader
+    is likeliest to undo: segments[0], never state.tokens. The two
+    agree under NO_COMMA except where extract_delimited has already
+    claimed a token, which is still in the stream here with only its
+    role set -- so "김민준씨 (Jimmy)" is the input that tells them
+    apart. See the note above the segment lookup in script_segment.
+
     That last token is the last of the NAME PART, which under NO_COMMA
     reaches into a maiden clause: maiden tokens are still main-stream
     here (extract_delimited has masked only bracketed content), so
@@ -386,8 +402,15 @@ def script_segment(state: ParseState) -> ParseState:
     # segments[0] is the NAME part under both remaining structures
     # (everything, under NO_COMMA); later segments are suffixes. Its
     # members are main-stream token indices by construction, so
-    # extracted nickname/maiden content is unreachable from here --
-    # no input can produce it, so no test pins it.
+    # extracted nickname/maiden content is unreachable from here.
+    # For the surname site that is merely tidy -- the first
+    # script-written token is the same either way. For the PEEL above
+    # it is load-bearing, and iterating state.tokens instead is a
+    # silent regression rather than a refactor: extracted content is
+    # still IN the token stream at this stage (only its role is set),
+    # so "김민준씨 (Jimmy)" would give the scan-back Jimmy as its site,
+    # which is no post-nominal to step over and ends in no tail, and
+    # the peel would be lost. Pinned by the case row of that name.
     i = next((i for i in state.segments[0]
               if effective_script(state.tokens[i].text) in scripts), None)
     # A post-nominal in the surname's own position is not a site to

tests/v2/cases.py

@@ -914,6 +914,32 @@ def __post_init__(self) -> None:
                "ko_honorific_glued_given_trailing_suffix, whose "
                "comma-less spelling of the same name reaches the same "
                "answer by the scan-back instead"),
+    Case("ko_honorific_glued_given_nickname", "김민준씨 (Jimmy)",
+         {"family": "김", "given": "민준", "suffix": "씨",
+          "nickname": "Jimmy"},
+         classification="fix(#308)",
+         notes="the other half of indexing the NAME part: extracted "
+               "content is still in the token stream at this stage but "
+               "NOT in segments[0], so the scan-back never reaches "
+               "Jimmy. Scanning the tokens instead would take Jimmy as "
+               "the site -- it is no post-nominal -- and lose the peel "
+               "entirely, with 씨 back in the given name. Nothing else "
+               "pins that choice: under NO_COMMA the two are otherwise "
+               "the same run"),
+    Case("ko_honorific_glued_family_comma", "田中さん, PhD",
+         {"family": "田中さん", "title": "PhD"},
+         classification="fix(#271)",
+         notes="the comma doctrine outranks the peel, and this is "
+               "where a reader meets that boundary: a one-word name "
+               "part before a comma is FAMILY_COMMA, which opts the "
+               "whole stage out, so 田中さん stays whole where the "
+               "comma-less 田中さん PhD gives family 田中 and suffix "
+               "'さん, PhD'. Deliberate -- whoever wrote the comma "
+               "already said where the family name ends -- and the "
+               "one spelling disagreement #308 does not remove, unlike "
+               "the 김민준씨 Jr. pair above. Classified to #271, not "
+               "#308: the fields here are the CJK order flip's, and "
+               "the peel never fires on this input"),
     Case("zh_honorific_glued_surname", "王先生",
          {"family": "王", "suffix": "先生"},
          locale="zh",

tests/v2/test_lexicon.py

@@ -608,3 +608,23 @@ def test_default_lexicon_honorific_tails_are_all_suffix_words() -> None:
     # not a content pin.
     d = Lexicon.default()
     assert d.honorific_tails <= d.suffix_words
+
+
+def test_removing_a_honorific_tail_is_the_peels_off_switch() -> None:
+    # The off-switch docs/customize.rst documents, at parse level:
+    # dropping a tail leaves the glued name unsplit while the SPACED
+    # spelling still routes the honorific, since suffix_words is a
+    # separate field and only the peel consults honorific_tails.
+    # honorific_tails is the one marker field a caller can empty
+    # without a matching base edit -- an orphan is what raises, and a
+    # removal cannot make one.
+    lex = Lexicon.default().remove(honorific_tails={"さん"})
+    p = Parser(lexicon=lex)
+    glued = p.parse("田中さん")
+    assert (glued.family, glued.suffix) == ("田中さん", "")
+    spaced = p.parse("田中 さん")
+    assert (spaced.family, spaced.suffix) == ("田中", "さん")
+    # the baseline the removal is against, so the test cannot pass by
+    # the peel being broken outright
+    on = Parser().parse("田中さん")
+    assert (on.family, on.suffix) == ("田中", "さん")

tools/differential/corpus_cjk.jsonl

@@ -37,6 +37,7 @@
 "田中 殿"
 "田中『ハナ』花子"
 "田中さん"
+"田中さん, PhD"
 "田中博士"
 "諸葛亮"
 "阿明"
@@ -56,6 +57,7 @@
 "김민준님"
 "김민준박사님"
 "김민준씨"
+"김민준씨 (Jimmy)"
 "김민준씨 Jr."
 "김선생님"
 "김씨"