feat: typographic nickname delimiters as defaults (closes #273)

derek73 · claude · derek73 · commit 7ee6e3a747fe · 2026-07-18T13:43:44.000-07:00
DEFAULT_NICKNAME_DELIMITERS grows by the eight conventions the issue
proposed: smart quotes, low-high (de/pl/cs/hu), right-right (sv/fi),
guillemets both directions (« Petit » inner spacing tolerated), CJK
corner brackets, fullwidth parentheses. Curly single quotes stay
excluded (U+2019 is the apostrophe in O'Connor, pinned). Both APIs
get them: the shim's sentinel map gains named keys
(smart_double_quotes, guillemets, ...) so the v1 keyed idioms
(pop/move/del) work on the new pairs like the original trio.

The conventions share characters in opposite roles ('"' closes the
German pair but opens the English one; '»' closes guillemets and
opens the reversed pair), which made extraction emit spurious
unbalanced-delimiter ambiguities: extract now tracks unmatched-open
offsets and drops candidates inside a region another pair
successfully masked. Genuine unbalanced opens still flag.

TDD throughout (extract suppression -&gt; defaults -&gt; shim sentinels,
each red first); 9 case rows through both runners; differential
harness exit 0 unchanged (no typographic quotes in the corpus);
release-log feat entry; reference/customize/migrate docs updated.

Co-Authored-By: Claude Fable 5 &lt;noreply@anthropic.com&gt;
diff --git a/docs/customize.rst b/docs/customize.rst
@@ -81,8 +81,9 @@ listed below.
      - ``frozenset[tuple[str, str]]``
      - Routes content enclosed by these delimiter pairs to
        ``nickname``. Defaults to
-       :data:`~nameparser.DEFAULT_NICKNAME_DELIMITERS` — quotes and
-       parentheses.
+       :data:`~nameparser.DEFAULT_NICKNAME_DELIMITERS` — straight
+       quotes and parentheses plus the typographic conventions (smart
+       quotes, guillemets, CJK brackets, ...).
    * - ``maiden_delimiters``
      - ``frozenset[tuple[str, str]]``
      - Routes content enclosed by these delimiter pairs to ``maiden``
diff --git a/docs/migrate.rst b/docs/migrate.rst
@@ -170,8 +170,9 @@ rendering argument, where the 2.0 equivalent isn't config at all):
      -
    * - ``nickname_delimiters``
      - ``Policy.nickname_delimiters``
-     - Was a three-sentinel dict; now a plain ``frozenset`` of
-       ``(open, close)`` pairs
+     - Was a dict of named sentinels; now a plain ``frozenset`` of
+       ``(open, close)`` pairs. Both APIs gained the #273 typographic
+       defaults (smart quotes, guillemets, CJK brackets, ...) in 2.0
    * - ``maiden_delimiters``
      - ``Policy.maiden_delimiters``
      - Same shape change as ``nickname_delimiters``. Precedence
diff --git a/docs/modules.rst b/docs/modules.rst
@@ -93,12 +93,17 @@ Delimiter defaults
 ^^^^^^^^^^^^^^^^^^
 
 .. py:data:: nameparser.DEFAULT_NICKNAME_DELIMITERS
-   :value: frozenset({("'", "'"), ('"', '"'), ("(", ")")})
-
-   The default :attr:`~nameparser.Policy.nickname_delimiters` set —
-   quotes and parentheses. Build on it for additive customizations,
-   e.g. ``nickname_delimiters=DEFAULT_NICKNAME_DELIMITERS |
-   {("«", "»")}``; to *reroute* a pair to ``maiden``, just list it in
+   :value: frozenset({("'", "'"), ('"', '"'), ("(", ")"), ("“", "”"), ("„", "“"), ("”", "”"), ("«", "»"), ("»", "«"), ("「", "」"), ("『", "』"), ("（", "）")})
+
+   The default :attr:`~nameparser.Policy.nickname_delimiters` set:
+   straight quotes and parentheses plus the typographic conventions —
+   smart quotes, German/Polish low-high quotes, Swedish right-right
+   quotes, guillemets in both directions, CJK corner brackets, and
+   fullwidth parentheses (#273). Curly *single* quotes are deliberately
+   absent: U+2019 is the typographic apostrophe ("O’Connor"). Build on
+   the constant for additive customizations, e.g.
+   ``nickname_delimiters=DEFAULT_NICKNAME_DELIMITERS | {("｟", "｠")}``;
+   to *reroute* a pair to ``maiden``, just list it in
    :attr:`~nameparser.Policy.maiden_delimiters` (it is dropped from
    the effective nickname set automatically).
 
diff --git a/docs/release_log.rst b/docs/release_log.rst
@@ -25,6 +25,7 @@ Release Log
     - Change suffix-delimiter rendering: with a custom ``Policy``/``Constants`` suffix delimiter configured (e.g. ``suffix_delimiter="/"``, ``"John Smith, RN/CRNA"``), 1.x split the token and rendered ``suffix="RN, CRNA"``; 2.0 keeps the no-space delimiter-core token whole (``suffix="RN/CRNA"``) -- role assignment is unchanged, only rendering differs (anti-#100, migration plan deviation 5; ``tests/v2/cases.py`` row ``suffix_delimiter_no_space_core``). Only fires with a non-default policy, so it does not appear in the (default-policy) differential corpus
     - Change ``comparison_key()``/``matches()`` (both APIs) to fold components with ``str.casefold()`` where 1.4 used ``str.lower()``: Unicode case-pair forms now compare equal -- ``"STRASSE"`` matches ``"Straße"``, and a Greek final-sigma variant matches its regular-sigma form. Strictly more permissive (anything 1.4 matched still matches); parse output is unaffected -- vocabulary normalization itself uses ``lower()``, v1-parity (``tests/v2/test_types.py`` row ``test_matches_casefolds_unicode_case_pairs``)
     - Change delimiter-overlap precedence in the 2.0 API: a pair listed in ``Policy.maiden_delimiters`` is dropped from the effective ``nickname_delimiters`` set (maiden wins), so ``Policy(maiden_delimiters={("(", ")")})`` alone routes parenthesized content to ``maiden`` -- no need to rebuild the nickname set. The default nickname set is now the public ``DEFAULT_NICKNAME_DELIMITERS`` constant. The 1.x facade keeps v1's nickname-wins precedence on overlap via a shim-side pre-subtraction, so no ``HumanName`` behavior changes (``tests/v2/cases.py`` row ``maiden_delimiters_win_when_shared``; ``tests/v2/test_config_shim.py`` row ``test_snapshot_overlap_keeps_v1_nickname_precedence``)
+    - Recognize typographic nickname delimiters by default in BOTH APIs (closes #273): smart quotes (``“Jack”``), German/Polish low-high quotes (``„Hansi“``), Swedish right-right quotes (``”Ann”``), guillemets in both directions (``«Petit»``, ``»Hansi«``, inner spacing tolerated), CJK corner brackets (``「タロ」``/``『ハナ』``), and fullwidth parentheses. In 1.x these leaked into ``middle`` as literal text. Curly *single* quotes stay excluded (U+2019 is the apostrophe in "O’Connor" -- pinned by ``curly_apostrophe_stays_literal``). The v1 keyed idioms work on the new named sentinels (``smart_double_quotes``, ``guillemets``, ...). A delimiter character consumed by another pair's extraction no longer emits a spurious ``unbalanced-delimiter`` ambiguity (``tests/v2/cases.py`` rows ``*_nickname``; ``tests/v2/pipeline/test_extract.py``). Not present in the differential corpus
 
     Everything else in the 486-name differential corpus (built from the
     v1 test banks as of commit ``2d5d8c2``, pre-dating the M12 test
diff --git a/nameparser/_config_shim.py b/nameparser/_config_shim.py
@@ -407,12 +407,24 @@ def __setstate__(self, state: dict[str, object]) -> None:
         self._on_change = None  # rewired by the owning Constants
 
 
-#: v1's three named delimiter buckets, translated to the ``Policy``
-#: (open, close) pairs they stand for (spec §3).
+#: The named delimiter buckets, translated to the ``Policy``
+#: (open, close) pairs they stand for (spec §3). The first three are
+#: v1's; the rest are the #273 typographic conventions, named so the
+#: v1 keyed idioms (pop/move/del) work on them like the originals.
+#: Keep in sync with DEFAULT_NICKNAME_DELIMITERS in _policy.py (pinned
+#: by the default-Constants equality test).
 _SENTINEL_PAIRS = {
     "quoted_word": ("'", "'"),
     "double_quotes": ('"', '"'),
     "parenthesis": ("(", ")"),
+    "smart_double_quotes": ("“", "”"),
+    "low_high_quotes": ("„", "“"),
+    "right_double_quotes": ("”", "”"),
+    "guillemets": ("«", "»"),
+    "reversed_guillemets": ("»", "«"),
+    "corner_brackets": ("「", "」"),
+    "white_corner_brackets": ("『", "』"),
+    "fullwidth_parenthesis": ("（", "）"),
 }
 
 #: derived, so the manager's accepted keys and _snapshot()'s
diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py
@@ -59,7 +59,9 @@ def extract_delimited(state: ParseState) -> ParseState:
     text = state.original
     extracted: list[tuple[Role, Span]] = []
     masked: list[Span] = []
-    ambiguities: list[PendingAmbiguity] = []
+    # candidates, not final: each carries the offset of the unmatched
+    # open so cross-pair overlaps can be filtered once all pairs ran
+    unbalanced: list[tuple[int, PendingAmbiguity]] = []
     # nickname first (v1 parse_nicknames order): when the same
     # delimiter pair sits in BOTH buckets, the nickname reading wins;
     # the documented bucket-move idiom removes it from nickname, so
@@ -79,11 +81,11 @@ def extract_delimited(state: ParseState) -> ParseState:
                        and not _close_ok(text, j, len(close))):
                     j = text.find(close, j + 1)
                 if j == -1:
-                    ambiguities.append(PendingAmbiguity(
+                    unbalanced.append((i, PendingAmbiguity(
                         AmbiguityKind.UNBALANCED_DELIMITER,
                         f"unmatched {open_!r} at offset {i}; treated as "
                         f"literal text",
-                    ))
+                    )))
                     if open_ == close:
                         # _close_ok is open-independent, so a failed
                         # close-walk means NO boundary-valid close
@@ -94,11 +96,11 @@ def extract_delimited(state: ParseState) -> ParseState:
                         scan = i + len(open_)
                         while (k := text.find(open_, scan)) != -1:
                             if _open_ok(text, k):
-                                ambiguities.append(PendingAmbiguity(
+                                unbalanced.append((k, PendingAmbiguity(
                                     AmbiguityKind.UNBALANCED_DELIMITER,
                                     f"unmatched {open_!r} at offset {k}; "
                                     f"treated as literal text",
-                                ))
+                                )))
                             scan = k + 1
                         break
                     pos = i + len(open_)
@@ -125,6 +127,14 @@ def extract_delimited(state: ParseState) -> ParseState:
                 pos = j + len(close)
     extracted.sort(key=lambda pair: pair[1])
     masked.sort()
+    # A delimiter character consumed by another pair's successful
+    # extraction is literal for every other pair: drop unbalanced
+    # candidates whose offset lies inside a masked region (#273 -- the
+    # conventions share characters in opposite roles: '“' closes „…“
+    # but opens “…”, '»' closes «…» but opens »…«).
+    ambiguities = tuple(
+        a for offset, a in unbalanced
+        if not any(m.start <= offset < m.end for m in masked))
     return dataclasses.replace(
         state, extracted=tuple(extracted), masked=tuple(masked),
-        ambiguities=state.ambiguities + tuple(ambiguities))
+        ambiguities=state.ambiguities + ambiguities)
diff --git a/nameparser/_policy.py b/nameparser/_policy.py
@@ -45,12 +45,25 @@ class PatronymicRule(StrEnum):
 
 _NAME_ROLES = frozenset({Role.GIVEN, Role.MIDDLE, Role.FAMILY})
 
-#: Policy.nickname_delimiters' default (v1 parity: quotes + parentheses).
-#: Public and named so customizations read as set math against a
-#: documented value -- e.g. ``DEFAULT_NICKNAME_DELIMITERS | {("«", "»")}``
-#: -- instead of a rebuilt literal the user had to go discover.
-DEFAULT_NICKNAME_DELIMITERS = frozenset(
-    {("'", "'"), ('"', '"'), ("(", ")")})
+#: Policy.nickname_delimiters' default. Public and named so
+#: customizations read as set math against a documented value -- e.g.
+#: ``DEFAULT_NICKNAME_DELIMITERS | {("｟", "｠")}`` -- instead of a
+#: rebuilt literal the user had to go discover. The v1 trio (straight
+#: quotes + parentheses) plus the typographic conventions (#273):
+#: smart quotes, low-high and right-right quotes, guillemets both
+#: directions, CJK corner brackets, fullwidth parentheses. Curly
+#: SINGLE quotes are deliberately absent: U+2019 is the typographic
+#: apostrophe ("O’Connor").
+DEFAULT_NICKNAME_DELIMITERS = frozenset({
+    ("'", "'"), ('"', '"'), ("(", ")"),      # v1 trio
+    ("“", "”"),                              # smart quotes (en, zh)
+    ("„", "“"),                              # low-high (de, pl, cs, hu)
+    ("”", "”"),                              # right-right (sv, fi)
+    ("«", "»"),                              # guillemets (fr, ru, it, el)
+    ("»", "«"),                              # reversed guillemets (de alt)
+    ("「", "」"), ("『", "』"),                # CJK corner brackets (ja)
+    ("（", "）"),                             # fullwidth parentheses (CJK)
+})
 
 
 def _reject_bare_string_order(value: object) -> None:
diff --git a/tests/test_constants.py b/tests/test_constants.py
@@ -545,11 +545,15 @@ def test_nickname_delimiters_default_builtins_resolve_live(self) -> None:
         # see test_overriding_builtin_regex_still_affects_nickname_parsing in
         # test_nicknames.py.
         c = Constants()
-        self.assertEqual(dict(c.nickname_delimiters), {
-            'quoted_word': 'quoted_word',
-            'double_quotes': 'double_quotes',
-            'parenthesis': 'parenthesis',
-        })
+        entries = dict(c.nickname_delimiters)
+        # v1 trio still present, still stored name-as-value
+        for name in ('quoted_word', 'double_quotes', 'parenthesis'):
+            self.assertEqual(entries[name], name)
+        # 2.0 adds the #273 typographic sentinels alongside them, same
+        # name-as-value scheme (full list pinned in tests/v2/
+        # test_config_shim.py against _SENTINEL_PAIRS)
+        assert all(value == name for name, value in entries.items())
+        assert 'smart_double_quotes' in entries
         self.assertEqual(dict(c.maiden_delimiters), {})
 
     def test_extra_nickname_delimiters_removed(self) -> None:
diff --git a/tests/v2/cases.py b/tests/v2/cases.py
@@ -126,6 +126,45 @@ def __post_init__(self) -> None:
                "one-liner replaces the bucket-move idiom; the v1 facade "
                "keeps v1's nickname-wins precedence via the shim's "
                "pre-subtraction (pinned in test_config_shim)"),
+    # #273: typographic nickname delimiters ship as defaults -- one row
+    # per new pair; expectations verified live 2026-07-19 before the
+    # pairs were added to DEFAULT_NICKNAME_DELIMITERS.
+    Case("smart_double_quotes_nickname", 'John “Jack” Kennedy',
+         {"given": "John", "family": "Kennedy", "nickname": "Jack"},
+         classification="feat(#273)"),
+    Case("low_high_quotes_nickname", 'Hans „Hansi“ Müller',
+         {"given": "Hans", "family": "Müller", "nickname": "Hansi"},
+         classification="feat(#273)",
+         notes="the closing '“' doubles as the English pair's opener; "
+               "no spurious unbalanced-delimiter ambiguity (pinned in "
+               "test_extract)"),
+    Case("guillemets_nickname_inner_spaces", 'Jean « Petit » Dupont',
+         {"given": "Jean", "family": "Dupont", "nickname": "Petit"},
+         classification="feat(#273)",
+         notes="French spacing: inner padding is trimmed from the "
+               "extracted nickname"),
+    Case("reversed_guillemets_nickname", 'Hans »Hansi« Müller',
+         {"given": "Hans", "family": "Müller", "nickname": "Hansi"},
+         classification="feat(#273)"),
+    Case("swedish_right_quotes_nickname", 'Anna ”Ann” Larsson',
+         {"given": "Anna", "family": "Larsson", "nickname": "Ann"},
+         classification="feat(#273)"),
+    Case("cjk_corner_bracket_nickname", '山田「タロ」太郎',
+         {"given": "山田", "family": "太郎", "nickname": "タロ"},
+         classification="feat(#273)",
+         notes="extraction also splits the unspaced remainder -- the "
+               "masked region acts as a token boundary"),
+    Case("cjk_white_corner_bracket_nickname", '田中『ハナ』花子',
+         {"given": "田中", "family": "花子", "nickname": "ハナ"},
+         classification="feat(#273)"),
+    Case("fullwidth_paren_nickname", 'John （Jack） Kennedy',
+         {"given": "John", "family": "Kennedy", "nickname": "Jack"},
+         classification="feat(#273)"),
+    Case("curly_apostrophe_stays_literal", 'Sean O’Connor',
+         {"given": "Sean", "family": "O’Connor"},
+         notes="U+2019 is the typographic apostrophe; curly single "
+               "quotes are deliberately NOT delimiters (#273 excludes "
+               "them)"),
     Case("family_segment_trailing_suffix", "Smith Jr., John",
          {"given": "John", "family": "Smith", "suffix": "Jr."},
          notes="v1: the family part may have suffixes in it "
diff --git a/tests/v2/pipeline/test_extract.py b/tests/v2/pipeline/test_extract.py
@@ -43,6 +43,32 @@ def test_unbalanced_delimiter_left_literal_with_ambiguity() -> None:
     assert out.ambiguities[0].kind is AmbiguityKind.UNBALANCED_DELIMITER
 
 
+def test_no_spurious_unbalanced_from_role_overlapping_pairs() -> None:
+    # #273: '“' closes the German „…“ pair but OPENS the English “…”
+    # pair, and '»' closes «…» but opens the reversed »…« pair. A
+    # delimiter character consumed by another pair's successful
+    # extraction is literal for every other pair -- it must not
+    # surface as an unbalanced-delimiter ambiguity.
+    policy = Policy(nickname_delimiters=frozenset(
+        {("“", "”"), ("„", "“"), ("«", "»"), ("»", "«")}))
+    for text, inner in (("Hans „Hansi“ Müller", "Hansi"),
+                        ("Jean «Petit» Dupont", "Petit")):
+        out = extract_delimited(_state(text, policy))
+        assert [text[s.start:s.end] for _, s in out.extracted] == [inner]
+        assert out.ambiguities == ()
+
+
+def test_genuine_unbalanced_still_flagged_alongside_overlapping_pairs() -> None:
+    # the suppression must not swallow REAL unbalanced opens: here the
+    # German open has no close anywhere, and no other pair extracts
+    policy = Policy(nickname_delimiters=frozenset(
+        {("“", "”"), ("„", "“")}))
+    out = extract_delimited(_state("Hans „Hansi Müller", policy))
+    assert out.extracted == ()
+    assert [a.kind for a in out.ambiguities] == [
+        AmbiguityKind.UNBALANCED_DELIMITER]
+
+
 def test_maiden_delimiters_route_to_maiden() -> None:
     policy = dataclasses.replace(
         Policy(),
diff --git a/tests/v2/test_config_shim.py b/tests/v2/test_config_shim.py
@@ -206,8 +206,14 @@ def test_constants_default_fields_present() -> None:
     assert c.force_mixed_case_capitalization is False
     assert c.string_format == "{title} {first} {middle} {last} {suffix} ({nickname})"
     assert c.suffix_delimiter is None
-    assert set(c.nickname_delimiters) == {"quoted_word", "double_quotes",
-                                          "parenthesis"}
+    # every named sentinel is a default nickname bucket (the v1 trio
+    # plus the #273 typographic pairs); derived from the map itself so
+    # a new sentinel can't leave this pin stale
+    from nameparser._config_shim import _SENTINEL_PAIRS
+
+    assert set(c.nickname_delimiters) == set(_SENTINEL_PAIRS)
+    assert {"quoted_word", "double_quotes", "parenthesis",
+            "smart_double_quotes", "guillemets"} <= set(_SENTINEL_PAIRS)
     assert len(c.maiden_delimiters) == 0
 
 
@@ -353,6 +359,19 @@ def test_snapshot_patronymic_and_middle_flags() -> None:
     assert policy.middle_as_family is True
 
 
+def test_typographic_delimiter_sentinels_present_and_movable() -> None:
+    # #273: the eight typographic pairs surface in the v1 API as new
+    # named sentinels, so the documented keyed idioms (pop/move/del)
+    # work on them exactly like the original trio
+    c = Constants()
+    assert "smart_double_quotes" in c.nickname_delimiters
+    assert "guillemets" in c.nickname_delimiters
+    c.maiden_delimiters["guillemets"] = c.nickname_delimiters.pop("guillemets")
+    _, policy, _ = c._snapshot()
+    assert ("«", "»") in policy.maiden_delimiters
+    assert ("«", "»") not in policy.nickname_delimiters
+
+
 def test_snapshot_delimiter_bucket_move() -> None:
     c = Constants()
     c.maiden_delimiters["parenthesis"] = c.nickname_delimiters.pop("parenthesis")
diff --git a/tests/v2/test_policy.py b/tests/v2/test_policy.py
@@ -99,10 +99,13 @@ def test_maiden_delimiters_win_over_nickname_defaults() -> None:
     p = Policy(maiden_delimiters=frozenset({("(", ")")}))
     assert ("(", ")") in p.maiden_delimiters
     assert ("(", ")") not in p.nickname_delimiters
-    # canonicalization: the explicit-removal spelling converges to the
-    # same value (equal AND same hash -- cache keys agree)
+    # canonicalization: the explicit-removal spelling (the documented
+    # set-math idiom) converges to the same value (equal AND same hash
+    # -- cache keys agree)
+    from nameparser import DEFAULT_NICKNAME_DELIMITERS
+
     explicit = Policy(
-        nickname_delimiters=frozenset({("'", "'"), ('"', '"')}),
+        nickname_delimiters=DEFAULT_NICKNAME_DELIMITERS - {("(", ")")},
         maiden_delimiters=frozenset({("(", ")")}),
     )
     assert p == explicit and hash(p) == hash(explicit)
