Skip to content

Commit d7df3ce

Browse files
derek73claude
andcommitted
docs: strip flags exclude characters from tokenization, not the input
'Removes ... from the input' contradicted the spans-into-original invariant: parse('John 😀 Smith').original keeps the emoji; the tokenizer just never includes those characters in any token, so they appear in no field or rendered view. Say that. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 4531538 commit d7df3ce

2 files changed

Lines changed: 9 additions & 5 deletions

File tree

docs/customize.rst

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,12 +97,13 @@ listed below.
9797
strict acronym form.
9898
* - ``strip_emoji``
9999
- ``bool``
100-
- Removes emoji from the input before parsing. Defaults to
100+
- Excludes emoji from tokenization — they appear in no field or
101+
rendered view, though ``original`` keeps them. Defaults to
101102
``True``.
102103
* - ``strip_bidi``
103104
- ``bool``
104-
- Removes bidirectional control characters from the input before
105-
parsing. Defaults to ``True``.
105+
- Excludes bidirectional control characters the same way.
106+
Defaults to ``True``.
106107

107108
.. doctest::
108109

nameparser/_policy.py

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,9 +87,12 @@ class Policy:
8787
#: Allows a post-comma segment to read as a suffix via the lenient
8888
#: initial-shaped test; False requires the strict acronym form.
8989
lenient_comma_suffixes: bool = True
90-
#: Removes emoji from the input before parsing.
90+
#: Excludes emoji from tokenization: they appear in no token,
91+
#: field, or rendered view. The original string keeps them (input
92+
#: is never modified -- spans stay true).
9193
strip_emoji: bool = True
92-
#: Removes bidirectional control characters before parsing.
94+
#: Excludes bidirectional control characters from tokenization the
95+
#: same way.
9396
strip_bidi: bool = True # =False replaces v1's opt-out CONSTANTS.regexes.bidi = False
9497

9598
# in the class body so @dataclass(slots=True) keeps them

0 commit comments

Comments
 (0)