Skip to content

Commit 022fd07

Browse files
derek73claude
andcommitted
Guard the remaining config data invariants at import time
prefixes.py asserts its own invariants at import so a bad edit fails immediately rather than drifting until a test happens to catch it. The other data modules had no such guard, and two of them had already drifted. Data fixes: - 'esq' was in both SUFFIX_ACRONYMS and SUFFIX_NOT_ACRONYMS. Those are the two halves of one distinction, not overlapping categories. _classify checks suffix_words first, so the acronym membership was unreachable -- removing it changes 0 of the 486 differential-corpus parses. Present in v1's data too; inherited, not introduced. - TITLES contained 'actor ' and 'television ' with trailing spaces, so "'actor' in TITLES" was False. Parsing was unaffected because both Lexicon and v1's SetManager normalize on ingest, which is exactly what let the typo survive. Also inherited from v1. Assertions added: - suffixes.py: SUFFIX_ACRONYMS_AMBIGUOUS <= SUFFIX_ACRONYMS, and SUFFIX_ACRONYMS disjoint from SUFFIX_NOT_ACRONYMS. - titles.py: FIRST_NAME_TITLES <= TITLES. True by construction today (TITLES = FIRST_NAME_TITLES | {...}), so it pins that against a future edit making TITLES standalone. - All six vocabulary modules plus CAPITALIZATION_EXCEPTIONS' keys: entries stored lowercase and whitespace-free. This is the check that found the TITLES typo. Deliberately NOT asserted: cross-category disjointness. TITLES and SUFFIX_ACRONYMS share 15 entries (lt, md, phd, cpt, ...) and every one is a legitimate homograph resolved by position, as are TITLES/PREFIXES and SUFFIX_ACRONYMS/PREFIXES. The predicates use .strip().lower() rather than _lexicon._normalize: config/ is the lower layer and _lexicon imports from it, so importing back would invert the layering. Each guard was verified to fire on a mutated copy of its own data -- an assertion that cannot fail is worth nothing. Note `assert` is stripped under `python -O`. These protect the shipped data during development; Lexicon re-checks the relationships at construction, which is what protects a caller's own vocabulary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 182249a commit 022fd07

7 files changed

Lines changed: 67 additions & 3 deletions

File tree

nameparser/config/bound_first_names.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,10 @@
2121
'أم', # "umm" (mother of), hamza spelling
2222
'ام', # "umm", hamza-less spelling
2323
}
24+
25+
26+
# See prefixes.py: entries are looked up in normalized form, so a stray
27+
# capital or trailing space makes one unreachable by direct membership
28+
# test even though the parser's own ingest normalizes it away.
29+
assert all(w == w.strip().lower() for w in BOUND_FIRST_NAMES), \
30+
"BOUND_FIRST_NAMES entries must be stored lowercase and whitespace-free"

nameparser/config/capitalization.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,9 @@
88
"""
99
Any pieces that are not capitalized by capitalizing the first letter.
1010
"""
11+
12+
13+
# See prefixes.py: keys are looked up in normalized form. Only the keys --
14+
# the values are the exact-cased replacements, so they are cased on purpose.
15+
assert all(k == k.strip().lower() for k in CAPITALIZATION_EXCEPTIONS), \
16+
"CAPITALIZATION_EXCEPTIONS keys must be stored lowercase and whitespace-free"

nameparser/config/conjunctions.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,10 @@
2626
"of" and "the" are also include to facilitate joining multiple titles,
2727
e.g. "President of the United States".
2828
"""
29+
30+
31+
# See prefixes.py: entries are looked up in normalized form, so a stray
32+
# capital or trailing space makes one unreachable by direct membership
33+
# test even though the parser's own ingest normalizes it away.
34+
assert all(w == w.strip().lower() for w in CONJUNCTIONS), \
35+
"CONJUNCTIONS entries must be stored lowercase and whitespace-free"

nameparser/config/maiden_markers.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,10 @@
3535
abbreviation "f." (collides with the initial "F." — only the full
3636
participles are safe).
3737
"""
38+
39+
40+
# See prefixes.py: entries are looked up in normalized form, so a stray
41+
# capital or trailing space makes one unreachable by direct membership
42+
# test even though the parser's own ingest normalizes it away.
43+
assert all(w == w.strip().lower() for w in MAIDEN_MARKERS), \
44+
"MAIDEN_MARKERS entries must be stored lowercase and whitespace-free"

nameparser/config/prefixes.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,3 +126,8 @@
126126
"NON_FIRST_NAME_PREFIXES must stay a subset of PREFIXES"
127127
assert not (NON_FIRST_NAME_PREFIXES & BOUND_FIRST_NAMES), \
128128
"NON_FIRST_NAME_PREFIXES must stay disjoint from BOUND_FIRST_NAMES"
129+
# Entries are looked up in normalized form; a stray capital or trailing
130+
# space makes one unreachable by direct membership test even though the
131+
# parser's own ingest normalizes it away.
132+
assert all(w == w.strip().lower() for w in PREFIXES), \
133+
"PREFIXES entries must be stored lowercase and whitespace-free"

nameparser/config/suffixes.py

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -391,7 +391,6 @@
391391
'emt-p',
392392
'enp',
393393
'erd',
394-
'esq',
395394
'evp',
396395
'faafp',
397396
'faan',
@@ -710,3 +709,20 @@
710709
when matching against these pieces.
711710
712711
"""
712+
713+
714+
# Guard the invariants the docstrings above promise, so a future edit that
715+
# breaks them fails at import time instead of silently drifting until a test
716+
# happens to catch it (same rationale as prefixes.py). Note `assert` is
717+
# stripped under `python -O`; Lexicon re-checks the relationships at
718+
# construction, which is what protects a caller's own vocabulary.
719+
assert SUFFIX_ACRONYMS_AMBIGUOUS <= SUFFIX_ACRONYMS, \
720+
"SUFFIX_ACRONYMS_AMBIGUOUS must stay a subset of SUFFIX_ACRONYMS"
721+
assert not (SUFFIX_ACRONYMS & SUFFIX_NOT_ACRONYMS), \
722+
"a suffix is an acronym or a word, not both: " \
723+
f"{sorted(SUFFIX_ACRONYMS & SUFFIX_NOT_ACRONYMS)}"
724+
# Entries are looked up in normalized form, so a stray capital or a
725+
# trailing space makes an entry unreachable by direct membership test even
726+
# though the parser's own ingest would paper over it.
727+
assert all(w == w.strip().lower() for w in SUFFIX_ACRONYMS | SUFFIX_NOT_ACRONYMS), \
728+
"suffix entries must be stored lowercase and whitespace-free"

nameparser/config/titles.py

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@
8989
'academic',
9090
'acolyte',
9191
'activist',
92-
'actor ',
92+
'actor',
9393
'actress',
9494
'adept',
9595
'adjutant',
@@ -664,7 +664,7 @@
664664
'teacher',
665665
'technical',
666666
'technologist',
667-
'television ',
667+
'television',
668668
'tenor',
669669
'theater',
670670
'theatre',
@@ -770,3 +770,19 @@
770770
'श्रीमती', # Shrimati (Mrs.)
771771
'डॉ', # Dr. abbreviation
772772
}
773+
774+
775+
# Guard the invariants at import time, so a bad edit fails here instead of
776+
# drifting silently until a test happens to catch it (see prefixes.py).
777+
# The subset rule holds by construction today -- TITLES is defined as
778+
# FIRST_NAME_TITLES | {...} -- so this pins it against a future edit that
779+
# makes TITLES a standalone set. Lexicon enforces the same rule on
780+
# caller-supplied vocabulary; `assert` is stripped under `python -O`.
781+
assert FIRST_NAME_TITLES <= TITLES, \
782+
"FIRST_NAME_TITLES must stay a subset of TITLES"
783+
# Entries are looked up in normalized form, so a stray capital or a
784+
# trailing space makes an entry unreachable by direct membership test
785+
# ('actor' in TITLES was False) even though the parser's own ingest
786+
# normalizes and papers over it. TITLES covers FIRST_NAME_TITLES.
787+
assert all(w == w.strip().lower() for w in TITLES), \
788+
"TITLES entries must be stored lowercase and whitespace-free"

0 commit comments

Comments
 (0)