Skip to content

Commit 17e101b

Browse files
derek73claude
andcommitted
Guard against complexity regressions, not just slow parses
Two quadratics shipped this cycle (_extract's mask-overlap scan, ParsedName's token-subset check) and both were caught in review rather than by a test. The existing perf tests cannot catch that class: they bound absolute cost on constant-size input, and their workload is a short name with no delimiters, so a stage quadratic in the number of delimiter pairs stays far inside the one-second bound. Measured: that workload runs in 0.067s against a 1.0s limit. Add a growth test instead -- time a repeated pathological unit at n and 4n, assert the ratio stays near the linear 4. Seven units, one per stage inner loop (matched pairs, the open==close path, both unmatched sweeps, long token streams, comma segments, title chains). Calibrated against the real quadratic rather than guessed, which mattered: the growth is mixed, so the textbook 16x never appears at a testable size. At n=200 the quadratic measures 7.7x and an 8.0 bound passes it -- the first version of this test was vacuous. At n=400 the populations separate (clean 4.1-4.3, quadratic 9.2), so the bound sits at 6.5. Verified both ways: passes clean, and reverting the _extract fix fails exactly the two shapes that build masked spans. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 8e77693 commit 17e101b

1 file changed

Lines changed: 69 additions & 1 deletion

File tree

tests/v2/test_benchmark.py

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,21 @@
11
"""Perf smoke (core spec §7 tail): parse cost stays v1-comparable
22
(microseconds per name). Deliberately generous bound -- guards against
3-
order-of-magnitude regressions, does not gate normal variance."""
3+
order-of-magnitude regressions, does not gate normal variance.
4+
5+
Two different things are measured here, and the distinction matters.
6+
The thousand-names tests below bound ABSOLUTE cost on constant-size
7+
input; the scaling test bounds GROWTH in input length. Neither
8+
subsumes the other, and the absolute tests are structurally blind to a
9+
complexity regression: their workload is a short name with no
10+
delimiters, so a stage that goes quadratic in the number of delimiter
11+
pairs stays far inside the one-second bound. Two such quadratics
12+
(_extract's mask-overlap scan, ParsedName's token-subset check) shipped
13+
and were caught in review rather than here -- hence the scaling test.
14+
"""
415
import time
516

17+
import pytest
18+
619
from nameparser import parse
720

821

@@ -26,3 +39,58 @@ def test_facade_thousand_names_under_a_second() -> None:
2639
HumanName(f"Dr. Juan{i} de la Vega III")
2740
elapsed = time.perf_counter() - start
2841
assert elapsed < 1.0, f"1000 facade parses took {elapsed:.2f}s"
42+
43+
44+
# Pathological shapes: each repeats a unit that drives one stage's inner
45+
# loop. Cheap per unit, so a superlinear stage shows up as growth rather
46+
# than as a slow parse.
47+
_SHAPES = {
48+
"delimiter_pairs": "(a) ", # extract: matched pairs -> masked spans
49+
"quote_pairs": '"a" ', # extract: the open==close path
50+
"unmatched_opens": "( ", # extract: the bulk unmatched-open sweep
51+
"unmatched_closes": ") ", # extract: the unmatched-close sweep
52+
"plain_tokens": "a ", # tokenize/assign: long token stream
53+
"commas": "a, ", # segment: many comma segments
54+
"titles": "Dr. ", # group: one long title chain
55+
}
56+
57+
_BASE = 400
58+
_FACTOR = 4
59+
# Calibrated, not guessed. A quadratic here is MIXED -- linear work
60+
# dominates at small sizes and the quadratic term takes over slowly --
61+
# so the textbook 16x never appears at a testable size, and picking the
62+
# operating point matters more than the bound. Measured against the
63+
# real _extract mask-overlap quadratic this test exists to catch:
64+
#
65+
# base clean ratio quadratic ratio
66+
# 200 3.9 - 4.3 7.7 <- under an 8.0 bound: useless
67+
# 400 4.1 - 4.3 9.2
68+
#
69+
# At 400 the two populations separate, so the bound sits between them:
70+
# ~1.5x headroom over the worst clean run (noise on a shared runner)
71+
# and ~1.4x under the quadratic. Lower this only with fresh numbers.
72+
_MAX_RATIO = 6.5
73+
74+
75+
def _best(text: str, repeats: int = 7) -> float:
76+
"""Minimum of several runs: on a shared runner the mean carries the
77+
noise of whatever else is running, while the minimum approaches the
78+
true cost."""
79+
parse(text) # warm the default parser cache
80+
best = float("inf")
81+
for _ in range(repeats):
82+
start = time.perf_counter()
83+
parse(text)
84+
best = min(best, time.perf_counter() - start)
85+
return best
86+
87+
88+
@pytest.mark.parametrize("unit", _SHAPES.values(), ids=list(_SHAPES))
89+
def test_parse_cost_grows_no_worse_than_linearly(unit: str) -> None:
90+
small = _best(unit * _BASE)
91+
large = _best(unit * (_BASE * _FACTOR))
92+
ratio = large / small
93+
assert ratio < _MAX_RATIO, (
94+
f"{unit!r} x{_BASE} took {small * 1e3:.2f}ms, "
95+
f"x{_BASE * _FACTOR} took {large * 1e3:.2f}ms -- {ratio:.1f}x for "
96+
f"{_FACTOR}x the input, which is superlinear (linear is ~{_FACTOR})")

0 commit comments

Comments
 (0)