Put a non-ASCII shape in the scaling benchmark (#308)

Every unit in _SHAPES was pure ASCII, so script_segment returned at its
isascii() bail before either the #308 peel or the #271 surname site ran:
the file's own docstring warns the absolute tests are structurally blind
to a complexity regression, and for this stage the scaling test was
blind too. A repeated 씨 walks the peel's site scan-back over the whole
run -- every token is a post-nominal -- and measures 4.0-4.3 at base
800, inside the clean column the ASCII shapes already occupy, so
_MAX_RATIO is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: derek73

AGENTS.md

@@ -186,7 +186,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_
 
 **`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_title_key` joins per-word `_normalize` and DROPS words that fold away; keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version".
 
-**Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over nine shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The nine shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`); measure before pruning one.
+**Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over ten shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The ten shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`, non-ASCII input only via `honorifics` — the other nine are pure ASCII, so `script_segment` returns at its bail and the CJK stages go unmeasured); measure before pruning one.
 
 **Expected-failure tests use `@pytest.mark.xfail`** — the conftest parametrized fixture breaks `@unittest.expectedFailure`; always use `@pytest.mark.xfail` instead.
 

tests/v2/test_benchmark.py

@@ -50,14 +50,22 @@ def test_facade_thousand_names_under_a_second() -> None:
 # before pruning one that looks redundant:
 #
 #   dimension                 covered by
-#   token count               all nine
+#   token count               all ten
 #   piece count               unmatched_*, plain_tokens, commas, titles
 #   SEGMENT count             commas ONLY -- deleting it leaves every
 #                             segment-keyed regression unguarded
 #   intra-piece accumulation  particles (one 799-token piece),
 #                             conjunctions (800) -- the merge() quadratic
 #   masked-span count         delimiter_pairs, quote_pairs (0 pieces:
 #                             everything is consumed as a delimited run)
+#   NON-ASCII input           honorifics ONLY -- every other unit here is
+#                             pure ASCII, so script_segment returns at
+#                             its isascii() bail and neither the #308
+#                             peel nor the #271 surname site is measured
+#                             at all. A repeated 씨 is what walks the
+#                             peel's site scan: every token is a
+#                             post-nominal, so the scan-back crosses the
+#                             whole run before declining
 _SHAPES = {
     "delimiter_pairs": "(a) ",      # extract: matched pairs -> masked spans
     "quote_pairs": '"a" ',          # extract: the open==close path
@@ -68,6 +76,7 @@ def test_facade_thousand_names_under_a_second() -> None:
     "titles": "Dr. ",               # group: one long title chain
     "particles": "van ",            # group: the prefix-chain inner loop
     "conjunctions": "and ",         # group: merge() accumulating one piece
+    "honorifics": "씨 ",             # script_segment: the peel's site scan
 }
 
 _BASE = 800
@@ -95,6 +104,9 @@ def test_facade_thousand_names_under_a_second() -> None:
 # a gap that is not there: at 800 the bound has ~1.4x over the worst
 # clean run (noise headroom on a shared runner) and ~1.3x under the
 # weakest quadratic. Re-derive BOTH numbers when adding a shape.
+# The tenth shape (honorifics) was measured into the clean column when
+# it arrived: 4.0-4.3 at base 800 across repeated runs, inside the range
+# the ASCII shapes already occupy, so neither number moved.
 _MAX_RATIO = 6.0