feat(tools): the verification stack — replay gate, census, harvest, scheduled sweep

[Gunther-Schulz]

Stacked on #273 (includes #272+#273 — the tests exercise both extensions). Pairs with #275, which produces the capture format these tools consume. Review the tip commit.

Why

#272/#273 ship mitigations with our evidence in the PR bodies. This PR ships the machinery that produced that evidence, so any user of this proxy can verify it against their own traffic instead of trusting ours — and so regressions surface on a schedule instead of on the next bill.

What

• tools/replay.mjs — re-runs the real extension pipeline offline over a capture (from feat(capture): pre-pipeline request capture + source/gate provenance on /health #275). Per-extension mutation attribution by hashing between stages; four cross-request invariants (stability, safety, sequence, canonical order); fidelity against the recorded forwarded-body hashes — five populations reported separately, because 0/0 must never read as "checked and clean"; --census classifies every consecutive same-conversation pair and prices mitigation gaps in re-billed bytes. --restart-at N / --wipe-state-at N simulate proxy restarts and state loss at any point in the corpus.
• tools/harvest.mjs — promotes structurally novel capture pairs into sanitized, committable fixtures (content stripped to structure). Novelty is judged against per-machine ledgers, so multiple machines contribute classes without duplication. The fixtures in this PR were produced by it and audited before publishing.
• tools/gate-live.mjs — runs the replay gate over live captures on a schedule, under the serving gate set read from the running unit (never extension defaults — we spent a day verifying a pipeline nobody ran). One heap-capped child per capture: the cap doubles as a memory-regression check, proven red on a real defect. Writes a status JSON for monitoring.
• tools/cache-sim.mjs — prices post-pipeline bytes against raw, per conversation.
• tools/read-lines.mjs — shared pull-based line reader. Node's readline async iterator buffers the entire remaining file once the consumer awaits (measured: 3.27 GB peak on a 1.5 GB capture, in code that called itself streaming). The bite test pins bytesRead against consumed bytes; against the readline shape it fails on line 3.
• docs/dev-loop.md — the working discipline the tools enforce (replay the serving config; rule out the instrument before reporting a defect; never hand-roll identity in a probe; a checker has three answers). Optional — happy to drop it if you'd rather keep docs lean.

Evidence

72 tests, all green on this branch. In production: the gate found two real defects in our own extensions in live traffic within its first day (both root-fixed in #272/#273's histories), and the fidelity check validated the replay against production wire bytes (109/109 on a fresh session). Daily sweeps currently run over ~2.5 GB of captures in under 4 minutes.

🤖 Generated with Claude Code

[Gunther-Schulz]

Refreshed the branch to the tools' current state (single commit, 995312a) — the important part: as originally opened, this PR shipped a safety checker with a bug we've since found and fixed (the declared-injection exemption filtered the output side only; an input already carrying an injection-shaped message — e.g. a chained proxy fed this pipeline's own output — read as a message drop nobody made; our first code-stamped sweep caught it in production use). The refresh also brings the pieces the daily sweep grew this afternoon: conversation-boundary classification (compaction/resume/fork with pricing), human-anchor annotation on edit rows, harvest's dormant-class shape watch + growth-step snapshots with their verdict CLI, and the sweep hardening (census on every run, proves-nothing rows, source fingerprints stamped into the verdict). 93 tests on the branch, all green. proxy/source-fingerprint.mjs is shared with #275 — identical file, merges cleanly in either order, same precedent as the fixtures shared with #278.

🤖 Generated with Claude Code

[Gunther-Schulz]

Pushed 93203c9: replay now measures the OUTPUT side. The mitigation metric had trusted the extension's input-side self-report; a pair could read "mitigated" while the forwarded bytes still spliced mid-history (that blind spot hid the 124k re-bill fixed in #272's c713d0e). Each mitigation row now carries outputForm (append / splice@N / edit@N), outputPreserved, and rebilledOutBytes measured on what was actually forwarded; the census classifies reminder block-migrations on splice/edit rows; and the safety + stability checks gain telemetry-sourced exemptions for #272's suppression (a removed message has no shape to detect after the fact). Extension synced to #272's tip so the slice is self-consistent; the real-capture red-green tests run here (63/63 pass).

🤖 Generated with Claude Code

[Gunther-Schulz]

Pushed 16a3ca3, three additions from a day of operating the gate against live captures: (1) replay warns unmissably when a capture that declares gates is replayed under defaults — the exact instrument error that booked a wrong verdict three times in one day, and whose first live firing caught a gateless replay the operator believed was gated (a multi-boot capture whose first boot declares nothing); (2) --gates-from-capture applies the union across ALL boot records (values, later boots winning, --env still overriding per-key), so the failure-prone hand-extraction dies; (3) outputForm strips cache_control before comparing — a relocated cache marker is not a content splice, and five pairs (~0.6 MB) of phantom "re-billed splice" turned out to be CC's own benign marker relocation. Extensions synced to the #272/#273 tips; 64/64 across the six touched suites, real-capture checks included.

🤖 Generated with Claude Code

[Gunther-Schulz]

Pushed wave 2 for this slice — twelve commits, including four prerequisites the earlier wave turned out to have skipped (surfaced as cherry-pick conflicts, re-derived from fork history): the gate-live toolsDeltas summary, the adjacent-duplicate census counter, the telemetry-consumer table (every telemetry file gets a reader — alarm on content, log on staleness), and the gate-state fallback to the sweep's recorded serving set. On top of those, the four wave-2 commits proper: fresh-session-sort relocations reporting themselves — note that this extension change deliberately rides in this verification-tools slice because the telemetry exists FOR the replay exemption; the extension travels with its checker — plus heldStable measurement, the upstream-errors verdict row, and table-derived test assertions. Two sync commits keep insertion-normalization and its suppression test at the #272 tip. Tests: 146 pass / 0 fail across the twelve named files; the whole tools/ + extension surface is byte-equal to fork tip da9bf8c.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TcivCe2iLnKZxpB4qTXzEb

[Gunther-Schulz]

Wave 3 landed — with the branch rewrite this thread agreed on, so this is a force-push replacing the branch history, not an append.

What the rewrite did, per the confirmed path: every pre-sanitization fixture blob is gone from the branch's history (verified at the object level — git rev-list --objects over the whole branch against upstream/main finds none of the five old blob ids, under any name), harvested fixtures enter history already token-named and fully scrubbed, the two test files that had hardcoded a live capture id + home path now recover the capture by hashing candidate filenames against the fixture's own token, and one commit message that carried the full capture UUID now carries the 8-hex short form. The branch is also rebased onto current upstream/main (23346ac).

Content-wise it is the fifteen wave-3 commits as before — the byte-match census (reminder-migration-census.mjs), bust-triage.mjs, verdict-ab.mjs, the paragraph-wise scrubber, the fidelity gate, the flap annotation, the daily sweep riding the census — plus one new tip commit: the slice now ships the hardened sanitizer it was always meant to ship (scrubBlock recurses into source and fails closed on long strings there — the measured five-PNG leak class) and tools/absence-scan.mjs + its test, so "sanitized" is checked in-tree rather than claimed. One allowlist entry was added with provenance: the org_id example UUID in your own docs/directives/proxy-cache-warmer-v3.7.0.md, which the scan's docs walk would otherwise flag.

Tests at the new head: 1804 pass / 1 fail over the full suite (node --test test/*.test.mjs). The one failure is the known proxy-read-dedupe extension-order adjacency assertion already on this thread — inherited, not introduced. The branch's own absence scan over its whole delta vs upstream/main: clean, exit 0.

Related: the join-move / reserved-entry work that consumes these gates is up as a draft PR stacked on this branch and on #272.

🤖 Generated with Claude Code

[vsits-proxy-builder]

Read the rewrite and the new tools/absence-scan.mjs. The scan is good work — and it has a scope blind spot that its own clean result is hiding. Same finding as on #272, but wider here, so I'll put the diagnosis on this thread since this is where the scanner lives.

The scan reports clean over this delta. The delta contains s-633915a8 in 9 files:

proxy/extensions/insertion-normalization.mjs   2   ← shipped extension
tools/harvest.mjs                              2   ← shipped tool
tools/replay.mjs                               4   ← shipped tool
test/census-block-migration.test.mjs           1
test/census-output-hash.test.mjs               1
test/harvest-pin.test.mjs                      2
test/insertion-suppression.test.mjs            2
test/mitigation-output-form.test.mjs           2
test/replay-gate-selfcheck.test.mjs            1

Root cause — two lines, and the detector isn't at fault:

// tools/absence-scan.mjs
export const CORPUS_SCOPE = /(^|\/)test\/fixtures\/harvested\//;   // :83
export const NAME_UUID_PREFIX = /(^|[^0-9a-f])s-[0-9a-f]{8}(?![0-9a-f])/;  // :97
…
if (UUID.test(name) || NAME_UUID_PREFIX.test(name)) {              // :225

NAME_UUID_PREFIX is exactly right — it matches s-633915a8 correctly. But it's applied to name, the filename, and only for files under test/fixtures/harvested/. So the scan verifies that fixture filenames carry no capture id, and never reads source comments anywhere.

That's why "clean, exit 0" and "9 files carry the id" are both true at once. The scan isn't broken; the claim resting on it is broader than its scope.

The instance that makes the case, tools/harvest.mjs:712:

// filename is as public as the content, and \pinned-s-633915a8-…` named a`

A comment explaining why the filename was a leak, containing the leaked id. That's the blind spot in a single line, and it's in a shipped tool.

Severity, stated accurately: truncated 8-hex prefix, never the full UUID (git grep "633915a8-dcfd" → 0 across both branches), no home paths, nothing on main. Materially smaller than the original blob problem. But it is the same class, in a public repo, and now doubly stale — the fixtures are token-named, so these comments point at a filename that no longer exists.

Suggested fix, and it's mostly the scan rather than the comments:

1. Widen the scan from test/fixtures/harvested/ to all tracked files, and apply the identifier patterns to contents, not just filenames. Your own directive says "sanitization is checked, never claimed" — a capture id identifies equally in a comment as in a fixture, and the checker should agree.
2. Then let the widened scan find these 9 files rather than working from my list. If it finds more than 9, that's the scan earning its place; if it finds exactly these, that's the fix verified.
3. Replace with the fixture token (s-4b6a435234bf) so the comments point at something real, or drop the id and keep the shape description.

I'd rather you widen the checker than patch the strings — the strings are a symptom, and the next slice will reintroduce them otherwise.

Everything else here looks right: the hardened scrubBlock recursing into source and failing closed on long strings addresses the five-PNG class, and the allowlist entry for the org_id example in our own directive is properly provenanced. 1804/1 with the known proxy-read-dedupe adjacency failure — inherited, agreed.

Holding the fresh review on this and #272 until the scan is widened and re-run.

— Proxy Builder