Skip to content

Commit 3d2095b

Browse files
test: real-pair checks fall back to pinned fixtures — rotation stops eating the evidence
Wires the fixture-fallback into both real-pair tests (test/insertion-suppression.test.mjs, test/mitigation-output-form.test.mjs): live capture present -> unchanged path (readCapture); capture absent -> fall back to the pinned fixture via readPinnedFixture if one exists; both absent -> skip with a stated reason, as before. Both readers yield the same [n, line] tuple shape, so the replay loop in each test is unchanged either way. Both the capture path and the fixture path are overridable via CACHE_FIX_TEST_CAPTURE_OVERRIDE / CACHE_FIX_TEST_FIXTURE_OVERRIDE so the fallback's own red-green test never has to touch the real capture file, which is read-only evidence shared with other concurrent work. Commits the first real pin: test/fixtures/harvested/pinned-s-633915a8-26-28 .json, the n=26->28 pair both real-pair tests already reference in their comments, produced with `node tools/harvest.mjs --pin s-633915a8 26..28` (holds the full prefix 0..28 per the previous commit's replay-from-start finding) and verified against the fixed scrubber from the prior commit: replaying it through the actual pipeline reproduces the live capture's exact values (mitigated=true, outputForm="append", suppressed=1 at index 31, 0 safety violations). test/harvest-pin.test.mjs gains the fallback's red-green proof: both real-pair test files are run as actual subprocesses (not re-derived) with env overrides -- capture+fixture both absent skips (never a false pass), capture absent + the real committed fixture present runs and PASSES (never a false fail). Both real-pair suites are green in both modes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> (cherry picked from commit 2dfe0f0)
1 parent 9702540 commit 3d2095b

4 files changed

Lines changed: 135 additions & 13 deletions

File tree

test/fixtures/harvested/pinned-s-4b6a435234bf-26-28.json

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

test/harvest-pin.test.mjs

Lines changed: 75 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,18 @@
66
// (s-633915a8, pair n=26->28) and SKIP once that capture rotates out of the
77
// per-machine retention window (~3 days, docs/dev-loop.md "Corpus
88
// hygiene"). `harvest --pin <key> <n..m>` freezes the sanitized range as a
9-
// committed, rotation-immune fixture that both real-pair tests can fall
10-
// back to (fallback wiring lands in a follow-up commit; this one covers the
11-
// pin mechanism itself — unit-level, on a tiny synthetic capture).
9+
// committed, rotation-immune fixture; both real-pair tests fall back to it
10+
// when the live capture is gone.
11+
//
12+
// Two things have to hold or the mechanism is worse than useless:
13+
// - the pin mechanism itself: it writes a sanitized, well-formed fixture
14+
// (unit-level, tiny synthetic capture);
15+
// - the FALLBACK actually works on the real files: capture-absent +
16+
// fixture-absent skips (never a false pass), capture-absent +
17+
// fixture-present runs and PASSES using the real committed fixture
18+
// (never a false fail) — checked by literally invoking the two
19+
// real-pair test files as subprocesses with env overrides, never by
20+
// re-deriving their assertions here.
1221

1322
import { test } from "node:test";
1423
import assert from "node:assert/strict";
@@ -189,3 +198,66 @@ test("readPinnedFixture: yields [n, line] tuples whose parsed records match what
189198
"indices are 0-based and contiguous, same shape readCapture's own [n, line] yields",
190199
);
191200
});
201+
202+
// =====================================================================
203+
// Fallback red-green — the actual real-pair tests, run as subprocesses
204+
// =====================================================================
205+
//
206+
// Not a re-derivation of what insertion-suppression.test.mjs and
207+
// mitigation-output-form.test.mjs assert: this literally invokes them with
208+
// env overrides (CACHE_FIX_TEST_CAPTURE_OVERRIDE /
209+
// CACHE_FIX_TEST_FIXTURE_OVERRIDE, both files) pointed at nonexistent paths
210+
// or at the real committed fixture, and reads their own TAP output — the
211+
// only way to know the fallback genuinely works end to end rather than
212+
// merely compiling. Never touches the real capture file
213+
// (~/.claude/cache-fix-captures/s-633915a8-...), which is read-only
214+
// evidence.
215+
216+
const REAL_PAIR_TESTS = [
217+
{ file: "mitigation-output-form.test.mjs", namePattern: "mitigation output-form: real capture n=26" },
218+
{ file: "insertion-suppression.test.mjs", namePattern: "real capture n=26->28: pin-and-suppress" },
219+
];
220+
const COMMITTED_FIXTURE = join(__dirname, "fixtures", "harvested", "pinned-s-4b6a435234bf-26-28.json");
221+
222+
// --test-reporter=tap: a stable, greppable "# pass N" / "# skipped N" / "#
223+
// fail N" summary — the default reporter's exact wording ("ℹ pass N", no
224+
// leading "#") is not a documented contract to grep against.
225+
//
226+
// NODE_TEST_CONTEXT / NODE_TEST_WORKER_ID must NOT reach the child: this
227+
// file itself runs under `node --test`, which sets both; inherited by a
228+
// NESTED `node --test` invocation, the child silently emits nothing to
229+
// stdout (observed directly — reporter output present unset, empty string
230+
// captured when inherited) rather than erroring, which would have looked
231+
// like a false "fallback broken" red instead of a harness artifact.
232+
function runRealPairTest({ file, namePattern }, env) {
233+
const childEnv = { ...process.env, ...env };
234+
delete childEnv.NODE_TEST_CONTEXT;
235+
delete childEnv.NODE_TEST_WORKER_ID;
236+
const result = execFileSync(
237+
process.execPath,
238+
["--test", "--test-reporter=tap", `--test-name-pattern=${namePattern}`, join(__dirname, file)],
239+
{ encoding: "utf-8", cwd: REPO, env: childEnv, stdio: ["ignore", "pipe", "pipe"] },
240+
);
241+
return result;
242+
}
243+
244+
for (const spec of REAL_PAIR_TESTS) {
245+
test(`fallback RED: ${spec.file} skips (not fails) when capture and fixture are both absent`, () => {
246+
const out = runRealPairTest(spec, {
247+
CACHE_FIX_TEST_CAPTURE_OVERRIDE: "/nonexistent/no-such-capture.jsonl",
248+
CACHE_FIX_TEST_FIXTURE_OVERRIDE: "/nonexistent/no-such-fixture.json",
249+
});
250+
assert.match(out, /# pass 0/);
251+
assert.match(out, /# skipped 1/);
252+
assert.match(out, /COULD NOT VERIFY/);
253+
});
254+
255+
test(`fallback GREEN: ${spec.file} runs and passes from the committed pinned fixture when the capture is absent`, () => {
256+
assert.ok(existsSync(COMMITTED_FIXTURE), "the committed n=26->28 fixture must exist for this check to mean anything");
257+
const out = runRealPairTest(spec, {
258+
CACHE_FIX_TEST_CAPTURE_OVERRIDE: "/nonexistent/no-such-capture.jsonl",
259+
});
260+
assert.match(out, /# pass 1/);
261+
assert.match(out, /# fail 0/);
262+
});
263+
}

test/insertion-suppression.test.mjs

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ import {
3535
pinnedBlockHashes,
3636
findSuppressibleDuplicate,
3737
} from "../proxy/extensions/insertion-normalization.mjs";
38+
import { readPinnedFixture } from "../tools/harvest.mjs";
3839

3940
const __dirname = dirname(fileURLToPath(import.meta.url));
4041
const REPO = join(__dirname, "..");
@@ -228,8 +229,20 @@ test("classifyPinned: an assistant-role standalone entry is never suppressed, ev
228229
// asserts the PRE-fix values (outputForm==="edit@31") and is NOT in this
229230
// change's write boundary; running it after this fix is expected to fail,
230231
// and that is surfaced in the closing report rather than fixed here.
232+
//
233+
// Fixture-fallback (BACKLOG.md "READY — harvest --pin freezes evidence
234+
// ranges as fixtures"): capture rotated away -> fall back to the pinned
235+
// fixture at test/fixtures/harvested/pinned-s-4b6a435234bf-26-28.json (`node
236+
// tools/harvest.mjs --pin <key> n..m`); both absent -> skip. Both paths are
237+
// overridable via env for the fallback's own red-green test
238+
// (test/harvest-pin.test.mjs) — never by editing the real capture, which is
239+
// read-only evidence shared with other work.
231240
const REAL_CAPTURE =
241+
process.env.CACHE_FIX_TEST_CAPTURE_OVERRIDE ??
232242
process.env.CACHE_FIX_TEST_CAPTURE_OVERRIDE ?? "";
243+
const PINNED_FIXTURE =
244+
process.env.CACHE_FIX_TEST_FIXTURE_OVERRIDE ??
245+
join(__dirname, "fixtures", "harvested", "pinned-s-4b6a435234bf-26-28.json");
233246
const GATES = {
234247
CACHE_FIX_FORWARD_PROXY: "on",
235248
CACHE_FIX_SESSION_MIRROR: "on",
@@ -258,8 +271,19 @@ const entry = (n, inMsgs, outMsgs, extra = {}) => ({
258271
test(
259272
"real capture n=26->28: pin-and-suppress turns the input-mitigated/output-spliced pair into a clean append, safety gate 0 violations",
260273
async (t) => {
261-
if (!existsSync(REAL_CAPTURE)) {
262-
t.skip(`capture rotated away (not found at ${REAL_CAPTURE}) — COULD NOT VERIFY`);
274+
// Fixture-fallback: capture present -> unchanged live-capture path;
275+
// capture absent -> pinned fixture if present; else skip. Both readers
276+
// yield the same [n, line] tuple shape, so the replay loop below is
277+
// identical either way.
278+
let source;
279+
if (existsSync(REAL_CAPTURE)) {
280+
source = null; // resolved below, once readCapture is loaded from tools/replay.mjs
281+
} else if (existsSync(PINNED_FIXTURE)) {
282+
source = readPinnedFixture(PINNED_FIXTURE);
283+
} else {
284+
t.skip(
285+
`capture rotated away (not found at ${REAL_CAPTURE}) and no pinned fixture at ${PINNED_FIXTURE} — COULD NOT VERIFY`,
286+
);
263287
return;
264288
}
265289

@@ -273,6 +297,7 @@ test(
273297
return;
274298
}
275299
const { findMitigationGaps, findSafetyViolations, safetyViolation, readCapture } = replayTools;
300+
if (source === null) source = readCapture(REAL_CAPTURE);
276301

277302
const scratch = await mkdtemp(join(tmpdir(), "insertion-suppression-"));
278303
const saved = {};
@@ -292,7 +317,7 @@ test(
292317

293318
const entries = [];
294319
let reqN = -1;
295-
for await (const [, line] of readCapture(REAL_CAPTURE)) {
320+
for await (const [, line] of source) {
296321
let rec;
297322
try {
298323
rec = JSON.parse(line);

test/mitigation-output-form.test.mjs

Lines changed: 31 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ import { join, dirname } from "node:path";
3636
import { fileURLToPath, pathToFileURL } from "node:url";
3737

3838
import { findMitigationGaps, readCapture } from "../tools/replay.mjs";
39+
import { readPinnedFixture } from "../tools/harvest.mjs";
3940

4041
const __dirname = dirname(fileURLToPath(import.meta.url));
4142
const REPO = join(__dirname, "..");
@@ -102,12 +103,24 @@ test("mitigation output-form: a genuine tail-append reconstruction reports appen
102103
//
103104
// The capture lives outside the repo, in the per-machine capture directory
104105
// that rotates on a quadratic clock (docs/dev-loop.md, "Corpus hygiene") —
105-
// it is not a committed fixture. If it has rotated away since this test was
106-
// written, the test SKIPS with a stated reason rather than reporting a
107-
// false pass or a false fail (docs/dev-loop.md, "A checker has THREE
108-
// answers"); see the closing report for the harvesting gap this leaves.
106+
// it is not a committed fixture. If it has rotated away, this test falls
107+
// back to a PINNED fixture (BACKLOG.md "READY — harvest --pin freezes
108+
// evidence ranges as fixtures"): `node tools/harvest.mjs --pin <key> n..m`
109+
// freezes the sanitized range as test/fixtures/harvested/pinned-<key>-n-m
110+
// .json, committed and therefore immune to capture rotation. Only if BOTH
111+
// the live capture and the pinned fixture are unavailable does the test
112+
// SKIP with a stated reason, rather than reporting a false pass or a false
113+
// fail (docs/dev-loop.md, "A checker has THREE answers").
114+
//
115+
// Both paths are overridable via env for the fallback's own red-green test
116+
// (test/harvest-pin.test.mjs) — never by editing the real capture, which is
117+
// read-only evidence shared with other work.
109118
const REAL_CAPTURE =
119+
process.env.CACHE_FIX_TEST_CAPTURE_OVERRIDE ??
110120
process.env.CACHE_FIX_TEST_CAPTURE_OVERRIDE ?? "";
121+
const PINNED_FIXTURE =
122+
process.env.CACHE_FIX_TEST_FIXTURE_OVERRIDE ??
123+
join(__dirname, "fixtures", "harvested", "pinned-s-4b6a435234bf-26-28.json");
111124
const GATES = {
112125
CACHE_FIX_FORWARD_PROXY: "on",
113126
CACHE_FIX_SESSION_MIRROR: "on",
@@ -125,8 +138,19 @@ const TARGET_N = 28;
125138
test(
126139
"mitigation output-form: real capture n=26->28 reports append/preserved once suppression and the cache_control strip both apply",
127140
async (t) => {
128-
if (!existsSync(REAL_CAPTURE)) {
129-
t.skip(`capture rotated away (not found at ${REAL_CAPTURE}) — COULD NOT VERIFY`);
141+
// Fixture-fallback: capture present -> unchanged live-capture path;
142+
// capture absent -> pinned fixture if present; else skip. Both readers
143+
// yield the same [n, line] tuple shape, so the replay loop below is
144+
// identical either way.
145+
let source;
146+
if (existsSync(REAL_CAPTURE)) {
147+
source = readCapture(REAL_CAPTURE);
148+
} else if (existsSync(PINNED_FIXTURE)) {
149+
source = readPinnedFixture(PINNED_FIXTURE);
150+
} else {
151+
t.skip(
152+
`capture rotated away (not found at ${REAL_CAPTURE}) and no pinned fixture at ${PINNED_FIXTURE} — COULD NOT VERIFY`,
153+
);
130154
return;
131155
}
132156

@@ -148,7 +172,7 @@ test(
148172

149173
const entries = [];
150174
let reqN = -1;
151-
for await (const [, line] of readCapture(REAL_CAPTURE)) {
175+
for await (const [, line] of source) {
152176
let rec;
153177
try {
154178
rec = JSON.parse(line);

0 commit comments

Comments
 (0)