Skip to content

Commit a1a5cf3

Browse files
olwangclaude
andcommitted
fix(selfhost): harden AST-parity gates — assert 0 run-failures, non-empty samples, doc release runtime
Addresses three review findings on the AST-parity harness: - Medium: ast_parity_corpus counted run-failures but never asserted on them, so the rss producer could start crashing on many files and the gate would still pass as long as the byte-exact floor held (contradicting the "0 run-failures" doc claim). Add assert_eq!(run_failures, 0, ...) — a crash now regresses the gate regardless of the floor. Verified in --release: 121 byte-exact, 0 run-failures. - Low: ast_sample_files() silently returns empty if selfhost/samples/ast is missing/unreadable, making ast_parity_samples pass vacuously. Assert files.len() >= AST_SAMPLE_MIN (6). - Low: the ignored corpus gate runs the reg-VM over ~560 files and is slow in a debug build (minutes). Document running it in --release (test docstring + plan doc) and point at ast_parity_samples as the fast non-ignored inner-loop gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 5fbecd6 commit a1a5cf3

2 files changed

Lines changed: 37 additions & 7 deletions

File tree

crates/rsscript/src/selfhost_parity.rs

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1478,6 +1478,14 @@ fn ast_parity_samples() {
14781478
let exe = compile_astdump().expect("rss astdump should compile");
14791479
let mut mismatches: Vec<String> = Vec::new();
14801480
let files = ast_sample_files();
1481+
// Guard against a vacuous pass: a missing/unreadable samples dir would make
1482+
// `files` empty and this test green while covering nothing.
1483+
assert!(
1484+
files.len() >= AST_SAMPLE_MIN,
1485+
"expected >= {AST_SAMPLE_MIN} curated AST samples under selfhost/samples/ast/, found {} \
1486+
(missing or unreadable dir makes this test pass vacuously)",
1487+
files.len()
1488+
);
14811489
for file in &files {
14821490
let rel = file
14831491
.strip_prefix(selfhost_dir())
@@ -1507,15 +1515,25 @@ fn ast_parity_samples() {
15071515
);
15081516
}
15091517

1518+
/// Minimum number of curated AST samples that must exist (guards `ast_parity_samples`
1519+
/// against a vacuous pass if the samples dir goes missing).
1520+
const AST_SAMPLE_MIN: usize = 6;
1521+
15101522
/// Floor for `ast_parity_corpus` — the number of corpus files whose rss AST dump
15111523
/// already matches the oracle byte-for-byte. Ratchets up as the producer's
15121524
/// coverage grows; a drop signals a regression. (Full parity = files.len().)
15131525
const AST_CORPUS_PARITY_FLOOR: usize = 121;
15141526

1515-
/// Step-2 measurement gate (ignored by default): how many of the 556 corpus files
1516-
/// the rss producer already reproduces byte-for-byte. Not full parity yet — this
1517-
/// ratchets a floor so coverage can only grow, and prints the current count so
1518-
/// the residual (SH-025) is visible.
1527+
/// Step-2 measurement gate (ignored by default): how many corpus files the rss
1528+
/// producer reproduces byte-for-byte. Not full parity yet — this ratchets a floor
1529+
/// so coverage can only grow, asserts the producer never crashes (0 run-failures),
1530+
/// and prints the current count so the residual (SH-025) is visible.
1531+
///
1532+
/// RUNTIME: this compiles the rss producer once and runs it over all ~560 corpus
1533+
/// files on the reg-VM; in a debug build that is slow (minutes). Run it in release
1534+
/// for a quick measurement:
1535+
/// `cargo test -p rsscript --release --lib selfhost_parity::ast_parity_corpus -- --ignored --nocapture`.
1536+
/// The fast inner-loop gate is `ast_parity_samples` (non-ignored, curated subset).
15191537
#[test]
15201538
#[ignore]
15211539
fn ast_parity_corpus() {
@@ -1550,6 +1568,14 @@ fn ast_parity_corpus() {
15501568
for rel in &sample_mismatches {
15511569
eprintln!("[mismatch] {rel}");
15521570
}
1571+
// The producer must never crash on corpus input — unsupported constructs are
1572+
// expected to mismatch (partial/`unknown-*` output), not error. A run-failure
1573+
// is a real regression even if `ok` still clears the floor.
1574+
assert_eq!(
1575+
run_failures, 0,
1576+
"rss AST producer had {run_failures} run-failures over {total} corpus files \
1577+
(it must degrade to a mismatch, never crash)"
1578+
);
15531579
assert!(
15541580
ok >= AST_CORPUS_PARITY_FLOOR,
15551581
"AST corpus parity regressed: {ok} byte-exact < floor {AST_CORPUS_PARITY_FLOOR}"

docs/self-hosting-stress-test-plan.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -280,9 +280,13 @@ likely want a sampled fast inner-loop gate plus the full corpus pre-push.
280280
recursive-descent rss parser that STREAMS the canonical dump (no handle-based AST is
281281
materialized). Parity harness: `ast_parity_tiny_sample` + `ast_parity_samples`
282282
(non-ignored, byte-exact vs the oracle over curated `samples/ast/*`) and
283-
`ast_parity_corpus` (`#[ignore]`, ratchets a floor over all 563 files). Current
284-
reach: **121/563 byte-exact, 0 run-failures** (the producer never crashes;
285-
unsupported constructs mismatch rather than panic). Covered: fns (generics/effects/
283+
`ast_parity_corpus` (`#[ignore]`, ratchets a floor over all 563 files and asserts
284+
**0 run-failures** so a producer crash regresses the gate even if the byte-exact
285+
count still clears the floor). That corpus gate runs the reg-VM over ~560 files and
286+
is slow in a debug build — run it in `--release` for a quick measurement; the fast
287+
non-ignored inner-loop gate is `ast_parity_samples`. Current reach: **121/563
288+
byte-exact, 0 run-failures** (the producer never crashes; unsupported constructs
289+
mismatch rather than panic). Covered: fns (generics/effects/
286290
return/body), struct/class/resource (opaque/generics/derives/handle-weak/defaults/
287291
drop), sum, const/type-alias/module/use, the core statements, and a precedence-exact
288292
expression parser (calls, field/index, array, try, effects, literals). Both risks are

0 commit comments

Comments
 (0)