Skip to content

Commit 7af4945

Browse files
Haofeiclaude
andcommitted
test: add metamorphic, example-exec, JIT-edge, and cargo-fuzz harnesses
Close the four remaining gaps in the dynamic-testing plan. #3 Metamorphic differential tests (tests/metamorphic.rs): for known-valid programs spanning the executable subset, apply semantics-preserving rewrites (reformat, trivia injection, unused local, top-level reorder, dead function) and require every backend still agrees AND produces the same output as the base; structure-preserving transforms must also leave the review classification unchanged. #7 Example-corpus differential execution (tests/examples_exec.rs): the static sweep already checks+lowers examples/scripts; this runs the pure synchronous subset across all backends and asserts agreement. The skip predicate is derived from the source (I/O, timing, async, plus one documented VM capability gap: struct-typed Map/Set keys in uop_interning). #5 JIT arithmetic-edge matrix (tests/jit_arithmetic_edges.rs): the edges the generative suite skips - i64 add/mul overflow, div/mod by zero, MIN/-1 div+mod (all trap cleanly), shift wrap, NaN, signed zero, float infinity (all agree). Each op lives in a pure helper called with read-args so the JIT actually compiles and guards it. #2 cargo-fuzz harness (fuzz/): a detached-workspace crate (never built by cargo build --workspace) with two libFuzzer targets - parse_check (front-end never-panics) and format_idempotent (format(format(x)) == format(x)). Adds common::differential::backends_agreed_stdout for the new sweeps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 9a3de6d commit 7af4945

10 files changed

Lines changed: 1807 additions & 2 deletions

File tree

crates/rsscript/tests/common/differential.rs

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,13 +155,15 @@ pub fn assert_backends_all_fail(file: &str, source: &str, args: &[&str]) {
155155
}
156156
}
157157

158-
/// Like [`assert_backends_agree`] but with an explicit backend set.
158+
/// Like [`assert_backends_agree`] but with an explicit backend set. Returns the
159+
/// stdout every backend agreed on (useful for comparing a program against a
160+
/// semantics-preserving variant of it).
159161
pub fn assert_backends_agree_on(
160162
file: &str,
161163
source: &str,
162164
args: &[&str],
163165
backends: &[Box<dyn Backend>],
164-
) {
166+
) -> String {
165167
let mut reference: Option<(&'static str, String)> = None;
166168
for backend in backends {
167169
let stdout = backend
@@ -182,4 +184,12 @@ pub fn assert_backends_agree_on(
182184
),
183185
}
184186
}
187+
reference.expect("at least one backend").1
188+
}
189+
190+
/// Run `source` on every backend, assert they all agree, and return the agreed
191+
/// stdout. The metamorphic-test entry point: it both enforces N-way parity and
192+
/// hands back the value to compare across equivalent program variants.
193+
pub fn backends_agreed_stdout(file: &str, source: &str, args: &[&str]) -> String {
194+
assert_backends_agree_on(file, source, args, &all_backends())
185195
}
Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
//! Differential execution of the example corpus.
2+
//!
3+
//! `checker_frontend::misc::examples_have_no_diagnostics_and_lower_to_runnable_packages`
4+
//! already proves every `examples/scripts/**/*.rss` type-checks and lowers to a
5+
//! `todo!`-free Rust package. That is the *static* half. This is the *dynamic*
6+
//! half: the synchronous, side-effect-free example scripts are run through every
7+
//! executable backend (VM interpreter, JIT, native tier when built, and the
8+
//! compiled-Rust path) and required to agree. It catches the "valid example,
9+
//! divergent backend" / doc-drift class without per-example expected-output
10+
//! files.
11+
//!
12+
//! Scripts that need real runtime resources (files, network, env, the clock) or
13+
//! that exercise the async scheduler are skipped here — they are non-trivial to
14+
//! run deterministically in a unit test and are covered by the dedicated
15+
//! `vm_eval_parity` fixtures (with temp dirs and local servers) and the static
16+
//! sweep above. The skip is decided from the source itself so a newly added pure
17+
//! example is picked up automatically.
18+
19+
mod common;
20+
21+
use common::differential::backends_agreed_stdout;
22+
23+
/// Namespaces / constructs whose behavior depends on real runtime resources or
24+
/// on scheduler timing, which this in-process differential cannot reproduce
25+
/// deterministically. A script touching any of these is left to the static
26+
/// check+lower sweep and the resource-backed `vm_eval_parity` tests.
27+
const NON_DETERMINISTIC_MARKERS: &[&str] = &[
28+
// Real I/O and host boundaries.
29+
"File.",
30+
"Directory.",
31+
"Http.",
32+
"WebSocket.",
33+
"Socket.",
34+
"Tcp",
35+
"Net.",
36+
"Process.",
37+
"Env.",
38+
"System.",
39+
// Wall-clock / timing.
40+
"Clock.",
41+
"Deadline.",
42+
"Instant.",
43+
"Timer.",
44+
"sleep",
45+
// Non-determinism.
46+
"Random.",
47+
"Rng.",
48+
// Async scheduler (ordering is covered by the async parity fixtures).
49+
"spawn ",
50+
"await ",
51+
"async ",
52+
"Channel",
53+
"TaskGroup",
54+
"Cancellation",
55+
"Select",
56+
];
57+
58+
fn needs_runtime_resources(source: &str) -> bool {
59+
NON_DETERMINISTIC_MARKERS
60+
.iter()
61+
.any(|marker| source.contains(marker))
62+
}
63+
64+
/// Example scripts that lower and run on the compiled backend but hit a known
65+
/// *capability* gap in the register VM (a clean `Runtime` error, not a crash),
66+
/// so they can't take part in the N-way agreement check. Each entry documents
67+
/// the specific gap. Path suffixes (matched with `ends_with`) to stay
68+
/// location-independent.
69+
const KNOWN_VM_UNSUPPORTED: &[&str] = &[
70+
// The register VM does not support struct-typed Map/Set keys; the compiled
71+
// backend does. This example interns `Uop` structs in a Map<Uop, _>/Set<Uop>.
72+
"examples/scripts/core/uop_interning.rss",
73+
];
74+
75+
fn known_vm_unsupported(name: &str) -> bool {
76+
KNOWN_VM_UNSUPPORTED
77+
.iter()
78+
.any(|suffix| name.replace('\\', "/").ends_with(suffix))
79+
}
80+
81+
#[test]
82+
fn example_scripts_agree_across_backends() {
83+
let mut executed = Vec::new();
84+
let mut skipped = Vec::new();
85+
86+
for path in common::recursive_paths_with_extension("examples/scripts", "rss") {
87+
let source = common::read_fixture(&path);
88+
let name = path.to_str().expect("utf-8 path");
89+
90+
if needs_runtime_resources(&source) || known_vm_unsupported(name) {
91+
skipped.push(name.to_string());
92+
continue;
93+
}
94+
95+
// Panics (with the diverging backend pair and the source) on any
96+
// failure or mismatch, so a divergence names the exact example.
97+
let _ = backends_agreed_stdout(name, &source, &[]);
98+
executed.push(name.to_string());
99+
}
100+
101+
// Guard against the predicate silently excluding everything (which would
102+
// turn this into a vacuous test): the pure `basic/` scripts alone exceed
103+
// this floor, so dropping below it signals the corpus or predicate drifted.
104+
assert!(
105+
executed.len() >= 6,
106+
"expected the differential sweep to execute several example scripts, \
107+
ran {} (skipped {}): executed={executed:?}",
108+
executed.len(),
109+
skipped.len()
110+
);
111+
}
Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
//! JIT arithmetic-edge differential matrix.
2+
//!
3+
//! The generative `backend_differential` suite deliberately *skips* the
4+
//! arithmetic edges (it `prop_assume!(oracle.is_some())`, dropping overflow, and
5+
//! its float strategy avoids division and NaN). Those edges are exactly where a
6+
//! JIT tier is most likely to diverge from the interpreter: native code that
7+
//! must trap on i64 overflow / divide-by-zero / `i64::MIN / -1`, mask an
8+
//! out-of-range shift, and reproduce IEEE-754 NaN / signed-zero semantics — or
9+
//! correctly bail to the interpreter at the guard.
10+
//!
11+
//! Each case runs through every backend (`vm-interpreter`, `vm-jit`, the native
12+
//! tier and its force-deopt twin when built, and `rust-compiled`) and asserts
13+
//! one of two contracts:
14+
//!
15+
//! * **trap edges** — every backend fails with a clean RSScript runtime error
16+
//! (no host panic, no garbage), via [`assert_backends_all_fail`]; and
17+
//! * **value edges** — every backend returns the identical observable value,
18+
//! via [`assert_backends_agree`].
19+
//!
20+
//! Each arithmetic operation lives in a small pure helper called with `read`
21+
//! arguments so the value is unknown at compile time and the JIT actually
22+
//! compiles (and guards) the operation rather than constant-folding it.
23+
24+
mod common;
25+
26+
use common::differential::{assert_backends_agree, assert_backends_all_fail};
27+
28+
/// `i64::MAX` is expressible as a literal; `i64::MIN` is built as
29+
/// `(0 - i64::MAX) - 1` because the bare literal `9223372036854775808` would
30+
/// overflow the `Int` lexer. Shared prelude installed in `main` before each call.
31+
const MIN_PRELUDE: &str = " let max = 9223372036854775807\n \
32+
let min = 0 - max - 1\n let neg_one = 0 - 1\n";
33+
34+
// --- Trap edges: every backend must fail cleanly -------------------------
35+
36+
#[test]
37+
fn jit_edge_integer_addition_overflow_traps_on_all_backends() {
38+
let source = "\
39+
fn add(a: Int, b: Int) -> Int {
40+
return a + b
41+
}
42+
43+
fn main() -> Unit {
44+
Log.write(message: read String.from_int(value: add(a: read 9223372036854775807, b: read 1)))
45+
return Unit
46+
}
47+
";
48+
assert_backends_all_fail("jit-edge-add-overflow.rss", source, &[]);
49+
}
50+
51+
#[test]
52+
fn jit_edge_integer_multiplication_overflow_traps_on_all_backends() {
53+
let source = "\
54+
fn mul(a: Int, b: Int) -> Int {
55+
return a * b
56+
}
57+
58+
fn main() -> Unit {
59+
Log.write(message: read String.from_int(value: mul(a: read 3037000500, b: read 3037000500)))
60+
return Unit
61+
}
62+
";
63+
assert_backends_all_fail("jit-edge-mul-overflow.rss", source, &[]);
64+
}
65+
66+
#[test]
67+
fn jit_edge_integer_division_by_zero_traps_on_all_backends() {
68+
let source = "\
69+
fn div(a: Int, b: Int) -> Int {
70+
return a / b
71+
}
72+
73+
fn main() -> Unit {
74+
Log.write(message: read String.from_int(value: div(a: read 10, b: read 0)))
75+
return Unit
76+
}
77+
";
78+
assert_backends_all_fail("jit-edge-div-zero.rss", source, &[]);
79+
}
80+
81+
#[test]
82+
fn jit_edge_integer_modulo_by_zero_traps_on_all_backends() {
83+
let source = "\
84+
fn rem(a: Int, b: Int) -> Int {
85+
return a % b
86+
}
87+
88+
fn main() -> Unit {
89+
Log.write(message: read String.from_int(value: rem(a: read 10, b: read 0)))
90+
return Unit
91+
}
92+
";
93+
assert_backends_all_fail("jit-edge-mod-zero.rss", source, &[]);
94+
}
95+
96+
#[test]
97+
fn jit_edge_int_min_divided_by_neg_one_traps_on_all_backends() {
98+
let source = format!(
99+
"\
100+
fn div(a: Int, b: Int) -> Int {{
101+
return a / b
102+
}}
103+
104+
fn main() -> Unit {{
105+
{MIN_PRELUDE} Log.write(message: read String.from_int(value: div(a: read min, b: read neg_one)))
106+
return Unit
107+
}}
108+
"
109+
);
110+
assert_backends_all_fail("jit-edge-min-div-neg1.rss", &source, &[]);
111+
}
112+
113+
#[test]
114+
fn jit_edge_int_min_modulo_neg_one_traps_on_all_backends() {
115+
let source = format!(
116+
"\
117+
fn rem(a: Int, b: Int) -> Int {{
118+
return a % b
119+
}}
120+
121+
fn main() -> Unit {{
122+
{MIN_PRELUDE} Log.write(message: read String.from_int(value: rem(a: read min, b: read neg_one)))
123+
return Unit
124+
}}
125+
"
126+
);
127+
assert_backends_all_fail("jit-edge-min-mod-neg1.rss", &source, &[]);
128+
}
129+
130+
// --- Value edges: every backend must agree -------------------------------
131+
132+
#[test]
133+
fn jit_edge_shift_amount_wraps_consistently() {
134+
// `wrapping_shl`/`wrapping_shr` mask the count to 0..63, so a shift of 64 is
135+
// a shift of 0. Every backend must agree on this defined wrap rather than
136+
// panic (debug `<<`) or produce a target-dependent value.
137+
let source = "\
138+
fn shl(value: Int, bits: Int) -> Int {
139+
return Int.shift_left(value: read value, bits: read bits)
140+
}
141+
142+
fn shr(value: Int, bits: Int) -> Int {
143+
return Int.shift_right(value: read value, bits: read bits)
144+
}
145+
146+
fn main() -> Unit {
147+
Log.write(message: read String.from_int(value: shl(value: read 1, bits: read 64)))
148+
Log.write(message: read String.from_int(value: shl(value: read 1, bits: read 63)))
149+
Log.write(message: read String.from_int(value: shr(value: read 256, bits: read 64)))
150+
Log.write(message: read String.from_int(value: shr(value: read 256, bits: read 4)))
151+
return Unit
152+
}
153+
";
154+
assert_backends_agree("jit-edge-shift-wrap.rss", source, &[]);
155+
}
156+
157+
#[test]
158+
fn jit_edge_float_nan_is_not_equal_to_itself() {
159+
// `0.0 / 0.0` is NaN; `NaN == NaN` is false on every backend. Reduced to a
160+
// Bool so float *formatting* (which is not under test here) can't diverge.
161+
let source = "\
162+
fn nan(a: Float, b: Float) -> Float {
163+
return a / b
164+
}
165+
166+
fn main() -> Unit {
167+
let value = nan(a: read 0.0, b: read 0.0)
168+
Log.write(message: read String.from_bool(value: value == value))
169+
return Unit
170+
}
171+
";
172+
assert_backends_agree("jit-edge-float-nan.rss", source, &[]);
173+
}
174+
175+
#[test]
176+
fn jit_edge_float_signed_zero_compares_equal() {
177+
// `-0.0 == 0.0` is true under IEEE-754; every backend must agree.
178+
let source = "\
179+
fn neg(value: Float) -> Float {
180+
return 0.0 - value
181+
}
182+
183+
fn main() -> Unit {
184+
let negative_zero = neg(value: read 0.0)
185+
Log.write(message: read String.from_bool(value: negative_zero == 0.0))
186+
return Unit
187+
}
188+
";
189+
assert_backends_agree("jit-edge-float-signed-zero.rss", source, &[]);
190+
}
191+
192+
#[test]
193+
fn jit_edge_float_division_by_zero_yields_infinity() {
194+
// `1.0 / 0.0` is +inf (no trap, unlike Int): every backend must agree it is
195+
// greater than any finite value. Reduced to a Bool to avoid inf formatting.
196+
let source = "\
197+
fn div(a: Float, b: Float) -> Float {
198+
return a / b
199+
}
200+
201+
fn main() -> Unit {
202+
let infinity = div(a: read 1.0, b: read 0.0)
203+
Log.write(message: read String.from_bool(value: infinity > 1000000.0))
204+
return Unit
205+
}
206+
";
207+
assert_backends_agree("jit-edge-float-inf.rss", source, &[]);
208+
}

0 commit comments

Comments
 (0)