test: add fast-check property suite#51
Open
voxpelli wants to merge 4 commits into
Open
Conversation
Three properties in test/properties.spec.js (100 runs each by default,
~50ms; FC_NUM_RUNS for deep sweeps, FC_SEED/FC_PATH replay a printed
counterexample):
- Differential vs native for-await: any values × bufferSize × error
position × errors mode × lenient/defensive source must yield the same
value multiset (fail-fast may cut early, never invent) and the same
identity-preserved terminal error. In the adoption spike this property
caught the post-done over-pull bug in 2-10 runs against the pre-fix
tree.
- Exactly-once delivery under generated abort geometry: fc.scheduler
orders callback resolutions while a microtask-hop-count arbitrary
(anchored at construction or at scheduler release — both matter)
positions an external abort; invariants are at most one rejection,
reason identity ∈ {callback error, abort reason}, done-forever after,
zero unhandled rejections. The generalized form of the drain-race
sweep in test/abort.spec.js; in the spike it reproduced that bug and
shrank it to the same minimal counterexample the hand-written sweep
pins.
- Protocol op-sequences vs a native AsyncGenerator as the model:
arbitrary next/return(v)/throw sequences must settle with identical
outcome shapes and rejection identities, with the two documented
eager-construction divergences (prefetch may start/finish the source
early) encoded as one-directional finally-parity.
Seeds stay random in CI: a property failure is a discovery (it prints
seed+path for deterministic replay), and every discovery gets pinned as
a plain example spec — regression protection never depends on run
count. CLAUDE.md documents the conventions: timer-free sources, no
fake timers, models are the review surface, hostile shapes stay in
hostile-results.spec.js. fast-check is a devDependency; the
zero-runtime-deps invariant is untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSyTJjbP823pjYfueuSsdu
A mutation-tested review found the three properties structurally unable to catch five real bug classes (each mutant survived 4k-20k runs while failing 2-17 example specs). The strengthening, validated by research against the WHATWG single-assignment abort algorithm, fast-check's documented timeout semantics, and the property-based-mutation-testing literature (Hughes' "How to Specify It!"; ICST 2023): - The abort property now uses the same commit-point oracle as the pinned drain-race sweep in test/abort.spec.js (cross-referenced both ways): the per-callback signal's immutable reason discriminates abort-won / fail-fast-won / no-commit, and each arm demands exactly the committed rejection — at most once AND at least once. A dispatched callback error with no out-racing shutdown must surface; a clean run must not reject; a pre-pull abort must reject the first pull. Every done result must be the exact 2-key shape (isCleanDone, extracted to test/utils.js and shared with expectSingleRejectionThenDone); a terminal must be reached; the scheduled abort anchor is now folded into the waitFor promise so it always fires (previously ~15% of those runs); abort geometry is one optional record (no dead shrink dimensions); empty sources are generated. - The differential property compares full multisets whenever no error surfaced (fail-fast included — silent value loss on a clean stream previously passed the property AND the whole example suite), uses a multiset-aware subset on the fail-fast error path (duplication was invisible to includes()), and restores the two model tripwires from the spike (oracle over-pull, oracle error identity). - The protocol property gains same-tick teeth: a terminal that settles before the library source's finally ran (the await-idempotence contract) now fails at the settling microtask, not via post-hoc flag comparison. - Infrastructure: per-run fast-check timeout plus interruptAfterTimeLimit/markInterruptAsFailure so a hang-class bug is reported with seed/path instead of dying as a bare mocha timeout; FC_NUM_RUNS/FC_SEED validated loudly (fast-check silently runs zero cases on numRuns 0/NaN); FC_PATH replay enforced to a single property (scope with mocha -g); mocha timeout scales with FC_NUM_RUNS; vacuity counters assert every conditional-oracle arm actually fired. Mutation validation with the committed spec run against mutant copies of index.js: 12 of 12 non-equivalent mutants killed within 2 000 runs (the five previous survivors included); the drain-race guard revert is now caught at 20k-run scale where it survived before (the sweep remains the deterministic pin); one mutant proved equivalent (144/144 example specs pass it — the removed line is unreachable bookkeeping). Correct code passes 20 000 runs per property with zero false positives, and the full check chain stays green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FSyTJjbP823pjYfueuSsdu
Records, at landing time, what the suite did and did not deliver: no new bugs found (all 2.0.0 findings came from hand-rolled probes first), prospective value against the proven-recurring post-done bug class, the vacuous first iteration that mutation testing caught, and the standing policy that mutation testing gates every future property — so future maintainers can judge the file on its record instead of its optics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FSyTJjbP823pjYfueuSsdu
voxpelli
force-pushed
the
claude/fast-check-properties-nixCB
branch
from
ca63349 to
96e51dd
Compare
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FSyTJjbP823pjYfueuSsdu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #50 — targets the 2.0.0 branch so the diff shows only the property suite; retarget to
main(or let GitHub auto-retarget) once #50 merges.What
Adds
fast-check(dev-only — the zero-runtime-deps invariant is untouched) andtest/properties.spec.jswith three properties, plus a sharedisCleanDonehelper intest/utils.js(also tighteningexpectSingleRejectionThenDone's tail check):for await— anyvalues × bufferSize × errorPos × errors-mode × lenient/defensive source: full multiset equality whenever no error surfaced (fail-fast included), multiset-aware subset on the fail-fast error path, identity-preserved terminal errors, and two model tripwires (oracle over-pull / oracle error identity) so a broken model can't masquerade as a library verdict.test/abort.spec.js(cross-referenced both ways: the sweep is the regression pin, this property is the accumulating search): the per-callback signal's immutable reason discriminates abort-won / fail-fast-won / error-obligated / clean runs, and each arm demands exactly the committed rejection — at most once and at least once — plus strict 2-key done shapes, a guaranteed terminal, and zero unhandled rejections. Scheduled-anchor aborts are folded into the scheduler'swaitForpromise so they always fire.AsyncGeneratoras the model — arbitrarynext/return(v)/throwsequences settle identically, with same-microtask teeth for the await-idempotence contract (a terminal may not settle before the source'sfinallyran) and the two documented eager-construction divergences encoded one-directionally.Mutation-validated (the acceptance gate)
The committed spec was run against mutant copies of
index.jsbuilt from real bug classes (each mutant fails 2–17 example specs, proving non-equivalence):Notable side-discovery: the fail-fast silent-value-loss mutant passes the entire example suite — the property suite is currently the only automated coverage for that contract class.
Conventions (documented in CLAUDE.md and the file header)
FC_SEED=… FC_PATH=… npx mocha test/properties.spec.js -g "<test name>"— enforced, since a shrink path is only meaningful for the property that produced it). Every discovery gets pinned as a plain example spec.timeout(a hanging library bug becomes a reported, shrunk failure instead of a bare mocha timeout losing the report) +interruptAfterTimeLimit/markInterruptAsFailurebounding the whole assert under the mocha cap, which itself scales withFC_NUM_RUNS.numRuns: 0/NaN, so badFC_NUM_RUNS/FC_SEEDvalues throw.test/hostile-results.spec.js.Verification
Full check chain (lint, tsc strict, knip, type-coverage, installed-check) + all 144 specs green; properties additionally validated by the mutation matrix above and a 20 000-run clean sweep. Design grounded in the WHATWG single-assignment abort algorithm (the commit-point oracle), fast-check's documented timeout semantics, and property-based mutation-testing methodology (Hughes' "How to Specify It!"; ICST 2023).
🤖 Generated with Claude Code
https://claude.ai/code/session_01FSyTJjbP823pjYfueuSsdu