test: deterministic seeds + multi-seed robustness sweeps for flaky tests

[Copilot]

Summary

Replaces the original "tag flaky tests as :experimental to keep CI green" approach with something stronger: each previously-flaky test gets a deterministic seeded baseline + a multi-seed robustness sweep, both running on every PR.

A single Random.seed!(0) is reproducible on one Julia version but the downstream numerics drift across the matrix (1.10 / 1.11 / 1.12). The robustness sweep (K=20 seeds with MersenneTwister(seed), per-test pass-rate threshold) survives that drift — ≥80% of seeds pass is a much stronger guarantee than one lucky seed passes.

The sweeps cost ~5 seconds in aggregate on a ~10-minute matrix run, so there's no opt-in gate. No env vars, no tags, no machinery.

What changed

test/runtests.jl — now one line of filter:

@run_package_tests filter = ti -> !occursin("/benchmark/", ti.filename)

The /benchmark/ filter matches Piccolissimo.jl's convention — the subtree has its own Project.toml + deps + workflow, so its @testitems shouldn't be picked up by the main test run. Side benefit: this preemptively unbreaks PR #93's CI (which currently fails because benchmark/convergence/convergence.jl is being discovered).

Two flaky tests reworked. Each now has an untagged deterministic baseline + an untagged sweep:

1. TimeConsistencyConstraint with free time optimization — was tagged :experimental (intermittent 0.066 < 1e-6). Now uses MersenneTwister(0) and a paired sweep at threshold 0.80.
2. NonlinearKnotPointConstraint - single variable with vector syntax — observed Julia-1.10 finite-diff failure on PR benchmarks: convergence suite v1 (Ipopt + MadNLP on X gate) #93. Same treatment; sweep threshold 0.65 (norm-based finite-diff is inherently noisier).

Test helpers grew an optional rng kwarg. bilinear_dynamics_and_trajectory and test_constraint accept rng::AbstractRNG (default Random.default_rng()) so seeded reproducibility is per-test, not via global RNG state. test_constraint also gained assert::Bool=true and returns a NamedTuple including jacobian_pass / hessian_pass — sweeps inspect those directly rather than triggering @test per seed.

Statistical strength of K=20 sweeps

True pass rate	P(passes at threshold 0.80)	P(passes at threshold 0.65)
0.95	0.998	~1.000
0.80	0.630	0.913
0.50	0.006	0.058

Thresholds are per-test, picked with buffer above the observed baseline rate. A regression that drops the true rate below ~0.5 fails with very high probability; a healthy test rarely false-fails.

Verified

Local Julia 1.12: 355 pass / 1 broken / 0 fail / 356 total / 10m02s

• TimeConsistency sweep: pass_rate = 1.00 (20/20)
• NonlinearKnotPoint sweep: pass_rate = 0.80 (16/20, threshold 0.65)

CI matrix: all green on Julia 1.10 / 1.11 / 1.12, codecov/patch, Documentation, Formatter.

Out of scope (separate work)

• Nightly Julia 1.13-rc1: blocked by upstream JET (v0.11.3 master pins julia = "1.12"). Resolves when JET adds 1.13 support; no force_latest_compatible_version change needed on our side.
• Shared sweep helper: when a 3rd sweep lands, the right shape of a robustness_sweep(K, threshold, predicate) helper will be obvious from three concrete callsites. Punting until then.
• Cross-package conventions: if @gennadiryan's upcoming upfront test-item discovery + caching work introduces a sanctioned sweep primitive, we'd migrate. The one-line filter in runtests.jl is intentionally trivial to absorb.

Test plan

• Default test suite passes locally on Julia 1.12
• Sweeps pass at their thresholds locally on Julia 1.12
• CI matrix (1.10 / 1.11 / 1.12) green with always-on sweeps
• codecov/patch green (sweep code is now exercised)

🤖 Generated with Claude Code

[gennadiryan]

@jack-champagne I have some things mocked up that will help us take advantage of this, but it comes with some additional potential overhauls to our CI. The idea is to run TestItemRunner's test item discovery routine up front to not only generate the list of test item locations, but also grab the associated code objects and cache them alongside the package itself. This would reduce CI latency spent rebuilding tests but would also allow for much more fine-grained control of which tests "apply" to a given CI run, since the tests would be getting filtered ahead of time. At present, the most obvious use case is filtering out GPU tests on certain runs (see https://github.com/harmoniqs/Piccolissimo.jl/commit/ade50e4a29225534163afe888ee7225a33d52d34), but there are many others. Another potential future use case is supporting separate development streams (e.g. in Piccolissimo, a main branch and a well-maintained altissimo_integration branch, where the former is not yet ready to merge into the latter, and where both need to be kept up to date with downstream upgrades in the meantime). More to follow (likely in a separate PR initially landing on Piccolo.jl, then propagated back down to here).

Yet another use case: temporarily gracefully skipping CI tests for particular extension packages due to that particular extension relying on a dep (QuantumToolbox.jl is the one I've got in mind today) that is behind on compat.

[codecov]

Codecov Report

❌ Patch coverage is 96.66667% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../constraints/linear/time_consistency_constraint.jl	92.30%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!