test: drop :experimental + :robustness tag machinery, run sweeps always

After the deterministic+sweep rework no testitems use :experimental anymore,
and the sweeps add ~5 seconds in aggregate on top of a ~10-minute matrix —
not worth a parallel filter framework.

Drops both INCLUDE_EXPERIMENTAL and INCLUDE_ROBUSTNESS env vars and the
ti.tags closure. The robustness sweeps now run on every PR (along with their
deterministic baselines).

Keeps a `/benchmark/` path filter following Piccolissimo.jl's convention —
the benchmark subtree has its own Project.toml + deps + workflow, so its
@testItems shouldn't be discovered by `@run_package_tests`. This also
preemptively unbreaks PR #93's CI (which fails today because
benchmark/convergence/convergence.jl is being picked up).

Verified locally on Julia 1.12: 355 pass / 1 broken / 0 fail.
  TimeConsistency sweep:    pass_rate = 1.0  (20/20)
  NonlinearKnotPoint sweep: pass_rate = 0.8  (16/20, threshold 0.65)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: jack-champagne

README.md

@@ -105,42 +105,32 @@ solve!(prob; max_iter=100)
 
 ## Testing
 
-To run the standard test suite (what CI gates every PR on):
 ```bash
 julia --project=. test/runtests.jl
 ```
 
-`@testitem`s are filtered by tag. Two opt-in tags expand coverage:
+`runtests.jl` runs every `@testitem` in `src/`, `ext/`, and `test/`. Tests in
+`benchmark/` are skipped — that subdirectory ships its own `Project.toml`
+(extra deps like `HarmoniqsBenchmarks`) and has a dedicated workflow.
 
-| Env var                     | Tag             | What it adds                                                                                                |
-| --------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------- |
-| `INCLUDE_EXPERIMENTAL=1`    | `:experimental` | Known-flaky tests held out of PR CI. Useful for local diagnosis while a fix is being worked out.            |
-| `INCLUDE_ROBUSTNESS=1`      | `:robustness`   | Multi-seed sweeps (K=20) that assert a minimum fraction of seeds pass within tolerance. Per-test threshold chosen with buffer above the observed baseline pass rate — typically 0.80 for clean tests, lower for inherently noisy ones (e.g. finite-difference comparisons). |
+### Stochastic / numerical primitives — two-layer testing
 
-```bash
-INCLUDE_EXPERIMENTAL=1 julia --project=. test/runtests.jl
-INCLUDE_ROBUSTNESS=1   julia --project=. test/runtests.jl
-```
+A single seeded `MersenneTwister` is reproducible on one Julia version but
+small downstream numerics can drift across the CI matrix (1.10 / 1.11 / 1.12).
+For tests that touch non-deterministic surfaces (solver convergence from
+random init, finite-difference derivative comparisons) we pair each test:
+
+1. **Deterministic baseline**: a single seeded trajectory + multiplier.
+   A failure is a real regression on a specific (Julia version, seed) pair.
+2. **Robustness sweep**: K=20 independent seeds; passes if a fraction of
+   seeds (per-test threshold, chosen with buffer above the observed baseline
+   rate — typically 0.80, lower for inherently noisy checks like norm-based
+   finite-diff) land within tolerance. Detects regressions that drop the
+   true pass rate well below the threshold with very high probability
+   (binomial), while staying insensitive to lucky/unlucky single draws.
 
-### Testing philosophy for stochastic / numerical primitives
-
-A single `Random.seed!(0)` covers reproducibility on one Julia version but
-can drift across the CI matrix (1.10 / 1.11 / 1.12). For tests that touch
-non-deterministic surfaces — solver convergence with random initial
-conditions, finite-difference derivative comparisons — we use a two-layer
-approach:
-
-1. **Deterministic baseline** (untagged, runs every PR): a single seeded
-   trajectory + multiplier. A failure here means a real regression on the
-   specific (Julia version, seed) pair.
-2. **Robustness sweep** (`:robustness`, opt-in / nightly): K independent
-   seeds; pass if ≥80% land within the tolerance. A K=20 sweep detects
-   regressions that drop the true pass rate below ~80% with very high
-   probability (binomial), while staying robust against random unlucky
-   draws on the deterministic baseline.
-
-When writing a new flaky test, prefer adding both rather than tagging
-`:experimental` indefinitely.
+The sweeps are cheap enough (a handful of seconds in aggregate) to run on
+every PR.
 
 ## Contributing
 

src/constraints/linear/time_consistency_constraint.jl

@@ -130,7 +130,7 @@ end
     @test abs(t[1]) < 1e-8
 end
 
-@testitem "TimeConsistencyConstraint free-time robustness sweep" tags=[:robustness] begin
+@testitem "TimeConsistencyConstraint free-time robustness sweep" begin
     include("../../../test/test_utils.jl")
     using NamedTrajectories
     using Random

src/constraints/nonlinear/knot_point_constraint.jl

@@ -340,7 +340,7 @@ end
     test_constraint(NLC2, traj; atol = 1e-3, rng = MersenneTwister(1))
 end
 
-@testitem "NonlinearKnotPointConstraint vector syntax robustness sweep" tags=[:robustness] begin
+@testitem "NonlinearKnotPointConstraint vector syntax robustness sweep" begin
     using DirectTrajOpt: CommonInterface
     using Random
 

test/runtests.jl

@@ -3,31 +3,7 @@ using TestItemRunner
 
 include("test_snippets.jl")
 
-# Tag taxonomy controlling which @testitems run in CI.
-#
-# Defaults (untagged):
-#     Always run. Fast, deterministic, must pass on every PR.
-#
-# :experimental
-#     Known-flaky or environment-sensitive. Excluded by default.
-#     Opt in with INCLUDE_EXPERIMENTAL=1 for local diagnosis.
-#     Goal: eventually rewrite as deterministic + :robustness pair, then drop the tag.
-#
-# :robustness
-#     Multi-seed sweeps that assert ≥80% of seeds pass within tolerance.
-#     Excluded by default because they re-solve a problem many times.
-#     Opt in with INCLUDE_ROBUSTNESS=1 (e.g. nightly / scheduled workflows).
-#     A regression that drops the true pass rate below ~80% will fail this gate
-#     with very high probability (binomial, K=20).
-#
-# The filter is a single closure so it stays trivial to absorb into a more
-# sophisticated upstream filter (e.g. upfront test-item discovery) later.
-const INCLUDE_EXPERIMENTAL = haskey(ENV, "INCLUDE_EXPERIMENTAL")
-const INCLUDE_ROBUSTNESS = haskey(ENV, "INCLUDE_ROBUSTNESS")
-
-@run_package_tests filter =
-    ti -> begin
-        tags = get(ti, :tags, Symbol[])
-        (INCLUDE_EXPERIMENTAL || !(:experimental in tags)) &&
-            (INCLUDE_ROBUSTNESS || !(:robustness in tags))
-    end
+# Exclude `benchmark/` testitems from the main test run — they live in a
+# subproject with its own Project.toml (different deps, e.g. HarmoniqsBenchmarks)
+# and are exercised by a dedicated workflow.
+@run_package_tests filter = ti -> !occursin("/benchmark/", ti.filename)