Skip to content

Add benchmark/Project.toml so the benchmark suite resolves StableRNGs#102

Merged
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:benchmark-add-stablerngs
Jul 14, 2026
Merged

Add benchmark/Project.toml so the benchmark suite resolves StableRNGs#102
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:benchmark-add-stablerngs

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Please ignore until reviewed by @ChrisRackauckas. Draft — opened by an agent.

Problem

The Benchmarks CI job fails on every PR that touches src/** (seen on #100, #101):

ERROR: LoadError: ArgumentError: Package StableRNGs not found in current path.

benchmark/benchmarks.jl does using StableRNGs, but there was no benchmark/Project.toml. AirspeedVelocity builds a temp environment for each benchmarked revision and, when that revision's checkout contains benchmark/Project.toml, copies it in as the environment (it auto-adds BenchmarkTools/JSON3 and the package under test, but nothing else). With no such file, StableRNGs never resolves.

Fix

Add benchmark/Project.toml declaring StableRNGs. That's all AirspeedVelocity needs (it supplies BenchmarkTools/JSON3 and the package itself).

Verification

Replicated AirspeedVelocity's env construction locally — copy benchmark/Project.toml into a fresh env, Pkg.develop the package, Pkg.add(["BenchmarkTools","JSON3"]) — then ran the unmodified benchmark/benchmarks.jl: StableRNGs resolves, the suite builds all groups (equality / per_query / batched / props_construct), and a benchmarkable runs.

Note on CI

This PR's own Benchmarks check can't go green (and won't trigger): the workflow runs benchpkg --rev=main,<PR> --bench-on=main, so it benchmarks using main's checkout, which won't have benchmark/Project.toml until this merges. The fix takes effect on the base branch — subsequent PRs' benchmark runs then resolve cleanly.

(Separately: the Benchmark.yml path filter watches bench/**, but the suite lives in benchmark/**; a one-line fix, left out here to keep this PR to just the Project.toml.)

@ChrisRackauckas-Claude ChrisRackauckas-Claude changed the title Add benchmark/Project.toml so the benchmark suite resolves its deps Make the benchmark suite self-contained (drop StableRNGs) Jul 13, 2026
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor Author

Note on the failing Benchmarks check — expected, and it clears on merge.

The SciML benchmark workflow invokes:

benchpkg "FindFirstFunctions" --rev=main,<PR sha> --bench-on=main

--bench-on=main makes AirspeedVelocity use main's benchmark/benchmarks.jl to benchmark both revisions. Since main's script still has using StableRNGs, the run fails at load (Running benchmarks for FindFirstFunctions@main: … Package StableRNGs not found) — before this PR's fixed script is ever used. So the benchmark check on this PR cannot pass until the fix is on the base branch; it's a chicken-and-egg for a PR that fixes the benchmark job itself.

The fix is verified out-of-band: in a bare environment (only FindFirstFunctions + BenchmarkTools, no StableRNGs) benchmark/benchmarks.jl on this branch loads, builds all groups, and runs a benchmarkable. Once merged, subsequent PRs' benchmark runs (which will --bench-on=main against the fixed script) resolve cleanly.

The `Benchmarks` CI job errored with `ArgumentError: Package StableRNGs not
found in current path`. AirspeedVelocity builds a temporary environment for
the benchmarked revision and, when a `benchmark/Project.toml` is present in
that revision's checkout, copies it in as the environment (it auto-adds
BenchmarkTools/JSON3 and the package under test, but nothing else). There was
no such file, so `using StableRNGs` in `benchmark/benchmarks.jl` could not
resolve.

Add `benchmark/Project.toml` declaring StableRNGs. Verified by replicating
AirspeedVelocity's env construction (copy the file, develop the package, add
BenchmarkTools/JSON3): the unmodified suite loads, builds all groups, and runs
a benchmarkable.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FvBXVKGujjeaCB3iLwsDeG
@ChrisRackauckas-Claude ChrisRackauckas-Claude changed the title Make the benchmark suite self-contained (drop StableRNGs) Add benchmark/Project.toml so the benchmark suite resolves StableRNGs Jul 14, 2026
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor Author

Updated: switched from vendoring an RNG to simply adding benchmark/Project.toml with StableRNGs — confirmed AirspeedVelocity auto-detects and uses a benchmarked revision's benchmark/Project.toml (Utils.jl: "Found Project.toml at …" → copies it into the env). My earlier "benchpkg ignores it" note was mistaken; those runs used --bench-on=main where main had no such file. This supersedes the previous comment.

@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review July 14, 2026 14:28
@ChrisRackauckas ChrisRackauckas merged commit e8458d4 into SciML:main Jul 14, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants