Current benchmarks get the results for master from Bencher. If we run both the comparison base (merge-base for PRs, previous commits for pushes to main) and the benchmarked version on the same runner, probably interleaved in randomised order, we should get much less noise. Bencher has direct support for this (find docs later).
Naively it would double the benchmark time, but that may be acceptable, or we may decrease per-route time to 20s or 15s.
See https://bencher.dev/docs/how-to/track-benchmarks/#relative-continuous-benchmarking
Blocked by #3459 currently.
Current benchmarks get the results for master from Bencher. If we run both the comparison base (merge-base for PRs, previous commits for pushes to main) and the benchmarked version on the same runner, probably interleaved in randomised order, we should get much less noise. Bencher has direct support for this (find docs later).
Naively it would double the benchmark time, but that may be acceptable, or we may decrease per-route time to 20s or 15s.
See https://bencher.dev/docs/how-to/track-benchmarks/#relative-continuous-benchmarking
Blocked by #3459 currently.