Skip to content

Commit 063d835

Browse files
docs: clarify benchmark gate guidance after review
1 parent 09bd0ab commit 063d835

2 files changed

Lines changed: 2 additions & 2 deletions

File tree

benchmarks/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ The `benchmarks` job on **ubuntu-latest** runs the full `tests/benchmarks/` suit
3939

4040
Pinned runner: `ubuntu-latest`, `--benchmark-min-rounds=5`.
4141

42-
Sub-millisecond cache lookups (`test_summary_cache_lookup`, `test_composer_map_cache_lookup`, `test_tab_summary_cache_lookup`) are already listed in `EXCLUDED_FROM_GATE` because shared runners show 24x spread. For remaining gated benches that turn flaky, raise `--slack` at baseline refresh time or add a targeted `EXCLUDED_FROM_GATE` entry.
42+
Sub-millisecond cache lookups (`test_summary_cache_lookup`, `test_composer_map_cache_lookup`, `test_tab_summary_cache_lookup`) are already listed in `EXCLUDED_FROM_GATE` because shared runners show 2-4x spread. When another gated bench turns flaky from runner variance, add a targeted `EXCLUDED_FROM_GATE` entry in `scripts/check_benchmark_regression.py`. Use `--slack` only when intentionally recalibrating all baselines via `scripts/reduce_baselines.py`, not to patch a single flaky benchmark.
4343

4444
`test_summary_cache_round_trip` is intentionally excluded from the gate: it calls `set_cached_projects` (file write) + `get_cached_projects` (file read) each round, so OS page-cache state on shared runners causes 3–5x variation between consecutive CI runs. The baseline entry is kept for observation only.
4545

scripts/check_benchmark_regression.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
# ZIP writes on ubuntu-latest (e.g. composers-10 ~4x spread; composers-50 hit
3333
# 1.6x vs Jul-15 baseline on master after unrelated merges). Not an application
3434
# regression. Baseline-only fixes fail when slow and fast runs cannot both sit
35-
# inside the 0.5x1.2x gate.
35+
# inside the 0.5x-1.2x gate.
3636
#
3737
# CI-gate maintenance only; prefer EXCLUDED_FROM_GATE over baselines.json churn
3838
# when the failure is runner variance, not a code slowdown.

0 commit comments

Comments
 (0)