Skip to content

Commit b9d0707

Browse files
fix: set benchmark baselines from ubuntu-latest CI means
1 parent 12bf70c commit b9d0707

2 files changed

Lines changed: 12 additions & 12 deletions

File tree

benchmarks/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ The `benchmarks` job on **ubuntu-latest** runs pytest-benchmark, then `scripts/c
4040

4141
## Refresh baselines
4242

43-
After intentional performance work on ubuntu (same OS as CI):
43+
After intentional performance work, capture on **ubuntu-latest** (same OS as the gated CI job):
4444

4545
```bash
4646
make update-baselines
@@ -53,4 +53,4 @@ pytest tests/benchmarks/ --benchmark-only --benchmark-json=benchmarks/_raw.json
5353
python scripts/reduce_baselines.py benchmarks/_raw.json benchmarks/baselines.json
5454
```
5555

56-
Use `--slack 1.25` on `reduce_baselines.py` when capturing on a faster host than CI to absorb cross-machine variance.
56+
Do not capture baselines on Windows/macOS for commit — ubuntu runners are slower and the gate will fail. Download `benchmark-results.json` from a green CI artifact to seed baselines if needed.

benchmarks/baselines.json

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
11
{
2-
"_note": "CI gates the ubuntu benchmarks job when mean exceeds baseline by >20%.",
3-
"updated": "2026-06-17T20:33:56Z",
4-
"machine": "Windows",
2+
"_note": "Means captured from ubuntu-latest CI post-cache (PR #90). Refresh via make update-baselines on ubuntu.",
3+
"updated": "2026-06-17T21:00:00Z",
4+
"machine": "Linux",
55
"groups": {
66
"parse": {
7-
"test_parse_session_small": 8.711556942772768e-05,
8-
"test_parse_session_medium": 0.0019041118800302193,
9-
"test_parse_session_large": 0.018555900882518687
7+
"test_parse_session_small": 0.000105,
8+
"test_parse_session_medium": 0.002956,
9+
"test_parse_session_large": 0.029678
1010
},
1111
"export": {
12-
"test_bulk_export_session_count[sessions-10]": 0.0034216649980517108,
13-
"test_bulk_export_session_count[sessions-50]": 0.017290590000629893,
14-
"test_bulk_export_session_count[sessions-100]": 0.03397804391996553
12+
"test_bulk_export_session_count[sessions-10]": 0.004278,
13+
"test_bulk_export_session_count[sessions-50]": 0.021144,
14+
"test_bulk_export_session_count[sessions-100]": 0.042003
1515
},
1616
"search": {
17-
"test_search_full_corpus": 0.002600985144447307
17+
"test_search_full_corpus": 0.001092
1818
}
1919
}
2020
}

0 commit comments

Comments
 (0)