Skip to content

Commit bd12eb0

Browse files
committed
test(bench): document CI variance data for No-op and 1-file rebuild thresholds
The NOISY_METRICS set docstring explained the 50% tolerance for fnDeps depth 1 with CI run citations, but left No-op rebuild and 1-file rebuild documented only by cross-reference ("same pattern as…"). Add concrete CI run IDs and baseline→measured numbers for both metrics across v3.11.0–3.12.0, drawn from the existing KNOWN_REGRESSIONS evidence already in this file, so future maintainers can verify the threshold is empirically grounded rather than arbitrary. Closes #1516
1 parent 6d242b4 commit bd12eb0

1 file changed

Lines changed: 31 additions & 0 deletions

File tree

tests/benchmarks/regression-guard.test.ts

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,37 @@ const NOISY_METRIC_THRESHOLD = BENCH_CANARY ? 1.0 : 0.5;
6565
* only when its baseline is consistently sub-30ms and CI variance has been
6666
* empirically shown to exceed 25%.
6767
*
68+
* - `No-op rebuild`: native baselines across releases span 15–30ms
69+
* (3.11.0: 15ms, 3.11.2: 19–25ms, 3.12.0: 23–30ms depending on suite).
70+
* Shared-runner jitter of ±10ms routinely lands +60–109% on these numbers:
71+
* - 3.11.0 baseline 18ms → 29ms (+61%) — run 26426483639
72+
* - 3.11.2 baseline 25ms → 45ms (+80%) and 19ms → 37ms (+95%)
73+
* — run 26792023287 (docs-only PR #1282, zero hot-path changes)
74+
* - 3.12.0 baseline 30ms → 48ms (+60%) and 23ms → 48ms (+109%)
75+
* — run 27457266151 (PR #1487 adds warmup runs, no-op path unchanged)
76+
* - 3.12.0 baseline 23ms → 114ms (+396%) — run 27455727444 (canary,
77+
* PR #1468 enclosing-caller fix; no-op exits before that code runs)
78+
* MIN_ABSOLUTE_DELTA (10ms) filters truly trivial jitter (e.g. 15→19ms),
79+
* but a runner under load can easily produce +20ms on a no-op, which is
80+
* both above the floor and above the 25% threshold on a ~20ms baseline.
81+
* The 50% tolerance absorbs this pattern while still blocking a genuine
82+
* regression that adds real work to the no-op path (which would show up
83+
* consistently across runs, not just on loaded runners).
84+
*
85+
* - `1-file rebuild`: native baselines span 64–115ms across 3.11.0–3.12.0.
86+
* CI jitter of ±20–50ms translates to +25–155% on sub-100ms measurements:
87+
* - 3.11.0 baseline 64ms → 109ms (+70%) — run 26706695868 (concurrent
88+
* PRs on same runner measured 64→80ms (+25%), confirming runner noise)
89+
* - 3.11.2 baseline 83ms → 212ms (+155%) — run 26793082961 (PR #1278
90+
* TypeScript resolver PR; locally measures 86ms, within baseline noise)
91+
* - 3.12.0 baseline 86ms → 131ms (+52%) — publish gate run (incremental
92+
* suite's identical metric PASSED in the same run, confirming cold-start
93+
* bias in the build suite rather than a structural regression)
94+
* The 50% threshold catches a genuine regression that adds real work (e.g.
95+
* an O(n) scan on every 1-file rebuild would consistently land +100ms+)
96+
* while absorbing the runner-load spikes documented above. Tracked in #1440
97+
* (add warmup runs to the build-benchmark 1-file tier to reduce this spread).
98+
*
6899
* - `fnDeps depth 1`: native baseline 28.7ms (v3.9.6). The fn_deps Rust
69100
* implementation, fnDepsData JS wrapper, and DB schema/indexes are all
70101
* byte-for-byte unchanged since v3.9.6 (verified by `git log v3.9.6..HEAD`

0 commit comments

Comments
 (0)