fix(bench): apply KNOWN_REGRESSIONS baseline fallback to release publish gate, not just dev (#2129)

* fix(bench): apply KNOWN_REGRESSIONS baseline fallback to release publish gate, not just dev

The v3.16.0 release tag (2e0e94f) failed its pre-publish benchmark gate on
Full build (+42%) and 1-file rebuild (+53-83%), blocking publish. Both
metrics are already tracked as growth-driven baseline drift (#2081) and
exempted via KNOWN_REGRESSIONS entries '3.15.0:Full build' /
'3.15.0:1-file rebuild' added in #2107 - but that exemption only fired
when assertNoRegressions' `latest.version` was the literal string 'dev'
(the per-PR gate label). publish.yml's pre-publish-benchmark job labels
its measurement with the real version being published (e.g. '3.16.0'),
so the fallback never applied on the one run it mattered most for,
and the gate failed on the exact regression it was supposed to be
exempt from.

Drop the `version === 'dev'` restriction so the baseline-anchored
fallback applies to any comparison against a known-regressed baseline,
whether the latest label is 'dev' or a real release version. Adds unit
tests exercising both cases directly against assertNoRegressions.

* fix(bench): widen the negative-case test delta so it holds under BENCH_CANARY too

The 'does not exempt' test used a 3521->5000 (+42%) delta, which clears
the default 25% Full build threshold but not the 50% BENCH_CANARY
threshold - under canary mode the check would get filtered out as "not
a regression" before ever reaching the KNOWN_REGRESSIONS fallback,
making the test pass without exercising the logic it's meant to guard.

* fix(bench): decouple KNOWN_REGRESSIONS baseline-fallback tests from live production data

Greptile review: the three always-on unit tests exercising the baseline
fallback hardcoded '3.15.0:Full build' against the module-level
KNOWN_REGRESSIONS set. That entry is documented for pruning once
v3.16.0's own benchmark data lands, which would break the first two
tests (expecting .not.toThrow()) the moment that routine cleanup PR
merges.

Add an optional knownRegressions parameter to assertNoRegressions
(defaulting to the module-level KNOWN_REGRESSIONS for production use)
and have the three unit tests pass a local, test-only fixture instead,
fully decoupling them from data that is expected to change over time.

Author: carlos-alm

tests/benchmarks/regression-guard.test.ts

@@ -217,21 +217,25 @@ const SKIP_VERSIONS = new Set(['3.8.0']);
  * Resolution keys use: "version:resolution <lang> precision" or "version:resolution <lang> recall".
  *
  * The `version` is the release where the regression was first observed.
- * When the per-PR gate runs `dev` against that release as baseline, the
- * exemption applies via the baseline-version fallback in assertNoRegressions
- * (and the resolution loop) — so a single `3.11.0:Foo` entry covers both
- * `3.11.0 vs 3.10.0` and every subsequent `dev vs 3.11.0` comparison until
- * the next release clears the regression and the entry is pruned.
+ * Any comparison whose baseline is that release gets the exemption via the
+ * baseline-version fallback in assertNoRegressions (and the resolution
+ * loop) — so a single `3.11.0:Foo` entry covers `3.11.0 vs 3.10.0`, every
+ * subsequent `dev vs 3.11.0` per-PR comparison, AND the eventual `3.12.0 vs
+ * 3.11.0` publish gate itself (the release that's supposed to fold the
+ * drift into a fresh baseline) — until that release's own benchmark data
+ * lands and the entry is pruned.
  *
- * Entries fire only when `latest.version` matches the prefix (or, for `dev`
- * latest, when `previous.version` matches via the baseline fallback). Once
- * a version is no longer the latest in committed history and no longer the
- * baseline used for `dev` comparisons, its entries become dead weight and
- * should be removed (last pruned: 3.9.0/3.9.1/3.9.2/3.9.6/3.10.0/3.11.0/3.11.1/
- * 3.11.2; the 3.12.0 and 3.13.0 entries — dataflow/no-op/full-build timing noise
- * and the erlang 0% drop — were pruned at the 3.15.0 release once the 3.15.0
- * benchmark baseline landed in PR #1702, which folds those deltas into the
- * baseline so dev-vs-3.15.0 comparisons no longer flag them).
+ * Entries fire when `latest.version` matches the prefix directly, or when
+ * `previous.version` (the baseline) matches via the baseline fallback —
+ * regardless of whether `latest` is `dev` (per-PR gate) or a real version
+ * (the publish gate for the next release). Once a version is no longer the
+ * latest in committed history and no longer used as a baseline for any
+ * comparison, its entries become dead weight and should be removed (last
+ * pruned: 3.9.0/3.9.1/3.9.2/3.9.6/3.10.0/3.11.0/3.11.1/3.11.2; the 3.12.0 and
+ * 3.13.0 entries — dataflow/no-op/full-build timing noise and the erlang 0%
+ * drop — were pruned at the 3.15.0 release once the 3.15.0 benchmark
+ * baseline landed in PR #1702, which folds those deltas into the baseline
+ * so dev-vs-3.15.0 comparisons no longer flag them).
  *
  * NOTE: WASM *timing* noise no longer needs per-version entries here — it is
  * handled structurally by WASM_TIMING_THRESHOLD (see above); native keeps the
@@ -501,21 +505,33 @@ function assertNoRegressions(
   version?: string,
   baselineVersion?: string,
   engine?: string,
+  // Defaults to the module-level KNOWN_REGRESSIONS (production behavior).
+  // Tests that exercise the fallback logic itself pass a local, stable
+  // fixture instead — so they don't break when real entries above are
+  // pruned by a future benchmark-recording cleanup PR (see the
+  // 'assertNoRegressions — KNOWN_REGRESSIONS baseline fallback' suite below).
+  knownRegressions: ReadonlySet<string> = KNOWN_REGRESSIONS,
 ) {
   const real = checks.filter(Boolean) as RegressionCheck[];
   const regressions = real.filter((c) => {
     if (c.pctChange <= thresholdFor(c.label, engine)) return false;
-    if (version && KNOWN_REGRESSIONS.has(`${version}:${c.label}`)) return false;
-    // When `latest` is the rolling 'dev' build, KNOWN_REGRESSIONS entries
-    // are anchored to the release where the regression was first observed
-    // (e.g. '3.9.6:No-op rebuild'), not to 'dev'. Fall back to the baseline
-    // version so a regression introduced before release N stays exempt for
-    // every PR comparing dev → N until release N+1 clears it.
-    if (
-      version === 'dev' &&
-      baselineVersion &&
-      KNOWN_REGRESSIONS.has(`${baselineVersion}:${c.label}`)
-    ) {
+    if (version && knownRegressions.has(`${version}:${c.label}`)) return false;
+    // KNOWN_REGRESSIONS entries are anchored to the release where the
+    // regression was first observed (e.g. '3.9.6:No-op rebuild'), not to
+    // whatever `latest` happens to be. Fall back to the baseline version so
+    // a regression introduced before release N stays exempt for every
+    // comparison against N — both per-PR gates (latest = 'dev') and the
+    // actual release N+1 publish gate itself (latest = the real N+1 version,
+    // e.g. '3.16.0') — until release N+1's own benchmark data lands as the
+    // new baseline and the entry goes stale (see the "KNOWN_REGRESSIONS
+    // entries are not stale" test below).
+    //
+    // This used to be gated on `version === 'dev'`, which meant the one run
+    // that most needs the exemption — the publish gate for the release that
+    // is supposed to fold the known drift into a fresh baseline — never got
+    // it, and failed on the exact regression it was meant to be exempt from
+    // (v3.16.0 publish, #2127; entries added by #2107 for exactly this).
+    if (baselineVersion && knownRegressions.has(`${baselineVersion}:${c.label}`)) {
       return false;
     }
     return true;
@@ -532,6 +548,72 @@ function assertNoRegressions(
   }
 }
 
+// Pure-logic tests for the KNOWN_REGRESSIONS baseline fallback. Unlike the
+// describe.runIf(RUN_REGRESSION_GUARD) suite below, these don't read real
+// benchmark report files — they exercise assertNoRegressions directly, so
+// they always run and don't need a recorded 'X.Y.Z vs baseline' pair to exist
+// in committed history.
+//
+// These tests use a local, test-only `knownRegressions` fixture (fictional
+// '9.9.9' version) rather than the module-level KNOWN_REGRESSIONS — the real
+// set is expected to have its '3.15.0:*' entries pruned once v3.16.0's own
+// benchmark data lands (see the KNOWN_REGRESSIONS docstring above), and
+// hardcoding these always-on tests against that live, evolving data would
+// break them the moment that routine cleanup PR lands.
+describe('assertNoRegressions — KNOWN_REGRESSIONS baseline fallback', () => {
+  const testKnownRegressions = new Set<string>(['9.9.9:Full build']);
+
+  test('exempts a real-release latest (not just dev) when its baseline matches a KNOWN_REGRESSIONS entry', () => {
+    // Reproduces the v3.16.0 publish-gate failure (#2127): a KNOWN_REGRESSIONS
+    // entry pinned to the prior release's baseline (e.g. '3.15.0:Full build',
+    // added by #2107 for repo-growth drift), but the publish gate labels
+    // `latest.version` with the real new version being published, never
+    // 'dev' — this must still hit the baseline fallback.
+    expect(() =>
+      assertNoRegressions(
+        [checkRegression('Full build', 5000, 3521)], // +42%, over the 25% threshold
+        '10.0.0',
+        '9.9.9',
+        'native',
+        testKnownRegressions,
+      ),
+    ).not.toThrow();
+  });
+
+  test('still exempts the per-PR dev-vs-baseline comparison', () => {
+    expect(() =>
+      assertNoRegressions(
+        [checkRegression('Full build', 5000, 3521)],
+        'dev',
+        '9.9.9',
+        'native',
+        testKnownRegressions,
+      ),
+    ).not.toThrow();
+  });
+
+  test('does not exempt a regression against a baseline with no matching KNOWN_REGRESSIONS entry', () => {
+    // This describe block runs unconditionally — including under
+    // BENCH_CANARY=1 (.github/workflows/perf-canary.yml), which widens
+    // REGRESSION_THRESHOLD from 0.25 to 0.5. The delta here must clear the
+    // regression threshold in *every* mode this test can run under, or the
+    // check gets filtered out as "not a regression" before the
+    // KNOWN_REGRESSIONS fallback is even reached — making the test flip-flop
+    // based on an ambient env var instead of the exemption logic it's meant
+    // to exercise. +100% clears both the 25% default and the 50% canary
+    // threshold with margin.
+    expect(() =>
+      assertNoRegressions(
+        [checkRegression('Full build', 8000, 4000)],
+        '10.0.0',
+        '9.8.0', // not covered by testKnownRegressions
+        'native',
+        testKnownRegressions,
+      ),
+    ).toThrow(/Full build/);
+  });
+});
+
 // ── Build benchmark data types ───────────────────────────────────────────
 
 interface BuildEngine {