Commit 11a81f2
committed
ci(release): gate npm publish on benchmark regressions
The regression guard previously ran inside `npm test` against the existing
benchmark history files. Because those files are only updated post-publish
(by the Benchmark workflow's auto-PRs), a regression introduced in vX could
ship to npm before its numbers were ever recorded — and the guard would
then fire on every dev push to main once the docs PR landed, blocking
unrelated work without ever having prevented the bad release.
Restructure so the gate runs where it can actually block:
- New `pre-publish-benchmark` job in publish.yml (release events only):
measures the just-built native artifact, writes new history entries,
runs the regression guard, and uploads the modified files. The `publish`
job depends on it, so a regression fails the workflow before npm sees
the new version. The history files are uploaded as an artifact.
- benchmark.yml's three measurement jobs (build/query/incremental) are
replaced by a single `record-benchmarks` job that downloads the
pre-publish artifact and opens one PR with the verified numbers.
`workflow_run.conclusion == 'success'` already gates this, so no PR
is opened for an aborted publish. The engine-parity gate runs here
as a soft signal (unchanged semantics). The embedding-benchmark job
is unchanged — no regression guard, can't fit in pre-publish.
- The regression-guard test is gated on `RUN_REGRESSION_GUARD=1` so the
default `npm test` run shows it as skipped rather than failing on
history that already passed gating at release time. CI sets the env
var in the pre-publish step.1 parent 8ed0b39 commit 11a81f2
4 files changed
Lines changed: 188 additions & 425 deletions
File tree
- .github/workflows
- tests/benchmarks
0 commit comments