Commit 20b8596
authored
ci(release): gate npm publish on benchmark regressions (#1040)
* ci(release): gate npm publish on benchmark regressions
The regression guard previously ran inside `npm test` against the existing
benchmark history files. Because those files are only updated post-publish
(by the Benchmark workflow's auto-PRs), a regression introduced in vX could
ship to npm before its numbers were ever recorded — and the guard would
then fire on every dev push to main once the docs PR landed, blocking
unrelated work without ever having prevented the bad release.
Restructure so the gate runs where it can actually block:
- New `pre-publish-benchmark` job in publish.yml (release events only):
measures the just-built native artifact, writes new history entries,
runs the regression guard, and uploads the modified files. The `publish`
job depends on it, so a regression fails the workflow before npm sees
the new version. The history files are uploaded as an artifact.
- benchmark.yml's three measurement jobs (build/query/incremental) are
replaced by a single `record-benchmarks` job that downloads the
pre-publish artifact and opens one PR with the verified numbers.
`workflow_run.conclusion == 'success'` already gates this, so no PR
is opened for an aborted publish. The engine-parity gate runs here
as a soft signal (unchanged semantics). The embedding-benchmark job
is unchanged — no regression guard, can't fit in pre-publish.
- The regression-guard test is gated on `RUN_REGRESSION_GUARD=1` so the
default `npm test` run shows it as skipped rather than failing on
history that already passed gating at release time. CI sets the env
var in the pre-publish step.
* fix(ci): cache npm in pre-publish-benchmark job (#1040)
Match the cache: "npm" setting used by every other job in publish.yml so
release runs reuse the npm cache instead of re-downloading dependencies.
* fix(ci): restore untracked-file detection in record-benchmarks (#1040)
Each of the three pre-consolidation jobs paired `git diff --quiet HEAD`
with `git ls-files --others --exclude-standard` so a first-run history
file (or one re-created after deletion) would still be picked up. The
consolidated check dropped that second guard, which would silently skip
the PR step if a benchmark history file was untracked. Restore the
guard so verified numbers always make it back to main.1 parent bb501e1 commit 20b8596
4 files changed
Lines changed: 190 additions & 423 deletions
File tree
- .github/workflows
- tests/benchmarks
0 commit comments