Skip to content

Commit 20b8596

Browse files
authored
ci(release): gate npm publish on benchmark regressions (#1040)
* ci(release): gate npm publish on benchmark regressions The regression guard previously ran inside `npm test` against the existing benchmark history files. Because those files are only updated post-publish (by the Benchmark workflow's auto-PRs), a regression introduced in vX could ship to npm before its numbers were ever recorded — and the guard would then fire on every dev push to main once the docs PR landed, blocking unrelated work without ever having prevented the bad release. Restructure so the gate runs where it can actually block: - New `pre-publish-benchmark` job in publish.yml (release events only): measures the just-built native artifact, writes new history entries, runs the regression guard, and uploads the modified files. The `publish` job depends on it, so a regression fails the workflow before npm sees the new version. The history files are uploaded as an artifact. - benchmark.yml's three measurement jobs (build/query/incremental) are replaced by a single `record-benchmarks` job that downloads the pre-publish artifact and opens one PR with the verified numbers. `workflow_run.conclusion == 'success'` already gates this, so no PR is opened for an aborted publish. The engine-parity gate runs here as a soft signal (unchanged semantics). The embedding-benchmark job is unchanged — no regression guard, can't fit in pre-publish. - The regression-guard test is gated on `RUN_REGRESSION_GUARD=1` so the default `npm test` run shows it as skipped rather than failing on history that already passed gating at release time. CI sets the env var in the pre-publish step. * fix(ci): cache npm in pre-publish-benchmark job (#1040) Match the cache: "npm" setting used by every other job in publish.yml so release runs reuse the npm cache instead of re-downloading dependencies. * fix(ci): restore untracked-file detection in record-benchmarks (#1040) Each of the three pre-consolidation jobs paired `git diff --quiet HEAD` with `git ls-files --others --exclude-standard` so a first-run history file (or one re-created after deletion) would still be picked up. The consolidated check dropped that second guard, which would silently skip the PR step if a benchmark history file was untracked. Restore the guard so verified numbers always make it back to main.
1 parent bb501e1 commit 20b8596

4 files changed

Lines changed: 190 additions & 423 deletions

File tree

0 commit comments

Comments
 (0)