Skip to content

fix(ci): make benchmark dispatch build and run measurements - #212

Merged
Antonov548 merged 1 commit into
mainfrom
fix/benchmark-workflow-ci
Jul 13, 2026
Merged

fix(ci): make benchmark dispatch build and run measurements#212
Antonov548 merged 1 commit into
mainfrom
fix/benchmark-workflow-ci

Conversation

@marcus-pousette

@marcus-pousette marcus-pousette commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

What

Make the dedicated benchmark workflow reliably reach measurements and retain evidence when a benchmark fails.

  • Resolve the PR number from the workflow_dispatch payload.
  • Build both checked-out workspaces before running their benchmark scripts.
  • Unset generic CI only for dedicated benchmark commands, so raw Playwright benchmark specs execute.
  • Add a safe dispatch selector for full or focused web measurements; /bench still runs the full suite.
  • Aggregate and upload partial JSON, command logs, and Playwright traces even after a base/head benchmark failure.
  • Publish the available results and each command outcome, then leave the job failed when a benchmark failed.

Why

The previous workflow had three dormant gates:

  1. It read an action-local input that was never supplied, so manual dispatch failed before checkout.
  2. It installed dependencies without building workspace packages and generated wa-sqlite assets.
  3. GitHub Actions set CI=true, while the raw browser benchmarks intentionally skip under normal CI.

Two real #164 dispatches exercised both outcome paths:

  • Run 29202455068 retained 65 partial rows and logs, updated the PR comment, and stayed red after an OPFS failure.
  • Run 29203415286 completed both sides, uploaded the full artifact, commented the results, and finished green.

The failed run led to the independent OPFS pathname fix in #213, which is exactly the kind of evidence the old workflow lost.

Scope / fast path

  • One commit; one workflow file, +71/-6.
  • No production, dependency, lockfile, benchmark, or test-source changes.
  • Suite names map through a fixed allowlist; no arbitrary script name reaches the shell.
  • Fork gating and same-repository security behavior are unchanged.
  • Normal CI still skips nondeterministic performance measurements.
  • Focused web dispatch avoids unrelated benchmark families during browser-only validation.

Verification

  • Workflow YAML parses.
  • Prettier and git diff --check pass.
  • Shell pipe failures propagate through tee.
  • Invalid suite values are rejected.
  • Real dispatches verified input resolution, symmetric builds, raw browser execution, focused selection, complete and partial artifact upload, PR commenting, and final success/failure propagation.
  • Normal GitHub CI is green.

@marcus-pousette marcus-pousette added bug Something isn't working performance Measurable runtime, storage, or responsiveness improvement area:web Browser, WASM, OPFS, workers, and playground runtime and removed performance Measurable runtime, storage, or responsiveness improvement labels Jul 12, 2026
@marcus-pousette
marcus-pousette force-pushed the fix/benchmark-workflow-ci branch from 22490e8 to 2431a2a Compare July 12, 2026 16:42
@marcus-pousette

Copy link
Copy Markdown
Collaborator Author

The first workflow_dispatch reproduced the broken PR input lookup. After switching to the dispatch event payload, the second validation run resolved PR #164 successfully and reached checkout: https://github.com/cybersemics/treecrdt/actions/runs/29200644118

@marcus-pousette marcus-pousette changed the title fix(ci): run raw browser benchmarks in the benchmark workflow fix(ci): make benchmark dispatch run real measurements Jul 12, 2026
@marcus-pousette
marcus-pousette force-pushed the fix/benchmark-workflow-ci branch from 2431a2a to 2a600c6 Compare July 12, 2026 16:48
@marcus-pousette marcus-pousette changed the title fix(ci): make benchmark dispatch run real measurements fix(ci): make benchmark dispatch build and run measurements Jul 12, 2026
@marcus-pousette
marcus-pousette force-pushed the fix/benchmark-workflow-ci branch from 8d24ef2 to 39ec94d Compare July 12, 2026 18:11
@Antonov548
Antonov548 merged commit b56ae4f into main Jul 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:web Browser, WASM, OPFS, workers, and playground runtime bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants