Switch CI benchmarks to relative continuous benchmarking

[alexeyr-ci2]

Summary

Closes #3492.

Switches the benchmark workflow from statistical continuous benchmarking (t-test thresholds against main's cross-runner history) to relative continuous benchmarking: each suite job now builds and benchmarks both the comparison base ref and the head ref on the same runner, then compares them directly. This removes the cross-runner variance that made shared-runner alerts noise-dominated (#4071, false positives #4038–#4044).

• New benchmark-suite-phase composite action runs one ref end to end (checkout + clean → install/build → assets → DB seed → server → bench → stop server); benchmark-suite.yml invokes it twice per job. Servers start under setsid so each phase can kill its whole process tree and free the port for the next phase.
• track_benchmarks.rb now makes two Bencher runs: base results to a throwaway per-run baseline branch (base-<run_id>-<suite-slug>, --start-point-reset; unique per suite/shard so concurrent shards can't anchor to each other's data), then head results anchored to it (--start-point <baseline> --start-point-reset --thresholds-reset) with percentage thresholds (25% one-sided per measure) instead of the t-test. The --start-point-hash retry fallback is gone with the hash itself.
• The comparison base is resolved once in detect-changes: PR merge-ref's first parent (current main, avoiding the stale-base.sha lag from Fix CI rerun mapping for split dummy app tests (#3909) #3919), event.before for main pushes, merge-base with main for dispatches. Only resolved when suites will actually run.
• The confirmation pipeline (still gated by BENCHMARK_REGRESSION_ISSUES_ENABLED) re-runs the whole base/head comparison on a fresh runner. Worth revisiting enabling it now that the cross-runner noise source is gone.
• The local dedicated-hardware runner keeps statistical t-test trend tracking (BencherRunner's default mode). Long-term dashboards now live on its testbed — relative runs reset the shared-runner series each run.

Stability (round 2 — response to the initial run)

The initial run flagged 15 routes 🔴/🟢 on an infra-only diff with swings up to +149% p50 — pure noise. Root cause was found, reproduced off-CI, and fixed; detected changes now also have to survive built-in reruns before they flag.

1. k6 noConnectionReuse: true — the dominant noise source. A keep-alive connection is pinned to whichever Puma worker process accepted it, and Linux's LIFO accept-queue wakeup skews reconnect placement, so the 10 VUs' per-worker split drifts randomly within a run (splits as bad as 9/1/0 observed — a whole worker idle for the sample). Each 30s sample landed in a random queueing "mode": stable-RPS/wild-p50 on saturated SSR routes, coupled RPS/p50 swings on fast routes, and the one suite with no Puma in front (node renderer, vegeta) rock-stable — all three initial-run patterns explained. Reproduced in a CI-topology Linux container (4 cores; k6 pinned to core 0; Puma 3 workers × 3 threads on cores 1–3; puma 6.6.1; k6 v2.0.0):

   mode	RPS spread (max/min−1)	RPS CV	p50 spread	p50 CV
   keep-alive (initial run)	38.2%	10.0%	36.0%	8.5%
   noConnectionReuse	9.3%	3.3%	9.2%	3.4%

   Connect cost is excluded from http_req_duration, and tcp_tw_reuse=2 recycles loopback TIME_WAIT ports (verified clean at 7.7k conn/s — 12× the busiest route).

2. Median-of-3 sampling — each route runs BENCHMARK_SAMPLES (CI: 3) k6 samples of 12s instead of a single 30s run; the median per measure ships to Bencher, so one disturbed sample can no longer set a route's number. Per-sample values ride in the display sidecar.

3. Sample confirmation (rerun-to-confirm) — a Bencher boundary crossing only stands if it reproduced across every sample (base and head sample ranges disjoint). Otherwise the summary table shows an unbolded ⚠️ instead of 🔴/🟢, the alert leaves the regression/candidate path, and the exit code normalizes. Pairs without ≥2 numeric samples on both sides fail open to single-sample behavior. The fresh-runner confirmation pipeline for main pushes is unchanged and now consumes only sample-confirmed alerts.

4. One instrument for both phases — both phases now execute the HEAD ref's stashed benchmarks/ tree (a new "Stash head benchmark harness" step + harness-dir action input), not each phase's checkout. The harness is the measuring instrument; running two different copies would turn any harness change (like this PR) into a phantom base-vs-head delta. App-side inputs (bin/prod*, gems, packs, renderer config, node-renderer protocol version) still come from each phase's own checkout — bench-node-renderer.rb resolves those through GITHUB_WORKSPACE.

Acceptance for this PR: its own benchmark run is an A/A test (base and head run identical app code), so the summary tables should show no 🔴/🟢 flags (at most ⚠️ downgrades) and small deltas. Do not merge on a run that still looks like the initial one.

Trade-offs

• Roughly doubles job wall-clock (two builds + two benchmark passes), plus ~10 min/job for sampling (3×12s + two extra k6 startups per route vs one 30s run).
• A PR adding a brand-new suite/app can't benchmark it (the base ref can't build it); drop the benchmark label for such PRs.
• Absolute numbers step once with the connection change (throughput now includes per-request connects). Irrelevant for the relative comparison; the local statistical trend series (run-local-benchmark.rb, unchanged single-sample 30s defaults) re-baselines within a few runs — noted in LOCAL_BENCHMARK.md.
• Interleaved/randomized base-head ordering from the issue is not implemented; sequential base-then-head matches the Bencher docs, and the container A/A runs bound within-job drift well under the 25% thresholds once connection stickiness is gone.

Workflow Change Audit

• Secrets: no new secrets; GITHUB_BASE_SHA env removed from the tracking step; GH_TOKEN added to the detect-changes base-resolution step (existing GITHUB_TOKEN).
• Permissions: unchanged.
• Triggers: unchanged.
• Third-party actions: none added; still bencherdev/bencher@v0.6.2, actions/*, grafana/setup-k6-action@v1, pnpm/action-setup@v6.
• New steps/inputs are literal-only: the harness stash copies benchmarks/ to runner.temp; harness-dir/BENCHMARK_SAMPLES/BENCHMARK_BASELINE_DISPLAY_JSON are constants. Untrusted inputs are still passed via env:/action inputs, never interpolated into scripts.

Verification

• All 348 benchmark specs pass (bundle exec rspec benchmarks/spec) — 20 new for sampling aggregation, sample-confirmation semantics (disjoint vs overlapping ranges, fail-open), ⚠️ rendering, exit-code normalization, and CLI wiring.
• RuboCop clean on benchmarks/; actionlint clean; Prettier clean on changed files.
• Mechanism validation in a CI-topology Linux container (table above): keep-alive vs noConnectionReuse A/B, 8×10s runs each; concurrent- and sequential-connect worker-split probes; port-churn stress at 7.7k conn/s with 0 failed requests.
• End-to-end bench.rb run (real k6, local server, BENCHMARK_SAMPLES=3): per-sample suffixed artifacts written, medians correctly selected, samples sidecar populated, statuses summed.
• End-to-end smoke test with a stub bencher binary produced exactly the two-command sequence from the Bencher docs (baseline --start-point-reset upload, then percentage-threshold comparison run).
• This PR carries the benchmark label so the new relative flow exercises itself on this run.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added relative base-versus-head continuous benchmarking, with a new base reference to compare against.
  • Added multi-sample benchmarking with per-sample artifacts and aggregated median metrics.
  • Added unconfirmed alert handling: overlapping results are flagged and shown in the report.
  • Added support for running benchmark phases across Rails and Node renderer environments; updated k6 to use fresh connections per request.
• Documentation
  • Expanded local benchmark guidance for repeated samples and interpretation updates.
• Tests
  • Updated and added coverage for sampling aggregation and unconfirmed alert behavior.

Confidence note

• Validated: 348 benchmark specs (bundle exec rspec benchmarks/spec, re-run green on the rebased head 9e430a4); RuboCop / actionlint / Prettier clean; A/A benchmark run 29187847192 on 75543d4 — 0 flagged rows across all 4 suites (initial run: 36), max |Δ| 18.5% (was 202%), 0 cells ≥ 25%; all 9 optimized hosted CI workflows green on 75543d4 (Generator, Playwright E2E, Integration, Pro Integration, Pro Package, RSpec gem, JS unit, Assets Precompile, Lint); required-pr-gate green; script/pr-merge-ledger 4580 --strict → complete_allowed: true (changelog: not_user_visible; codex P1 dispositioned not_applicable with evidence).
• Evidence: stability before/after comparison; CodeRabbit APPROVED, claude-review clean (non-blocking notes), greptile 4/5, 0 unresolved review threads on 75543d4.
• UNKNOWN: none — the rebased head 9e430a4 re-ran everything green, including a second independent A/A benchmark run (29190965437): again 0 flagged rows across all 4 suites, max |Δ| 20.1%, 0 cells ≥ 25%. Final-cycle review threads (6) triaged: 2 rebutted with evidence, 4 deferred to Follow-up: Exercise GitHub Actions changes from PR #4580 #4613 with rationale; 0 unresolved.
• Residual risk: multi-minute CPU-steal windows on shared runners can still shift a stretch of adjacent routes in one phase (observed +13–18% on four adjacent Core routes, well under thresholds); fully cancelling that would need interleaved base/head phases — documented trade-off, revisit only if real runs show it crossing thresholds.

[coderabbitai]

Walkthrough

The PR adds same-runner base/head benchmarking, repeated per-route k6 samples with median aggregation, baseline-aware Bencher modes, sample confirmation for alerts, and workflow plumbing for base SHA resolution and result artifacts.

Changes

Relative benchmark execution

Layer / File(s)	Summary
Benchmark phase action .github/actions/benchmark-suite-phase/action.yml	Adds a reusable action that prepares a ref, runs the benchmark harness, manages Rails or node-renderer processes, cleans up, and validates results.
Base and head workflow orchestration .github/workflows/benchmark-suite.yml, .github/workflows/benchmark.yml	Resolves base_sha, preserves the head harness, runs base and head phases, uploads base results, and passes baseline paths to tracking.
Repeated benchmark sampling benchmarks/bench.rb, benchmarks/k6.ts, benchmarks/bench-node-renderer.rb, benchmarks/LOCAL_BENCHMARK.md	Adds repeated samples, per-sample duration and artifacts, median/status aggregation, fresh connections, workspace-root resolution, and local-run documentation.
Bencher modes and sample confirmation benchmarks/lib/bencher_*.rb, benchmarks/lib/benchmark_table.rb, benchmarks/lib/bmf_helpers.rb	Adds relative Bencher modes, baseline/head threshold handling, sample sidecars, unconfirmed alert classification, exit normalization, and table rendering.
Baseline/head tracking orchestration benchmarks/lib/track_benchmarks/*, benchmarks/track_benchmarks.rb	Plans baseline and head branches, uploads baseline before head, applies sample confirmation, and validates both benchmark JSON inputs.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Workflow
  participant PhaseAction
  participant BenchmarkServer
  participant BenchScript
  participant Bencher
  Workflow->>PhaseAction: run base phase
  PhaseAction->>BenchmarkServer: build and start target
  PhaseAction->>BenchScript: execute repeated samples
  BenchScript-->>PhaseAction: write benchmark results
  Workflow->>PhaseAction: run head phase
  PhaseAction->>BenchmarkServer: build and start target
  PhaseAction->>BenchScript: execute repeated samples
  BenchScript-->>Bencher: provide head and baseline artifacts
  Bencher-->>Workflow: classify confirmed or unconfirmed alerts

Loading

Possibly related issues

• Issue #4321: Updates the same benchmark tracking file and removes legacy delegation wrappers covered by this change.
• Issue #3169: Addresses benchmark flakiness through multi-sample base/head comparisons and sample-confirmation logic.
• Issue #4073: Supports dedicated-runner benchmarking and trend/alerting reliability through workflow and sampling changes.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 49.32% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes implement same-runner base/head comparison and Bencher relative benchmarking as requested in #3492.
Out of Scope Changes check	✅ Passed	The diff stays focused on benchmark CI, harness, stability, and reporting updates with no clear unrelated additions.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: CI benchmarks now use relative continuous benchmarking.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch alexeyr/3492/relative-continuous-benchmarking

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[claude]

Review: Switch CI benchmarks to relative continuous benchmarking

Overview

This PR replaces the cross-runner statistical (t-test) benchmark comparison with relative continuous benchmarking: each suite job now builds/benchmarks both the resolved comparison base ref and the head ref on the same runner (via the new benchmark-suite-phase composite action, invoked twice), then track_benchmarks.rb submits base results to a throwaway per-run baseline branch and compares head results against it with percentage thresholds. This directly targets the cross-runner noise (#4071) that caused false-positive regression issues (#4038–#4044). The change is large but mechanically well-scoped: one new composite action, workflow wiring for a base_sha input, and a corresponding refactor of BencherRunner/BranchArgs/Cli into an explicit baseline+head two-run model.

Code quality

• The comments throughout (action.yml, bencher_runner.rb, branch_args.rb) are unusually good at explaining why, not just what — e.g. the invariant that benchmark-suite-phase must only be invoked while HEAD is checked out (since uses: ./... resolves from the working tree), and why per-run branch names must be suite/shard-unique to avoid concurrent shards anchoring to each other's baseline.
• BencherRunner's MODES/mode_args/tracked_measures refactor cleanly generalizes the old single-mode (t-test) design to three explicit modes without duplicating the CLI-arg-building logic.
• Good defensive design: the baseline run configures no thresholds and can't itself alert, so Cli#run_baseline treats any non-zero exit as an operational failure and aborts before the head run could compare against a baseline that was never actually recorded.
• The removal of the --start-point-hash retry-fallback machinery (dead now that relative comparison doesn't rely on a possibly-missing historical start point) is a nice simplification, not just an add.

Potential issues / minor observations (non-blocking)

1. benchmark.yml push-event base fallback — when github.event.before is the zero SHA (the "push" case for a rewritten/force-pushed history), the fallback is git rev-parse HEAD^. Since push only triggers on main and detect-changes checks out with fetch-depth: 50, this should virtually always succeed, but it would fail loudly under set -euo pipefail if main's local shallow history didn't contain a parent (e.g. an unusual history rewrite deeper than 50 commits). Given the rarity, this is fine as-is, but worth a one-line comment noting the depth-50 assumption if this ever needs to be revisited.
2. Doubled job wall-clock is explicitly called out as an accepted trade-off in the PR description, so not a surprise, but worth confirming CI queue/cost impact is acceptable long-term, especially combined with the confirmation-rerun pipeline once BENCHMARK_REGRESSION_ISSUES_ENABLED is turned back on.
3. 25% percentage threshold is deliberately loose per the comments, with a plan to tighten "once real runs show the same-runner spread." Since this is a single-sample-vs-single-sample comparison, both false negatives (small real regressions hidden under 25%) and occasional false positives from CPU contention are possible until that tuning happens — already acknowledged, just flagging it as something to watch once this rolls out.
4. There's a small amount of redundancy in benchmark-suite.yml: Restore head checkout does git checkout --force "$GITHUB_SHA", and then the very next step's own composite-action checkout logic checks out the same ref again. Harmless (needed for the action-resolution invariant), just noting it's not purely a no-op-elimination.

Test coverage

Solid — bencher_runner_spec.rb explicitly pins CLI args per mode (including the "unknown mode raises" case), branch_args_spec.rb covers all three event types plus the concurrent-shard baseline-branch-uniqueness invariant, and cli_spec.rb covers both the happy path (baseline → head) and the baseline-failure abort path. The new spec asserting RELATIVE_THRESHOLDS and THRESHOLDS share the same measures/directions is a good safety net against the two threshold sets silently diverging.

Security

No new secrets, no new third-party actions, and the new gh api compare call (workflow_dispatch base resolution) only needs contents: read, which the job already has. Untrusted inputs continue to flow through env:/action inputs rather than being interpolated into shell strings.

Overall this is a well-reasoned, thoroughly tested infrastructure change with clear rationale and honest trade-off documentation. No blocking issues found.

[alexeyr-ci2]

The initial run results show very unimpressive stability, must be solved before merging.

Should also include detected regressions/improvements being rerun to confirm.

[justin808]

@alexeyr-ci2 Agreed — the initial run was unusable (15 🔴/🟢 flags on an infra-only diff, up to +149% p50). I root-caused it off-CI, fixed it, and added the rerun-to-confirm you asked for, in 75543d4.

Root cause (validated, not guessed): k6's keep-alive connections pin to whichever Puma worker process accepts them, and Linux's LIFO accept wakeup skews reconnect placement — so the 10 VUs' split across the 3 workers drifts randomly within a run (I observed splits as bad as 9/1/0, i.e. one worker idle for the whole sample). Each 30s sample therefore lands in a random queueing "mode". That one mechanism explains all three patterns in the initial run: stable-RPS/wild-p50 on saturated SSR routes, coupled RPS+p50 swings on the fast client-side routes, and the vegeta node-renderer suite (no Puma in front) staying ±5%.

Reproduced in a CI-topology Linux container (4 cores, k6 pinned to core 0, Puma 3×3 on cores 1–3, puma 6.6.1, k6 v2.0.0), 8×10s runs per mode:

mode	RPS spread (max/min−1)	p50 spread
keep-alive (what the initial run used)	38.2%	36.0%
noConnectionReuse: true (this fix)	9.3%	9.2%

38% run-to-run spread against 25% thresholds is exactly the false-flag generator we saw.

What changed:

1. k6.ts sets noConnectionReuse: true (connect cost is excluded from http_req_duration; loopback tcp_tw_reuse=2 handles the port churn — verified clean at 12× our busiest route's rate).
2. Each route now runs 3×12s samples per phase (vs one 30s run) and the median ships to Bencher, so a single disturbed sample can't set a route's number.
3. Rerun-to-confirm, as you asked: the 3 samples per side are the reruns. A boundary crossing only stands if it reproduced in every one (base and head sample ranges disjoint); otherwise the table shows an unbolded ⚠️ instead of 🔴/🟢 and the alert is dropped from the regression/candidate path. The fresh-runner confirmation pipeline for main pushes now consumes only sample-confirmed alerts.
4. Both phases now run the head ref's stashed harness (one measuring instrument) — otherwise this very PR would have compared old-harness base numbers against new-harness head numbers and the harness change itself would read as a performance delta.

Acceptance: this PR's own benchmark run is an A/A test (base and head run identical app code), so the new tables should show no 🔴/🟢 flags and small deltas. I'll post the comparison against the initial run when the suites finish — and per the same standard, this doesn't merge unless the run actually demonstrates the improvement.

[claude]

Review: Switch CI benchmarks to relative continuous benchmarking

Overall this is a very well-executed PR — extensive inline documentation, thoughtful edge-case handling, and matching spec coverage (branch-args, bencher_report sample confirmation, CLI orchestration, table rendering). I read through the composite action, the workflow diffs, and the full benchmarks/lib changes; I didn't find correctness bugs. Notes below are mostly minor/confirmatory.

What it does

• Replaces cross-runner statistical (t-test) CI benchmarking with same-runner relative comparison: each suite job now builds+benchmarks both the base ref and head ref via a new benchmark-suite-phase composite action, then track_benchmarks.rb uploads base results to a throwaway per-run baseline branch and head results compared against it with percentage thresholds.
• Adds repeated k6 sampling (BENCHMARK_SAMPLES=3 in CI) with median aggregation and sample-based "unconfirmed" downgrading of boundary crossings that don't reproduce across samples.
• Resolves the comparison base SHA once in detect-changes (merge-ref first parent for PRs, avoiding the stale base.sha lag from Fix CI rerun mapping for split dummy app tests (#3909) #3919).
• Removes the old --start-point-hash/retry-without-hash machinery entirely, replaced by the two-report baseline/head flow.
• k6.ts now sets noConnectionReuse: true to fix a per-worker connection-split variance issue, with measured before/after numbers in the comment.

Strengths

• apply_sample_confirmation! / overlapping_measures in bencher_report.rb fail open correctly (pairs without ≥2 numeric samples on both sides stay confirmed) — good defensive default, and it's exercised by bencher_report_spec.rb's "fails open" cases.
• Boundary#mirror reuse for both t-test and percentage thresholds is correct: baseline*(1±p) is symmetric about baseline in the same way the t-test's prediction interval is, so mirroring by 2*baseline - limit holds for both.
• Process-group handling (setsid + kill -TERM -- "-$PID") and the port-free wait loop between phases look correct — the earlier single-run version only killed the direct PID, which would have orphaned foreman-managed children across phases.
• run_baseline aborting via exit before the head run submits anything prevents comparing against a baseline that was never recorded — good ordering, and it's the first thing verified in cli_spec.rb.
• Untrusted GitHub context values (PR base sha, before-sha, dispatch ref) are consistently passed through env: rather than interpolated directly into run: shell blocks, avoiding script-injection risk.
• Naming for sample-confirmation joins (BmfCollector#add's "#{prefix}#{name}#{suffix}") is the same string Bencher echoes back as benchmark.name, so the head/base sample dictionaries and alert-pair matching line up correctly without a normalization step.

Minor observations (non-blocking)

1. .github/workflows/benchmark.yml "Resolve benchmark comparison base" (workflow_dispatch, non-main branch): base=$(gh api .../compare/main...$DISPATCH_REF --jq .merge_base_commit.sha) runs under set -euo pipefail, so a failed gh api call now aborts the step immediately rather than falling back to main the way the old workflow_dispatch_args Ruby helper did. That's arguably the right call given this PR's "fail loud" philosophy elsewhere, but it is a behavior change for manual dispatches worth calling out explicitly if not intentional.
2. No job-level timeout-minutes was added to run-suite even though the job now runs the build+benchmark sequence twice (once per phase). With benchmark_timeout_minutes: 120 for the Pro suites, two phases could in theory consume close to 240 minutes just in the timed benchmark-execution steps, still under GitHub's 360-minute default job timeout, but worth double-checking against actual observed run times now that this lands on CI.
3. benchmarks/lib/bencher_runner.rb's mode_args falls through to the statistical branch via a bare else rather than when :statistical; harmless since MODES is validated in initialize, but an explicit when :statistical would make the three-way branch self-documenting without relying on the reader checking MODES.

Verification

I read the full diff (27 files) and the resulting benchmarks/lib/bencher_report.rb, bencher_runner.rb, track_benchmarks/cli.rb, and bench.rb in their post-merge state, plus the composite action and both workflow files. I was not able to execute bundle exec rspec in this sandbox to independently confirm the "328 specs pass" claim, but the new specs I read (sample confirmation fail-open cases, mode-specific CLI arg building, branch-plan collision avoidance) look correctly targeted at the behavior they claim to cover.

[greptile-apps]

Greptile Summary

This PR changes CI benchmarking to compare base and head on the same runner. The main changes are:

• A new composite action for one benchmark phase.
• Base and head benchmark phases in the reusable suite workflow.
• Relative Bencher uploads with per-run baseline branches.
• Multi-sample k6 aggregation and unconfirmed-alert rendering.
• Updated benchmark docs and specs.

Confidence Score: 4/5

The main-push alert handoff needs a small fix before merging.

• The base/head workflow structure is coherent.
• The process cleanup and result paths look consistent.
• A downgraded Bencher alert can still be handled with the wrong exit-code state on main pushes.

benchmarks/lib/track_benchmarks/cli.rb

Important Files Changed

Filename	Overview
.github/actions/benchmark-suite-phase/action.yml	Adds the per-ref checkout, build, server lifecycle, benchmark execution, cleanup, and result-stash phase.
.github/workflows/benchmark-suite.yml	Runs base and head phases with a stashed head benchmark harness and passes baseline artifacts to tracking.
.github/workflows/benchmark.yml	Resolves the benchmark comparison base and passes it into initial and confirmation suite runs.
benchmarks/bench.rb	Adds repeated k6 samples, median aggregation, status merging, and per-sample display data.
benchmarks/lib/bencher_report.rb	Adds sample-confirmation logic that downgrades boundary crossings when base and head sample ranges overlap.
benchmarks/lib/track_benchmarks/cli.rb	Orchestrates baseline and head Bencher runs and applies sample confirmation before reporting.
benchmarks/lib/bencher_runner.rb	Adds relative baseline and head modes with percentage thresholds while preserving statistical mode.
benchmarks/lib/track_benchmarks/branch_args.rb	Builds per-run baseline and head branch plans for push, pull request, dispatch, and confirmation runs.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
  participant Detect as detect-changes
  participant Suite as benchmark-suite.yml
  participant Phase as benchmark-suite-phase
  participant Track as track_benchmarks.rb
  participant Bencher as Bencher

  Detect->>Suite: base_sha and suite matrix
  Suite->>Phase: benchmark base ref with HEAD harness
  Phase-->>Suite: base results in runner temp
  Suite->>Phase: benchmark head ref with HEAD harness
  Phase-->>Suite: head results in workspace
  Suite->>Track: base and head result files
  Track->>Bencher: upload base baseline branch
  Track->>Bencher: upload head comparison run
  Track->>Track: downgrade overlapping sample alerts

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
  participant Detect as detect-changes
  participant Suite as benchmark-suite.yml
  participant Phase as benchmark-suite-phase
  participant Track as track_benchmarks.rb
  participant Bencher as Bencher

  Detect->>Suite: base_sha and suite matrix
  Suite->>Phase: benchmark base ref with HEAD harness
  Phase-->>Suite: base results in runner temp
  Suite->>Phase: benchmark head ref with HEAD harness
  Phase-->>Suite: head results in workspace
  Suite->>Track: base and head result files
  Track->>Bencher: upload base baseline branch
  Track->>Bencher: upload head comparison run
  Track->>Track: downgrade overlapping sample alerts

Loading

Comments Outside Diff (1)

1. benchmarks/lib/track_benchmarks/cli.rb, line 37-39 (link)

   Unconfirmed Alerts Fail Main

   When a main-push Bencher alert is downgraded by sample confirmation, bencher_exit_code is already normalized to 0 here. Passing that value into the candidate writer loses the original Bencher failure code, so a main run with only unconfirmed alerts can still go down the wrong handoff path instead of cleanly treating the alert as downgraded.

   Context Used: CLAUDE.md (source)

_{Reviews (1): Last reviewed commit: "Stabilize relative benchmarking: fix wor..." | Re-trigger Greptile}

[claude]

Review: Switch CI benchmarks to relative continuous benchmarking

Reviewed the diff (gh pr diff 4580) across the new composite action, the two workflow files, and the Ruby benchmark harness/tracking code.

Overview

This is a well-engineered, thoroughly-documented rework of the benchmark CI pipeline: same-runner base-vs-head comparison instead of cross-runner statistical trend tracking, plus median-of-N sampling and a sample-confirmation gate to kill noise. The PR description's root-cause analysis of the k6 keep-alive/Puma-worker-affinity noise source (noConnectionReuse: true) is a genuinely good catch, backed by concrete measurement data.

Strengths

• Root-cause fix, not a band-aid: the noConnectionReuse change directly addresses the LIFO accept-queue/keep-alive pinning issue rather than just loosening thresholds.
• One-instrument discipline: stashing the HEAD ref's benchmarks/ tree and running both phases from it (while app-side inputs still come from each phase's own checkout) is a subtle but important correctness property — it stops a harness change (like this PR itself) from reading as a fake performance delta. Well documented in both the composite action header and the workflow inline comments.
• Fail-open sample confirmation: BencherReport#apply_sample_confirmation! only downgrades a crossing when both sides have ≥2 numeric samples and ranges overlap; anything else (missing sidecar, single-sample, non-numeric) preserves today's behavior instead of silently suppressing real regressions.
• Per-run, per-suite/shard-unique baseline branches (base-<run_id>-<suite-slug>) correctly prevent concurrent shards from cross-contaminating each other's Bencher baseline.
• Test coverage is extensive and targeted (sample aggregation, median edge cases, status merging, sample-confirmation disjoint/overlap/fail-open cases, exit-code normalization, CLI wiring) — this is exactly the kind of logic that's easy to get subtly wrong, and it's well covered.
• Workflow security hygiene: inputs are passed via env:/action with: rather than interpolated directly into run: script bodies, avoiding the classic Actions script-injection pattern.

Issue found (see inline comment)

.github/workflows/benchmark.yml, "Resolve benchmark comparison base" step: the workflow_dispatch branch's gh api .../compare/main...$DISPATCH_REF --jq .merge_base_commit.sha call is unguarded under set -euo pipefail, unlike the pull_request branch's equivalent external call, which is wrapped in an if ... && ... ; then ... else ... fallback. A transient API failure here will hard-abort the whole detect-changes job instead of degrading gracefully the way the previous Ruby implementation (workflow_dispatch_args) did. Left inline suggestion on the specific lines.

Minor / non-blocking observations

• Bencher branch accumulation: every job run creates a new throwaway base-<run_id>-<suite> (and confirm-base-<run_id>-<suite>) branch on Bencher that's never cleaned up. Given the run frequency, this could bloat the Bencher project's branch list over time — probably fine if Bencher auto-archives inactive branches, but worth a quick check if the dashboard becomes cluttered.
• Doubled job wall-clock: acknowledged in the PR description as a trade-off. Worth double-checking the "pro" suites' 120-minute benchmark_timeout_minutes (from generate_matrix.rb) plus install/build/DB-seed overhead, now paid twice per job, doesn't creep toward the default 360-minute GitHub Actions job timeout (no explicit timeout-minutes: is set on the benchmark job itself) — right now there appears to be comfortable headroom, but worth keeping an eye on if suite timeouts are ever raised.

Overall this looks like a solid, well-validated improvement to a genuinely noisy CI signal. The one correctness issue found is CI-only in impact and easy to fix.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 75543d47df

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[justin808]

Stability verdict: A/A run on the fixed harness is clean

Run 29187847192 (head 75543d4) completed all four suites. Like the initial run, this is an A/A test — base and head run identical app code — so every flag would be a false positive. Before/after:

suite	initial run — flagged rows / max |Δ| / cells >10%	this run — flagged rows / max |Δ| / cells >10%
Core (42 rows)	16 🔴🟢 / 149.0% / 43 of 84	0 / 18.5% / 8 of 83
Pro shard 1/2 (38 rows)	9 🔴🟢 / 202.2% / 30 of 72	0 / 7.8% / 0 of 72
Pro shard 2/2 (37 rows)	11 🔴🟢 / 109.8% / 50 of 74	0 / 5.6% / 0 of 69
Pro Node Renderer (2 rows)	0 / 6.1% / 0 of 4	0 / 1.1% / 0 of 4
total	36 flagged rows (42 flag cells), 123 of 234 cells >10%	0 flagged rows, 8 of 228 cells >10%, 0 cells ≥25%

• No route crossed any 25% threshold, so the new sample-confirmation layer didn't even need to downgrade anything (no ⚠️ rows) — but it's armed for the day a disturbance does cross.
• The only residual >10% deltas are four adjacent client-side routes in Core, all shifted the same direction (+13–18% RPS / −11–16% p50): a single few-minute disturbance window on the shared runner during the base phase's pass over that stretch, not per-route noise. All comfortably inside the threshold. Fully cancelling multi-minute phase-local drift would take interleaved base/head phases — noted in the PR as the accepted residual.
• The confirmation/regression pipeline jobs correctly skipped (no alerts to confirm).

Requesting hosted CI next (workflow changes trip the generator-sensitivity gate). Note the ready-for-hosted-ci label event will kick off a second benchmark run and refresh the summary tables — a bonus second A/A sample; this comment preserves the first run's numbers.

[justin808]

+ci-run-hosted

[github-actions]

Hosted CI Requested

Triggered 9 workflow(s) for 75543d47df0b.
Mode: optimized hosted CI (path-selected by script/ci-changes-detector).
Added ready-for-hosted-ci, so future commits will keep running optimized hosted CI until +ci-stop-hosted is used.

View progress in the Actions tab.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e430a435d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Pro Node Renderer Benchmark Summary

Benchmark	RPS	p50(ms)	p90(ms)	Status
Pro Node Renderer: simple_eval (non-RSC)	2086.96 ▲2.3% (2039.19)	4.38 ▼0.7% (4.41)	5.5	200=62617
Pro Node Renderer: react_ssr (non-RSC)	1794.15 ▲2.7% (1747.15)	4.96 ▼5.9% (5.27)	6.38	200=53827

▲/▼ non-zero change vs baseline · 0.0% exact/near-zero match · 🔴 significant regression · 🟢 significant improvement (tracked measures) · ⚠️ crossed threshold but base/head samples overlap (unconfirmed) · (n) = baseline

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/actions/benchmark-suite-phase/action.yml:
- Around line 335-375: Add the GitHub Actions `if: ${{ always() }}` condition to
the “Stop benchmark target (${{ inputs.phase }})” step so its cleanup script
runs even when the preceding benchmark execution fails. Keep the existing
process-group termination and port validation logic unchanged.
- Around line 87-105: Validate the PHASE input in the benchmark checkout step
before it is used or written to GITHUB_ENV, accepting only base or head and
rejecting all other values, including newline-containing input. Anchor the
validation to the existing PHASE environment variable and preserve the current
checkout and cleanup flow for valid values.

In `@benchmarks/lib/bencher_report.rb`:
- Around line 254-278: Update overlapping_measures to resolve each measure
pair’s Boundary and retain confirmation only when every comparable head sample
crosses the same effective limit; do not rely solely on disjoint head/base
ranges. Preserve the existing range check as an optimization if useful, handle
the lower-limit example so 95 prevents confirmation, and add a spec covering
this disjoint-but-not-all-crossing case.

In `@benchmarks/spec/bencher_report_spec.rb`:
- Around line 453-461: Update the fixture in the “matches sample measure keys
across slug/name normalization” example to use distinct measure slug and
display-name values from alerting_report. Populate the sample maps with the
appropriate slug and name representations so the expectation verifies matching
through measure.name rather than only case-folding.
- Around line 444-451: Update the example around report.significance to first
assert that the parsed "/foo: Core" rps result with :lower direction is
classified as :improvement, then retain the existing assertion that it becomes
:unconfirmed after confirmed is applied. This must verify the intended
improvement-to-unconfirmed transition rather than only its final state.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3d8edc9a-c3dd-44f2-833f-178f5a40e51b

📥 Commits

Reviewing files that changed from the base of the PR and between 75543d4 and 9e430a4.

📒 Files selected for processing (27)

• .github/actions/benchmark-suite-phase/action.yml
• .github/workflows/benchmark-suite.yml
• .github/workflows/benchmark.yml
• benchmarks/LOCAL_BENCHMARK.md
• benchmarks/bench-node-renderer.rb
• benchmarks/bench.rb
• benchmarks/k6.ts
• benchmarks/lib/bencher_report.rb
• benchmarks/lib/bencher_runner.rb
• benchmarks/lib/benchmark_table.rb
• benchmarks/lib/bmf_helpers.rb
• benchmarks/lib/track_benchmarks/bencher_run.rb
• benchmarks/lib/track_benchmarks/branch_args.rb
• benchmarks/lib/track_benchmarks/cli.rb
• benchmarks/lib/track_benchmarks/config.rb
• benchmarks/lib/track_benchmarks/confirmation.rb
• benchmarks/lib/track_benchmarks/summary.rb
• benchmarks/spec/bench_spec.rb
• benchmarks/spec/bencher_report_spec.rb
• benchmarks/spec/bencher_runner_spec.rb
• benchmarks/spec/benchmark_table_spec.rb
• benchmarks/spec/bmf_collector_spec.rb
• benchmarks/spec/track_benchmarks/branch_args_spec.rb
• benchmarks/spec/track_benchmarks/cli_spec.rb
• benchmarks/spec/track_benchmarks/summary_spec.rb
• benchmarks/spec/track_benchmarks_spec.rb
• benchmarks/track_benchmarks.rb

🚧 Files skipped from review as they are similar to previous changes (21)

• benchmarks/spec/track_benchmarks/branch_args_spec.rb
• benchmarks/k6.ts
• benchmarks/spec/benchmark_table_spec.rb
• benchmarks/bench-node-renderer.rb
• benchmarks/lib/track_benchmarks/bencher_run.rb
• benchmarks/lib/track_benchmarks/summary.rb
• benchmarks/spec/bmf_collector_spec.rb
• benchmarks/spec/track_benchmarks/summary_spec.rb
• benchmarks/spec/bencher_runner_spec.rb
• benchmarks/lib/track_benchmarks/config.rb
• .github/workflows/benchmark.yml
• benchmarks/lib/benchmark_table.rb
• benchmarks/track_benchmarks.rb
• benchmarks/LOCAL_BENCHMARK.md
• benchmarks/spec/track_benchmarks/cli_spec.rb
• benchmarks/lib/bmf_helpers.rb
• .github/workflows/benchmark-suite.yml
• benchmarks/bench.rb
• benchmarks/lib/bencher_runner.rb
• benchmarks/lib/track_benchmarks/branch_args.rb
• benchmarks/lib/track_benchmarks/cli.rb

[github-actions]

Core Benchmark Summary

Benchmark	RPS	p50(ms)	p90(ms)	Status
/: Core	3.31 0.0% (3.31)	2778.32 ▼0.3% (2787.15)	3399.94	200=138
/client_side_hello_world: Core	466.39 ▲1.0% (461.84)	20.41 ▲0.2% (20.37)	26.33	200=16831
/client_side_rescript_hello_world: Core	475.62 ▼1.0% (480.61)	20.13 ▲3.0% (19.55)	25.94	200=17126
/client_side_hello_world_shared_store: Core	451.37 ▼1.8% (459.81)	21.11 ▲3.5% (20.4)	27.33	200=16314
/client_side_hello_world_shared_store_controller: Core	445.65 ▼2.2% (455.76)	21.3 ▲3.5% (20.57)	27.72	200=16212
/client_side_hello_world_shared_store_defer: Core	442.89 ▼3.2% (457.34)	21.49 ▲4.3% (20.6)	27.62	200=16137
/server_side_hello_world_shared_store: Core	14.21 0.0% (14.21)	688.3 ▼2.0% (702.16)	877.02	200=534
/server_side_hello_world_shared_store_controller: Core	14.25 0.0% (14.25)	702.07 ▲0.9% (696.14)	863.8	200=539
/server_side_hello_world_shared_store_defer: Core	14.17 ▼0.6% (14.26)	695.51 0.0% (695.54)	874.18	200=534
/server_side_hello_world: Core	28.61 ▼0.6% (28.78)	346.58 ▼0.7% (348.89)	400.11	200=1060
/server_side_hello_world_hooks: Core	28.73 ▲0.8% (28.5)	348.31 ▲0.1% (348.13)	390.51	200=1057
/server_side_hello_world_props: Core	28.94 0.0% (28.93)	343.6 ▼0.4% (344.9)	393.38	200=1067
/client_side_log_throw: Core	459.35 ▼4.4% (480.57)	20.81 ▲6.3% (19.58)	26.68	200=16652
/server_side_log_throw: Core	28.01 ▲1.2% (27.69)	354.16 ▼1.4% (359.08)	413.58	200=1036
/server_side_log_throw_plain_js: Core	28.02 ▼1.4% (28.43)	355.96 ▲1.4% (350.88)	410.88	200=1038
/server_side_log_throw_raise: Core	28.6 ▲1.1% (28.3)	350.28 ▼0.9% (353.4)	400.27	3xx=1055
/server_side_log_throw_raise_invoker: Core	542.23 ▼1.8% (551.91)	17.54 ▲3.0% (17.03)	22.64	200=19540
/server_side_hello_world_es5: Core	28.87 ▲0.5% (28.74)	343.66 ▼0.4% (345.2)	397.82	200=1065
/server_side_redux_app: Core	27.65 ▼0.2% (27.7)	360.37 ▲0.4% (359.03)	420.99	200=1022
/server_side_hello_world_with_options: Core	28.71 ▼1.1% (29.03)	344.35 0.0% (344.27)	396.01	200=1058
/server_side_redux_app_cached: Core	484.96 ▼1.7% (493.56)	19.73 ▲3.5% (19.06)	26.18	200=17529
/client_side_manual_render: Core	473.5 ▼0.4% (475.3)	20.16 ▲2.1% (19.74)	25.84	200=16977
/render_js: Core	30.9 ▼0.2% (30.95)	322.52 ▲0.1% (322.16)	368.51	200=1140
/react_router: Core	27.09 0.0% (27.1)	364.89 ▼0.7% (367.46)	417.37	200=1005
/pure_component: Core	28.86 ▼1.2% (29.22)	344.27 ▲0.4% (343.0)	388.64	200=1067
/react_compiler_example: Core	28.67 ▲0.2% (28.6)	349.41 ▲0.5% (347.66)	390.8	200=1058
/css_modules_images_fonts_example: Core	28.53 ▼0.3% (28.62)	348.76 ▲0.2% (348.19)	398.43	200=1055
/turbolinks_cache_disabled: Core	471.55 ▼3.2% (486.91)	20.14 ▲4.5% (19.27)	26.37	200=17073
/rendered_html: Core	28.86 ▼1.0% (29.16)	345.23 ▲1.1% (341.41)	392.4	200=1067
/xhr_refresh: Core	14.89 ▲0.5% (14.82)	671.51 ▲2.2% (657.26)	843.26	200=559
/react_helmet: Core	28.02 ▲0.1% (27.99)	355.01 ▼0.1% (355.51)	405.51	200=1038
/broken_app: Core	27.72 ▼0.1% (27.74)	359.35 ▲0.3% (358.14)	407.88	200=1028
/image_example: Core	28.48 ▲0.2% (28.41)	350.23 ▲0.2% (349.63)	399.37	200=1048
/font_optimization_example: Core	539.16 ▲0.5% (536.26)	17.69 ▲1.5% (17.43)	22.52	200=19435
/client_side_activity: Core	466.47 ▼2.0% (476.01)	20.46 ▲4.2% (19.64)	26.32	200=16862
/server_side_activity: Core	28.51 ▼0.7% (28.72)	351.77 ▲2.0% (344.96)	397.65	200=1051
/turbo_frame_tag_hello_world: Core	512.16 ▼0.3% (513.77)	18.64 ▲2.6% (18.17)	24.07	200=18423
/manual_render_test: Core	484.66 ▲1.2% (479.07)	19.73 ▲0.8% (19.57)	25.09	200=17465
/root_error_callbacks: Core	29.09 ▲1.1% (28.76)	342.82 ▼1.0% (346.44)	387.18	200=1075
/hydration_scheduling: Core	10.08 ▲1.4% (9.94)	964.13 ▼1.7% (980.79)	1245.92	200=390
/rails_form: Core	474.17 ▲0.3% (472.62)	20.07 ▲1.4% (19.79)	26.04	200=17046
/typed_rails_action: Core	470.94 ▲0.3% (469.32)	20.27 ▲1.7% (19.93)	25.98	200=16909

▲/▼ non-zero change vs baseline · 0.0% exact/near-zero match · 🔴 significant regression · 🟢 significant improvement (tracked measures) · ⚠️ crossed threshold but base/head samples overlap (unconfirmed) · (n) = baseline

[github-actions]

Pro (shard 2/2) Benchmark Summary

Benchmark	RPS	p50(ms)	p90(ms)	Status
/empty: Pro	1188.63 ▲1.2% (1174.26)	8.05 ▼1.6% (8.18)	9.78	200=41452
/ssr_shell_error: Pro	137.3 ▲1.9% (134.69)	70.05 ▼2.1% (71.54)	90.21	200=4955
/ssr_sync_error: Pro	139.41 ▲1.9% (136.81)	68.72 ▼1.9% (70.08)	88.79	200=4891,3xx=19
/rsc_component_error: Pro	136.08 ▲1.3% (134.34)	70.68 ▼0.7% (71.21)	91.46	200=4901
/non_existing_stream_react_component: Pro	155.65 ▲1.8% (152.84)	61.72 ▼0.8% (62.23)	79.2	200=5629
/server_side_redux_app_cached: Pro	313.95 ▲2.6% (306.01)	30.65 ▼1.6% (31.16)	38.76	200=11309
/loadable: Pro	131.8 ▲2.0% (129.25)	73.36 ▼0.9% (74.06)	95.09	200=4552,3xx=27
/apollo_graphql: Pro	123.6 ▲3.1% (119.89)	77.41 ▼3.6% (80.28)	103.01	200=4493
/console_logs_in_async_server: Pro	3.51 ▲2.6% (3.42)	2151.74 ▲0.3% (2146.12)	4095.31	200=159
/stream_error_demo: Pro	3.68 ▼1.9% (3.75)	2037.34 ▼0.1% (2039.52)	3963.31	200=157
/stream_async_components: Pro	134.07 ▲2.8% (130.39)	71.1 ▼3.8% (73.94)	92.97	200=4806
/rsc_posts_page_over_http: Pro	110.68 ▼1.0% (111.84)	81.83 ▼4.2% (85.45)	109.88	200=4068,3xx=32
/rsc_echo_props: Pro	50.82 ▼1.3% (51.47)	190.72 ▲0.9% (189.06)	237.59	200=1859
/client_side_fouc_probe: Pro	301.74 ▲1.1% (298.36)	31.87 0.0% (31.87)	41.23	200=10765
/async_on_server_sync_on_client_client_render: Pro	298.84 ▲4.2% (286.73)	31.82 ▼4.1% (33.18)	41.17	200=10721
/server_router_client_render: Pro	289.97 ▲2.0% (284.25)	33.11 ▼1.9% (33.74)	42.92	200=10508
/unwrapped_rsc_route_stream_render: Pro	151.93 ▲0.2% (151.61)	63.17 ▼0.2% (63.29)	81.01	200=5480
/async_render_function_returns_component: Pro	152.68 ▲1.4% (150.56)	60.48 ▼0.4% (60.71)	81.68	200=5499,3xx=18
/native_metadata: Pro	155.26 ▲3.6% (149.87)	61.6 ▼3.3% (63.7)	81.24	200=5594
/hybrid_metadata_streaming: Pro	151.77 ▲2.6% (147.96)	62.99 ▼3.2% (65.04)	82.29	200=5491
/cache_demo: Pro	104.95 0.0% (104.94)	88.02 ▼0.8% (88.75)	118.85	200=3607
/client_side_hello_world: Pro	293.77 ▼1.9% (299.53)	32.55 ▲1.4% (32.1)	42.26	200=10672
/client_side_hello_world_shared_store_controller: Pro	287.32 ▲3.5% (277.53)	33.15 ▼3.9% (34.51)	43.4	200=10310
/server_side_hello_world_shared_store: Pro	106.35 ▲1.7% (104.62)	85.84 ▼3.3% (88.79)	116.32	200=3798,3xx=19
/server_side_hello_world_shared_store_defer: Pro	111.44 ▲2.0% (109.21)	86.81 ▼1.4% (88.05)	109.63	200=4046
/server_side_hello_world_hooks: Pro	166.63 ▲2.3% (162.9)	57.71 ▼1.1% (58.33)	73.53	200=5989
/server_side_log_throw: Pro	153.81 ▲1.4% (151.74)	61.71 ▼2.5% (63.32)	82.48	200=5309,3xx=15
/source_mapped_prerender_error_probe: Pro	208.21 ▲4.3% (199.57)	46.31 ▼4.8% (48.66)	60.89	3xx=7469
/server_side_log_throw_raise_invoker: Pro	341.89 ▼1.8% (348.3)	27.76 ▲0.9% (27.52)	36.5	200=12314
/server_side_redux_app: Pro	142.16 0.0% (142.09)	65.77 ▼1.9% (67.02)	89.85	200=5148,3xx=20
/server_side_redux_app_cached: Pro	316.8 ▲3.5% (306.01)	30.44 ▼2.3% (31.16)	38.39	200=11391
/render_js: Pro	326.99 ▲1.2% (323.14)	29.31 ▼1.4% (29.74)	36.88	200=11837
/pure_component: Pro	152.38 ▼2.2% (155.85)	60.64 ▼0.7% (61.08)	81.08	200=5640
/turbolinks_cache_disabled: Pro	310.1 ▲2.6% (302.11)	30.93 ▼2.4% (31.69)	40.28	200=11207
/xhr_refresh: Pro	116.03 ▲3.4% (112.22)	83.24 ▼1.7% (84.64)	105.83	200=4181
/broken_app: Pro	154.26 ▲1.8% (151.53)	60.19 ▼3.6% (62.44)	83.25	200=5394,3xx=15
/server_render_with_timeout: Pro	67.33 ▲1.7% (66.2)	140.65 ▲0.3% (140.23)	173.32	200=2459

▲/▼ non-zero change vs baseline · 0.0% exact/near-zero match · 🔴 significant regression · 🟢 significant improvement (tracked measures) · ⚠️ crossed threshold but base/head samples overlap (unconfirmed) · (n) = baseline

[github-actions]

Pro (shard 1/2) Benchmark Summary

Benchmark	RPS	p50(ms)	p90(ms)	Status
/: Pro	54.7 ▲15.2% (47.48)	175.9 ▼14.3% (205.23)	221.94	200=2034
/error_scenarios_hub: Pro	339.9 ▲7.4% (316.44)	27.84 ▼7.1% (29.98)	36.97	200=12324
/ssr_async_error: Pro	3.68 ▼1.6% (3.74)	2035.6 ▼0.4% (2042.93)	3965.69	200=158
/ssr_async_prop_error: Pro	1.24 ▼3.9% (1.29)	5040.89 ▼0.1% (5044.28)	10064.71	200=77
/non_existing_react_component: Pro	145.35 ▼4.8% (152.71)	65.66 ▲6.9% (61.41)	87.31	200=5149,3xx=17
/non_existing_rsc_payload: Pro	147.63 ▼14.9% (173.54)	64.92 ▲18.6% (54.73)	84.02	200=5364
/cached_react_helmet: Pro	315.29 ▼13.3% (363.73)	30.61 ▲16.5% (26.28)	38.39	200=11316
/cached_redux_component: Pro	315.63 ▼9.2% (347.47)	30.47 ▲11.0% (27.46)	38.37	200=11397
/lazy_apollo_graphql: Pro	130.61 ▲8.4% (120.53)	74.22 ▼6.0% (78.95)	96.5	200=4485,3xx=23
/redis_receiver: Pro	90.83 ▲5.0% (86.47)	100.69 ▼3.5% (104.34)	154.57	200=3291
/stream_shell_error_demo: Pro	153.04 ▲0.8% (151.81)	63.07 ▲0.5% (62.78)	81.51	200=5498
/test_incremental_rendering: Pro	3.69 0.0% (3.69)	2027.85 ▼0.2% (2031.97)	3977.24	200=155
/rsc_posts_page_over_redis: Pro	91.88 ▼1.3% (93.08)	104.19 0.0% (104.17)	138.45	200=3305
/rsc_fouc_probe: Pro	150.62 ▲3.6% (145.36)	63.61 ▼3.2% (65.71)	81.99	200=5474
/async_on_server_sync_on_client: Pro	2.32 0.0% (2.32)	3030.66 ▼0.2% (3036.31)	5994.42	200=105
/server_router: Pro	138.95 ▼12.3% (158.35)	69.05 ▲14.5% (60.31)	88.98	200=5074
/unwrapped_rsc_route_client_render: Pro	306.97 ▼15.5% (363.13)	31.19 ▲20.1% (25.96)	40.53	200=11118
/async_render_function_returns_string: Pro	155.42 ▼10.4% (173.47)	61.41 ▲11.6% (55.01)	80.34	200=5553,3xx=26
/async_components_demo: Pro	7.94 ▲0.1% (7.93)	1074.53 0.0% (1074.34)	1783.71	200=320
/stream_native_metadata: Pro	163.78 ▼1.1% (165.52)	58.69 ▲0.8% (58.22)	76.33	200=5904
/rsc_native_metadata: Pro	8.01 ▼1.0% (8.09)	1035.54 ▼0.2% (1038.12)	1927.83	200=313
/react_intl_rsc_demo: Pro	84.43 ▲0.8% (83.78)	115.1 ▲0.4% (114.6)	152.05	200=3000
/client_side_hello_world_shared_store: Pro	338.91 ▲3.2% (328.35)	28.14 ▼2.5% (28.86)	36.59	200=12164
/client_side_hello_world_shared_store_defer: Pro	333.59 ▲2.2% (326.42)	28.56 ▼0.8% (28.78)	37.58	200=11952
/server_side_hello_world_shared_store_controller: Pro	120.49 ▼3.1% (124.31)	78.93 ▲2.5% (77.0)	103.42	200=4403
/server_side_hello_world: Pro	170.83 ▼1.0% (172.56)	56.13 ▲3.1% (54.45)	73.63	200=6175
/client_side_log_throw: Pro	340.95 ▲1.1% (337.12)	27.16 ▼2.8% (27.93)	37.48	200=12458
/server_side_log_throw_plain_js: Pro	366.39 ▲1.8% (359.8)	26.08 ▼1.5% (26.47)	33.29	200=13339
/server_side_log_throw_raise: Pro	231.14 ▲0.5% (230.04)	41.93 ▲1.0% (41.53)	55.17	3xx=8133
/server_side_hello_world_es5: Pro	175.18 ▼0.1% (175.32)	54.41 0.0% (54.42)	72.54	200=6134,3xx=9
/server_side_hello_world_with_options: Pro	170.31 ▼0.9% (171.93)	56.16 ▲1.1% (55.53)	74.99	200=6134
/client_side_manual_render: Pro	347.61 ▲4.7% (332.13)	27.15 ▼4.1% (28.32)	36.12	200=12713
/react_router: Pro	208.45 ▼1.1% (210.84)	46.64 ▲1.2% (46.09)	61.42	200=7369
/css_modules_images_fonts_example: Pro	159.86 ▼5.9% (169.92)	58.84 ▲7.6% (54.66)	78.59	200=5758,3xx=14
/rendered_html: Pro	163.38 ▼7.5% (176.66)	58.67 ▲8.9% (53.88)	77.83	200=5877
/react_helmet: Pro	114.51 ▼1.8% (116.59)	82.0 ▲2.0% (80.42)	105.77	200=4121
/image_example: Pro	167.9 ▲3.7% (161.93)	56.72 ▼4.4% (59.32)	74.64	200=5924,3xx=6
/posts_page: Pro	127.81 ▲3.8% (123.17)	74.97 ▼2.5% (76.93)	97.09	200=4590

▲/▼ non-zero change vs baseline · 0.0% exact/near-zero match · 🔴 significant regression · 🟢 significant improvement (tracked measures) · ⚠️ crossed threshold but base/head samples overlap (unconfirmed) · (n) = baseline