Skip to content

Commit d41fde3

Browse files
hyperpolymathclaude
hyperpolymath
and
claude
committed
perf(d-4 bootstrap): workflow_dispatch automation for baseline collection
Phase D-4 of the single-lane HCG tier-2 channel (standards#91 / #99) is "real baseline numbers populated in bench/baseline.json and the perf-regression gate armed by flipping _status to active". The rebaseline ritual in docs/perf-contract.md § Baseline lifecycle defines step 2 as `just bench-collect` on a CI-equivalent target, but the published reference per the same doc is `ubuntu-latest` GHA runners — yet the ritual was authored as a manual local step that requires the operator to have an Elixir 1.19 / OTP 28 toolchain on their machine. That gap blocked D-4 from being executable by anyone without a matching local dev env. This PR adds the missing on-the-published-target automation. It does NOT collect numbers or flip _status itself; those are still the maintainer's deliberate acts. It provides the means. What landed ─────────── * `.github/workflows/perf-rebaseline.yml` — manual workflow_dispatch workflow that runs `mix run bench/gateway_latency.exs` on ubuntu-latest (same target perf-regression.yml uses, so numbers are comparable), pipes results through bench/rebaseline.exs, opens a `perf: rebaseline (standards#99)` PR with the regenerated bench/baseline.json. Uses the same SHA-pinned actions and the same `runner.os-perf-${hashFiles(mix.lock)}` cache key as perf-regression.yml so the first rebaseline run primes off the warm cache. Deliberately NO concurrency cancel-in-progress (an operator-initiated rebaseline should complete on its own; this workflow has no obsolescence relationship the way per-PR runs do). * `bench/rebaseline.exs` — reads bench/results.json + bench/baseline.json, writes a new bench/baseline.json with real p50/p95/p99/ips per scenario. Preserves _comment, _schema_version, tolerance, and per-scenario _comment_* fields. Leaves _status as "scaffold-placeholder" — flipping to "active" is the maintainer's separate review step. Field order is preserved end-to-end via Jason.OrderedObject (decode with `objects: :ordered_objects`, encode back; Jason 1.4+ already in mix.lock) so the rebaseline diff is review-grade (numbers move; structure does not). * `Justfile` — adds `just rebaseline` (runs harness + regeneration script) so the same two-step sequence the workflow runs is also available locally for operators previewing a rebaseline. The `just bench-collect` message is updated to point at `just rebaseline`. * `docs/perf-contract.md` — splits § Baseline lifecycle into "Automated (preferred — D-4 bootstrap)" and "Manual (for local previews or operators without GHA access)". Both paths leave _status as scaffold-placeholder; flipping to active is called out as a separate deliberate decision in either path. What is deliberately NOT in this PR ──────────────────────────────────── * The real baseline numbers themselves — those land in the generated `perf: rebaseline (standards#99)` PR after the maintainer dispatches the workflow. * Flipping bench/baseline.json `_status` to "active" — maintainer judgement on noise/spread; may land in the same generated PR or in a follow-up. * Tightening tolerance ratios — also a maintainer judgement, post-D-4. * Schema-drift hardening in bench/compare.exs (new scenario in results but not baseline silently passes in active mode) — separate defensive D-3 follow-up, not coupled to D-4 collection. Refs hyperpolymath/standards#91 Refs hyperpolymath/standards#99 (NOT Closes #99: joint-close is owner-only, and D-4 baseline collection + the _status flip still pend under #99 even after this lands. This PR makes D-4 executable; the generated rebaseline PR + the flip-to-active PR jointly close it.) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 54a97a6 commit d41fde3

4 files changed

Lines changed: 393 additions & 3 deletions

File tree

‎.github/workflows/perf-rebaseline.yml‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# SPDX-License-Identifier: MPL-2.0
2+
# Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
3+
#
4+
# perf-rebaseline.yml — Phase D-4 baseline-collection automation
5+
# (standards#99 of the standards#91 single-lane HCG channel).
6+
#
7+
# What it does
8+
# ────────────
9+
# Manual workflow_dispatch only. Runs bench/gateway_latency.exs on the
10+
# published reference target (ubuntu-latest, per docs/perf-contract.md
11+
# § Targets), pipes bench/results.json through bench/rebaseline.exs to
12+
# produce a regenerated bench/baseline.json with real p50/p95/p99/ips
13+
# per scenario, then opens a `perf: rebaseline (standards#99)` PR for
14+
# maintainer review.
15+
#
16+
# Why it exists
17+
# ─────────────
18+
# docs/perf-contract.md § Baseline lifecycle defines the rebaseline
19+
# ritual as `just bench-collect` on a CI-equivalent target. The
20+
# published reference is ubuntu-latest GHA, but the ritual was authored
21+
# as a manual local step, which requires an Elixir 1.19 / OTP 28
22+
# toolchain set up locally on the operator's machine. This workflow
23+
# moves step 2 of the ritual (the actual collection run) onto the
24+
# published reference target itself, so the rebaseline can be initiated
25+
# from any GitHub UI without a local toolchain. The numbers it produces
26+
# are then comparable to perf-regression.yml's gate, which runs on the
27+
# same target.
28+
#
29+
# The generated PR leaves bench/baseline.json `_status` as
30+
# "scaffold-placeholder". The maintainer reviews the numbers (steps
31+
# 3-4 of the ritual) and either flips `_status` → "active" in the same
32+
# PR (arming perf-regression.yml's gate immediately) or in a follow-up
33+
# after a confidence-building window. Splitting "land real numbers"
34+
# from "arm the gate" stays compatible with the runbook's D-4 / D-3
35+
# checklist split in
36+
# boj-server/docs/integration/hcg-tier2-rollout-runbook.md § 1.1.
37+
#
38+
# What it deliberately does NOT do
39+
# ────────────────────────────────
40+
# - Flip `_status` to "active" itself — that's a maintainer judgement
41+
# on the noise/spread of the collected numbers, not an automation.
42+
# - Push directly to main — never; always opens a PR.
43+
# - Tighten tolerance ratios — also a maintainer judgement once
44+
# intra-run variance is characterised.
45+
46+
name: Perf Rebaseline
47+
48+
on:
49+
workflow_dispatch:
50+
inputs:
51+
ref:
52+
description: 'Branch or SHA to collect from (default: main)'
53+
required: false
54+
default: main
55+
56+
# Deliberately NO concurrency cancel-in-progress here: a workflow_dispatch
57+
# rebaseline is operator-initiated and serial dispatches should each
58+
# complete on their own. (perf-regression.yml has cancel-in-progress
59+
# because PR re-pushes obsolete prior runs; this workflow has no such
60+
# obsolescence relationship.)
61+
62+
permissions:
63+
contents: write
64+
pull-requests: write
65+
66+
jobs:
67+
rebaseline:
68+
name: Collect baseline and open PR
69+
runs-on: ubuntu-latest
70+
timeout-minutes: 20
71+
72+
steps:
73+
- name: Checkout repository
74+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
75+
with:
76+
ref: ${{ github.event.inputs.ref }}
77+
# PR-creating workflows need fresh history so the branch push
78+
# carries enough context for the merge-base computation.
79+
fetch-depth: 0
80+
81+
- name: Setup Elixir/OTP
82+
uses: erlef/setup-beam@fc68ffb90438ef2936bbb3251622353b3dcb2f93 # v1.18.2
83+
with:
84+
# Pinned to match .tool-versions; bump both together.
85+
# MUST also match perf-regression.yml so numbers are comparable.
86+
elixir-version: '1.19'
87+
otp-version: '28'
88+
89+
- name: Cache deps
90+
uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
91+
with:
92+
# Reuse perf-regression.yml's cache key so the first rebaseline
93+
# after a perf-regression run primes off the warm cache.
94+
path: |
95+
deps
96+
_build
97+
key: ${{ runner.os }}-perf-${{ hashFiles('mix.lock') }}
98+
restore-keys: |
99+
${{ runner.os }}-perf-
100+
101+
- name: Install deps
102+
run: |
103+
mix local.hex --force
104+
mix local.rebar --force
105+
mix deps.get
106+
107+
- name: Compile
108+
run: mix compile --warnings-as-errors
109+
110+
- name: Run bench harness
111+
run: mix run bench/gateway_latency.exs | tee bench/console.log
112+
113+
- name: Regenerate baseline.json from results.json
114+
env:
115+
REBASELINE_RUN_ID: ${{ github.run_id }}
116+
REBASELINE_RUN_REF: ${{ github.event.inputs.ref }}
117+
run: mix run bench/rebaseline.exs
118+
119+
- name: Upload bench artefacts
120+
if: always()
121+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
122+
with:
123+
name: perf-rebaseline-results
124+
path: |
125+
bench/results.json
126+
bench/console.log
127+
retention-days: 30
128+
129+
- name: Open rebaseline PR
130+
env:
131+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
132+
run: |
133+
set -euo pipefail
134+
BRANCH="perf/rebaseline-${{ github.run_id }}"
135+
git config user.name 'github-actions[bot]'
136+
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
137+
git checkout -b "$BRANCH"
138+
git add bench/baseline.json
139+
if git diff --cached --quiet; then
140+
echo '::warning::No baseline.json delta from this run — nothing to PR.'
141+
exit 0
142+
fi
143+
RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
144+
git commit -m 'perf: rebaseline (standards#99)' \
145+
-m "Auto-collected from \`bench/gateway_latency.exs\` on \`ubuntu-latest\` (the published reference target per \`docs/perf-contract.md\` § Targets)." \
146+
-m "Workflow run: $RUN_URL"
147+
git push -u origin "$BRANCH"
148+
gh pr create \
149+
--base main \
150+
--head "$BRANCH" \
151+
--title 'perf: rebaseline (standards#99)' \
152+
--body "$(cat <<EOF
153+
## Summary
154+
155+
Auto-generated by \`.github/workflows/perf-rebaseline.yml\` (workflow run [#${{ github.run_id }}]($RUN_URL)). Replaces the scaffold-placeholder TODO values in \`bench/baseline.json\` with real \`p50\` / \`p95\` / \`p99\` / \`ips\` per scenario, collected by \`bench/gateway_latency.exs\` on the published reference target (\`ubuntu-latest\`, per \`docs/perf-contract.md\` § Targets).
156+
157+
\`Refs hyperpolymath/standards#91\`
158+
\`Refs hyperpolymath/standards#99\`
159+
160+
## What's in this PR
161+
162+
- \`bench/baseline.json\`: real per-scenario percentiles + ips; refreshed \`_generated_at\` / \`_generated_by\`.
163+
- \`_status\` remains \`scaffold-placeholder\`. The maintainer flips it to \`active\` in this PR (arming the gate immediately) or in a follow-up after a confidence-building window — see \`docs/perf-contract.md\` § Baseline lifecycle.
164+
- \`tolerance\` ratios unchanged.
165+
166+
## Review checklist
167+
168+
- [ ] Numbers are within an order of magnitude of expectations for each scenario (sanity).
169+
- [ ] Spread looks reasonable (no scenario with p99 >> p95 in a way that suggests a noisy outlier; if so, consider re-running before merging).
170+
- [ ] Decide whether to flip \`_status\` → \`active\` here (one-PR D-4 + D-3 close) or in a follow-up (D-4 lands; D-3 flip later).
171+
172+
Bench artefacts (results.json, console.log) are attached to the workflow run as the \`perf-rebaseline-results\` artefact for 30 days.
173+
EOF
174+
)"

‎Justfile‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,18 @@ bench:
110110
bench-collect:
111111
mix run bench/gateway_latency.exs
112112
@echo "Results written to bench/results.json"
113-
@echo "To rebaseline: review numbers, then update bench/baseline.json in a dedicated PR."
113+
@echo "To rebaseline: run 'just rebaseline' or update bench/baseline.json by hand in a dedicated PR."
114+
115+
# Run the harness and regenerate bench/baseline.json from the result
116+
# (Phase D-4 rebaseline ritual; see docs/perf-contract.md). Leaves
117+
# `_status` as scaffold-placeholder — the maintainer reviews and flips
118+
# to "active" in the rebaseline PR. The CI workflow
119+
# .github/workflows/perf-rebaseline.yml runs the same two steps on
120+
# the published reference target (ubuntu-latest); use this recipe to
121+
# preview the regeneration locally before dispatching the workflow.
122+
rebaseline:
123+
mix run bench/gateway_latency.exs
124+
mix run bench/rebaseline.exs
114125

115126
# ═══════════════════════════════════════════════════════════════════════════════
116127
# LINT & FORMAT

‎bench/rebaseline.exs‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# SPDX-License-Identifier: MPL-2.0
2+
# Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
3+
#
4+
# bench/rebaseline.exs — Phase D-4 baseline regeneration helper
5+
# (standards#99 of the standards#91 single-lane HCG channel).
6+
#
7+
# Reads bench/results.json (produced by bench/gateway_latency.exs) and
8+
# bench/baseline.json, then writes a regenerated bench/baseline.json
9+
# that replaces the per-scenario TODO values with the real p50/p95/p99/
10+
# ips from results.json — preserving _comment, _schema_version, tolerance,
11+
# per-scenario _comment_* fields, and (deliberately) _status.
12+
#
13+
# `_status` is left as "scaffold-placeholder". The rebaseline PR is the
14+
# review gate; the maintainer flips `_status` → "active" in the PR
15+
# (arming perf-regression.yml's gate) or in a follow-up. See
16+
# docs/perf-contract.md § Baseline lifecycle and the workflow file
17+
# .github/workflows/perf-rebaseline.yml for the surrounding ritual.
18+
#
19+
# Field ordering is preserved via Jason.OrderedObject so the diff
20+
# against the prior baseline is review-grade (numbers move; structure
21+
# does not).
22+
#
23+
# Runs:
24+
#
25+
# • Driven by .github/workflows/perf-rebaseline.yml on ubuntu-latest
26+
# (the published reference target per docs/perf-contract.md).
27+
# • Locally after `just bench-collect`:
28+
#
29+
# just rebaseline
30+
#
31+
# or directly:
32+
#
33+
# mix run bench/rebaseline.exs
34+
35+
defmodule Bench.Rebaseline do
36+
alias Jason.OrderedObject
37+
38+
@results_path "bench/results.json"
39+
@baseline_path "bench/baseline.json"
40+
41+
def run do
42+
with {:ok, results_raw} <- File.read(@results_path),
43+
{:ok, baseline_raw} <- File.read(@baseline_path),
44+
{:ok, results} <- Jason.decode(results_raw),
45+
{:ok, baseline} <- Jason.decode(baseline_raw, objects: :ordered_objects) do
46+
new_baseline = rebaseline(baseline, results)
47+
json = Jason.encode!(new_baseline, pretty: true)
48+
File.write!(@baseline_path, json <> "\n")
49+
report(new_baseline)
50+
else
51+
{:error, :enoent} ->
52+
IO.puts(
53+
:stderr,
54+
"ERROR: #{@results_path} or #{@baseline_path} missing. " <>
55+
"Did `mix run bench/gateway_latency.exs` run first?"
56+
)
57+
58+
System.halt(2)
59+
60+
{:error, reason} ->
61+
IO.puts(:stderr, "ERROR reading baseline/results: #{inspect(reason)}")
62+
System.halt(2)
63+
end
64+
end
65+
66+
# ── Rebaseline logic ───────────────────────────────────────────────────────
67+
68+
defp rebaseline(%OrderedObject{} = baseline, results) do
69+
results_stats = Map.get(results, "statistics", %{})
70+
existing_scenarios = oget(baseline, "scenarios", %OrderedObject{values: []})
71+
72+
rebaselined_scenarios =
73+
results_stats
74+
|> Enum.sort_by(&elem(&1, 0))
75+
|> Enum.map(fn {name, stats} ->
76+
{name, scenario_entry(oget(existing_scenarios, name, %OrderedObject{values: []}), stats)}
77+
end)
78+
|> then(fn pairs -> %OrderedObject{values: pairs} end)
79+
80+
baseline
81+
|> oput("_generated_at", DateTime.utc_now() |> DateTime.to_iso8601())
82+
|> oput("_generated_by", generated_by())
83+
|> oput("scenarios", rebaselined_scenarios)
84+
end
85+
86+
defp scenario_entry(%OrderedObject{values: existing_pairs}, stats) do
87+
comments =
88+
existing_pairs
89+
|> Enum.filter(fn {k, _v} -> String.starts_with?(k, "_comment") end)
90+
91+
numbers = [
92+
{"p50_us", us(stats, "50")},
93+
{"p95_us", us(stats, "95")},
94+
{"p99_us", us(stats, "99")},
95+
{"ips", round2(Map.get(stats, "ips"))}
96+
]
97+
98+
%OrderedObject{values: comments ++ numbers}
99+
end
100+
101+
defp us(stats, p) do
102+
case get_in(stats, ["percentiles", p]) do
103+
nil -> nil
104+
ns when is_number(ns) -> round2(ns / 1_000.0)
105+
end
106+
end
107+
108+
defp round2(nil), do: nil
109+
defp round2(n) when is_integer(n), do: Float.round(n / 1.0, 2)
110+
defp round2(n) when is_float(n), do: Float.round(n, 2)
111+
112+
defp generated_by do
113+
run_id = System.get_env("REBASELINE_RUN_ID")
114+
ref = System.get_env("REBASELINE_RUN_REF")
115+
116+
case run_id do
117+
nil ->
118+
"local: mix run bench/rebaseline.exs"
119+
120+
run ->
121+
".github/workflows/perf-rebaseline.yml (ubuntu-latest; ref=#{ref || "main"}; run=#{run})"
122+
end
123+
end
124+
125+
# ── OrderedObject helpers ──────────────────────────────────────────────────
126+
#
127+
# Jason.OrderedObject does not implement the full Access protocol, so
128+
# tiny get/put helpers keep the rebaseline code shape close to the
129+
# equivalent Map.get / Map.put it would otherwise use.
130+
131+
defp oget(%OrderedObject{values: pairs}, key, default) do
132+
case List.keyfind(pairs, key, 0) do
133+
{^key, v} -> v
134+
nil -> default
135+
end
136+
end
137+
138+
defp oput(%OrderedObject{values: pairs}, key, value) do
139+
new_pairs =
140+
case List.keymember?(pairs, key, 0) do
141+
true -> List.keyreplace(pairs, key, 0, {key, value})
142+
false -> pairs ++ [{key, value}]
143+
end
144+
145+
%OrderedObject{values: new_pairs}
146+
end
147+
148+
# ── Reporter ───────────────────────────────────────────────────────────────
149+
150+
defp report(%OrderedObject{} = baseline) do
151+
IO.puts("")
152+
IO.puts("bench/baseline.json regenerated.")
153+
IO.puts(" _status: #{inspect(oget(baseline, "_status", nil))}")
154+
IO.puts(" _generated_at: #{oget(baseline, "_generated_at", "?")}")
155+
IO.puts(" _generated_by: #{oget(baseline, "_generated_by", "?")}")
156+
IO.puts("")
157+
158+
scenarios = oget(baseline, "scenarios", %OrderedObject{values: []})
159+
160+
IO.puts("Per-scenario numbers (µs / ips):")
161+
162+
Enum.each(scenarios.values, fn {name, %OrderedObject{values: pairs}} ->
163+
m = Map.new(pairs)
164+
165+
IO.puts(
166+
" #{name}: p50=#{fmt(m["p50_us"])} p95=#{fmt(m["p95_us"])} " <>
167+
"p99=#{fmt(m["p99_us"])} ips=#{fmt(m["ips"])}"
168+
)
169+
end)
170+
171+
IO.puts("")
172+
173+
IO.puts(
174+
"Next: review numbers in the rebaseline PR; flip `_status` → \"active\" " <>
175+
"to arm the perf-regression gate (see docs/perf-contract.md § Baseline lifecycle)."
176+
)
177+
end
178+
179+
defp fmt(nil), do: "—"
180+
defp fmt(n) when is_number(n), do: to_string(n)
181+
end
182+
183+
Bench.Rebaseline.run()

0 commit comments

Comments
 (0)