You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Phase D-2 (PR #14) left two cost surfaces folded into the
"exact route allow (proxy 200)" scenario: the Phase A contract-header
construction in Proxy.build_backend_headers/1, and the transport-level
mTLS handshake. A baseline collected from D-2 would have attributed
both to the proxy-200 number, so a regression in either would only
show up as an aggregate slowdown -- the comparator couldn't tell us
which leg moved.
Phase D-3 (this change, hyperpolymath/standards#99 sub-deliverable)
pulls each into its own Benchee scenario so D-4 baseline collection
attributes them independently:
* "trust-header rewrite (Proxy.build_backend_headers)" - direct
call to a new @doc false Proxy.__benchmark_build_backend_headers__/1
seam over the existing private build_backend_headers/1. No policy
lookup, no Gateway.call/2 pipeline, no network I/O. Isolates the
cost surface the Phase A contract invariant lives on: the gateway
sets X-Trust-Level / X-Request-ID as authoritative values that
shadow any client-supplied header, and the cost of building that
final header map is what this scenario measures.
* "mTLS handshake (test CA)" - raw :ssl acceptor + :ssl.connect/4
+ immediate :ssl.close/1, using a test CA chain generated in-memory
at bench startup via :public_key.pkix_test_data/1. Each iteration
is one fresh handshake (verify_peer + cert chain validation +
key exchange) closed before any application bytes are exchanged.
Bounds the connection-spike SLO Phase E rollout has to budget for.
Why in-memory rather than reusing the committed Phase B fixture in
test/fixtures/mtls/: the fixture ships only *.crt files -- *.key is
gitignored at the repo root. Reusing the committed certs would require
committing the matching keys (or carving a fixture exception in
.gitignore), which the estate security posture prefers we don't do
for a bench fixture. The chain shape (test CA -> server peer + test CA
-> client peer, verify_peer enforced) is identical to Phase B's
fixture; only the key material differs, and it never touches disk.
bench/baseline.json gains the two new scenario keys (both TODO; the
_status stays "scaffold-placeholder" because D-3 still doesn't collect
real numbers -- D-4 does, via the `perf: rebaseline` ritual in
docs/perf-contract.md). _schema_version bumps 0.1.0 -> 0.2.0 to signal
the shape change to anyone diffing the file or running compare.exs
against an older results.json.
docs/perf-contract.md moves both items from "Out of scope" to in scope,
adds them to the scenarios table and the targets table, and re-anchors
the remaining out-of-scope list on D-4 (baseline collection), Phase E
(historical dashboard), and a possible follow-up amortised-handshake
scenario if D-4 numbers show the proxy-200 + handshake bracket is
loose. Phase D-3 is the last harness-shape change before D-4
collection: scenario keys are now stable, the comparator's per-scenario
diff stays correctly keyed across the flip from scaffold to active.
Unblocks: standards#99 D-4 (real baseline collection + gate arming)
and, downstream, the remaining items in section 1.1 of boj-server's
docs/integration/hcg-tier2-rollout-runbook.md.
Refs hyperpolymath/standards#91
Refs hyperpolymath/standards#99
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: bench/baseline.json
+17-3Lines changed: 17 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
{
2
-
"_comment": "Phase D scaffold-placeholder baseline (standards#99). Numbers are TODO. Phase D-1 (PR #12) wired the Benchee harness + this file + the CI gate; Phase D-2 (this PR) added a real in-process loopback backend so the 'exact route allow (proxy 200)' scenario measures real dial-and-read cost instead of port-1 connection-refused. Phase D-4 collects real numbers via the `perf: rebaseline` ritual in docs/perf-contract.md and flips `_status` to 'active' to arm the CI gate. DO NOT trust these numbers; the perf-regression CI gate is intentionally non-blocking until that flip.",
3
-
"_schema_version": "0.1.0-scaffold",
2
+
"_comment": "Phase D scaffold-placeholder baseline (standards#99). Numbers are TODO. Phase D-1 (PR #12) wired the Benchee harness + this file + the CI gate; Phase D-2 (PR #14) added a real in-process loopback backend so the 'exact route allow (proxy 200)' scenario measures real dial-and-read cost instead of port-1 connection-refused; Phase D-3 (this update) added two scenarios -- 'trust-header rewrite (Proxy.build_backend_headers)' isolating the Phase A contract-header construction cost, and 'mTLS handshake (test CA)' isolating per-handshake cost using the Phase B real-CA fixture. Phase D-4 collects real numbers via the `perf: rebaseline` ritual in docs/perf-contract.md and flips `_status` to 'active' to arm the CI gate. DO NOT trust these numbers; the perf-regression CI gate is intentionally non-blocking until that flip.",
3
+
"_schema_version": "0.2.0-scaffold",
4
4
"_generated_at": "TODO(Phase D-4): replace with ISO8601 timestamp from collection run",
"_comment_d3": "Phase D-3 scenario. Isolates the Phase A contract-header construction cost (X-Trust-Level / X-Request-ID / X-Forwarded-*). Direct call to Proxy.__benchmark_build_backend_headers__/1 on a pre-built conn with assigns[:trust_level] = :authenticated and assigns[:request_id] set. No policy lookup, no network. Expected to be the fastest of the five scenarios.",
28
+
"p50_us": "TODO",
29
+
"p95_us": "TODO",
30
+
"p99_us": "TODO",
31
+
"ips": "TODO"
32
+
},
33
+
"mTLS handshake (test CA)": {
34
+
"_comment_d3": "Phase D-3 scenario. Per-handshake cost on a raw :ssl acceptor; cert/key chain generated in-memory via :public_key.pkix_test_data/1 (the committed Phase B fixture in test/fixtures/mtls ships only *.crt -- *.key is gitignored, so the chain is rebuilt at bench startup with the same verify_peer + test-CA shape). Each iteration :ssl.connect/4 with the client-side config + immediate :ssl.close/1. Bounds the connection-spike SLO for Phase E rollout; amortised per-request cost is bracketed by this + the 'exact route allow (proxy 200)' scenario.",
35
+
"p50_us": "TODO",
36
+
"p95_us": "TODO",
37
+
"p99_us": "TODO",
38
+
"ips": "TODO"
25
39
}
26
40
},
27
41
"tolerance": {
28
-
"_comment": "Multiplicative regression tolerance applied to each percentile. 1.20 = a PR is allowed to be up to 20% slower than baseline before the gate fails. Phase D-4 will revisit these once we have signal on intra-run variance.",
42
+
"_comment": "Multiplicative regression tolerance applied to each percentile. 1.20 = a PR is allowed to be up to 20% slower than baseline before the gate fails. Phase D-4 will revisit these once we have signal on intra-run variance. D-3 scenarios use the same tolerances pending D-4 calibration; the mTLS handshake scenario is likely the noisiest (TLS RNG draws + cert chain validation timing) and may need a looser p99 cap once we see the spread.",
0 commit comments