Skip to content

Commit da84d75

Browse files
hyperpolymathclaude
hyperpolymath
and
claude
authored
perf(d-2): wire in-process loopback backend so proxy-200 scenario measures real dial-and-read (#14)
The Phase D-1 scaffold (PR #12) dialled http://127.0.0.1:1 on the allow path so the gateway returned 502 quickly. That isolated the in-gateway cost up to the proxy dial, but it did not measure the cost the gateway actually pays in production: the Req-based outbound HTTP request, the response read, and the forwarding back to the caller. A baseline collected against that scaffold would have been useless for catching proxy-path regressions or for sizing the Phase E rollout SLA. Phase D-2 (this change, hyperpolymath/standards#99 sub-deliverable; runbook checklist item 1.1 in boj-server's docs/integration/hcg-tier2-rollout-runbook.md) replaces the dial-port-1 hack with an in-process Plug.Cowboy listener that responds 200 with a tiny JSON body. The pattern mirrors the MockBoj backend already used by test/e2e_boj_integration_test.exs (just on port 19_877 instead of 19_876 so the two can never collide). The scenario is renamed from "exact route allow (proxy short-circuits)" to "exact route allow (proxy 200)" in both the harness and bench/baseline.json so the comparator's per-scenario diff stays correctly keyed. bench/baseline.json `_status` stays "scaffold-placeholder" because D-2 does not collect real numbers — D-4 does, via the `perf: rebaseline` ritual in docs/perf-contract.md. The CI gate remains non-blocking until that flip. With D-2 landed, the prerequisites for D-4 baseline collection are: (a) the harness now exercises a production-shaped allow path, (b) the scenario names are stable, (c) D-3 mTLS-handshake and trust-header rewrite scenarios are additive and won't disturb the existing three. Unblocks: standards#99 D-4 (real baseline collection + gate arming) and, downstream, the Phase E rollout-prerequisite checklist in boj-server's hcg-tier2-rollout-runbook.md section 1.1. Refs hyperpolymath/standards#91 Refs hyperpolymath/standards#99 Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 029fe35 commit da84d75

3 files changed

Lines changed: 130 additions & 70 deletions

File tree

‎bench/baseline.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"_comment": "Phase D scaffold placeholder baseline (standards#99). Numbers are TODO. Real collection in Phase D-4 — see docs/perf-contract.md §Baseline collection. DO NOT trust these numbers; the perf-regression CI gate is intentionally non-blocking until this file is populated by a baseline-collection PR.",
2+
"_comment": "Phase D scaffold-placeholder baseline (standards#99). Numbers are TODO. Phase D-1 (PR #12) wired the Benchee harness + this file + the CI gate; Phase D-2 (this PR) added a real in-process loopback backend so the 'exact route allow (proxy 200)' scenario measures real dial-and-read cost instead of port-1 connection-refused. Phase D-4 collects real numbers via the `perf: rebaseline` ritual in docs/perf-contract.md and flips `_status` to 'active' to arm the CI gate. DO NOT trust these numbers; the perf-regression CI gate is intentionally non-blocking until that flip.",
33
"_schema_version": "0.1.0-scaffold",
44
"_generated_at": "TODO(Phase D-4): replace with ISO8601 timestamp from collection run",
55
"_generated_by": "TODO(Phase D-4): replace with `bench/gateway_latency.exs` git SHA + machine fingerprint",
@@ -17,7 +17,7 @@
1717
"p99_us": "TODO",
1818
"ips": "TODO"
1919
},
20-
"exact route allow (proxy short-circuits)": {
20+
"exact route allow (proxy 200)": {
2121
"p50_us": "TODO",
2222
"p95_us": "TODO",
2323
"p99_us": "TODO",

‎bench/gateway_latency.exs‎

Lines changed: 99 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,22 @@
11
# SPDX-License-Identifier: PMPL-1.0-or-later
22
# Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
33
#
4-
# bench/gateway_latency.exs — Phase D benchmark harness scaffold (standards#99)
4+
# bench/gateway_latency.exs — Phase D benchmark harness (standards#99)
55
#
6-
# This file is the SKELETON for the Phase D published-latency contract. The
7-
# real measurements (collection, baseline numbers, regression thresholds) are
8-
# multi-session work and are deliberately NOT included here — see
9-
# docs/perf-contract.md and bench/baseline.json for the placeholder constants
10-
# and TODO markers that the follow-up sessions must replace.
6+
# Phase D-1 (standards#99 scaffold, http-capability-gateway#12) wired the
7+
# Benchee-driven gateway pipeline measurement and the JSON output the
8+
# perf-regression CI gate diffs against bench/baseline.json. The "exact
9+
# route allow" scenario in D-1 dialled `http://127.0.0.1:1` (refused
10+
# immediately) so the timing only captured the in-gateway cost up to the
11+
# proxy dial, not the dial-and-read cost.
12+
#
13+
# Phase D-2 (this file): adds an in-process Plug.Cowboy loopback backend
14+
# that responds 200 cheaply, so the allow scenario measures the real
15+
# end-to-end cost the gateway pays in production — header rewrite +
16+
# Req-based proxy + response forwarding. This is the cost surface the
17+
# Phase E rollout SLA (p99 latency at production endpoints) is measured
18+
# against; without it the baseline numbers a Phase D-4 collection would
19+
# produce are not comparable to production reality.
1120
#
1221
# Run locally:
1322
#
@@ -19,41 +28,75 @@
1928
#
2029
# CI invocation lives in .github/workflows/perf-regression.yml.
2130
#
22-
# ── Scenarios (Phase D scaffold scope) ───────────────────────────────────────
31+
# ── Scenarios (Phase D-2 scope) ──────────────────────────────────────────────
2332
#
24-
# 1. health endpoint — fastest path, no policy lookup, no proxy
25-
# 2. policy deny — 405 fast-path (unknown verb, policy table hit, no proxy)
26-
# 3. exact-route allow — O(1) policy hit; backend stubbed (no real network)
33+
# 1. health endpoint — fastest path, no policy lookup, no proxy
34+
# 2. policy deny (405 fast-path) — unknown verb, policy table hit, no proxy
35+
# 3. exact route allow (proxy 200) — O(1) policy hit; proxies to the
36+
# in-process loopback backend which
37+
# responds 200 with a tiny JSON body
2738
#
2839
# Each scenario is reported under p50 / p95 / p99 latency, matching the SLO
2940
# names in docs/perf-contract.md. Throughput (req/s) is reported as a
3041
# secondary signal; the SLO is latency, not throughput.
3142
#
32-
# ── What this scaffold deliberately does NOT do ──────────────────────────────
43+
# ── What this harness still deliberately does NOT do ─────────────────────────
3344
#
34-
# • Talk to a real backend (Phase D-2 will add the loopback backend fixture)
3545
# • Exercise the mTLS handshake path (Phase D-3; reuses Phase B fixture)
36-
# • Capture the X-Trust-Level rewrite cost (Phase D-3)
46+
# • Capture the X-Trust-Level rewrite cost as a dedicated scenario
47+
# (Phase D-3 — the cost is currently folded into the proxy-200 scenario)
3748
# • Compare against a populated baseline.json (Phase D-4 collects baseline)
3849
# • Run distributed (single-node only; estate concurrency-pool guard in #122)
3950
#
40-
# Those are all separate, sequenced post-merge tasks under the single-lane
41-
# HCG channel (standards#91). This PR delivers the scaffold only.
51+
# Those are sequenced post-D-2 tasks under the single-lane HCG channel
52+
# (standards#91).
4253

43-
# `mix run` loads all deps onto the code path, including Benchee
44-
# (declared as :dev/:test dep in mix.exs).
54+
# `mix run` loads all deps onto the code path, including Benchee and
55+
# Plug.Cowboy (both declared in mix.exs).
4556
#
4657
# Boot the OTP supervision tree so RateLimiter ETS, K9Contract, etc. are
4758
# initialised — Gateway.call/2 depends on those processes existing.
4859
{:ok, _} = Application.ensure_all_started(:http_capability_gateway)
4960

5061
alias HttpCapabilityGateway.{Gateway, PolicyCompiler}
5162

63+
# ── Loopback backend fixture (Phase D-2) ────────────────────────────────────
64+
#
65+
# In-process Plug.Cowboy listener on a fixed loopback port. Mirrors the
66+
# pattern in test/e2e_boj_integration_test.exs (MockBoj) so the bench and
67+
# the seam test share the same backend shape. The body is intentionally
68+
# tiny: real BoJ responses are larger, but the bench is measuring the
69+
# gateway's overhead, not the backend's response cost. A larger body
70+
# would add noise from socket-write timings that aren't gateway-attributable.
71+
#
72+
# Port choice: 19_877 — one above the E2E test's 19_876 so the two can
73+
# coexist if a future runner ever interleaves them. The CI runner is
74+
# clean per job; conflict is not expected.
75+
defmodule HttpCapabilityGateway.Bench.LoopbackBackend do
76+
use Plug.Router
77+
78+
plug(:match)
79+
plug(:dispatch)
80+
81+
match _ do
82+
conn
83+
|> Plug.Conn.put_resp_content_type("application/json")
84+
|> Plug.Conn.send_resp(200, ~s({"ok":true}))
85+
end
86+
end
87+
88+
backend_port = 19_877
89+
backend_url = "http://127.0.0.1:#{backend_port}"
90+
91+
{:ok, _backend_pid} =
92+
Plug.Cowboy.http(HttpCapabilityGateway.Bench.LoopbackBackend, [], port: backend_port)
93+
5294
# ── Test fixture: a small, deterministic policy ──────────────────────────────
5395
#
54-
# Real Phase D-2 collection will use a fixture closer to the BoJ production
55-
# policy shape. For the scaffold we just need a policy that exercises the
56-
# three scenarios above.
96+
# Real Phase D-4 collection will use a fixture closer to the BoJ production
97+
# policy shape (covering all routes in config/gateway-policy-boj-example.yaml
98+
# in the boj-server repo). For D-1/D-2 we just need a policy that exercises
99+
# the three scenarios above.
57100
policy = %{
58101
"dsl_version" => "1",
59102
"governance" => %{
@@ -62,12 +105,7 @@ policy = %{
62105
%{
63106
"path" => "/api/bench",
64107
"verbs" => ["GET"],
65-
# NB: backend URL is never dialled in scaffold mode; the deny/health
66-
# scenarios short-circuit before proxy. The allow scenario uses a
67-
# local loopback that intentionally refuses — the gateway returns
68-
# 502 quickly, which still captures the in-gateway cost we care about.
69-
# TODO(Phase D-2): wire a real loopback fixture (Bandit / cowboy_test).
70-
"backend" => "http://127.0.0.1:1",
108+
"backend" => backend_url,
71109
"exposure" => "public"
72110
}
73111
]
@@ -78,6 +116,7 @@ policy = %{
78116
{:ok, table} = PolicyCompiler.compile(policy, delete_old: false)
79117
Application.put_env(:http_capability_gateway, :policy_table, table)
80118
Application.put_env(:http_capability_gateway, :stealth_profiles, %{})
119+
Application.put_env(:http_capability_gateway, :backend_url, backend_url)
81120

82121
# Generous rate limits so the harness is not measuring 429s.
83122
Application.put_env(:http_capability_gateway, :rate_limits, %{
@@ -94,30 +133,37 @@ import Plug.Test, only: [conn: 2]
94133

95134
# ── Benchee scenarios ────────────────────────────────────────────────────────
96135

97-
Benchee.run(
98-
%{
99-
"health endpoint" => fn ->
100-
conn(:get, "/health") |> Gateway.call([])
101-
end,
102-
"policy deny (405 fast-path)" => fn ->
103-
c = conn(:get, "/api/bench")
104-
%{c | method: "PROPFIND"} |> Gateway.call([])
105-
end,
106-
"exact route allow (proxy short-circuits)" => fn ->
107-
conn(:get, "/api/bench") |> Gateway.call([])
108-
end
109-
},
110-
# Phase D-1 (this scaffold): keep timings short so the workflow stays
111-
# under the 6-minute estate CI budget. Phase D-2 will widen warmup/time.
112-
warmup: 1,
113-
time: 2,
114-
memory_time: 0,
115-
percentiles: [50, 95, 99],
116-
# JSON output for the CI regression gate to diff against baseline.json.
117-
formatters: [
118-
Benchee.Formatters.Console,
119-
{Benchee.Formatters.JSON, file: "bench/results.json"}
120-
],
121-
# Stable, reproducible names that match docs/perf-contract.md.
122-
print: [benchmarking: true, fast_warning: false, configuration: true]
123-
)
136+
try do
137+
Benchee.run(
138+
%{
139+
"health endpoint" => fn ->
140+
conn(:get, "/health") |> Gateway.call([])
141+
end,
142+
"policy deny (405 fast-path)" => fn ->
143+
c = conn(:get, "/api/bench")
144+
%{c | method: "PROPFIND"} |> Gateway.call([])
145+
end,
146+
"exact route allow (proxy 200)" => fn ->
147+
conn(:get, "/api/bench") |> Gateway.call([])
148+
end
149+
},
150+
# Phase D-1/D-2: keep timings short so the workflow stays under the
151+
# 6-minute estate CI budget. Phase D-4 may widen warmup/time once the
152+
# rebaseline ritual is in place.
153+
warmup: 1,
154+
time: 2,
155+
memory_time: 0,
156+
percentiles: [50, 95, 99],
157+
# JSON output for the CI regression gate to diff against baseline.json.
158+
formatters: [
159+
Benchee.Formatters.Console,
160+
{Benchee.Formatters.JSON, file: "bench/results.json"}
161+
],
162+
# Stable, reproducible names that match docs/perf-contract.md.
163+
print: [benchmarking: true, fast_warning: false, configuration: true]
164+
)
165+
after
166+
# Stop the loopback backend so the script exits cleanly. Plug.Cowboy
167+
# registers the listener under the plug module's name by default.
168+
_ = Plug.Cowboy.shutdown(HttpCapabilityGateway.Bench.LoopbackBackend.HTTP)
169+
end

‎docs/perf-contract.md‎

Lines changed: 29 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -3,22 +3,29 @@
33

44
# Performance Contract
55

6-
> **Status: scaffold (Phase D-1).** This document names the SLOs the
6+
> **Status: scaffold (Phase D-2).** This document names the SLOs the
77
> gateway will publish and the regression-alert gate will enforce. The
88
> concrete numbers are deliberately left as `TODO` — Phase D-4 of the
99
> single-lane HCG channel (standards#91) collects the baseline and fills
10-
> them in. The CI gate is non-blocking until that point.
10+
> them in. The CI gate is non-blocking until that point. Phase D-2
11+
> (loopback backend fixture) is the last harness-shape change before
12+
> D-4 collection; further D-3 work adds the mTLS-handshake and trust-
13+
> header rewrite scenarios but does not change the existing three.
1114
1215
Tracks: `Refs hyperpolymath/standards#99` (Phase D of `standards#91`).
1316

1417
## Scope of this contract
1518

16-
This contract covers in-gateway latency only — the cost of the gateway's
19+
This contract covers in-gateway latency — the cost of the gateway's
1720
own pipeline (trust resolution, policy lookup, rate limiter, circuit
18-
breaker, header rewrite) up to the point where it would dial the
19-
backend. Backend latency, network RTT, and TLS handshake amortisation are
20-
out of scope here; mTLS handshake cost is tracked separately and lands in
21-
Phase D-3.
21+
breaker, header rewrite). The `exact route allow (proxy 200)` scenario
22+
also includes the dial-and-read cost against an in-process loopback
23+
backend (added in Phase D-2), since the production gateway always pays
24+
that cost on an allow path; the loopback strips out real-network RTT and
25+
real-backend processing time so what remains is the gateway-attributable
26+
proxy overhead. Real production backend latency, real-network RTT, and
27+
mTLS handshake amortisation are out of scope here; mTLS handshake cost
28+
is tracked separately and lands in Phase D-3.
2229

2330
The three named scenarios match the three Benchee scenarios in
2431
`bench/gateway_latency.exs`:
@@ -27,7 +34,7 @@ The three named scenarios match the three Benchee scenarios in
2734
|-------------------------------------------|-----------------------------------------------------------------------------------|
2835
| `health endpoint` | Cheapest path. No policy lookup, no proxy. Floor of any plug-pipeline overhead. |
2936
| `policy deny (405 fast-path)` | Verb-rejection path. Policy table hit, no proxy. Cost of the deny short-circuit. |
30-
| `exact route allow (proxy short-circuits)`| Allow path with backend dial that intentionally refuses, isolating in-gateway cost.|
37+
| `exact route allow (proxy 200)` | Allow path that proxies to an in-process loopback backend returning 200. Captures the real dial-and-read cost the gateway pays in production. |
3138

3239
## Published latency SLOs
3340

@@ -43,13 +50,13 @@ Reported per scenario, per CI run:
4350
|-------------------------------------------|------------|------------|------------|
4451
| `health endpoint` | TODO | TODO | TODO |
4552
| `policy deny (405 fast-path)` | TODO | TODO | TODO |
46-
| `exact route allow (proxy short-circuits)`| TODO | TODO | TODO |
53+
| `exact route allow (proxy 200)` | TODO | TODO | TODO |
4754

4855
Units: microseconds (µs). Hardware reference: `ubuntu-latest` GitHub
4956
runners — the CI environment IS the published reference, deliberately
5057
chosen because it is the environment every reviewer can reproduce
51-
without local hardware variance. Phase D-2 will revisit whether a
52-
dedicated runner is needed once we see the spread.
58+
without local hardware variance. Phase D-4 (baseline collection) will
59+
revisit whether a dedicated runner is needed once we see the spread.
5360

5461
## Regression-alert tolerance
5562

@@ -82,17 +89,24 @@ Updating the baseline is a deliberate act:
8289
4. Reviewer approves the new numbers (or rejects if the regression is
8390
real and unjustified). Never silently rebaseline in an unrelated PR.
8491

85-
## Out of scope for Phase D-1
92+
## Out of scope for Phase D-2
8693

87-
The scaffold deliberately does NOT include:
94+
The scaffold deliberately does NOT yet include:
8895

89-
- Real loopback backend fixture (Phase D-2 — measures the dial-and-read cost).
9096
- mTLS handshake amortisation (Phase D-3 — reuses the Phase B real-CA fixture).
9197
- Trust-header rewrite cost on the `Proxy.build_backend_headers/1` path
92-
introduced in Phase C (Phase D-3).
98+
introduced in Phase C, as a dedicated scenario (Phase D-3 — the cost is
99+
currently folded into `exact route allow (proxy 200)`).
93100
- Real baseline numbers and rebaselined tolerances (Phase D-4).
94101
- Dashboard publication of historical numbers (Phase E — `standards#100`).
95102

103+
D-2 (this iteration) landed the in-process `Plug.Cowboy` loopback backend
104+
in `bench/gateway_latency.exs` and renamed the `exact route allow`
105+
scenario from `(proxy short-circuits)` (D-1: dialled `:1`, refused) to
106+
`(proxy 200)`. `bench/baseline.json` carries the renamed scenario key;
107+
the `_status` remains `scaffold-placeholder` because the rename does not
108+
collect a real baseline — D-4 does.
109+
96110
These are sequenced post-merge tasks under the single-lane HCG channel;
97111
see `standards#91` for the order-of-operations and `standards#100` for
98112
Phase E.

0 commit comments

Comments
 (0)