perf(d-3): add explicit amortised-handshake Benchee scenario (mTLS N-over-keep-alive)

Closes the bracket scenarios 3 + 5 left loose: per-iteration cost is
(handshake + N * request) / N which approximates the per-request cost
the Phase E rollout sees once the handshake is amortised across a
kept-alive pool.

Adds:
  - bench/gateway_latency.exs: scenario 6
    "mTLS amortised (test CA, N requests over kept-alive)" plus a
    dedicated kept-alive acceptor on port 19_879 running an echo loop
    (separate listener from scenario 5 so its close-on-handshake shape
    doesn't kill the kept-alive connection). N=16; payload is a tiny
    8-byte ASCII frame.
  - bench/baseline.json: new scenario key with TODO placeholders;
    _schema_version bumped to 0.3.0-scaffold.
  - docs/perf-contract.md: scenarios table + targets table now list six
    scenarios; the previously deferred amortised follow-up is moved
    out of "Out of scope" into in-scope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: hyperpolymath

bench/baseline.json

@@ -1,6 +1,6 @@
 {
-  "_comment": "Phase D scaffold-placeholder baseline (standards#99). Numbers are TODO. Phase D-1 (PR #12) wired the Benchee harness + this file + the CI gate; Phase D-2 (PR #14) added a real in-process loopback backend so the 'exact route allow (proxy 200)' scenario measures real dial-and-read cost instead of port-1 connection-refused; Phase D-3 (this update) added two scenarios -- 'trust-header rewrite (Proxy.build_backend_headers)' isolating the Phase A contract-header construction cost, and 'mTLS handshake (test CA)' isolating per-handshake cost using the Phase B real-CA fixture. Phase D-4 collects real numbers via the `perf: rebaseline` ritual in docs/perf-contract.md and flips `_status` to 'active' to arm the CI gate. DO NOT trust these numbers; the perf-regression CI gate is intentionally non-blocking until that flip.",
-  "_schema_version": "0.2.0-scaffold",
+  "_comment": "Phase D scaffold-placeholder baseline (standards#99). Numbers are TODO. Phase D-1 (PR #12) wired the Benchee harness + this file + the CI gate; Phase D-2 (PR #14) added a real in-process loopback backend so the 'exact route allow (proxy 200)' scenario measures real dial-and-read cost instead of port-1 connection-refused; Phase D-3 (PR #22) added two scenarios -- 'trust-header rewrite (Proxy.build_backend_headers)' isolating the Phase A contract-header construction cost, and 'mTLS handshake (test CA)' isolating per-handshake cost using the Phase B real-CA fixture. Phase D-3 follow-up (this update) adds 'mTLS amortised (test CA, N requests over kept-alive)' which holds one connection open and does N=16 send/recv round-trips per iteration, closing the bracket that scenarios 3 + 5 left loose. Phase D-4 collects real numbers via the `perf: rebaseline` ritual in docs/perf-contract.md and flips `_status` to 'active' to arm the CI gate. DO NOT trust these numbers; the perf-regression CI gate is intentionally non-blocking until that flip.",
+  "_schema_version": "0.3.0-scaffold",
   "_generated_at": "TODO(Phase D-4): replace with ISO8601 timestamp from collection run",
   "_generated_by": "TODO(Phase D-4): replace with `bench/gateway_latency.exs` git SHA + machine fingerprint",
   "_status": "scaffold-placeholder",
@@ -36,10 +36,17 @@
       "p95_us": "TODO",
       "p99_us": "TODO",
       "ips": "TODO"
+    },
+    "mTLS amortised (test CA, N requests over kept-alive)": {
+      "_comment_d3": "Phase D-3 follow-up scenario. One :ssl.connect/4 followed by N=16 send/recv round-trips on the open connection, then :ssl.close/1. Per-iteration cost is (handshake + N * request) / N, approximating the per-request cost the Phase E rollout sees once the handshake is amortised across a kept-alive pool. Closes the bracket scenarios 3 ('exact route allow (proxy 200)') and 5 ('mTLS handshake (test CA)') left loose. Uses a dedicated kept-alive acceptor on port 19_879 with an echo loop so the connection stays open across the N round-trips; payload is a tiny 8-byte frame so per-request cost is dominated by connection round-trip, not payload write/read.",
+      "p50_us": "TODO",
+      "p95_us": "TODO",
+      "p99_us": "TODO",
+      "ips": "TODO"
     }
   },
   "tolerance": {
-    "_comment": "Multiplicative regression tolerance applied to each percentile. 1.20 = a PR is allowed to be up to 20% slower than baseline before the gate fails. Phase D-4 will revisit these once we have signal on intra-run variance. D-3 scenarios use the same tolerances pending D-4 calibration; the mTLS handshake scenario is likely the noisiest (TLS RNG draws + cert chain validation timing) and may need a looser p99 cap once we see the spread.",
+    "_comment": "Multiplicative regression tolerance applied to each percentile. 1.20 = a PR is allowed to be up to 20% slower than baseline before the gate fails. Phase D-4 will revisit these once we have signal on intra-run variance. D-3 scenarios use the same tolerances pending D-4 calibration; the mTLS handshake scenario is likely the noisiest (TLS RNG draws + cert chain validation timing) and may need a looser p99 cap once we see the spread. The amortised scenario is expected to sit close to N * proxy-200 in per-iteration terms (since the handshake is paid once per 16 round-trips); its variance should be tighter than scenario 5's but looser than scenario 3's.",
     "p50_max_ratio": 1.20,
     "p95_max_ratio": 1.30,
     "p99_max_ratio": 1.50

bench/gateway_latency.exs

@@ -66,6 +66,15 @@
 #                                                   client-internal cert against a raw
 #                                                   :ssl acceptor; isolates per-handshake
 #                                                   cost from the proxy hot-path
+#   6. mTLS amortised (test CA, N requests over kept-alive)
+#                                                 — D-3 follow-up — one :ssl.connect/4
+#                                                   followed by N tiny send/recv round-trips
+#                                                   on the open connection, then close.
+#                                                   Approximates the per-request cost the
+#                                                   Phase E rollout sees once the handshake
+#                                                   is amortised across a kept-alive pool.
+#                                                   Closes the bracket scenario 3 + scenario
+#                                                   5 left loose.
 #
 # Each scenario is reported under p50 / p95 / p99 latency, matching the SLO
 # names in docs/perf-contract.md. Throughput (req/s) is reported as a
@@ -76,10 +85,7 @@
 #   • Compare against a populated baseline.json (Phase D-4 collects baseline
 #     numbers on a CI-equivalent target and flips `_status` to `active`)
 #   • Run distributed (single-node only; estate concurrency-pool guard in #122)
-#   • Capture amortised cost of N requests over one kept-alive TLS connection
-#     as a dedicated scenario (the per-handshake cost in scenario 5 plus the
-#     proxy-200 cost in scenario 3 already bracket it; if D-4 numbers show
-#     the bracket is loose, a follow-up adds an explicit amortised scenario)
+# (Scenario 6 in D-3 closes the previously deferred amortised follow-up.)
 #
 # Those are sequenced post-D-3 tasks under the single-lane HCG channel
 # (standards#91).
@@ -284,6 +290,86 @@ acceptor_pid =
     accept_loop.(accept_loop)
   end)
 
+# ── mTLS amortised acceptor fixture (Phase D-3, scenario 6) ──────────────────
+#
+# Second :ssl listener dedicated to the amortised scenario. Each accepted
+# connection is held open and the server echoes back every frame the
+# client sends; the connection terminates when the client closes its
+# side. The amortised bench scenario opens ONE connection, does N
+# send/recv round-trips, then closes, so this acceptor sees one
+# handshake + N echo cycles + one close per Benchee iteration.
+#
+# Separate listener (port 19_879) instead of reusing 19_878: scenario 5's
+# acceptor is shaped to close immediately after handshake, which would
+# kill the kept-alive connection scenario 6 needs. Keeping the two
+# fixtures independent also means D-4 can run them back-to-back without
+# the scenario-5 acceptor's close-on-handshake racing scenario 6's
+# send/recv loop.
+#
+# N (kept_alive_request_count below): chosen as 16 — a small bounded
+# constant in the same order of magnitude as typical HTTP/1.1 keep-alive
+# pool reuse counts before pool rotation, large enough that
+# (handshake + N * request) / N visibly differs from the per-handshake
+# cost in scenario 5 (so the bracket scenarios 5 + 3 left loose is
+# visibly tightened), small enough that one iteration completes well
+# within the Benchee per-iteration budget.
+#
+# Payload (kept_alive_payload below): a tiny 8-byte ASCII frame. The
+# point is to measure per-request *connection* cost (round-trip on an
+# established TLS session), not bulk-data throughput; the payload is
+# kept deliberately small so socket-write timings don't dominate.
+kept_alive_request_count = 16
+kept_alive_payload = "bench-d3"
+
+{:ok, kept_alive_listen_socket} = :ssl.listen(19_879, tls_listen_opts)
+
+kept_alive_acceptor_pid =
+  spawn_link(fn ->
+    accept_loop = fn loop ->
+      case :ssl.transport_accept(kept_alive_listen_socket, 5_000) do
+        {:ok, transport_socket} ->
+          case :ssl.handshake(transport_socket, 5_000) do
+            {:ok, tls_socket} ->
+              # Echo loop: read whatever the client sends, send it back,
+              # continue until the client closes. The bench client does
+              # exactly N send/recv pairs and then closes, so this loop
+              # exits cleanly on {:error, :closed} once :ssl.close/1 has
+              # been called on the client side.
+              echo_loop = fn echo ->
+                case :ssl.recv(tls_socket, byte_size(kept_alive_payload), 5_000) do
+                  {:ok, data} ->
+                    case :ssl.send(tls_socket, data) do
+                      :ok -> echo.(echo)
+                      {:error, _reason} -> :ssl.close(tls_socket)
+                    end
+
+                  {:error, :closed} ->
+                    :ok
+
+                  {:error, _reason} ->
+                    :ssl.close(tls_socket)
+                end
+              end
+
+              echo_loop.(echo_loop)
+
+            {:error, _reason} ->
+              :ok
+          end
+
+          loop.(loop)
+
+        {:error, :timeout} ->
+          loop.(loop)
+
+        {:error, :closed} ->
+          :ok
+      end
+    end
+
+    accept_loop.(accept_loop)
+  end)
+
 # ── Benchee scenarios ────────────────────────────────────────────────────────
 
 try do
@@ -304,6 +390,22 @@ try do
       end,
       "mTLS handshake (test CA)" => fn ->
         {:ok, sock} = :ssl.connect(~c"127.0.0.1", tls_port, tls_client_opts, 5_000)
+        :ssl.close(sock)
+      end,
+      "mTLS amortised (test CA, N requests over kept-alive)" => fn ->
+        # ONE handshake, then N tiny send/recv round-trips on the open
+        # connection, then close. Per-iteration cost is
+        # (handshake + N * request) / N which approximates the per-request
+        # cost in a kept-alive setting. N is the module-level constant
+        # `kept_alive_request_count` (see the amortised acceptor fixture
+        # above for why 16 was chosen).
+        {:ok, sock} = :ssl.connect(~c"127.0.0.1", 19_879, tls_client_opts, 5_000)
+
+        Enum.each(1..kept_alive_request_count, fn _ ->
+          :ok = :ssl.send(sock, kept_alive_payload)
+          {:ok, _echoed} = :ssl.recv(sock, byte_size(kept_alive_payload), 5_000)
+        end)
+
         :ssl.close(sock)
       end
     },
@@ -333,4 +435,9 @@ after
   Process.unlink(acceptor_pid)
   :ssl.close(listen_socket)
   Process.exit(acceptor_pid, :shutdown)
+
+  # Same teardown shape for the amortised acceptor (Phase D-3 scenario 6).
+  Process.unlink(kept_alive_acceptor_pid)
+  :ssl.close(kept_alive_listen_socket)
+  Process.exit(kept_alive_acceptor_pid, :shutdown)
 end

docs/perf-contract.md

@@ -9,8 +9,10 @@
 > single-lane HCG channel (standards#91) collects the baseline and fills
 > them in. The CI gate is non-blocking until that point. Phase D-3 (this
 > revision) is the last harness-shape change before D-4 collection: it
-> adds the `trust-header rewrite` and `mTLS handshake` scenarios so the
-> baseline attributes those costs independently of the proxy-200 path.
+> adds the `trust-header rewrite`, `mTLS handshake`, and `mTLS amortised`
+> scenarios so the baseline attributes header-rewrite cost, per-handshake
+> cost, and per-request kept-alive cost independently of the proxy-200
+> path.
 
 Tracks: `Refs hyperpolymath/standards#99` (Phase D of `standards#91`).
 
@@ -24,12 +26,13 @@ backend (added in Phase D-2), since the production gateway always pays
 that cost on an allow path; the loopback strips out real-network RTT and
 real-backend processing time so what remains is the gateway-attributable
 proxy overhead. The `mTLS handshake` scenario (added in Phase D-3)
-covers per-handshake transport cost in isolation. Real production
-backend latency, real-network RTT, and amortised mTLS cost across a
-kept-alive connection remain out of scope — the latter is bracketed by
-the `proxy 200` + `mTLS handshake` scenarios pending D-4 signal.
+covers per-handshake transport cost in isolation, and the
+`mTLS amortised` scenario (added in the Phase D-3 follow-up) covers the
+per-request cost when N=16 requests share one kept-alive TLS connection,
+closing the bracket that `proxy 200` + `mTLS handshake` left loose. Real
+production backend latency and real-network RTT remain out of scope.
 
-The five named scenarios match the five Benchee scenarios in
+The six named scenarios match the six Benchee scenarios in
 `bench/gateway_latency.exs`:
 
 | Scenario                                                  | What it measures                                                                  |
@@ -39,6 +42,7 @@ The five named scenarios match the five Benchee scenarios in
 | `exact route allow (proxy 200)`                           | Allow path that proxies to an in-process loopback backend returning 200. Captures the real dial-and-read cost the gateway pays in production. |
 | `trust-header rewrite (Proxy.build_backend_headers)`      | Direct call to the `Proxy.__benchmark_build_backend_headers__/1` benchmark seam on a pre-built conn (`assigns[:trust_level] = :authenticated`, `assigns[:request_id]` set). Isolates the Phase A contract-header construction (X-Trust-Level / X-Request-ID / X-Forwarded-*) from policy lookup and network I/O. |
 | `mTLS handshake (test CA)`                                | `:ssl.connect/4` against a raw `:ssl` acceptor + immediate close. Test CA chain is generated in-memory at bench startup via `:public_key.pkix_test_data/1` — the committed Phase B fixture ships only `*.crt` files (matching `*.key` is gitignored), so the chain is rebuilt with the same `verify_peer` + test-CA shape rather than reusing the on-disk material. Per-handshake cost (cert chain validation + key exchange) in isolation; no HTTP pipeline. Bounds the connection-spike SLO for Phase E rollout. |
+| `mTLS amortised (test CA, N requests over kept-alive)`    | One `:ssl.connect/4` followed by N=16 tiny send/recv round-trips on the open connection, then `:ssl.close/1`. Uses a dedicated kept-alive acceptor (separate listener from the handshake-only scenario) with an echo loop, so the connection stays open across the N round-trips. Per-iteration cost is `(handshake + N * request) / N`, which approximates the per-request cost the Phase E rollout sees once the handshake is amortised across a kept-alive pool. Closes the bracket scenarios 3 + 5 left loose. |
 
 ## Published latency SLOs
 
@@ -57,6 +61,7 @@ Reported per scenario, per CI run:
 | `exact route allow (proxy 200)`                           | TODO       | TODO       | TODO       |
 | `trust-header rewrite (Proxy.build_backend_headers)`      | TODO       | TODO       | TODO       |
 | `mTLS handshake (test CA)`                                | TODO       | TODO       | TODO       |
+| `mTLS amortised (test CA, N requests over kept-alive)`    | TODO       | TODO       | TODO       |
 
 Units: microseconds (µs). Hardware reference: `ubuntu-latest` GitHub
 runners — the CI environment IS the published reference, deliberately
@@ -103,15 +108,9 @@ The scaffold deliberately does NOT yet include:
   `perf: rebaseline` ritual described above produces these on a
   CI-equivalent target and flips `bench/baseline.json` `_status` to
   `active`).
-- A dedicated "mTLS amortised" scenario covering the per-request cost
-  when N requests share one kept-alive TLS connection. The combined
-  bracket of `mTLS handshake (test CA)` (one-off cost) and
-  `exact route allow (proxy 200)` (per-request cost with the handshake
-  already paid) bounds it; if D-4 numbers show that bracket is too
-  loose to be useful, a follow-up adds the explicit amortised scenario.
 - Dashboard publication of historical numbers (Phase E — `standards#100`).
 
-D-3 (this iteration) added two new scenarios to
+D-3 (this iteration) added three new scenarios to
 `bench/gateway_latency.exs`:
 
 - `trust-header rewrite (Proxy.build_backend_headers)` — direct call
@@ -126,11 +125,20 @@ D-3 (this iteration) added two new scenarios to
   same `verify_peer` + test-CA shape rather than reusing the on-disk
   material. Each iteration is one fresh handshake closed immediately;
   isolates per-handshake cost from the proxy hot-path.
-
-`bench/baseline.json` gains the two new scenario keys (both `TODO`,
+- `mTLS amortised (test CA, N requests over kept-alive)` — dedicated
+  kept-alive acceptor (separate listener) running an echo loop. Each
+  Benchee iteration opens one connection, does N=16 tiny send/recv
+  round-trips on the held-open connection, and then closes. Per-iteration
+  cost is `(handshake + N * request) / N`, which approximates the
+  per-request cost in a kept-alive setting. This was previously deferred
+  as out-of-scope and bracketed by `mTLS handshake` + `proxy 200`; the
+  follow-up adds it as an explicit scenario so D-4 baseline collection
+  captures it directly rather than via the bracket.
+
+`bench/baseline.json` gains the three new scenario keys (all `TODO`,
 `_status` remains `scaffold-placeholder`) and bumps `_schema_version`
-to `0.2.0-scaffold` to signal the shape change. D-4 will land real
-numbers across all five scenarios in a single dedicated
+to `0.3.0-scaffold` to signal the shape change. D-4 will land real
+numbers across all six scenarios in a single dedicated
 `perf: rebaseline` PR.
 
 D-2 (the previous iteration) landed the in-process `Plug.Cowboy`