feat: extend chaos testing to proxy/record mode

Pre-flight chaos (drop/disconnect) prevents upstream contact.
Post-response chaos (malformed) corrupts relay after recording.
SSE bypass tracked via aimock_chaos_bypassed_total metric.
Explicit source label (fixture/proxy) on all chaos metrics.
All handlers migrated to new applyChaos signature.

Author: jpr5

CHANGELOG.md

@@ -1,5 +1,11 @@
 # @copilotkit/aimock
 
+## [Unreleased]
+
+### Added
+
+- **Chaos testing in proxy mode** — Pre-flight chaos (drop/disconnect) prevents upstream contact; post-response chaos (malformed) corrupts relay body after recording the real upstream response. SSE bypass tracked via `aimock_chaos_bypassed_total` metric. Explicit `source` label (`fixture`/`proxy`/`internal`) on all chaos Prometheus counters and journal entries.
+
 ## [1.17.0] - 2026-05-04
 
 ### Added

docs/chaos-testing/index.html

@@ -70,7 +70,10 @@ <h2>Failure Modes</h2>
               <td>HTTP 500</td>
               <td>
                 Returns a 500 error with
-                <code>{"error":{"message":"Chaos: request dropped","code":"chaos_drop"}}</code>
+                <code
+                  >{"error":{"message":"Chaos: request
+                  dropped","type":"server_error","code":"chaos_drop"}}</code
+                >
               </td>
             </tr>
             <tr>
@@ -205,7 +208,7 @@ <h2>Per-Request Headers</h2>
     <span class="str">"Content-Type"</span>: <span class="str">"application/json"</span>,
     <span class="str">"x-aimock-chaos-disconnect"</span>: <span class="str">"1.0"</span>,
   },
-  <span class="prop">body</span>: <span class="fn">JSON.stringify</span>({ <span class="prop">model</span>: <span class="str">"gpt-4"</span>, <span class="prop">messages</span>: [...] }),
+  <span class="prop">body</span>: <span class="fn">JSON.stringify</span>({ <span class="prop">model</span>: <span class="str">"gpt-4"</span>, <span class="prop">messages</span>: [{ <span class="prop">role</span>: <span class="str">"user"</span>, <span class="prop">content</span>: <span class="str">"hello"</span> }] }),
 });</code></pre>
         </div>
 
@@ -218,7 +221,7 @@ <h2>CLI Flags</h2>
               <div class="code-block-header">
                 CLI chaos flags <span class="lang-tag">shell</span>
               </div>
-              <pre><code>$ npx -p @copilotkit/aimock llmock --fixtures ./fixtures \
+              <pre><code>$ npx -p @copilotkit/aimock aimock --fixtures ./fixtures \
   --chaos-drop 0.1 \
   --chaos-malformed 0.05 \
   --chaos-disconnect 0.02</code></pre>
@@ -240,6 +243,55 @@ <h2>CLI Flags</h2>
           </div>
         </div>
 
+        <h2>Proxy Mode</h2>
+        <p>
+          When aimock is configured as a record/replay proxy (<code>--record</code>), chaos applies
+          to proxied requests too &mdash; so a staging environment pointed at real upstream APIs
+          still sees the failure modes your tests expect. Chaos is rolled <em>once per request</em>,
+          after fixture matching, with the same headers&nbsp;&gt;&nbsp;fixture&nbsp;&gt;&nbsp;server
+          precedence.
+        </p>
+        <table class="endpoint-table">
+          <thead>
+            <tr>
+              <th>Mode</th>
+              <th>When upstream is contacted</th>
+              <th>What the client sees</th>
+            </tr>
+          </thead>
+          <tbody>
+            <tr>
+              <td><code>drop</code></td>
+              <td>Never &mdash; upstream not contacted</td>
+              <td>HTTP 500 chaos body; upstream is not called</td>
+            </tr>
+            <tr>
+              <td><code>disconnect</code></td>
+              <td>Never &mdash; upstream not contacted</td>
+              <td>Connection destroyed; upstream is not called</td>
+            </tr>
+            <tr>
+              <td><code>malformed</code></td>
+              <td>Called &mdash; post-response</td>
+              <td>
+                Request proxies normally; the upstream response is captured, then the body is
+                replaced with invalid JSON before relay. The recorded fixture (if recording) keeps
+                the real upstream response &mdash; chaos is a live-traffic decoration, not a fixture
+                mutation.
+              </td>
+            </tr>
+          </tbody>
+        </table>
+        <p>
+          <strong>SSE bypass.</strong> If upstream returns
+          <code>Content-Type: text/event-stream</code>, aimock streams chunks to the client
+          progressively. By the time <code>malformed</code> would fire, the bytes are already on the
+          wire &mdash; the chaos action cannot be applied. This bypass is observable via the
+          <code>aimock_chaos_bypassed_total</code> counter (see Prometheus Metrics below) and a
+          warning in the server log, so a configured chaos rate doesn't silently drop to 0% on SSE
+          traffic. Streaming mutation is planned for a future phase.
+        </p>
+
         <h2>Journal Tracking</h2>
         <p>
           When chaos triggers, the journal entry includes a <code>chaosAction</code> field recording
@@ -255,6 +307,7 @@ <h2>Journal Tracking</h2>
   "path": "/v1/chat/completions",
   "response": {
     "status": 500,
+    "source": "fixture",
     "fixture": { "...": "elided for brevity" },
     "chaosAction": "drop"
   }
@@ -269,15 +322,31 @@ <h2>Journal Tracking</h2>
         <h2>Prometheus Metrics</h2>
         <p>
           When metrics are enabled (<code>--metrics</code>), each chaos trigger increments the
-          <code>aimock_chaos_triggered_total</code> counter with an <code>action</code> label:
+          <code>aimock_chaos_triggered_total</code> counter, tagged with <code>action</code> and
+          <code>source</code>. <code>source="fixture"</code> means a fixture matched (or would have,
+          before chaos intervened); <code>source="proxy"</code> means the request was on the proxy
+          dispatch path.
         </p>
 
         <div class="code-block">
           <div class="code-block-header">Metrics output <span class="lang-tag">text</span></div>
           <pre><code># TYPE aimock_chaos_triggered_total counter
-aimock_chaos_triggered_total{action="drop"} 3
-aimock_chaos_triggered_total{action="malformed"} 1
-aimock_chaos_triggered_total{action="disconnect"} 2</code></pre>
+aimock_chaos_triggered_total{action="drop",source="fixture"} 3
+aimock_chaos_triggered_total{action="malformed",source="fixture"} 1
+aimock_chaos_triggered_total{action="disconnect",source="proxy"} 2</code></pre>
+        </div>
+
+        <p>
+          When a chaos action is rolled but can't be applied &mdash; today, only
+          <code>malformed</code> on an SSE proxy response &mdash; the bypass is recorded in a
+          separate counter so operators can distinguish "chaos didn't roll" from "chaos rolled but
+          was bypassed":
+        </p>
+
+        <div class="code-block">
+          <div class="code-block-header">Bypass counter <span class="lang-tag">text</span></div>
+          <pre><code># TYPE aimock_chaos_bypassed_total counter
+aimock_chaos_bypassed_total{action="malformed",source="proxy",reason="sse_streamed"} 4</code></pre>
         </div>
       </main>
       <aside class="page-toc" id="page-toc"></aside>

src/__tests__/chaos-fixture-mode.test.ts

@@ -0,0 +1,117 @@
+import { describe, it, expect, afterEach } from "vitest";
+import { createServer, type ServerInstance } from "../server.js";
+
+// minimal helpers duplicated to keep this test isolated
+import * as http from "node:http";
+
+function post(url: string, body: unknown): Promise<{ status: number; body: string }> {
+  return new Promise((resolve, reject) => {
+    const data = JSON.stringify(body);
+    const parsed = new URL(url);
+    const req = http.request(
+      {
+        hostname: parsed.hostname,
+        port: parsed.port,
+        path: parsed.pathname,
+        method: "POST",
+        headers: {
+          "Content-Type": "application/json",
+          "Content-Length": Buffer.byteLength(data),
+        },
+      },
+      (res) => {
+        const chunks: Buffer[] = [];
+        res.on("data", (c: Buffer) => chunks.push(c));
+        res.on("end", () => {
+          resolve({ status: res.statusCode ?? 0, body: Buffer.concat(chunks).toString() });
+        });
+      },
+    );
+    req.on("error", reject);
+    req.write(data);
+    req.end();
+  });
+}
+
+let server: ServerInstance | undefined;
+
+afterEach(async () => {
+  if (server) {
+    await new Promise<void>((resolve) => server!.server.close(() => resolve()));
+    server = undefined;
+  }
+});
+
+const CHAT_REQUEST = {
+  model: "gpt-4",
+  messages: [{ role: "user", content: "What is the capital of France?" }],
+};
+
+describe("chaos (fixture mode)", () => {
+  it("chaos short-circuits even when fixture would match", async () => {
+    const fixture = {
+      match: { userMessage: "capital of France" },
+      response: { content: "Paris" },
+    };
+
+    server = await createServer([fixture], {
+      port: 0,
+      chaos: { dropRate: 1.0 },
+    });
+
+    const resp = await post(`${server.url}/v1/chat/completions`, CHAT_REQUEST);
+
+    expect(resp.status).toBe(500);
+    const body = JSON.parse(resp.body);
+    expect(body).toMatchObject({ error: { code: "chaos_drop" } });
+  });
+
+  it("rolls chaos once per request: drop journals the matched fixture, not null", async () => {
+    // Pins the single-roll behavior: chaos evaluation happens AFTER fixture
+    // matching, so when drop fires on a request that matches a fixture, the
+    // journal entry reflects the match (not null, as the old double-roll
+    // pre-flight path would have recorded).
+    const fixture = {
+      match: { userMessage: "capital of France" },
+      response: { content: "Paris" },
+    };
+
+    server = await createServer([fixture], {
+      port: 0,
+      chaos: { dropRate: 1.0 },
+    });
+
+    const resp = await post(`${server.url}/v1/chat/completions`, CHAT_REQUEST);
+    expect(resp.status).toBe(500);
+
+    const last = server.journal.getLast();
+    expect(last?.response.chaosAction).toBe("drop");
+    expect(last?.response.fixture).toBe(fixture);
+    // Match count reflects that the fixture did participate in the decision
+    expect(server.journal.getFixtureMatchCount(fixture)).toBe(1);
+  });
+
+  it("disconnect journals the matched fixture with status 0", async () => {
+    // Symmetric to the drop test above. Disconnect's status is 0 (no response
+    // ever written before res.destroy()) which is a slightly unusual shape;
+    // pin it so future refactors don't silently change it to e.g. 500.
+    const fixture = {
+      match: { userMessage: "capital of France" },
+      response: { content: "Paris" },
+    };
+
+    server = await createServer([fixture], {
+      port: 0,
+      chaos: { disconnectRate: 1.0 },
+    });
+
+    // Client sees a socket destroy mid-request → post() rejects
+    await expect(post(`${server.url}/v1/chat/completions`, CHAT_REQUEST)).rejects.toThrow();
+
+    const last = server.journal.getLast();
+    expect(last?.response.chaosAction).toBe("disconnect");
+    expect(last?.response.status).toBe(0);
+    expect(last?.response.fixture).toBe(fixture);
+    expect(server.journal.getFixtureMatchCount(fixture)).toBe(1);
+  });
+});

src/__tests__/metrics.test.ts

@@ -549,7 +549,7 @@ describe("integration: /metrics endpoint", () => {
     expect(infMatch![1]).toBe(countMatch![1]);
   });
 
-  it("increments chaos counter when chaos triggers", async () => {
+  it("increments chaos counter when chaos triggers (fixture source)", async () => {
     const fixtures: Fixture[] = [
       {
         match: { userMessage: "hello" },
@@ -565,7 +565,43 @@ describe("integration: /metrics endpoint", () => {
 
     const res = await httpGet(`${instance.url}/metrics`);
     expect(res.body).toContain("aimock_chaos_triggered_total");
-    expect(res.body).toMatch(/aimock_chaos_triggered_total\{[^}]*action="drop"[^}]*\} 1/);
+    // Require both labels: action AND source. The source label is part of the
+    // public metric contract (added when chaos was extended to proxy mode) and
+    // an unasserted label is a regression hazard — future callers that forget
+    // to pass source would serialize `source=""` and pass a bare action match.
+    expect(res.body).toMatch(
+      /aimock_chaos_triggered_total\{[^}]*action="drop"[^}]*source="fixture"[^}]*\} 1/,
+    );
+  });
+
+  it('chaos counter carries source="proxy" on proxy path', async () => {
+    // Counterpart to the fixture-source test: proves the source label flips
+    // correctly when the chaos roll belongs to the proxy dispatch branch.
+    // Together these two tests pin both label values of the source dimension.
+    const upstream = await createServer(
+      [{ match: { userMessage: "hi" }, response: { content: "upstream" } }],
+      { port: 0 },
+    );
+    try {
+      instance = await createServer([], {
+        metrics: true,
+        chaos: { dropRate: 1.0 },
+        record: {
+          providers: { openai: upstream.url },
+          fixturePath: "/tmp/aimock-metrics-proxy-source",
+          proxyOnly: true,
+        },
+      });
+
+      await httpPost(`${instance.url}/v1/chat/completions`, chatRequest("hi"));
+
+      const res = await httpGet(`${instance.url}/metrics`);
+      expect(res.body).toMatch(
+        /aimock_chaos_triggered_total\{[^}]*action="drop"[^}]*source="proxy"[^}]*\} 1/,
+      );
+    } finally {
+      await new Promise<void>((resolve) => upstream.server.close(() => resolve()));
+    }
   });
 
   it("increments chaos counter on Anthropic /v1/messages endpoint", async () => {
@@ -588,7 +624,9 @@ describe("integration: /metrics endpoint", () => {
 
     const res = await httpGet(`${instance.url}/metrics`);
     expect(res.body).toContain("aimock_chaos_triggered_total");
-    expect(res.body).toMatch(/aimock_chaos_triggered_total\{[^}]*action="drop"[^}]*\} 1/);
+    expect(res.body).toMatch(
+      /aimock_chaos_triggered_total\{[^}]*action="drop"[^}]*source="fixture"[^}]*\} 1/,
+    );
   });
 
   it("tracks fixtures loaded gauge", async () => {

src/__tests__/multimedia-record.test.ts

@@ -105,7 +105,7 @@ describe("multimedia record: image response detection", () => {
       };
 
       const { req, res } = createMockReqRes("/v1/images/generations");
-      const proxied = await proxyAndRecord(
+      const outcome = await proxyAndRecord(
         req,
         res,
         request,
@@ -115,7 +115,7 @@ describe("multimedia record: image response detection", () => {
         { record, logger },
       );
 
-      expect(proxied).toBe(true);
+      expect(outcome).toBe("relayed");
       expect(fixtures).toHaveLength(1);
       const fixture = fixtures[0];
       expect(fixture.match.endpoint).toBe("image");