Skip to content

Commit b9b4edc

Browse files
authored
Add per-request X-AIMock-Strict header support (#178)
## Summary - Adds `resolveStrictMode()` helper that reads the `X-AIMock-Strict` request header and overrides the server-wide `--strict` flag per-request (`true`/`1` = strict, `false`/`0` = lenient) - Updates all 22+ handler sites across every provider (OpenAI, Anthropic, Gemini, Bedrock, Ollama, Cohere, ElevenLabs, fal.ai, WebSocket handlers) to use per-request strict resolution - Records `strictOverride` in journal entries when the header overrides the server default - Follows the existing `X-AIMock-Chaos-*` header precedence pattern This enables the same aimock instance to serve both **strict fixture-only probes** (harness sends `X-AIMock-Strict: true`) and **live demo traffic** (no header, proxies to real LLMs) simultaneously. ## Test plan - [x] Unit tests for `resolveStrictMode()` — all header values, fallback to server default - [x] Unit tests for `strictOverrideField()` — override vs no-override cases - [x] Integration tests — strict server + lenient header yields 404, lenient server + strict header yields 503 - [x] Integration tests — journal records `strictOverride` only when header overrides default - [x] 2891 tests pass, 0 fail - [x] `grep -rn "defaults\.strict" src/` returns 0 matches (all replaced)
2 parents 4a0628f + 28d16a7 commit b9b4edc

28 files changed

Lines changed: 528 additions & 86 deletions

.claude-plugin/plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "aimock",
3-
"version": "1.21.0",
3+
"version": "1.22.0",
44
"description": "Fixture authoring guidance for @copilotkit/aimock — LLM, multimedia, MCP, A2A, AG-UI, vector, and service mocking",
55
"author": {
66
"name": "CopilotKit"

CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22

33
## [Unreleased]
44

5+
## [1.22.0] - 2026-05-11
6+
7+
### Added
8+
9+
- **Per-request strict mode via `X-AIMock-Strict` header** — overrides the server-wide `--strict` flag per request (`true`/`1` = strict, `false`/`0` = lenient). When strict: fixture miss returns 503; when lenient: fixture miss proxies to real provider. Follows the `X-AIMock-Chaos-*` precedence pattern. Journal entries record `strictOverride` when the header overrides the server default. Enables the same aimock instance to serve both deterministic test probes and live demo traffic simultaneously.
10+
511
### Fixed
612

713
- **Progressive relay for NDJSON and Bedrock binary event streams** — Ollama NDJSON and Bedrock binary event streams were fully buffered before relay, triggering downstream idle timeouts; now relayed progressively as chunks arrive

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or
5454
- **Multimedia APIs**[image generation](https://aimock.copilotkit.dev/images) (DALL-E, Imagen), [text-to-speech](https://aimock.copilotkit.dev/speech), [audio transcription](https://aimock.copilotkit.dev/transcription), [video generation](https://aimock.copilotkit.dev/video)
5555
- **[MCP](https://aimock.copilotkit.dev/mcp-mock) / [A2A](https://aimock.copilotkit.dev/a2a-mock) / [AG-UI](https://aimock.copilotkit.dev/agui-mock) / [Vector](https://aimock.copilotkit.dev/vector-mock)** — Mock every protocol your AI agents use
5656
- **[Chaos Testing](https://aimock.copilotkit.dev/chaos-testing)** — 500 errors, malformed JSON, mid-stream disconnects at any probability
57+
- **Per-Request Strict Mode**`X-AIMock-Strict` header overrides the server-level `--strict` flag per request (`true`/`1` = strict, `false`/`0` = lenient)
5758
- **[Drift Detection](https://aimock.copilotkit.dev/drift-detection)** — Daily CI validation against real APIs
5859
- **[Streaming Physics](https://aimock.copilotkit.dev/streaming-physics)** — Configurable `ttft`, `tps`, and `jitter`
5960
- **[WebSocket APIs](https://aimock.copilotkit.dev/websocket)** — OpenAI Realtime, Responses WS, Gemini Live

package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@copilotkit/aimock",
3-
"version": "1.21.0",
3+
"version": "1.22.0",
44
"description": "Mock infrastructure for AI application testing — LLM APIs, image generation, text-to-speech, transcription, audio generation, video generation, MCP tools, A2A agents, AG-UI event streams, vector databases, search, rerank, and moderation. One package, one port, zero dependencies.",
55
"license": "MIT",
66
"keywords": [
Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
import { describe, it, expect, afterEach } from "vitest";
2+
import http from "node:http";
3+
import { resolveStrictMode, strictOverrideField } from "../helpers.js";
4+
import { createServer, type ServerInstance } from "../server.js";
5+
import type { Fixture, ChatCompletionRequest } from "../types.js";
6+
7+
// ---------------------------------------------------------------------------
8+
// Helpers
9+
// ---------------------------------------------------------------------------
10+
11+
async function httpPost(
12+
url: string,
13+
body: object,
14+
headers?: Record<string, string>,
15+
): Promise<{ status: number; body: string }> {
16+
return new Promise((resolve, reject) => {
17+
const req = http.request(
18+
url,
19+
{
20+
method: "POST",
21+
headers: {
22+
"Content-Type": "application/json",
23+
...headers,
24+
},
25+
},
26+
(res) => {
27+
const chunks: Buffer[] = [];
28+
res.on("data", (c) => chunks.push(c));
29+
res.on("end", () =>
30+
resolve({
31+
status: res.statusCode!,
32+
body: Buffer.concat(chunks).toString(),
33+
}),
34+
);
35+
},
36+
);
37+
req.on("error", reject);
38+
req.write(JSON.stringify(body));
39+
req.end();
40+
});
41+
}
42+
43+
function chatRequest(userContent: string): ChatCompletionRequest {
44+
return {
45+
model: "gpt-4",
46+
messages: [{ role: "user", content: userContent }],
47+
};
48+
}
49+
50+
// ---------------------------------------------------------------------------
51+
// Unit tests: resolveStrictMode
52+
// ---------------------------------------------------------------------------
53+
54+
describe("resolveStrictMode", () => {
55+
it("returns server default when no header", () => {
56+
expect(resolveStrictMode(false)).toBe(false);
57+
expect(resolveStrictMode(true)).toBe(true);
58+
expect(resolveStrictMode(undefined)).toBe(false);
59+
});
60+
61+
it("returns server default when headers are empty", () => {
62+
expect(resolveStrictMode(false, {})).toBe(false);
63+
expect(resolveStrictMode(true, {})).toBe(true);
64+
});
65+
66+
it('header "true" overrides server default false', () => {
67+
expect(resolveStrictMode(false, { "x-aimock-strict": "true" })).toBe(true);
68+
});
69+
70+
it('header "1" overrides server default false', () => {
71+
expect(resolveStrictMode(false, { "x-aimock-strict": "1" })).toBe(true);
72+
});
73+
74+
it('header "false" overrides server default true', () => {
75+
expect(resolveStrictMode(true, { "x-aimock-strict": "false" })).toBe(false);
76+
});
77+
78+
it('header "0" overrides server default true', () => {
79+
expect(resolveStrictMode(true, { "x-aimock-strict": "0" })).toBe(false);
80+
});
81+
82+
it("ignores unrecognised header values and falls back to server default", () => {
83+
expect(resolveStrictMode(true, { "x-aimock-strict": "yes" })).toBe(true);
84+
expect(resolveStrictMode(false, { "x-aimock-strict": "maybe" })).toBe(false);
85+
});
86+
});
87+
88+
// ---------------------------------------------------------------------------
89+
// Unit tests: strictOverrideField
90+
// ---------------------------------------------------------------------------
91+
92+
describe("strictOverrideField", () => {
93+
it("returns strictOverride when header overrides server default", () => {
94+
expect(strictOverrideField(false, { "x-aimock-strict": "true" })).toEqual({
95+
strictOverride: true,
96+
});
97+
expect(strictOverrideField(true, { "x-aimock-strict": "false" })).toEqual({
98+
strictOverride: false,
99+
});
100+
});
101+
102+
it("returns empty object when no override", () => {
103+
expect(strictOverrideField(false, {})).toEqual({});
104+
expect(strictOverrideField(true, { "x-aimock-strict": "true" })).toEqual({});
105+
expect(strictOverrideField(false)).toEqual({});
106+
expect(strictOverrideField(undefined)).toEqual({});
107+
});
108+
});
109+
110+
// ---------------------------------------------------------------------------
111+
// Integration tests: per-request strict header
112+
// ---------------------------------------------------------------------------
113+
114+
describe("X-AIMock-Strict header integration", () => {
115+
let server: ServerInstance;
116+
const helloFixture: Fixture = {
117+
match: { userMessage: "hello" },
118+
response: { content: "Hi there!" },
119+
};
120+
121+
afterEach(async () => {
122+
if (server) {
123+
await new Promise<void>((resolve) => server.server.close(() => resolve()));
124+
}
125+
});
126+
127+
it("server with --strict + header false returns 404 not 503", async () => {
128+
server = await createServer([helloFixture], { port: 0, strict: true });
129+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"), {
130+
"X-AIMock-Strict": "false",
131+
});
132+
expect(res.status).toBe(404);
133+
const body = JSON.parse(res.body);
134+
expect(body.error.message).toBe("No fixture matched");
135+
});
136+
137+
it("server without --strict + header true returns 503 not 404", async () => {
138+
server = await createServer([helloFixture], { port: 0 });
139+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"), {
140+
"X-AIMock-Strict": "true",
141+
});
142+
expect(res.status).toBe(503);
143+
const body = JSON.parse(res.body);
144+
expect(body.error.message).toBe("Strict mode: no fixture matched");
145+
});
146+
147+
it("header works on chat completions endpoint with matched fixture", async () => {
148+
server = await createServer([helloFixture], { port: 0 });
149+
// Header should not affect matched requests — fixture still serves
150+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("hello"), {
151+
"X-AIMock-Strict": "true",
152+
});
153+
expect(res.status).toBe(200);
154+
});
155+
156+
it("header 1 enables strict mode", async () => {
157+
server = await createServer([helloFixture], { port: 0 });
158+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"), {
159+
"X-AIMock-Strict": "1",
160+
});
161+
expect(res.status).toBe(503);
162+
});
163+
164+
it("header 0 disables strict mode", async () => {
165+
server = await createServer([helloFixture], { port: 0, strict: true });
166+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"), {
167+
"X-AIMock-Strict": "0",
168+
});
169+
expect(res.status).toBe(404);
170+
});
171+
172+
it("absent header falls back to server default", async () => {
173+
// No header at all — server default (strict: true) applies
174+
server = await createServer([helloFixture], { port: 0, strict: true });
175+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"));
176+
expect(res.status).toBe(503);
177+
const body = JSON.parse(res.body);
178+
expect(body.error.message).toBe("Strict mode: no fixture matched");
179+
});
180+
181+
it("absent header falls back to server default (non-strict)", async () => {
182+
// No header at all — server default (strict: false) applies
183+
server = await createServer([helloFixture], { port: 0 });
184+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"));
185+
expect(res.status).toBe(404);
186+
const body = JSON.parse(res.body);
187+
expect(body.error.message).toBe("No fixture matched");
188+
});
189+
190+
it("records strictOverride in journal when header overrides server default", async () => {
191+
server = await createServer([helloFixture], { port: 0 });
192+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"), {
193+
"X-AIMock-Strict": "true",
194+
});
195+
expect(res.status).toBe(503);
196+
const entries = server.journal.getAll();
197+
expect(entries.length).toBe(1);
198+
expect(entries[0].response.strictOverride).toBe(true);
199+
});
200+
201+
it("does not record strictOverride when header matches server default", async () => {
202+
server = await createServer([helloFixture], { port: 0, strict: true });
203+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"), {
204+
"X-AIMock-Strict": "true",
205+
});
206+
expect(res.status).toBe(503);
207+
const entries = server.journal.getAll();
208+
expect(entries.length).toBe(1);
209+
expect(entries[0].response.strictOverride).toBeUndefined();
210+
});
211+
212+
it("records strictOverride=false when header disables strict on strict server", async () => {
213+
server = await createServer([helloFixture], { port: 0, strict: true });
214+
const res = await httpPost(`${server.url}/v1/chat/completions`, chatRequest("unmatched"), {
215+
"X-AIMock-Strict": "false",
216+
});
217+
expect(res.status).toBe(404);
218+
const entries = server.journal.getAll();
219+
expect(entries.length).toBe(1);
220+
expect(entries[0].response.strictOverride).toBe(false);
221+
});
222+
});

src/bedrock-converse.ts

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ import {
2727
flattenHeaders,
2828
getTestId,
2929
resolveResponse,
30+
resolveStrictMode,
31+
strictOverrideField,
3032
} from "./helpers.js";
3133
import { matchFixture } from "./router.js";
3234
import { writeErrorResponse } from "./sse-writer.js";
@@ -616,19 +618,24 @@ export async function handleConverse(
616618
return;
617619
}
618620
}
619-
const strictStatus = defaults.strict ? 503 : 404;
620-
const strictMessage = defaults.strict
621+
const effectiveStrict = resolveStrictMode(defaults.strict, req.headers);
622+
const strictStatus = effectiveStrict ? 503 : 404;
623+
const strictMessage = effectiveStrict
621624
? "Strict mode: no fixture matched"
622625
: "No fixture matched";
623-
if (defaults.strict) {
626+
if (effectiveStrict) {
624627
logger.error(`STRICT: No fixture matched for ${req.method ?? "POST"} ${urlPath}`);
625628
}
626629
journal.add({
627630
method: req.method ?? "POST",
628631
path: urlPath,
629632
headers: flattenHeaders(req.headers),
630633
body: completionReq,
631-
response: { status: strictStatus, fixture: null },
634+
response: {
635+
status: strictStatus,
636+
fixture: null,
637+
...strictOverrideField(defaults.strict, req.headers),
638+
},
632639
});
633640
writeErrorResponse(
634641
res,
@@ -881,19 +888,24 @@ export async function handleConverseStream(
881888
return;
882889
}
883890
}
884-
const strictStatus = defaults.strict ? 503 : 404;
885-
const strictMessage = defaults.strict
891+
const effectiveStrict = resolveStrictMode(defaults.strict, req.headers);
892+
const strictStatus = effectiveStrict ? 503 : 404;
893+
const strictMessage = effectiveStrict
886894
? "Strict mode: no fixture matched"
887895
: "No fixture matched";
888-
if (defaults.strict) {
896+
if (effectiveStrict) {
889897
logger.error(`STRICT: No fixture matched for ${req.method ?? "POST"} ${urlPath}`);
890898
}
891899
journal.add({
892900
method: req.method ?? "POST",
893901
path: urlPath,
894902
headers: flattenHeaders(req.headers),
895903
body: completionReq,
896-
response: { status: strictStatus, fixture: null },
904+
response: {
905+
status: strictStatus,
906+
fixture: null,
907+
...strictOverrideField(defaults.strict, req.headers),
908+
},
897909
});
898910
writeErrorResponse(
899911
res,

src/bedrock.ts

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ import {
3838
flattenHeaders,
3939
getTestId,
4040
resolveResponse,
41+
resolveStrictMode,
42+
strictOverrideField,
4143
} from "./helpers.js";
4244
import { matchFixture } from "./router.js";
4345
import { writeErrorResponse } from "./sse-writer.js";
@@ -447,19 +449,24 @@ export async function handleBedrock(
447449
return;
448450
}
449451
}
450-
const strictStatus = defaults.strict ? 503 : 404;
451-
const strictMessage = defaults.strict
452+
const effectiveStrict = resolveStrictMode(defaults.strict, req.headers);
453+
const strictStatus = effectiveStrict ? 503 : 404;
454+
const strictMessage = effectiveStrict
452455
? "Strict mode: no fixture matched"
453456
: "No fixture matched";
454-
if (defaults.strict) {
457+
if (effectiveStrict) {
455458
logger.error(`STRICT: No fixture matched for ${req.method ?? "POST"} ${urlPath}`);
456459
}
457460
journal.add({
458461
method: req.method ?? "POST",
459462
path: urlPath,
460463
headers: flattenHeaders(req.headers),
461464
body: completionReq,
462-
response: { status: strictStatus, fixture: null },
465+
response: {
466+
status: strictStatus,
467+
fixture: null,
468+
...strictOverrideField(defaults.strict, req.headers),
469+
},
463470
});
464471
writeErrorResponse(
465472
res,
@@ -1062,19 +1069,24 @@ export async function handleBedrockStream(
10621069
return;
10631070
}
10641071
}
1065-
const strictStatus = defaults.strict ? 503 : 404;
1066-
const strictMessage = defaults.strict
1072+
const effectiveStrict = resolveStrictMode(defaults.strict, req.headers);
1073+
const strictStatus = effectiveStrict ? 503 : 404;
1074+
const strictMessage = effectiveStrict
10671075
? "Strict mode: no fixture matched"
10681076
: "No fixture matched";
1069-
if (defaults.strict) {
1077+
if (effectiveStrict) {
10701078
logger.error(`STRICT: No fixture matched for ${req.method ?? "POST"} ${urlPath}`);
10711079
}
10721080
journal.add({
10731081
method: req.method ?? "POST",
10741082
path: urlPath,
10751083
headers: flattenHeaders(req.headers),
10761084
body: completionReq,
1077-
response: { status: strictStatus, fixture: null },
1085+
response: {
1086+
status: strictStatus,
1087+
fixture: null,
1088+
...strictOverrideField(defaults.strict, req.headers),
1089+
},
10781090
});
10791091
writeErrorResponse(
10801092
res,

0 commit comments

Comments
 (0)