Skip to content

Commit 0b04f4e

Browse files
authored
feat: add match.systemMessage fixture matcher (v1.20.0) (#173)
## Summary Adds a new `match.systemMessage` fixture matcher that gates a fixture on a substring (or regexp) found inside the concatenated text of every `role: "system"` message in the request. Existing matchers (`userMessage`, `hasToolResult`, `toolName`, `sequenceIndex`, `turnIndex`) only inspect the user prompt and tool-flow shape. When a calling app exposes UI controls that mutate the agent context (e.g. a CopilotKit demo with a name / timezone / preferences pane that feeds `useAgentContext`), the user prompt is identical across state changes, and a substring `userMessage` match keeps winning. The fixture's baked response then leaks the *old* state values, producing confusingly-wrong output that looks like a fixture-replay bug to whoever is using the demo. This matcher closes that hole at the matcher layer instead of forcing fixture authors to make their canned responses state-agnostic. ## API JSON form: ```json { "match": { "userMessage": "What do you know about me from my context?", "systemMessage": "name=Atai" }, "response": { "content": "Hi Atai, I know your timezone and recent activity." } } ``` Programmatic form accepts `string | RegExp`. Combined with `userMessage` (or any other matcher) the standard **AND** semantics apply — all specified fields must match. ## Semantics - Scans every `role: "system"` message in the request (not just the last one) — hosts that build a system context from multiple sources (persona + agent-context entries + tool guidance) routinely emit several system messages per request. - Joins their text with `\n` so a single substring or regexp sees the whole context as one body. - Multi-part content (`[{type: "text", text: "..."}]`) is extracted via the existing `getTextContent` helper, matching `userMessage` behaviour. - Case-sensitive string matching, honors `requestTransform`'s exact-match mode, mirroring `userMessage`. - No match (no system messages, or substring/regexp miss) → fixture falls through to the next fixture or upstream proxy, as expected. ## Files - `src/types.ts` — `FixtureMatch.systemMessage: string | RegExp`; `FixtureFileEntry.match.systemMessage: string` for JSON - `src/router.ts` — new `getSystemText()` helper + matcher block placed after `userMessage` and before `toolCallId` - `src/fixture-loader.ts` — `entryToFixture` passthrough, type validation (must be string), inclusion in catch-all discriminator set - `src/__tests__/router.test.ts` — 11 new tests covering string, regexp, multi-system, array content, combined-with-userMessage, fall-through, no-system-messages, plus `getSystemText` unit tests - `src/__tests__/fixture-loader.test.ts` — `entryToFixture` passthrough test + non-string validation error test - `README.md`, `skills/write-fixtures/SKILL.md` — documented as a peer of `userMessage` - `CHANGELOG.md`, `package.json`, `.claude-plugin/plugin.json`, `charts/aimock/Chart.yaml` — version 1.19.5 → 1.20.0 (new feature → minor bump). The pre-existing `[Unreleased]` entries (drift-test vacuous-assertion fix, proxy relay status normalization) ride along under the 1.20.0 heading per existing changelog convention. ## Test plan - [x] `pnpm run lint` — clean - [x] `pnpm run build` — clean (tsdown) - [x] `pnpm exec vitest run src/__tests__/router.test.ts src/__tests__/fixture-loader.test.ts` — 218 / 218 pass - [x] `pnpm run test` — 2856 / 2857 pass; 1 failure is a pre-existing Windows-path-separator assertion in `fixtures-remote.test.ts` (verified failing on `origin/main` before any change) — passes on Linux CI - [x] `pnpm run format:check` — clean except for `.claude/commands/write-fixtures.md`, which is a Windows-checkout artifact of a `120000` symlink (verified `git ls-files -s`); on Linux prettier follows the symlink to the real target ## Why minor and not patch New backwards-compatible matcher field — fixtures without `systemMessage` are unaffected. SemVer minor.
2 parents a9996c7 + a8b12e9 commit 0b04f4e

11 files changed

Lines changed: 249 additions & 6 deletions

File tree

.claude-plugin/plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "aimock",
3-
"version": "1.19.5",
3+
"version": "1.20.0",
44
"description": "Fixture authoring guidance for @copilotkit/aimock — LLM, multimedia, MCP, A2A, AG-UI, vector, and service mocking",
55
"author": {
66
"name": "CopilotKit"

CHANGELOG.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# @copilotkit/aimock
22

3-
## [Unreleased]
3+
## [1.20.0] - 2026-05-11
44

55
### Fixed
66

@@ -9,6 +9,7 @@
99

1010
### Added
1111

12+
- **`match.systemMessage` fixture matcher** — gate a fixture on a substring (or regexp) found inside the concatenated text of every `system` role message in the request. Hosts that plumb dynamic context (persona, agent-context entries, dynamic config) through system messages can now narrow a fixture to a specific context state; when the caller changes that state the fixture stops matching and the request falls through to the next fixture or upstream proxy instead of silently returning a stale baked response. JSON form: `"match": { "userMessage": "Who am I?", "systemMessage": "name=Atai" }`. Programmatic form accepts `string | RegExp`.
1213
- **Status code normalization tests** — 5 tests verifying proxy relay normalization (201→200, 429→502, 503→502, 401→502, SSE 429→502) with fixture preservation assertions; 2 existing tests updated to expect normalized 502
1314

1415
## [1.19.5] - 2026-05-09

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or
4949
## Features
5050

5151
- **[Record & Replay](https://aimock.copilotkit.dev/record-replay)** — Proxy real APIs, save as fixtures, replay deterministically forever
52-
- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `sequenceIndex`, or custom predicates
52+
- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `sequenceIndex`, `systemMessage` (gate on host-supplied agent context), or custom predicates
5353
- **[12 LLM Providers](https://aimock.copilotkit.dev/docs)** — OpenAI Chat, OpenAI Responses, OpenAI Realtime, Claude, Gemini, Gemini Live, Gemini Interactions, Azure, Bedrock, Vertex AI, Ollama, Cohere — full streaming support
5454
- **Multimedia APIs**[image generation](https://aimock.copilotkit.dev/images) (DALL-E, Imagen), [text-to-speech](https://aimock.copilotkit.dev/speech), [audio transcription](https://aimock.copilotkit.dev/transcription), [video generation](https://aimock.copilotkit.dev/video)
5555
- **[MCP](https://aimock.copilotkit.dev/mcp-mock) / [A2A](https://aimock.copilotkit.dev/a2a-mock) / [AG-UI](https://aimock.copilotkit.dev/agui-mock) / [Vector](https://aimock.copilotkit.dev/vector-mock)** — Mock every protocol your AI agents use

charts/aimock/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@ name: aimock
33
description: Mock infrastructure for AI application testing (OpenAI, Anthropic, Gemini, MCP, A2A, vector)
44
type: application
55
version: 0.1.0
6-
appVersion: "1.19.5"
6+
appVersion: "1.20.0"

package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@copilotkit/aimock",
3-
"version": "1.19.5",
3+
"version": "1.20.0",
44
"description": "Mock infrastructure for AI application testing — LLM APIs, image generation, text-to-speech, transcription, audio generation, video generation, MCP tools, A2A agents, AG-UI event streams, vector databases, search, rerank, and moderation. One package, one port, zero dependencies.",
55
"license": "MIT",
66
"keywords": [

skills/write-fixtures/SKILL.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ aimock is a zero-dependency mock infrastructure for AI apps. Fixture-driven. Mul
2323
| ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
2424
| `userMessage` | `string` | Substring of last `role: "user"` message text |
2525
| `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text |
26+
| `systemMessage` | `string` | Substring of the concatenated text of every `role: "system"` message in the request. Use to gate a fixture on host-supplied context (persona, agent-context entries) so changes to that context cause the fixture to fall through instead of returning a stale baked response |
27+
| `systemMessage` | `RegExp` | Pattern test on the concatenated system-message text |
2628
| `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) |
2729
| `inputText` | `RegExp` | Pattern test on embedding input text |
2830
| `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`) |

src/__tests__/fixture-loader.test.ts

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -940,6 +940,22 @@ describe("validateFixtures", () => {
940940
expect(results.filter((r) => r.message.includes("hasToolResult"))).toHaveLength(0);
941941
});
942942

943+
// --- match.systemMessage type checks ---
944+
945+
it("error: systemMessage is a number", () => {
946+
const fixtures = [makeFixture({ match: { userMessage: "test", systemMessage: 42 as never } })];
947+
const results = validateFixtures(fixtures);
948+
expect(results.some((r) => r.severity === "error" && r.message.includes("systemMessage"))).toBe(
949+
true,
950+
);
951+
});
952+
953+
it("no error: systemMessage is a string", () => {
954+
const fixtures = [makeFixture({ match: { userMessage: "test", systemMessage: "Atai" } })];
955+
const results = validateFixtures(fixtures);
956+
expect(results.filter((r) => r.message.includes("systemMessage"))).toHaveLength(0);
957+
});
958+
943959
// --- Warning checks ---
944960

945961
it("warning: duplicate userMessage", () => {
@@ -1479,6 +1495,15 @@ describe("auto-stringify JSON objects in fixture entries", () => {
14791495
expect((fixture.response as TextResponse).content).toBe("Hello, world!");
14801496
});
14811497

1498+
it("passes systemMessage through entryToFixture", () => {
1499+
const entry: FixtureFileEntry = {
1500+
match: { userMessage: "test", systemMessage: "name=Atai" },
1501+
response: { content: "ok" },
1502+
};
1503+
const fixture = entryToFixture(entry);
1504+
expect(fixture.match.systemMessage).toBe("name=Atai");
1505+
});
1506+
14821507
it("stringifies nested objects in arguments", () => {
14831508
const entry: FixtureFileEntry = {
14841509
match: { userMessage: "test" },

src/__tests__/router.test.ts

Lines changed: 159 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { describe, it, expect } from "vitest";
2-
import { matchFixture, getLastMessageByRole, getTextContent } from "../router.js";
2+
import { matchFixture, getLastMessageByRole, getSystemText, getTextContent } from "../router.js";
33
import type { ChatCompletionRequest, ChatMessage, ContentPart, Fixture } from "../types.js";
44

55
// ---------------------------------------------------------------------------
@@ -237,6 +237,164 @@ describe("matchFixture — userMessage (RegExp)", () => {
237237
});
238238
});
239239

240+
// ---------------------------------------------------------------------------
241+
// getSystemText
242+
// ---------------------------------------------------------------------------
243+
244+
describe("getSystemText", () => {
245+
it("returns empty string when there are no system messages", () => {
246+
expect(getSystemText([{ role: "user", content: "hi" }])).toBe("");
247+
});
248+
249+
it("returns the single system message text", () => {
250+
expect(
251+
getSystemText([
252+
{ role: "system", content: "You are helpful." },
253+
{ role: "user", content: "hi" },
254+
]),
255+
).toBe("You are helpful.");
256+
});
257+
258+
it("joins multiple system messages with newlines in order", () => {
259+
expect(
260+
getSystemText([
261+
{ role: "system", content: "first" },
262+
{ role: "user", content: "ignored" },
263+
{ role: "system", content: "second" },
264+
]),
265+
).toBe("first\nsecond");
266+
});
267+
268+
it("extracts text from array-of-parts system content", () => {
269+
expect(
270+
getSystemText([{ role: "system", content: [{ type: "text", text: "from parts" }] }]),
271+
).toBe("from parts");
272+
});
273+
});
274+
275+
// ---------------------------------------------------------------------------
276+
// matchFixture — systemMessage
277+
// ---------------------------------------------------------------------------
278+
279+
describe("matchFixture — systemMessage (string)", () => {
280+
it("matches when a system message contains the substring", () => {
281+
const fixture = makeFixture({ systemMessage: "Atai" });
282+
const req = makeReq({
283+
messages: [
284+
{ role: "system", content: "User name is Atai. Timezone America/Los_Angeles." },
285+
{ role: "user", content: "Who am I?" },
286+
],
287+
});
288+
expect(matchFixture([fixture], req)).toBe(fixture);
289+
});
290+
291+
it("does not match when no system message contains the substring", () => {
292+
const fixture = makeFixture({ systemMessage: "Atai" });
293+
const req = makeReq({
294+
messages: [
295+
{ role: "system", content: "User name is Alem." },
296+
{ role: "user", content: "Who am I?" },
297+
],
298+
});
299+
expect(matchFixture([fixture], req)).toBeNull();
300+
});
301+
302+
it("does not match when there are no system messages", () => {
303+
const fixture = makeFixture({ systemMessage: "anything" });
304+
const req = makeReq({ messages: [{ role: "user", content: "hi" }] });
305+
expect(matchFixture([fixture], req)).toBeNull();
306+
});
307+
308+
it("matches across the joined text of multiple system messages", () => {
309+
const fixture = makeFixture({ systemMessage: "Atai" });
310+
const req = makeReq({
311+
messages: [
312+
{ role: "system", content: "Persona: helpful." },
313+
{ role: "system", content: "Context: name=Atai" },
314+
{ role: "user", content: "Who am I?" },
315+
],
316+
});
317+
expect(matchFixture([fixture], req)).toBe(fixture);
318+
});
319+
320+
it("matches when system content is array-of-parts", () => {
321+
const fixture = makeFixture({ systemMessage: "Atai" });
322+
const req = makeReq({
323+
messages: [
324+
{ role: "system", content: [{ type: "text", text: "name=Atai" }] },
325+
{ role: "user", content: "Who am I?" },
326+
],
327+
});
328+
expect(matchFixture([fixture], req)).toBe(fixture);
329+
});
330+
331+
it("combines with userMessage — both must match", () => {
332+
const fixture = makeFixture({ userMessage: "Who am I", systemMessage: "Atai" });
333+
const matching = makeReq({
334+
messages: [
335+
{ role: "system", content: "name=Atai" },
336+
{ role: "user", content: "Who am I?" },
337+
],
338+
});
339+
expect(matchFixture([fixture], matching)).toBe(fixture);
340+
341+
const userOnly = makeReq({
342+
messages: [
343+
{ role: "system", content: "name=Alem" },
344+
{ role: "user", content: "Who am I?" },
345+
],
346+
});
347+
expect(matchFixture([fixture], userOnly)).toBeNull();
348+
349+
const systemOnly = makeReq({
350+
messages: [
351+
{ role: "system", content: "name=Atai" },
352+
{ role: "user", content: "Different prompt" },
353+
],
354+
});
355+
expect(matchFixture([fixture], systemOnly)).toBeNull();
356+
});
357+
358+
it("falls through to the next fixture on systemMessage miss", () => {
359+
const specific = makeFixture(
360+
{ userMessage: "Who am I", systemMessage: "Atai" },
361+
{ content: "Hi Atai" },
362+
);
363+
const fallback = makeFixture({ userMessage: "Who am I" }, { content: "Hi user" });
364+
const req = makeReq({
365+
messages: [
366+
{ role: "system", content: "name=Alem" },
367+
{ role: "user", content: "Who am I?" },
368+
],
369+
});
370+
expect(matchFixture([specific, fallback], req)).toBe(fallback);
371+
});
372+
});
373+
374+
describe("matchFixture — systemMessage (RegExp)", () => {
375+
it("matches when the joined system text satisfies the regexp", () => {
376+
const fixture = makeFixture({ systemMessage: /name=Atai/ });
377+
const req = makeReq({
378+
messages: [
379+
{ role: "system", content: "ctx: name=Atai, tz=PST" },
380+
{ role: "user", content: "Who am I?" },
381+
],
382+
});
383+
expect(matchFixture([fixture], req)).toBe(fixture);
384+
});
385+
386+
it("does not match when the regexp misses", () => {
387+
const fixture = makeFixture({ systemMessage: /name=Atai/ });
388+
const req = makeReq({
389+
messages: [
390+
{ role: "system", content: "ctx: name=Alem" },
391+
{ role: "user", content: "Who am I?" },
392+
],
393+
});
394+
expect(matchFixture([fixture], req)).toBeNull();
395+
});
396+
});
397+
240398
// ---------------------------------------------------------------------------
241399
// matchFixture — toolCallId
242400
// ---------------------------------------------------------------------------

src/fixture-loader.ts

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ export function entryToFixture(entry: FixtureFileEntry): Fixture {
5353
return {
5454
match: {
5555
userMessage: entry.match.userMessage,
56+
systemMessage: entry.match.systemMessage,
5657
inputText: entry.match.inputText,
5758
toolCallId: entry.match.toolCallId,
5859
toolName: entry.match.toolName,
@@ -618,6 +619,13 @@ export function validateFixtures(fixtures: Fixture[]): ValidationResult[] {
618619
message: `match.hasToolResult must be a boolean, got ${typeof f.match.hasToolResult}`,
619620
});
620621
}
622+
if (f.match.systemMessage !== undefined && typeof f.match.systemMessage !== "string") {
623+
results.push({
624+
severity: "error",
625+
fixtureIndex: i,
626+
message: `match.systemMessage must be a string, got ${typeof f.match.systemMessage}`,
627+
});
628+
}
621629

622630
// --- Warning checks ---
623631

@@ -644,6 +652,7 @@ export function validateFixtures(fixtures: Fixture[]): ValidationResult[] {
644652
const hasDiscriminator =
645653
match.endpoint !== undefined ||
646654
match.userMessage !== undefined ||
655+
match.systemMessage !== undefined ||
647656
match.inputText !== undefined ||
648657
match.responseFormat !== undefined ||
649658
match.toolCallId !== undefined ||

src/router.ts

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,23 @@ export function getLastMessageByRole(messages: ChatMessage[], role: string): Cha
1515
return null;
1616
}
1717

18+
/**
19+
* Concatenate the text content of every `system` role message in order.
20+
* Hosts that build a system context from multiple sources (persona, agent
21+
* context entries, tool guidance) often emit several system messages in one
22+
* request; this joins them with newlines so a substring matcher sees the
23+
* whole context as one body.
24+
*/
25+
export function getSystemText(messages: ChatMessage[]): string {
26+
const parts: string[] = [];
27+
for (const m of messages) {
28+
if (m.role !== "system") continue;
29+
const text = getTextContent(m.content);
30+
if (text) parts.push(text);
31+
}
32+
return parts.join("\n");
33+
}
34+
1835
/**
1936
* Extract the text content from a message's content field.
2037
* Handles both plain string content and array-of-parts content
@@ -96,6 +113,26 @@ export function matchFixture(
96113
}
97114
}
98115

116+
// systemMessage — case-sensitive substring (or regexp) match against the
117+
// joined text of every system message in the request. Use to gate a
118+
// fixture on host-supplied context (e.g. agent-context entries) so that
119+
// when the calling app changes that context the fixture stops matching
120+
// and the request falls through to the next fixture or upstream proxy.
121+
if (match.systemMessage !== undefined) {
122+
const text = getSystemText(effective.messages);
123+
if (!text) continue;
124+
if (typeof match.systemMessage === "string") {
125+
if (useExactMatch) {
126+
if (text !== match.systemMessage) continue;
127+
} else {
128+
if (!text.includes(match.systemMessage)) continue;
129+
}
130+
} else {
131+
match.systemMessage.lastIndex = 0;
132+
if (!match.systemMessage.test(text)) continue;
133+
}
134+
}
135+
99136
// toolCallId — a toolCallId fixture answers the model's response to a tool
100137
// result, which by API contract only happens when the conversation's LAST
101138
// message is a tool result. If a newer user (or other) turn follows the

0 commit comments

Comments
 (0)