Skip to content

Commit 6b46392

Browse files
jpr5jpr5
committed
Format fixture-loader tests and SKILL.md
1 parent 44f6e3f commit 6b46392

2 files changed

Lines changed: 69 additions & 71 deletions

File tree

‎skills/write-fixtures/SKILL.md‎

Lines changed: 56 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -19,35 +19,35 @@ aimock is a zero-dependency mock infrastructure for AI apps. Fixture-driven. Mul
1919

2020
## Match Field Reference
2121

22-
| Field | Type | Matches Against |
23-
| ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------- |
24-
| `userMessage` | `string` | Substring of last `role: "user"` message text |
25-
| `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text |
26-
| `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) |
27-
| `inputText` | `RegExp` | Pattern test on embedding input text |
28-
| `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`) |
29-
| `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message |
30-
| `model` | `string` | Exact match on `req.model` |
31-
| `model` | `RegExp` | Pattern test on `req.model` |
32-
| `responseFormat` | `string` | Exact match on `req.response_format.type` (`"json_object"`, `"json_schema"`) |
33-
| `sequenceIndex` | `number` | Matches only when this fixture's match count equals the given index (0-based) |
22+
| Field | Type | Matches Against |
23+
| ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
24+
| `userMessage` | `string` | Substring of last `role: "user"` message text |
25+
| `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text |
26+
| `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) |
27+
| `inputText` | `RegExp` | Pattern test on embedding input text |
28+
| `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`) |
29+
| `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message |
30+
| `model` | `string` | Exact match on `req.model` |
31+
| `model` | `RegExp` | Pattern test on `req.model` |
32+
| `responseFormat` | `string` | Exact match on `req.response_format.type` (`"json_object"`, `"json_schema"`) |
33+
| `sequenceIndex` | `number` | Matches only when this fixture's match count equals the given index (0-based) |
3434
| `turnIndex` | `number` | Stateless conversation-depth matching. Counts `role: "assistant"` messages in the request; matches when that count equals the value. `turnIndex: 0` = first turn (no prior assistant messages). Use instead of `sequenceIndex` for shared/deployed instances where stateful counters break under concurrency |
35-
| `hasToolResult` | `boolean` | Stateless tool-message presence matching. `true` matches when any `role: "tool"` message exists in the request; `false` matches when none exist. Provider-consistent across all aimock handlers (OpenAI, Claude, Gemini, Bedrock, Ollama, Cohere) |
36-
| `endpoint` | `string` | Restrict to endpoint type: `"chat"`, `"image"`, `"speech"`, `"transcription"`, `"video"`, `"embedding"` |
37-
| `predicate` | `(req: ChatCompletionRequest) => boolean` | Custom function — full access to request |
35+
| `hasToolResult` | `boolean` | Stateless tool-message presence matching. `true` matches when any `role: "tool"` message exists in the request; `false` matches when none exist. Provider-consistent across all aimock handlers (OpenAI, Claude, Gemini, Bedrock, Ollama, Cohere) |
36+
| `endpoint` | `string` | Restrict to endpoint type: `"chat"`, `"image"`, `"speech"`, `"transcription"`, `"video"`, `"embedding"` |
37+
| `predicate` | `(req: ChatCompletionRequest) => boolean` | Custom function — full access to request |
3838

3939
**AND logic**: all specified fields must match. Empty match `{}` = catch-all.
4040

4141
Multi-part content (e.g., `[{type: "text", text: "hello"}]`) is automatically extracted — `userMessage` matching works regardless of content format.
4242

4343
### When to Use Each Multi-turn Matching Approach
4444

45-
| Approach | Stateless? | Best For |
46-
| ---------------- | ---------- | ------------------------------------------------------------------------------------------------------- |
47-
| `turnIndex` | Yes | Shared/deployed instances; matches on conversation depth (count of assistant messages in request) |
48-
| `hasToolResult` | Yes | Simplest option for 2-step tool flows — boolean: are there tool results in the request? |
49-
| `sequenceIndex` | No | Single-client unit tests with repeated identical requests (server-side counter, breaks under concurrency) |
50-
| `toolCallId` | Yes | Matching specific tool result IDs in the conversation history |
45+
| Approach | Stateless? | Best For |
46+
| --------------- | ---------- | --------------------------------------------------------------------------------------------------------- |
47+
| `turnIndex` | Yes | Shared/deployed instances; matches on conversation depth (count of assistant messages in request) |
48+
| `hasToolResult` | Yes | Simplest option for 2-step tool flows — boolean: are there tool results in the request? |
49+
| `sequenceIndex` | No | Single-client unit tests with repeated identical requests (server-side counter, breaks under concurrency) |
50+
| `toolCallId` | Yes | Matching specific tool result IDs in the conversation history |
5151

5252
**Prefer stateless approaches** (`turnIndex`, `hasToolResult`) for shared aimock instances (deployed via Docker, used by multiple test runners). Use `sequenceIndex` only in isolated single-client unit tests where the counter won't be corrupted by concurrent requests.
5353

@@ -678,42 +678,42 @@ const mock = await LLMock.create({ port: 0 }); // creates + starts in one call
678678

679679
## API Quick Reference
680680

681-
| Method | Purpose |
682-
| --------------------------------------- | ------------------------------------------- |
683-
| `addFixture(f)` | Append fixture (last priority) |
684-
| `addFixtures(f[])` | Append multiple |
685-
| `prependFixture(f)` | Insert at front (highest priority) |
686-
| `clearFixtures()` | Remove all fixtures |
687-
| `getFixtures()` | Read current fixture list |
688-
| `on(match, response, opts?)` | Shorthand for `addFixture` |
689-
| `onMessage(pattern, response, opts?)` | Match by user message |
690-
| `onEmbedding(pattern, response, opts?)` | Match by embedding input text |
691-
| `onJsonOutput(pattern, json, opts?)` | Match by user message with `responseFormat` |
692-
| `onToolCall(name, response, opts?)` | Match by tool name in `tools[]` |
693-
| `onToolResult(id, response, opts?)` | Match by `tool_call_id` |
681+
| Method | Purpose |
682+
| ---------------------------------------- | ------------------------------------------- |
683+
| `addFixture(f)` | Append fixture (last priority) |
684+
| `addFixtures(f[])` | Append multiple |
685+
| `prependFixture(f)` | Insert at front (highest priority) |
686+
| `clearFixtures()` | Remove all fixtures |
687+
| `getFixtures()` | Read current fixture list |
688+
| `on(match, response, opts?)` | Shorthand for `addFixture` |
689+
| `onMessage(pattern, response, opts?)` | Match by user message |
690+
| `onEmbedding(pattern, response, opts?)` | Match by embedding input text |
691+
| `onJsonOutput(pattern, json, opts?)` | Match by user message with `responseFormat` |
692+
| `onToolCall(name, response, opts?)` | Match by tool name in `tools[]` |
693+
| `onToolResult(id, response, opts?)` | Match by `tool_call_id` |
694694
| `onTurn(turn, pattern, response, opts?)` | Match by turn index + user message |
695-
| `nextRequestError(status, body?)` | One-shot error, auto-removes |
696-
| `loadFixtureFile(path)` | Load JSON fixture file |
697-
| `loadFixtureDir(path)` | Load all JSON files in directory |
698-
| `start()` | Start server, returns URL |
699-
| `stop()` | Stop server |
700-
| `reset()` | Clear fixtures + journal + match counts |
701-
| `resetMatchCounts()` | Clear sequence match counts only |
702-
| `getRequests()` | All journal entries |
703-
| `getLastRequest()` | Most recent journal entry |
704-
| `clearRequests()` | Clear journal only |
705-
| `setChaos(opts)` | Set server-level chaos rates |
706-
| `clearChaos()` | Remove server-level chaos |
707-
| `onSearch(pattern, results)` | Match search requests by query |
708-
| `onRerank(pattern, results)` | Match rerank requests by query |
709-
| `onModerate(pattern, result)` | Match moderation requests by input |
710-
| `onImage(pattern, response)` | Match image generation by prompt |
711-
| `onSpeech(pattern, response)` | Match TTS by input text |
712-
| `onTranscription(response)` | Match audio transcription |
713-
| `onVideo(pattern, response)` | Match video generation by prompt |
714-
| `mount(path, handler)` | Mount a Mountable (VectorMock, etc.) |
715-
| `url` / `baseUrl` | Server URL (throws if not started) |
716-
| `port` | Server port number |
695+
| `nextRequestError(status, body?)` | One-shot error, auto-removes |
696+
| `loadFixtureFile(path)` | Load JSON fixture file |
697+
| `loadFixtureDir(path)` | Load all JSON files in directory |
698+
| `start()` | Start server, returns URL |
699+
| `stop()` | Stop server |
700+
| `reset()` | Clear fixtures + journal + match counts |
701+
| `resetMatchCounts()` | Clear sequence match counts only |
702+
| `getRequests()` | All journal entries |
703+
| `getLastRequest()` | Most recent journal entry |
704+
| `clearRequests()` | Clear journal only |
705+
| `setChaos(opts)` | Set server-level chaos rates |
706+
| `clearChaos()` | Remove server-level chaos |
707+
| `onSearch(pattern, results)` | Match search requests by query |
708+
| `onRerank(pattern, results)` | Match rerank requests by query |
709+
| `onModerate(pattern, result)` | Match moderation requests by input |
710+
| `onImage(pattern, response)` | Match image generation by prompt |
711+
| `onSpeech(pattern, response)` | Match TTS by input text |
712+
| `onTranscription(response)` | Match audio transcription |
713+
| `onVideo(pattern, response)` | Match video generation by prompt |
714+
| `mount(path, handler)` | Mount a Mountable (VectorMock, etc.) |
715+
| `url` / `baseUrl` | Server URL (throws if not started) |
716+
| `port` | Server port number |
717717

718718
Sequential responses use `on()` with `sequenceIndex` in the match — there is no dedicated convenience method.
719719

‎src/__tests__/fixture-loader.test.ts‎

Lines changed: 13 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -805,27 +805,25 @@ describe("validateFixtures", () => {
805805
it("error: turnIndex is negative", () => {
806806
const fixtures = [makeFixture({ match: { userMessage: "test", turnIndex: -1 } })];
807807
const results = validateFixtures(fixtures);
808-
expect(
809-
results.some((r) => r.severity === "error" && r.message.includes("turnIndex")),
810-
).toBe(true);
808+
expect(results.some((r) => r.severity === "error" && r.message.includes("turnIndex"))).toBe(
809+
true,
810+
);
811811
});
812812

813813
it("error: turnIndex is a float", () => {
814814
const fixtures = [makeFixture({ match: { userMessage: "test", turnIndex: 1.5 } })];
815815
const results = validateFixtures(fixtures);
816-
expect(
817-
results.some((r) => r.severity === "error" && r.message.includes("turnIndex")),
818-
).toBe(true);
816+
expect(results.some((r) => r.severity === "error" && r.message.includes("turnIndex"))).toBe(
817+
true,
818+
);
819819
});
820820

821821
it("error: turnIndex is a string", () => {
822-
const fixtures = [
823-
makeFixture({ match: { userMessage: "test", turnIndex: "zero" as never } }),
824-
];
822+
const fixtures = [makeFixture({ match: { userMessage: "test", turnIndex: "zero" as never } })];
825823
const results = validateFixtures(fixtures);
826-
expect(
827-
results.some((r) => r.severity === "error" && r.message.includes("turnIndex")),
828-
).toBe(true);
824+
expect(results.some((r) => r.severity === "error" && r.message.includes("turnIndex"))).toBe(
825+
true,
826+
);
829827
});
830828

831829
it("no error: turnIndex is 0 (falsy but valid)", () => {
@@ -845,9 +843,9 @@ describe("validateFixtures", () => {
845843
makeFixture({ match: { userMessage: "test", hasToolResult: "yes" as never } }),
846844
];
847845
const results = validateFixtures(fixtures);
848-
expect(
849-
results.some((r) => r.severity === "error" && r.message.includes("hasToolResult")),
850-
).toBe(true);
846+
expect(results.some((r) => r.severity === "error" && r.message.includes("hasToolResult"))).toBe(
847+
true,
848+
);
851849
});
852850

853851
it("no error: hasToolResult is false (falsy but valid)", () => {

0 commit comments

Comments
 (0)