Skip to content

Commit a44f73f

Browse files
jpr5jpr5
authored
feat: snapshot-style fixture recording by test ID (closes #155) (#157)
## Summary When `X-Test-Id` is present, recorded fixtures are organized by test instead of timestamps: ``` fixtures/recorded/ agent-chat--handles-tool-call/ openai.json agent-chat--streams-correctly/ openai.json ``` - **Slugified test IDs** as directory names — readable, grepable, stable across runs - **Merge-append** on re-run — new fixtures join existing ones in the same file - **Timestamp fallback** when no `X-Test-Id` or empty slug — backwards compatible - **Path traversal safe** — slugifier strips all dangerous characters - **Corrupted file recovery** — invalid JSON silently replaced Closes #155 — feature request by @jantimon ## Test plan - [x] `pnpm test` — 2769 passed, 36 skipped - [x] `npx tsc --noEmit` — clean - [x] `pnpm run lint` / `pnpm run format:check` — clean - [x] CR converged (R1: 1 finding already fixed, R2: 0 findings, 7 agents per round) - [x] Adversarial review: all 6 issue asks verified delivered
2 parents d7dfea8 + e1d4604 commit a44f73f

7 files changed

Lines changed: 629 additions & 7 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# @copilotkit/aimock
22

3+
## [Unreleased]
4+
5+
### Added
6+
7+
- **Snapshot-style recording** — When `X-Test-Id` is present, recorded fixtures are saved to `<fixturePath>/<slugified-testId>/<provider>.json` instead of timestamp-based filenames. Multiple fixtures for the same test+provider merge into one file. Stable paths enable meaningful PR diffs and easy test-to-fixture mapping. (Feature request by @jantimon, issue #155)
8+
39
## [1.18.0] - 2026-05-04
410

511
### Added

‎docs/examples/index.html‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -368,6 +368,42 @@ <h3>Record &amp; Replay</h3>
368368
}</code></pre>
369369
</div>
370370

371+
<h3>Snapshot Recording (per-test fixtures)</h3>
372+
<p>
373+
Send <code>X-Test-Id</code> from your test runner to organize recorded fixtures into
374+
per-test directories. Here is a Playwright example:
375+
</p>
376+
<div class="code-block">
377+
<div class="code-block-header">playwright/setup.ts <span class="lang-tag">ts</span></div>
378+
<pre><code><span class="kw">import</span> { test } <span class="kw">from</span> <span class="str">"@playwright/test"</span>;
379+
380+
test.<span class="fn">beforeEach</span>(<span class="kw">async</span> ({ <span class="op">page</span> }, <span class="op">testInfo</span>) <span class="kw">=&gt;</span> {
381+
<span class="cm">// Route all LLM traffic through aimock with a test ID header</span>
382+
<span class="kw">await</span> <span class="op">page</span>.<span class="fn">setExtraHTTPHeaders</span>({
383+
<span class="str">"X-Test-Id"</span>: <span class="op">testInfo</span>.<span class="prop">title</span>,
384+
});
385+
});</code></pre>
386+
</div>
387+
<p>
388+
The resulting fixture directory layout groups each test&rsquo;s recordings by provider:
389+
</p>
390+
<div class="code-block">
391+
<div class="code-block-header">
392+
Recorded fixture tree <span class="lang-tag">text</span>
393+
</div>
394+
<pre><code>fixtures/recorded/
395+
should-greet-the-user/
396+
openai.json
397+
anthropic.json
398+
should-handle-tool-calls/
399+
openai.json</code></pre>
400+
</div>
401+
<p>
402+
See
403+
<a href="/record-replay#snapshot-style-recording">Snapshot-Style Recording</a> for the
404+
full workflow, including replay and drift detection.
405+
</p>
406+
371407
<!-- ─── Full Suite ─────────────────────────────────────────── -->
372408

373409
<h2>Full Suite</h2>

‎docs/fixtures/index.html‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -493,6 +493,16 @@ <h3>From a directory</h3>
493493
<span class="op">mock</span>.<span class="fn">loadFixtureDir</span>(<span class="str">"./fixtures"</span>);</code></pre>
494494
</div>
495495

496+
<div class="info-box">
497+
<p>
498+
<strong>Snapshot-style recording:</strong> When recording with <code>X-Test-Id</code>,
499+
fixtures are automatically organized into per-test directories
500+
(<code>&lt;fixturePath&gt;/&lt;test-slug&gt;/&lt;provider&gt;.json</code>). See
501+
<a href="/record-replay#snapshot-style-recording">Snapshot-Style Recording</a> for
502+
details.
503+
</p>
504+
</div>
505+
496506
<h3>Programmatically</h3>
497507
<div class="code-block">
498508
<div class="code-block-header">programmatic.ts <span class="lang-tag">ts</span></div>

‎docs/record-replay/index.html‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -465,6 +465,88 @@ <h2>Fixture Auto-Generation</h2>
465465
fixture is saved to disk with a warning but not registered in memory.
466466
</p>
467467

468+
<h2 id="snapshot-style-recording">Snapshot-Style Recording</h2>
469+
<p>
470+
When the <code>X-Test-Id</code> header is present on a request, aimock uses
471+
<strong>snapshot-style recording</strong> instead of the default timestamp-based
472+
filenames. Fixtures are organized by test, producing stable file paths that work well with
473+
version control and PR diffs.
474+
</p>
475+
476+
<h3>Directory structure</h3>
477+
<p>
478+
The test ID is slugified into a directory name, and each provider gets its own file within
479+
that directory:
480+
</p>
481+
482+
<div class="code-block">
483+
<div class="code-block-header">
484+
Snapshot directory layout <span class="lang-tag">text</span>
485+
</div>
486+
<pre><code>fixtures/recorded/
487+
agent-chat--handles-tool-call/
488+
openai.json # All OpenAI fixtures for this test
489+
anthropic.json # All Anthropic fixtures for this test
490+
simple-test/
491+
openai.json</code></pre>
492+
</div>
493+
494+
<p>
495+
The slugify rules: Common test file prefixes (<code>.spec.ts</code>,
496+
<code>.test.tsx</code>, <code>.e2e.js</code>, etc.) are automatically stripped from the
497+
test ID before slugifying, so <code>my-app.spec.ts › greeting</code> becomes
498+
<code>greeting</code>. Then Playwright's <code>&nbsp;›&nbsp;</code> separator becomes
499+
<code>--</code>, non-word characters become <code>-</code>, runs of 3+ dashes collapse to
500+
<code>--</code>, and the result is lowercased. For example,
501+
<code>"agent chat › handles tool call"</code> becomes
502+
<code>agent-chat--handles-tool-call</code>.
503+
</p>
504+
505+
<h3>Merge behavior on re-run</h3>
506+
<p>
507+
When you re-run a test, the new fixture is <strong>appended</strong> to the existing
508+
<code>&lt;provider&gt;.json</code> file rather than overwriting it. This preserves
509+
multi-turn conversations in a single file. If the existing file is corrupted (invalid
510+
JSON), it is silently replaced.
511+
</p>
512+
513+
<h3>Sending <code>X-Test-Id</code> from test frameworks</h3>
514+
515+
<div class="code-block">
516+
<div class="code-block-header">Playwright <span class="lang-tag">ts</span></div>
517+
<pre><code><span class="cm">// Playwright exposes testInfo.titlePath which joins suite + test titles</span>
518+
<span class="kw">import</span> { test } <span class="kw">from</span> <span class="str">"@playwright/test"</span>;
519+
520+
test(<span class="str">"handles tool call"</span>, <span class="kw">async</span> ({ page }, testInfo) => {
521+
<span class="cm">// titlePath = ["agent chat", "handles tool call"]</span>
522+
<span class="kw">const</span> testId = testInfo.titlePath.join(<span class="str">" › "</span>);
523+
<span class="cm">// Set on your OpenAI/Anthropic client config as a default header:</span>
524+
<span class="cm">// headers: { "X-Test-Id": testId }</span>
525+
});</code></pre>
526+
</div>
527+
528+
<div class="code-block">
529+
<div class="code-block-header">Vitest <span class="lang-tag">ts</span></div>
530+
<pre><code><span class="kw">import</span> { describe, it } <span class="kw">from</span> <span class="str">"vitest"</span>;
531+
532+
describe(<span class="str">"agent chat"</span>, () => {
533+
it(<span class="str">"handles tool call"</span>, <span class="kw">async</span> () => {
534+
<span class="cm">// Pass X-Test-Id on each LLM request:</span>
535+
<span class="kw">const</span> resp = <span class="kw">await</span> fetch(<span class="str">"http://localhost:4010/v1/chat/completions"</span>, {
536+
headers: { <span class="str">"X-Test-Id"</span>: <span class="str">"agent chat › handles tool call"</span> },
537+
<span class="cm">// ...body</span>
538+
});
539+
});
540+
});</code></pre>
541+
</div>
542+
543+
<h3>Fallback behavior</h3>
544+
<p>
545+
When no <code>X-Test-Id</code> header is present (or the value is
546+
<code>__default__</code>), recording falls back to the standard timestamp-based filename:
547+
<code>&lt;provider&gt;-&lt;timestamp&gt;-&lt;uuid&gt;.json</code>.
548+
</p>
549+
468550
<h2>Fixture Lifecycle</h2>
469551
<ul>
470552
<li>

0 commit comments

Comments
 (0)