Skip to content

Commit ed4a463

Browse files
AlemTuzlakjpr5
AlemTuzlak
authored and
jpr5
committed
docs: add multimedia endpoint docs, comparison matrix, drift rules, record/replay
1 parent 1b5d2a6 commit ed4a463

8 files changed

Lines changed: 1064 additions & 1 deletion

File tree

‎docs/fixtures/index.html‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,26 @@ <h2>Response Types</h2>
162162
<td>embedding[]</td>
163163
<td>Vector of numbers</td>
164164
</tr>
165+
<tr>
166+
<td>Image</td>
167+
<td>image.url or images[].url</td>
168+
<td>Generated image URL(s) or base64 data</td>
169+
</tr>
170+
<tr>
171+
<td>Speech</td>
172+
<td>audio</td>
173+
<td>Base64-encoded audio data</td>
174+
</tr>
175+
<tr>
176+
<td>Transcription</td>
177+
<td>transcription.text, words?, segments?</td>
178+
<td>Transcribed text with optional timestamps</td>
179+
</tr>
180+
<tr>
181+
<td>Video</td>
182+
<td>video.url, video.duration?</td>
183+
<td>Generated video URL with async polling</td>
184+
</tr>
165185
</tbody>
166186
</table>
167187

@@ -239,6 +259,10 @@ <h3>Programmatically</h3>
239259
<span class="op">mock</span>.<span class="fn">onMessage</span>(<span class="str">"hello"</span>, { <span class="prop">content</span>: <span class="str">"Hi!"</span> });
240260
<span class="op">mock</span>.<span class="fn">onToolCall</span>(<span class="str">"get_weather"</span>, { <span class="prop">content</span>: <span class="str">"72F"</span> });
241261
<span class="op">mock</span>.<span class="fn">onEmbedding</span>(<span class="str">"my text"</span>, { <span class="prop">embedding</span>: [<span class="num">0.1</span>, <span class="num">0.2</span>] });
262+
<span class="op">mock</span>.<span class="fn">onImage</span>(<span class="str">"sunset"</span>, { <span class="prop">image</span>: { <span class="prop">url</span>: <span class="str">"https://example.com/sunset.png"</span> } });
263+
<span class="op">mock</span>.<span class="fn">onSpeech</span>(<span class="str">"hello"</span>, { <span class="prop">audio</span>: <span class="str">"SGVsbG8="</span> });
264+
<span class="op">mock</span>.<span class="fn">onTranscription</span>(<span class="str">"audio.mp3"</span>, { <span class="prop">transcription</span>: { <span class="prop">text</span>: <span class="str">"Hello"</span> } });
265+
<span class="op">mock</span>.<span class="fn">onVideo</span>(<span class="str">"cats"</span>, { <span class="prop">video</span>: { <span class="prop">url</span>: <span class="str">"https://example.com/cats.mp4"</span> } });
242266
<span class="op">mock</span>.<span class="fn">onJsonOutput</span>(<span class="str">"data"</span>, { <span class="prop">key</span>: <span class="str">"value"</span> });
243267
<span class="op">mock</span>.<span class="fn">onToolResult</span>(<span class="str">"call_123"</span>, { <span class="prop">content</span>: <span class="str">"Done"</span> });
244268

‎docs/images/index.html‎

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
<!doctype html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8" />
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
6+
<title>Image Generation — aimock</title>
7+
<link rel="icon" type="image/svg+xml" href="../favicon.svg" />
8+
<link rel="preconnect" href="https://fonts.googleapis.com" />
9+
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
10+
<link
11+
href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:ital,wght@0,300;0,400;0,500;0,600;0,700;1,400&family=Instrument+Sans:wght@400;500;600;700&display=swap"
12+
rel="stylesheet"
13+
/>
14+
<link rel="stylesheet" href="../style.css" />
15+
</head>
16+
<body>
17+
<nav class="top-nav">
18+
<div class="nav-inner">
19+
<div style="display: flex; align-items: center; gap: 1rem">
20+
<button
21+
class="sidebar-toggle"
22+
onclick="document.querySelector('.sidebar').classList.toggle('open')"
23+
aria-label="Toggle sidebar"
24+
>
25+
&#9776;
26+
</button>
27+
<a href="/" class="nav-brand"> <span class="prompt">$</span> aimock </a>
28+
</div>
29+
<ul class="nav-links">
30+
<li><a href="/">Home</a></li>
31+
<li><a href="/docs" style="color: var(--accent)">Docs</a></li>
32+
<li>
33+
<a href="https://github.com/CopilotKit/aimock" class="gh-link" target="_blank"
34+
><svg width="16" height="16" viewBox="0 0 16 16" fill="currentColor">
35+
<path
36+
d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"
37+
/>
38+
</svg>
39+
GitHub</a
40+
>
41+
</li>
42+
</ul>
43+
</div>
44+
</nav>
45+
46+
<div class="docs-layout">
47+
<aside class="sidebar" id="sidebar"></aside>
48+
49+
<main class="docs-content">
50+
<h1>Image Generation</h1>
51+
<p class="lead">
52+
The image generation endpoints support both OpenAI
53+
<code>POST /v1/images/generations</code> and Gemini Imagen
54+
<code>POST /v1beta/models/{model}:predict</code> formats. Return single or multiple images
55+
as URLs or base64-encoded data.
56+
</p>
57+
58+
<h2>Endpoints</h2>
59+
<table class="endpoint-table">
60+
<thead>
61+
<tr>
62+
<th>Method</th>
63+
<th>Path</th>
64+
<th>Format</th>
65+
</tr>
66+
</thead>
67+
<tbody>
68+
<tr>
69+
<td>POST</td>
70+
<td>/v1/images/generations</td>
71+
<td>JSON (OpenAI)</td>
72+
</tr>
73+
<tr>
74+
<td>POST</td>
75+
<td>/v1beta/models/{model}:predict</td>
76+
<td>JSON (Gemini Imagen)</td>
77+
</tr>
78+
</tbody>
79+
</table>
80+
81+
<h2>Unit Test: Single Image URL</h2>
82+
<p>
83+
Using the programmatic API with vitest, register a fixture and assert on the response.
84+
</p>
85+
86+
<div class="code-block">
87+
<div class="code-block-header">image-url.test.ts <span class="lang-tag">ts</span></div>
88+
<pre><code><span class="kw">import</span> { <span class="type">LLMock</span> } <span class="kw">from</span> <span class="str">"@copilotkit/aimock"</span>;
89+
<span class="kw">import</span> { <span class="fn">describe</span>, <span class="fn">it</span>, <span class="fn">expect</span>, <span class="fn">beforeAll</span>, <span class="fn">afterAll</span> } <span class="kw">from</span> <span class="str">"vitest"</span>;
90+
91+
<span class="kw">let</span> <span class="op">mock</span>: <span class="type">LLMock</span>;
92+
93+
<span class="fn">beforeAll</span>(<span class="kw">async</span> () <span class="kw">=&gt;</span> {
94+
<span class="op">mock</span> = <span class="kw">new</span> <span class="type">LLMock</span>();
95+
<span class="kw">await</span> <span class="op">mock</span>.<span class="fn">start</span>();
96+
});
97+
98+
<span class="fn">afterAll</span>(<span class="kw">async</span> () <span class="kw">=&gt;</span> {
99+
<span class="kw">await</span> <span class="op">mock</span>.<span class="fn">stop</span>();
100+
});
101+
102+
<span class="fn">it</span>(<span class="str">"returns a single image URL"</span>, <span class="kw">async</span> () <span class="kw">=&gt;</span> {
103+
<span class="op">mock</span>.<span class="fn">onImage</span>(<span class="str">"a sunset over mountains"</span>, {
104+
<span class="prop">image</span>: { <span class="prop">url</span>: <span class="str">"https://example.com/sunset.png"</span> },
105+
});
106+
107+
<span class="kw">const</span> <span class="op">res</span> = <span class="kw">await</span> <span class="fn">fetch</span>(<span class="str">`${mock.url}/v1/images/generations`</span>, {
108+
<span class="prop">method</span>: <span class="str">"POST"</span>,
109+
<span class="prop">headers</span>: { <span class="str">"Content-Type"</span>: <span class="str">"application/json"</span> },
110+
<span class="prop">body</span>: <span class="type">JSON</span>.<span class="fn">stringify</span>({
111+
<span class="prop">model</span>: <span class="str">"dall-e-3"</span>,
112+
<span class="prop">prompt</span>: <span class="str">"a sunset over mountains"</span>,
113+
<span class="prop">n</span>: <span class="num">1</span>,
114+
<span class="prop">size</span>: <span class="str">"1024x1024"</span>,
115+
}),
116+
});
117+
118+
<span class="kw">const</span> <span class="op">body</span> = <span class="kw">await</span> <span class="op">res</span>.<span class="fn">json</span>();
119+
<span class="fn">expect</span>(<span class="op">body</span>.<span class="prop">data</span>[<span class="num">0</span>].<span class="prop">url</span>).<span class="fn">toBe</span>(<span class="str">"https://example.com/sunset.png"</span>);
120+
});</code></pre>
121+
</div>
122+
123+
<h2>Unit Test: Multiple Images</h2>
124+
125+
<div class="code-block">
126+
<div class="code-block-header">
127+
image-multiple.test.ts <span class="lang-tag">ts</span>
128+
</div>
129+
<pre><code><span class="fn">it</span>(<span class="str">"returns multiple images"</span>, <span class="kw">async</span> () <span class="kw">=&gt;</span> {
130+
<span class="op">mock</span>.<span class="fn">onImage</span>(<span class="str">"cats"</span>, {
131+
<span class="prop">images</span>: [
132+
{ <span class="prop">url</span>: <span class="str">"https://example.com/cat1.png"</span> },
133+
{ <span class="prop">url</span>: <span class="str">"https://example.com/cat2.png"</span> },
134+
],
135+
});
136+
137+
<span class="kw">const</span> <span class="op">res</span> = <span class="kw">await</span> <span class="fn">fetch</span>(<span class="str">`${mock.url}/v1/images/generations`</span>, {
138+
<span class="prop">method</span>: <span class="str">"POST"</span>,
139+
<span class="prop">headers</span>: { <span class="str">"Content-Type"</span>: <span class="str">"application/json"</span> },
140+
<span class="prop">body</span>: <span class="type">JSON</span>.<span class="fn">stringify</span>({
141+
<span class="prop">model</span>: <span class="str">"dall-e-3"</span>,
142+
<span class="prop">prompt</span>: <span class="str">"cats playing"</span>,
143+
<span class="prop">n</span>: <span class="num">2</span>,
144+
}),
145+
});
146+
147+
<span class="kw">const</span> <span class="op">body</span> = <span class="kw">await</span> <span class="op">res</span>.<span class="fn">json</span>();
148+
<span class="fn">expect</span>(<span class="op">body</span>.<span class="prop">data</span>).<span class="fn">toHaveLength</span>(<span class="num">2</span>);
149+
<span class="fn">expect</span>(<span class="op">body</span>.<span class="prop">data</span>[<span class="num">0</span>].<span class="prop">url</span>).<span class="fn">toBe</span>(<span class="str">"https://example.com/cat1.png"</span>);
150+
<span class="fn">expect</span>(<span class="op">body</span>.<span class="prop">data</span>[<span class="num">1</span>].<span class="prop">url</span>).<span class="fn">toBe</span>(<span class="str">"https://example.com/cat2.png"</span>);
151+
});</code></pre>
152+
</div>
153+
154+
<h2>Unit Test: Base64 Response</h2>
155+
156+
<div class="code-block">
157+
<div class="code-block-header">image-base64.test.ts <span class="lang-tag">ts</span></div>
158+
<pre><code><span class="fn">it</span>(<span class="str">"returns base64-encoded image"</span>, <span class="kw">async</span> () <span class="kw">=&gt;</span> {
159+
<span class="op">mock</span>.<span class="fn">onImage</span>(<span class="str">"logo"</span>, {
160+
<span class="prop">image</span>: { <span class="prop">b64_json</span>: <span class="str">"iVBORw0KGgoAAAANSUhEUg..."</span> },
161+
});
162+
163+
<span class="kw">const</span> <span class="op">res</span> = <span class="kw">await</span> <span class="fn">fetch</span>(<span class="str">`${mock.url}/v1/images/generations`</span>, {
164+
<span class="prop">method</span>: <span class="str">"POST"</span>,
165+
<span class="prop">headers</span>: { <span class="str">"Content-Type"</span>: <span class="str">"application/json"</span> },
166+
<span class="prop">body</span>: <span class="type">JSON</span>.<span class="fn">stringify</span>({
167+
<span class="prop">model</span>: <span class="str">"dall-e-3"</span>,
168+
<span class="prop">prompt</span>: <span class="str">"a company logo"</span>,
169+
<span class="prop">response_format</span>: <span class="str">"b64_json"</span>,
170+
}),
171+
});
172+
173+
<span class="kw">const</span> <span class="op">body</span> = <span class="kw">await</span> <span class="op">res</span>.<span class="fn">json</span>();
174+
<span class="fn">expect</span>(<span class="op">body</span>.<span class="prop">data</span>[<span class="num">0</span>].<span class="prop">b64_json</span>).<span class="fn">toBeDefined</span>();
175+
});</code></pre>
176+
</div>
177+
178+
<h2>Unit Test: Gemini Imagen Format</h2>
179+
180+
<div class="code-block">
181+
<div class="code-block-header">image-gemini.test.ts <span class="lang-tag">ts</span></div>
182+
<pre><code><span class="fn">it</span>(<span class="str">"handles Gemini Imagen predict endpoint"</span>, <span class="kw">async</span> () <span class="kw">=&gt;</span> {
183+
<span class="op">mock</span>.<span class="fn">onImage</span>(<span class="str">"landscape"</span>, {
184+
<span class="prop">image</span>: { <span class="prop">url</span>: <span class="str">"https://example.com/landscape.png"</span> },
185+
});
186+
187+
<span class="kw">const</span> <span class="op">res</span> = <span class="kw">await</span> <span class="fn">fetch</span>(
188+
<span class="str">`${mock.url}/v1beta/models/imagen-3.0-generate-002:predict`</span>,
189+
{
190+
<span class="prop">method</span>: <span class="str">"POST"</span>,
191+
<span class="prop">headers</span>: { <span class="str">"Content-Type"</span>: <span class="str">"application/json"</span> },
192+
<span class="prop">body</span>: <span class="type">JSON</span>.<span class="fn">stringify</span>({
193+
<span class="prop">instances</span>: [{ <span class="prop">prompt</span>: <span class="str">"a beautiful landscape"</span> }],
194+
<span class="prop">parameters</span>: { <span class="prop">sampleCount</span>: <span class="num">1</span> },
195+
}),
196+
}
197+
);
198+
199+
<span class="kw">const</span> <span class="op">body</span> = <span class="kw">await</span> <span class="op">res</span>.<span class="fn">json</span>();
200+
<span class="fn">expect</span>(<span class="op">body</span>.<span class="prop">predictions</span>).<span class="fn">toBeDefined</span>();
201+
});</code></pre>
202+
</div>
203+
204+
<h2>JSON Fixture</h2>
205+
206+
<div class="code-block">
207+
<div class="code-block-header">
208+
fixtures/images.json <span class="lang-tag">json</span>
209+
</div>
210+
<pre><code>{
211+
<span class="key">"fixtures"</span>: [
212+
{
213+
<span class="key">"match"</span>: { <span class="key">"userMessage"</span>: <span class="str">"sunset"</span> },
214+
<span class="key">"response"</span>: {
215+
<span class="key">"image"</span>: { <span class="key">"url"</span>: <span class="str">"https://example.com/sunset.png"</span> }
216+
}
217+
},
218+
{
219+
<span class="key">"match"</span>: { <span class="key">"userMessage"</span>: <span class="str">"cats"</span> },
220+
<span class="key">"response"</span>: {
221+
<span class="key">"images"</span>: [
222+
{ <span class="key">"url"</span>: <span class="str">"https://example.com/cat1.png"</span> },
223+
{ <span class="key">"url"</span>: <span class="str">"https://example.com/cat2.png"</span> }
224+
]
225+
}
226+
}
227+
]
228+
}</code></pre>
229+
</div>
230+
231+
<h2>Response Format</h2>
232+
<p>Matches the OpenAI <code>/v1/images/generations</code> response format:</p>
233+
<ul>
234+
<li><code>created</code> &mdash; Unix timestamp</li>
235+
<li>
236+
<code>data[].url</code> &mdash; URL of the generated image (when using URL format)
237+
</li>
238+
<li>
239+
<code>data[].b64_json</code> &mdash; base64-encoded image data (when using b64_json
240+
format)
241+
</li>
242+
<li>
243+
<code>data[].revised_prompt</code> &mdash; the prompt as revised by the model (optional)
244+
</li>
245+
</ul>
246+
247+
<div class="info-box">
248+
<p>
249+
Image fixtures use <code>match.userMessage</code> which maps to the
250+
<code>prompt</code> field in the request body. The <code>prompt</code> matcher checks
251+
for substring matches.
252+
</p>
253+
</div>
254+
255+
<h2>Record &amp; Replay</h2>
256+
<p>
257+
When no fixture matches an incoming request, aimock can proxy it to the real API and
258+
record the response as a fixture for future replays. Enable recording with the
259+
<code>--record</code> flag or via <code>RecordConfig</code> in the programmatic API.
260+
Recorded image fixtures capture the <code>url</code> or <code>b64_json</code> from the
261+
provider response and save them to disk, so subsequent runs replay instantly without
262+
hitting the real API.
263+
</p>
264+
265+
<div class="code-block">
266+
<div class="code-block-header">CLI <span class="lang-tag">sh</span></div>
267+
<pre><code>npx aimock --record --provider-openai https://api.openai.com</code></pre>
268+
</div>
269+
</main>
270+
<aside class="page-toc" id="page-toc"></aside>
271+
</div>
272+
<footer class="docs-footer">
273+
<div class="footer-inner">
274+
<div class="footer-left"><span>$</span> aimock &middot; MIT License</div>
275+
<ul class="footer-links">
276+
<li><a href="https://github.com/CopilotKit/aimock" target="_blank">GitHub</a></li>
277+
<li>
278+
<a href="https://www.npmjs.com/package/@copilotkit/aimock" target="_blank">npm</a>
279+
</li>
280+
</ul>
281+
</div>
282+
</footer>
283+
<script src="../sidebar.js"></script>
284+
<script src="../cli-tabs.js"></script>
285+
</body>
286+
</html>

0 commit comments

Comments
 (0)