Skip to content

Commit 1de6f7f

Browse files
committed
docs: reframe drift detection for users + add drift report output
- Headline: 'Your mocks never go stale' - Lead reframed for users, not maintainers - Restored drift report output block from v1.5.0 homepage - Removed 'Running Drift Tests' key export section - Feature card: 'Fixtures stay accurate as providers evolve'
1 parent 2bf6bc3 commit 1de6f7f

2 files changed

Lines changed: 95 additions & 27 deletions

File tree

docs/drift-detection.html

Lines changed: 83 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -281,20 +281,91 @@ <h4>All three agree</h4>
281281
</div>
282282
</div>
283283

284-
<h2>Running Drift Tests</h2>
284+
<h2>What Drift Looks Like</h2>
285285

286-
<div class="code-block">
287-
<div class="code-block-header">Run drift tests <span class="lang-tag">shell</span></div>
288-
<pre><code><span class="cm"># Set API keys for providers you want to test</span>
289-
<span class="kw">export</span> OPENAI_API_KEY=sk-...
290-
<span class="kw">export</span> ANTHROPIC_API_KEY=sk-ant-...
291-
<span class="kw">export</span> GOOGLE_API_KEY=AI...
292-
293-
<span class="cm"># Run all drift tests</span>
294-
pnpm test:drift
286+
<style>
287+
.drift-report {
288+
background: var(--bg-deep);
289+
border: 1px solid var(--border);
290+
border-radius: 8px;
291+
padding: 1.25rem 1.5rem;
292+
margin-top: 2rem;
293+
font-family: var(--font-mono);
294+
font-size: 0.75rem;
295+
line-height: 1.8;
296+
color: var(--text-secondary);
297+
overflow-x: auto;
298+
}
299+
.drift-report .report-header {
300+
color: var(--text-primary);
301+
font-weight: 600;
302+
margin-bottom: 0.75rem;
303+
font-size: 0.8rem;
304+
}
305+
.drift-report .severity-critical {
306+
color: var(--error);
307+
}
308+
.drift-report .severity-warning {
309+
color: var(--warning);
310+
}
311+
.drift-report .severity-ok {
312+
color: var(--accent);
313+
}
314+
.drift-report .field-path {
315+
color: var(--blue);
316+
}
317+
.drift-report .drift-label {
318+
color: var(--text-primary);
319+
}
320+
.drift-report .report-summary {
321+
color: var(--text-dim);
322+
}
323+
.drift-report .field-label {
324+
color: var(--text-dim);
325+
}
326+
.drift-report .divider {
327+
border-top: 1px solid var(--border);
328+
margin: 0.6rem 0;
329+
}
330+
</style>
295331

296-
<span class="cm"># Run for a specific provider</span>
297-
pnpm test:drift -- --grep "OpenAI Chat"</code></pre>
332+
<div class="drift-report">
333+
<div class="report-header">$ pnpm test:drift</div>
334+
<span class="severity-critical">[critical]</span>
335+
<span class="drift-label">AIMOCK DRIFT</span> &mdash; field in SDK + real API but missing
336+
from mock<br />
337+
<span class="field-label">Path:</span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="field-path"
338+
>choices[].message.refusal</span
339+
><br />
340+
<span class="field-label">SDK:</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;null &nbsp;&nbsp;
341+
<span class="field-label">Real:</span> null &nbsp;&nbsp;
342+
<span class="field-label">Mock:</span> &lt;absent&gt;<br />
343+
<div class="divider"></div>
344+
<span class="severity-critical">[critical]</span>
345+
<span class="drift-label">TYPE MISMATCH</span> &mdash; real API and mock disagree on
346+
type<br />
347+
<span class="field-label">Path:</span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="field-path"
348+
>content[].input</span
349+
><br />
350+
<span class="field-label">SDK:</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;object &nbsp;&nbsp;
351+
<span class="field-label">Real:</span> object &nbsp;&nbsp;
352+
<span class="field-label">Mock:</span> string<br />
353+
<div class="divider"></div>
354+
<span class="severity-warning">[warning]</span>
355+
<span class="drift-label">PROVIDER ADDED FIELD</span> &mdash; in real API but not in SDK
356+
or mock<br />
357+
<span class="field-label">Path:</span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="field-path"
358+
>choices[].message.annotations</span
359+
><br />
360+
<span class="field-label">SDK:</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;absent&gt;
361+
&nbsp;&nbsp; <span class="field-label">Real:</span> array &nbsp;&nbsp;
362+
<span class="field-label">Mock:</span> &lt;absent&gt;<br />
363+
<div class="divider"></div>
364+
<span class="severity-ok" style="font-size: 0.85rem">&#10003;</span>
365+
<span class="report-summary"
366+
>2 critical (test fails) &middot; 1 warning (logged) &middot; detected before any user
367+
reported it</span
368+
>
298369
</div>
299370

300371
<h2>Test Files</h2>

docs/index.html

Lines changed: 12 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1363,10 +1363,7 @@ <h3>Chaos Testing</h3>
13631363
<div class="feature-card fade-in" style="transition-delay: 0.4s">
13641364
<div class="feature-icon">&#128202;</div>
13651365
<h3>Drift Detection</h3>
1366-
<p>
1367-
Daily CI validation against real APIs. Know immediately when provider behavior changes
1368-
break your fixtures.
1369-
</p>
1366+
<p>Fixtures stay accurate as providers evolve. Fixes ship before your tests break.</p>
13701367
</div>
13711368
</div>
13721369
</div>
@@ -1375,35 +1372,35 @@ <h3>Drift Detection</h3>
13751372
<!-- ─── Section 6: Drift Detection / Reliability ──────────────── -->
13761373
<section class="section-reliability" id="reliability">
13771374
<div class="container">
1378-
<h2 class="fade-in">Verified against real APIs, every day</h2>
1375+
<h2 class="fade-in">Your mocks never go stale</h2>
13791376
<p class="reliability-sub fade-in" style="transition-delay: 0.1s">
1380-
aimock's drift detection runs daily against live provider APIs. When response formats
1381-
change, you know immediately &mdash; not when your tests break in production.
1377+
LLM providers change their response formats without warning. aimock catches it within 24
1378+
hours and ships the fix before your tests break.
13821379
</p>
13831380

13841381
<div class="drift-flow fade-in" style="transition-delay: 0.2s">
13851382
<div class="drift-step">
13861383
<div class="drift-step-num">1</div>
13871384
<h3>Real API calls</h3>
13881385
<p>
1389-
Daily CI hits actual OpenAI, Anthropic, Gemini endpoints to capture current response
1390-
formats.
1386+
Daily CI hits actual OpenAI, Anthropic, Gemini endpoints to detect response format
1387+
changes.
13911388
</p>
13921389
</div>
13931390
<div class="drift-step">
13941391
<div class="drift-step-num">2</div>
1395-
<h3>Response validation</h3>
1392+
<h3>Compare &amp; detect</h3>
13961393
<p>
1397-
Compares real responses against aimock's fixture format. Schema changes are caught
1398-
instantly.
1394+
Response schemas are validated against aimock's builders. Any drift is flagged
1395+
automatically.
13991396
</p>
14001397
</div>
14011398
<div class="drift-step">
14021399
<div class="drift-step-num">3</div>
1403-
<h3>Auto-remediation</h3>
1400+
<h3>Fix &amp; ship</h3>
14041401
<p>
1405-
Drift detected &rarr; PR opened &rarr; fixtures, skills, and docs updated
1406-
automatically. Zero manual effort.
1402+
Drift detected &rarr; fixtures, builders, skills, and docs updated &rarr; PR shipped.
1403+
Always current, within a day.
14071404
</p>
14081405
</div>
14091406
</div>

0 commit comments

Comments
 (0)