Skip to content

Commit 75dbb72

Browse files
authored
Timing-aware recording and replay with --replay-speed multiplier (#223)
## Summary Records per-frame arrival timestamps during proxy recording and replays them at configurable speed. Requested by Atai. - **Recording:** SSE frame parser captures per-frame timestamps. NDJSON uses `\n` delimiter. Bedrock binary EventStream uses 4-byte length-prefix frame detection. Stores `recordedTimings { ttftMs, interChunkDelaysMs, totalDurationMs }` on fixtures. - **Replay:** `calculateDelay` extended with `recordedTimings` + `replaySpeed` params. Precedence: streamingProfile > recordedTimings > global latency. Speed multiplier divides all delays. - **CLI:** `--replay-speed 2.0` (2x faster). Per-fixture `replaySpeed` override. - **Coverage:** All 11 streaming paths — 8 SSE/NDJSON/EventStream writers + 3 WebSocket handlers. All providers including Bedrock. - **Validation:** Per-fixture `replaySpeed <= 0` stripped with warning. NaN/negative values in `recordedTimings` sanitized. Approximate timing reproduction based on recorded TTFT and inter-frame cadence. Replay chunk count may differ from recording — TTFT and average pace are preserved, not per-token fidelity. **Spec:** https://www.notion.so/3643aa38185281328fe6ddf27407c6ed ## Test plan - [x] 3097 tests pass (9 new across 2 test files) - [x] TypeScript clean - [x] Lint + format + build clean - [x] CR Round 1: 1 finding fixed (Bedrock/Ollama defaults fallback) - [x] CR Round 2: 7/7 confirmed 0 findings - [x] CR Round 3 adversarial: 6 findings — fixed eventIndex reset, NaN validation, Bedrock capture, docs language - [x] Positive control tests for precedence and replaySpeed - [x] Backward compat verified 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents a20e4f6 + 3c0a6f3 commit 75dbb72

28 files changed

Lines changed: 1130 additions & 44 deletions

.claude-plugin/marketplace.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
"source": {
1010
"source": "npm",
1111
"package": "@copilotkit/aimock",
12-
"version": "^1.25.0"
12+
"version": "^1.26.0"
1313
},
1414
"description": "Fixture authoring skill for @copilotkit/aimock — LLM, multimedia (image/TTS/transcription/video), MCP, A2A, AG-UI, vector, embeddings, structured output, sequential responses, streaming physics, record/replay, agent loop patterns, and debugging"
1515
}

.claude-plugin/plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "aimock",
3-
"version": "1.25.0",
3+
"version": "1.26.0",
44
"description": "Fixture authoring guidance for @copilotkit/aimock — LLM, multimedia, MCP, A2A, AG-UI, vector, and service mocking",
55
"author": {
66
"name": "CopilotKit"

CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,15 @@
44

55
### Added
66

7+
- **Timing-aware recording and replay** — proxy recording captures per-frame
8+
arrival timestamps as `recordedTimings` on fixtures. Replay uses recorded
9+
timings for approximate timing reproduction based on recorded TTFT and
10+
inter-frame cadence instead of the synthetic model. Replay chunk count may
11+
differ from recording chunk count — TTFT and average pace are preserved,
12+
not per-token fidelity. `--replay-speed N` multiplier applies to all delay
13+
sources (recorded timings, streaming profiles, global latency). Per-fixture
14+
`replaySpeed` override. Covers SSE, NDJSON, Bedrock EventStream, and
15+
WebSocket protocols.
716
- **Gemini `embedContent` endpoint**`POST /v1beta/models/{model}:embedContent`
817
with deterministic fallback embeddings and fixture matching
918
- **`/v1/images/edit` and `/v1/images/variations` endpoints** — multipart

README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or
4949
## Features
5050

5151
- **[Record & Replay](https://aimock.copilotkit.dev/record-replay)** — Proxy real APIs, save as fixtures, replay deterministically forever
52+
- **Timing-aware recording and replay** — Recorded fixtures capture per-frame arrival timestamps; replay uses recorded timings for approximate timing reproduction based on recorded TTFT and inter-frame cadence (replay chunk count may differ from recording — TTFT and average pace are preserved, not per-token fidelity) with configurable `--replay-speed` multiplier
5253
- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `sequenceIndex`, `systemMessage` (gate on host-supplied agent context), or custom predicates
5354
- **[14 LLM Providers](https://aimock.copilotkit.dev/docs)** — OpenAI Chat, OpenAI Responses, OpenAI Realtime (GA + Beta shim), Claude, Gemini (REST + embedContent), Gemini Live, Gemini Interactions, Azure, Bedrock, Vertex AI, Ollama (chat + embeddings), Cohere (chat + embed), ElevenLabs TTS — full streaming support
5455
- **Multimedia APIs**[image generation](https://aimock.copilotkit.dev/images) (DALL-E, Imagen), [image editing](https://aimock.copilotkit.dev/images) (/v1/images/edit), [text-to-speech](https://aimock.copilotkit.dev/speech) (OpenAI + ElevenLabs), [audio transcription](https://aimock.copilotkit.dev/transcription), [audio translation](https://aimock.copilotkit.dev/transcription) (/v1/audio/translations), [video generation](https://aimock.copilotkit.dev/video), [fal.ai](https://aimock.copilotkit.dev/fal-ai) (image / video / audio with queue lifecycle)
@@ -101,6 +102,9 @@ npx -p @copilotkit/aimock llmock --record --provider-openai https://api.openai.c
101102
npx -p @copilotkit/aimock llmock --record --provider-openai https://api.openai.com \
102103
--body-timeout-ms 180000
103104

105+
# Replay recorded fixtures at 2× speed
106+
npx -p @copilotkit/aimock llmock -p 4010 -f ./fixtures --replay-speed 2
107+
104108
# Convert fixtures from other tools
105109
npx @copilotkit/aimock convert vidaimock ./templates/ ./fixtures/
106110
npx @copilotkit/aimock convert mockllm ./config.yaml ./fixtures/

charts/aimock/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@ name: aimock
33
description: Mock infrastructure for AI application testing (OpenAI, Anthropic, Gemini, MCP, A2A, vector)
44
type: application
55
version: 0.1.0
6-
appVersion: "1.25.0"
6+
appVersion: "1.26.0"

package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@copilotkit/aimock",
3-
"version": "1.25.0",
3+
"version": "1.26.0",
44
"description": "Mock infrastructure for AI application testing — LLM APIs, image generation, text-to-speech, transcription, audio generation, video generation, MCP tools, A2A agents, AG-UI event streams, vector databases, search, rerank, and moderation. One package, one port, zero dependencies.",
55
"license": "MIT",
66
"keywords": [

0 commit comments

Comments
 (0)