|
| 1 | +# E2E Test Fixture Workflow |
| 2 | + |
| 3 | +E2E tests run against `@copilotkit/aimock` (`LLMock`) — a local HTTP server that replays recorded LLM responses. This makes tests free, deterministic, and CI-friendly. |
| 4 | + |
| 5 | +## How aimock matching works |
| 6 | + |
| 7 | +Fixtures are matched by **substring**: `incoming_last_user_message.includes(fixture.match.userMessage)`. A fixture fires if its match string appears _anywhere_ in the last user message of the API request. |
| 8 | + |
| 9 | +**Critical**: the last user message always contains `<environment_details>` with the current time. Never use a match string that includes a timestamp — it will stop matching on the next run. |
| 10 | + |
| 11 | +Record mode uses **record-on-miss**: if an existing fixture already matches a request, aimock serves it and does **not** re-record. Only unmatched requests are proxied to the real API and saved as `openai-*.json` files. |
| 12 | + |
| 13 | +## Adding a fixture for a new test |
| 14 | + |
| 15 | +1. Write the test in `src/suite/`. Use short, stable, unique text in the task prompt. |
| 16 | + |
| 17 | +2. Clear any stale auto-recorded files first (they accumulate across record runs): |
| 18 | + |
| 19 | + ```sh |
| 20 | + git clean -fx apps/vscode-e2e/fixtures/ |
| 21 | + ``` |
| 22 | + |
| 23 | + The `-x` flag is required because `openai-*.json` files are gitignored — `git clean -f` alone silently skips them. |
| 24 | + |
| 25 | +3. Record fixtures (requires an OpenRouter API key with credits): |
| 26 | + |
| 27 | + ```sh |
| 28 | + OPENROUTER_API_KEY=<key> pnpm --filter @roo-code/vscode-e2e test:record |
| 29 | + ``` |
| 30 | + |
| 31 | + This proxies unmatched requests to OpenRouter and writes `fixtures/openai-*.json`. Background |
| 32 | + calls from the extension will also be recorded here — that's expected, ignore them. |
| 33 | +
|
| 34 | +4. Find the auto-recorded file for your test: |
| 35 | +
|
| 36 | + ```sh |
| 37 | + grep -l "your unique prompt text" apps/vscode-e2e/fixtures/openai-*.json |
| 38 | + ``` |
| 39 | +
|
| 40 | +5. Inspect it to find the `response` block (tool calls the LLM made). |
| 41 | +
|
| 42 | +6. Create a named fixture file, e.g. `fixtures/my-feature.json`, with a **short stable match string**: |
| 43 | +
|
| 44 | + ```json |
| 45 | + { |
| 46 | + "fixtures": [ |
| 47 | + { |
| 48 | + "match": { "userMessage": "your unique prompt text" }, |
| 49 | + "response": { |
| 50 | + "toolCalls": [ |
| 51 | + { "name": "attempt_completion", "arguments": "{\"result\":\"...\"}", "id": "call_001" } |
| 52 | + ] |
| 53 | + } |
| 54 | + } |
| 55 | + ] |
| 56 | + } |
| 57 | + ``` |
| 58 | +
|
| 59 | + The match string should be unique enough to identify this request but contain **no timestamps, file paths, or environment details**. |
| 60 | +
|
| 61 | +7. Delete the `openai-*.json` files — they're gitignored and can't be replayed. |
| 62 | +
|
| 63 | +8. Verify in mock mode (no API key needed): |
| 64 | + ```sh |
| 65 | + pnpm --filter @roo-code/vscode-e2e test:ci:mock |
| 66 | + ``` |
| 67 | +
|
| 68 | +## Multi-turn tests |
| 69 | +
|
| 70 | +If the LLM calls a tool first (e.g. `read_file`) and then calls `attempt_completion` after seeing the result, you need two fixtures: |
| 71 | +
|
| 72 | +- **Turn 1**: match on the task prompt → respond with the tool call |
| 73 | +- **Turn 2**: match on a stable part of the tool _result_ → respond with `attempt_completion` |
| 74 | +
|
| 75 | +The tool result is provided by the extension (not the mock), so its content is deterministic if test files have stable names. Use a stable substring from the tool result as the turn-2 match string. |
| 76 | +
|
| 77 | +## 404 errors in logs are expected |
| 78 | +
|
| 79 | +Background API calls from the extension (usage collection, initialization) hit aimock with no matching fixture and return 404. These do **not** affect test results — the tests still pass. You'll see `[OpenRouter] API error: { message: '404 No fixture matched' }` in the output; this is normal. |
| 80 | +
|
| 81 | +## Running tests |
| 82 | +
|
| 83 | +| Command | Purpose | |
| 84 | +| ------------------------------------------------------------------------- | ------------------------------------------------------------------ | |
| 85 | +| `pnpm --filter @roo-code/vscode-e2e test:ci:mock` | Replay mode — no API key needed, uses fixtures | |
| 86 | +| `OPENROUTER_API_KEY=<key> pnpm --filter @roo-code/vscode-e2e test:record` | Record mode — proxies to real API, writes `openai-*.json` | |
| 87 | +| `OPENROUTER_API_KEY=<key> pnpm --filter @roo-code/vscode-e2e test:ci` | Real-API mode — runs against live OpenRouter (for drift detection) | |
| 88 | +
|
| 89 | +## Programmatic fixtures (regex matching) |
| 90 | +
|
| 91 | +For requests that can't be matched by a stable substring (e.g. "starts with `<environment_details>` but not preceded by a user message"), add a programmatic fixture in `src/runTest.ts` using `mock.addFixture()` with a `RegExp` match. These are only available in replay mode and are not recorded. |
0 commit comments