Skip to content

Commit 2c72188

Browse files
committed
docs(e2e): clarify subtask drain invariants from code review
1 parent b3f3347 commit 2c72188

3 files changed

Lines changed: 24 additions & 8 deletions

File tree

apps/vscode-e2e/AGENTS.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,17 @@ Example:
121121
122122
The `model` field can be added to either match when a test targets a specific model.
123123
124+
## Delaying a fixture response (simulating a slow or hung provider)
125+
126+
Use `streamingProfile: { ttft: <ms> }` on a fixture, not a flat `latency: <ms>`, when a test needs
127+
to simulate a slow or hung provider (e.g. to cancel an in-flight request mid-stream). `ttft` delays
128+
only the first SSE chunk, so the pending window is exactly the configured value. Flat `latency`
129+
delays _every_ chunk, and aimock never observes client disconnects — after a test cancels the
130+
request, a flat-latency stream keeps flushing chunks server-side for `chunks × latency` before
131+
reaching the dead socket, which can interleave with the next test's request against the same mock
132+
server. See `SUBTASK_API_HANG_RESPONSE_LATENCY_MS` in `fixtures/subtasks.ts` for an example,
133+
including the bounded post-test drain the calling suite uses to wait out that window.
134+
124135
## 404 errors in logs are expected
125136

126137
Background API calls from the extension (usage collection, initialization) hit aimock with no matching fixture and return 404. These do **not** affect test results — the tests still pass. You'll see `[OpenRouter] API error: { message: '404 No fixture matched' }` in the output; this is normal.

apps/vscode-e2e/src/fixtures/subtasks.ts

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ export const SUBTASK_API_HANG_PARENT_RESULT = "API hang parent resumed"
3535

3636
// How long the API-hang child's first mocked response stays pending before its first SSE
3737
// chunk. Shared with the subtask suite so its post-test drain waits exactly one window.
38+
// Correctness depends on no flat `latency` (fixture or LLMock default) being set on that
39+
// fixture — a flat latency would apply to every chunk after the first, not just the ttft.
3840
export const SUBTASK_API_HANG_RESPONSE_LATENCY_MS = 15_000
3941

4042
// Abandon-subtask scenario (#559) — separate markers to avoid sequenceIndex collisions with the
@@ -270,9 +272,8 @@ export function addSubtaskFixtures(mock: InstanceType<typeof LLMock>) {
270272
// applies `latency` to EVERY chunk and never observes client disconnects, so after the test
271273
// cancels, a flat-latency stream would stay pending server-side for chunks × latency before
272274
// flushing to the dead socket. With ttft the pending window is exactly
273-
// SUBTASK_API_HANG_RESPONSE_LATENCY_MS, which is what the suite's post-test drain waits out.
274-
// This relies on no flat `latency` (fixture or LLMock default) being set — a flat latency
275-
// would still apply to every chunk after the first.
275+
// SUBTASK_API_HANG_RESPONSE_LATENCY_MS (see its doc comment for the no-flat-latency
276+
// invariant this relies on), which is what the suite's post-test drain waits out.
276277
streamingProfile: { ttft: SUBTASK_API_HANG_RESPONSE_LATENCY_MS },
277278
response: {
278279
toolCalls: [

apps/vscode-e2e/src/suite/subtasks.test.ts

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,12 @@ const findAimockRequest = (entries: AimockJournalEntry[], expectedText: string,
7070
)
7171
})
7272

73-
// Returns the journal timestamp of the matching request so callers can anchor
74-
// post-test drains to the exact request this test created.
75-
const waitForAimockRequestContaining = async (expectedText: string, excludeText?: string) => {
73+
// Waits for a matching request to appear in the aimock journal and returns its journal
74+
// timestamp, so callers can anchor post-test drains to the exact request this test created.
75+
const waitForAimockRequestContaining = async (
76+
expectedText: string,
77+
excludeText?: string,
78+
): Promise<number | undefined> => {
7679
let matchedAt: number | undefined
7780

7881
await waitFor(async () => {
@@ -83,8 +86,9 @@ const waitForAimockRequestContaining = async (expectedText: string, excludeText?
8386
return matchedAt
8487
}
8588

86-
// Grace period after the delayed window for aimock to flush the stream's remaining
87-
// chunks to the dead socket.
89+
// Grace period after the delayed window for aimock to flush the stream's remaining chunks to
90+
// the dead socket. 500ms is an empirical margin for that flush plus socket teardown; if this
91+
// suite becomes flaky again on slow CI runners, widen this value first.
8892
const SUBTASK_API_HANG_DRAIN_GRACE_MS = 500
8993

9094
// aimock does not observe client disconnects: after the API-hang child request is cancelled,

0 commit comments

Comments
 (0)