Skip to content

Commit fe5181b

Browse files
authored
Developer-session e2e: chat-theater recordings, focus timeline, one-mp4 films (#962)
* Developer-session e2e: real OpenCode chat on camera via a replay brain A scenario now runs the way a developer actually tests the product: chat with the real OpenCode TUI in a recorded PTY, the agent registers an API and drops a connect link in the chat (real MCP tool execution), a browser opens the link and pastes a real emulator-minted key while the terminal session stays open, and the chat resumes to verify the connection with a live send confirmed by the provider emulator's request ledger. - e2e/src/clients/replay-brain.ts: local OpenAI-wire chat-completions server (SSE + tool calls) scripted per scenario by transcript inspection, so approval pauses and OpenCode side requests don't derail it. Tool names resolve by suffix against whatever the agent offers. - clients/opencode.ts: makeOpenCodeHome optionally wires a 'replay' provider (openai-compatible) at the brain and pre-allows permissions. - scenarios/connect-handoff-session.test.ts: one continuous terminal.cast timeline with the browser hop forked concurrently; chat-brain.json transcript written as a run artifact even on failure. * Chat theater: agent-chat recordings over real mcporter calls, no inference The developer-session recording no longer needs a third-party agent binary or a faked LLM: the 'agent' is a chat presentation (agent-chat-tui.ts, a renderer in the recorded PTY) driven by chat-theater.ts while the scenario makes REAL mcporter MCP calls — OAuth, execute, approval pause/resume all genuine, each on-screen tool spinner bracketing the actual call it narrates. Runs anywhere bun runs (no opencode capability), ~28s on cloud. replay-brain.ts stays for the distinct tier where the third-party client's own protocol behavior is under test (OpenCode/Claude Code); real-inference evals remain a separate axis (performance distributions, not pass/fail scenarios). * Viewer: a run with both recordings plays as one session in time order Runs from chat-theater scenarios produce a terminal.cast (the chat) AND a session.mp4 (the browser hop). The run page previously picked one (video ?? cast), hiding the chat that led to the browser. Now both render under a single 'session' tab in story order — 1 · the chat, in the terminal; 2 · the browser hop, mid-chat — with autoplay reserved for video-only runs. * The Desk: film a scenario on one virtual desktop, one recording e2e/desk/run.sh builds a Linux image (Xvfb + openbox + xterm + headed Chromium + ffmpeg) and runs a scenario inside it with E2E_DESK=1: the chat renderer opens in a visible UTF-8 xterm (events over a FIFO instead of a PTY), the browser hop is a real headed window raised over it, and a single x11grab films the whole screen for the whole run — no splicing. The film replaces session.mp4 in the scenario's run dir; the scenario file itself is unchanged (chat-theater switches transports on E2E_DESK). First run builds the image and fills a node_modules volume; subsequent films of the connect-handoff session take ~2 minutes end to end. * film.ts: the session as one mp4 — full-screen cuts, no virtual desktop bun scripts/film.ts runs/<target>/<slug> turns a chat-theater run's two real recordings into one video that plays like a screen capture of a developer tabbing between full-screen windows: the cast (rendered without idle compression so cast time == video time) is cut at the browser hop — located by the scenario's narrator line, falling back to the largest output gap — and the browser recording fills the gap. Both segments are the genuine recordings; only the cut is editorial, exactly like tabbing. The film registers itself in result.json and the viewer plays film.mp4 as the session when present. Runs anywhere (agg + ffmpeg) — no Docker, no display server; the Desk remains for when actual window layering matters. * Focus is derived from operations: surfaces emit the run's focus timeline Acting on a window IS focusing it: a Playwright step marks the browser focused, a chat-theater event marks the terminal focused. Surfaces record these transitions (plus each recording's start anchor) into the run's timeline.json as a side effect of normal driving — no scenario changes, no declared focus anywhere. film.ts now builds its cut list from that timeline (any number of hops, each act mapped into its recording's own clock via the anchors), keeping the narrator-line/largest-gap heuristic only as a fallback for runs without one. The film is the operations, edited by where they happened. * Developer-session scenario adds Resend by served spec URL only The emulator serves its own OpenAPI document, so the scenario stops hand-writing a Resend subset and an explicit auth template — addSpec gets just the spec URL, exactly what an agent would send, and the platform derives base URL and the paste-a-token method from the spec itself. * Chat theater tool calls render like a real agent TUI chat.tool now takes the call itself — tool name, the REAL input (the sandbox code the scenario executes, shown as the indented preview block agent TUIs draw), and a result-line renderer — instead of a prose label. The renderer paints name → input block → live spinner → ✓ with the real duration and the real result's first line. The recording reads like an actual agent session: what was called, with what, how long it took, what came back. * pr-media prefers film.mp4 — the whole session, not just the browser hop * Auto-film runs that produce both recordings film.mp4 was a manual post-step, so any re-run left the viewer showing the parts (cast + browser video) instead of the spliced session. The runner now invokes scripts/film.ts after every successful run that has both terminal.cast and session.mp4 — best-effort, so missing agg/ffmpeg degrades back to the parts rather than failing the run. Every developer- session run's page now plays the single film by default. * Pin vendor/emulate at the OpenAPI-serving emulators Every REST emulator now serves its own OpenAPI document; the connect scenarios register integrations from those live spec URLs instead of hand-written specs.
1 parent f15f33d commit fe5181b

16 files changed

Lines changed: 1232 additions & 15 deletions

File tree

e2e/AGENTS.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,3 +133,32 @@ When handing results to the user, follow the evidence contract in the root
133133
values, not booleans — `expect(list).toContain(x)`, never
134134
`expect(list.includes(x)).toBe(true)` — so failures show the data.
135135
- Keep it deterministic: no sleeps; wait on conditions.
136+
137+
## Developer-session recordings (chat theater + desk)
138+
139+
Some scenarios are meant to be WATCHED — they show the product the way a
140+
developer actually uses it. Three tiers, pick deliberately:
141+
142+
1. **Chat theater** (`src/clients/chat-theater.ts`): the default for
143+
product-flow recordings. The "agent" is a chat renderer in a recorded
144+
PTY; every tool spinner brackets a REAL mcporter MCP call (OAuth,
145+
execute, approval resume). No inference, no third-party binary.
146+
Exemplar: `scenarios/connect-handoff-session.test.ts`. Artifacts:
147+
`terminal.cast` (the chat) + `session.mp4` (browser hops); the viewer
148+
plays them in story order.
149+
2. **Replay brain + real client** (`src/clients/replay-brain.ts`): when the
150+
third-party CLIENT's behavior is under test (OpenCode/Claude Code
151+
protocol handling). A scripted OpenAI-wire server plays the LLM; the
152+
real client does everything else. Script by transcript inspection, never
153+
turn counting.
154+
3. **Real-inference evals**: a different axis (performance distributions,
155+
not pass/fail). Not in this suite.
156+
157+
**The Desk** (`desk/`): films a scenario on one virtual Linux desktop — the
158+
chat renderer in a visible xterm, the browser as a real headed window, one
159+
ffmpeg x11grab. The film replaces session.mp4 in the run dir; the scenario
160+
file is unchanged (chat-theater switches transports on `E2E_DESK=1`).
161+
162+
```
163+
e2e/desk/run.sh [scenario] [project] # docker; first run builds + installs
164+
```

e2e/desk/Dockerfile

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# The Desk: a virtual Linux desktop that e2e scenarios are FILMED on — one
2+
# Xvfb display showing a real terminal window (the chat) and a real headed
3+
# Chromium (the browser hop), captured by a single ffmpeg x11grab into one
4+
# session.mp4. No splicing: the film is the screen.
5+
#
6+
# The repo is bind-mounted at /repo at runtime; node_modules and caches live
7+
# in named volumes (host artifacts are macOS-native and can't be reused).
8+
FROM node:22-bookworm
9+
10+
RUN apt-get update && apt-get install -y --no-install-recommends \
11+
xvfb openbox xterm ffmpeg xdotool x11-utils x11-xserver-utils \
12+
fonts-dejavu-core fonts-noto-color-emoji \
13+
curl unzip ca-certificates git \
14+
&& rm -rf /var/lib/apt/lists/*
15+
16+
# Bun — the repo's package manager and script runner.
17+
RUN curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.13" \
18+
&& ln -s /root/.bun/bin/bun /usr/local/bin/bun \
19+
&& ln -s /root/.bun/bin/bun /usr/local/bin/bunx
20+
21+
# Chromium + system deps baked into the image, pinned to the repo's
22+
# playwright version so the runtime install check is a no-op.
23+
ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
24+
RUN npx -y playwright@1.60.0 install --with-deps chromium && rm -rf /root/.npm
25+
26+
WORKDIR /repo
27+
ENTRYPOINT ["bash", "e2e/desk/entry.sh"]

e2e/desk/entry.sh

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
#!/usr/bin/env bash
2+
# Desk entrypoint (runs inside the desk image): boot a display, roll one
3+
# camera on it, run the scenario with E2E_DESK=1 so its windows land on the
4+
# display, then drop the film into the scenario's run dir as session.mp4.
5+
set -euo pipefail
6+
7+
SCENARIO="${DESK_SCENARIO:-scenarios/connect-handoff-session.test.ts}"
8+
PROJECT="${DESK_PROJECT:-cloud}"
9+
SIZE="${DESK_SIZE:-1440x900}"
10+
11+
echo "[desk] installing dependencies (cached in a volume after first run)…"
12+
bun install --frozen-lockfile
13+
14+
echo "[desk] display :99 at ${SIZE}"
15+
Xvfb :99 -screen 0 "${SIZE}x24" -nolisten tcp &
16+
export DISPLAY=:99
17+
sleep 0.5
18+
openbox &
19+
xsetroot -solid "#0b0b10" || true
20+
21+
mkdir -p /desk-out
22+
echo "[desk] camera rolling"
23+
ffmpeg -loglevel error -f x11grab -framerate 24 -video_size "$SIZE" -i :99 \
24+
-c:v libx264 -preset veryfast -crf 24 -pix_fmt yuv420p -y /desk-out/desk.mp4 &
25+
FFMPEG_PID=$!
26+
27+
set +e
28+
(cd e2e && E2E_DESK=1 npx vitest run --project "$PROJECT" "$SCENARIO")
29+
STATUS=$?
30+
set -e
31+
32+
sleep 1
33+
kill -INT "$FFMPEG_PID" 2>/dev/null || true
34+
wait "$FFMPEG_PID" 2>/dev/null || true
35+
36+
# The desk film IS the session recording — it replaces the browser-only
37+
# video the surface recorded inside the run.
38+
RUN_DIR=$(ls -dt e2e/runs/"$PROJECT"/*/ 2>/dev/null | head -1)
39+
if [ -n "$RUN_DIR" ] && [ -f /desk-out/desk.mp4 ]; then
40+
cp /desk-out/desk.mp4 "${RUN_DIR}session.mp4"
41+
echo "[desk] film → ${RUN_DIR}session.mp4"
42+
fi
43+
exit "$STATUS"

e2e/desk/run.sh

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env bash
2+
# Film a scenario on the Desk (one virtual desktop, one recording).
3+
# e2e/desk/run.sh [scenario-path] [project]
4+
# First run builds the image and fills the node_modules volume; later runs
5+
# reuse both.
6+
set -euo pipefail
7+
cd "$(dirname "$0")/../.."
8+
9+
docker build -t executor-e2e-desk e2e/desk
10+
11+
docker run --rm \
12+
-v "$PWD":/repo \
13+
-v executor-desk-node-modules:/repo/node_modules \
14+
-v executor-desk-bun-cache:/root/.bun/install/cache \
15+
-e DESK_SCENARIO="${1:-scenarios/connect-handoff-session.test.ts}" \
16+
-e DESK_PROJECT="${2:-cloud}" \
17+
executor-e2e-desk
Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
// The connect handoff as a DEVELOPER SESSION — the way a human actually
2+
// tests this: an agent chat in a real terminal where the agent wires up the
3+
// API over MCP and drops a connect link, a browser hop to paste the key,
4+
// then back to the chat to prove the connection works with a live send.
5+
//
6+
// No inference, no third-party agent binary: the "agent" is the chat
7+
// theater (src/clients/chat-theater.ts) presenting REAL mcporter MCP calls
8+
// — OAuth, execute, approval pause/resume all genuine, every tool spinner
9+
// on screen bracketing the actual call it narrates. The provider on the
10+
// other side is real too (resend.emulators.dev); its request ledger is the
11+
// final evidence.
12+
import { randomBytes } from "node:crypto";
13+
import { join } from "node:path";
14+
15+
import { expect } from "@effect/vitest";
16+
import { Effect } from "effect";
17+
18+
import { scenario } from "../src/scenario";
19+
import { Browser, Cli, Mcp, RunDir, Target } from "../src/services";
20+
import { withChatTheater } from "../src/clients/chat-theater";
21+
import type { McpSession } from "../src/surfaces/mcp";
22+
23+
const EMULATOR_BASE = "https://resend.emulators.dev";
24+
25+
const unique = (prefix: string) => `${prefix}_${randomBytes(4).toString("hex")}`;
26+
27+
// The emulator serves its own OpenAPI document (bearer auth, base URL in
28+
// `servers`) — adding by URL with nothing else is exactly what an agent
29+
// does, and the platform derives the paste-a-token auth method from the
30+
// spec's security scheme.
31+
const EMULATOR_SPEC_URL = `${EMULATOR_BASE}/openapi.json`;
32+
33+
const addSpecCode = (slug: string) => `
34+
const added = await tools.executor.openapi.addSpec({
35+
spec: { kind: "url", url: ${JSON.stringify(EMULATOR_SPEC_URL)} },
36+
slug: ${JSON.stringify(slug)},
37+
});
38+
return added.ok ? { ok: true, slug: added.data.slug, toolCount: added.data.toolCount } : { ok: false, error: added.error };
39+
`;
40+
41+
const createHandoffCode = (slug: string) => `
42+
const handoff = await tools.executor.coreTools.connections.createHandoff({
43+
integration: ${JSON.stringify(slug)},
44+
owner: "org",
45+
label: "Resend",
46+
});
47+
return handoff.ok ? { ok: true, url: handoff.data.url } : { ok: false, error: handoff.error };
48+
`;
49+
50+
const sendEmailCode = (slug: string, subject: string) => `
51+
const found = await tools.search({ namespace: ${JSON.stringify(slug)}, query: "send email", limit: 5 });
52+
const path = found.items[0]?.path;
53+
if (!path) return { ok: false, error: "no send tool found" };
54+
let t = tools;
55+
for (const seg of path.split(".")) t = t[seg];
56+
const sent = await t({
57+
body: {
58+
from: "onboarding@example.com",
59+
to: "dev-session@example.com",
60+
subject: ${JSON.stringify(subject)},
61+
html: "<p>connect-handoff developer session</p>",
62+
},
63+
});
64+
return { ok: sent.ok, path, result: sent.ok ? sent.data : sent.error };
65+
`;
66+
67+
/** Run `execute`, auto-approving a paused execution (policy elicitation)
68+
* once, and parse the sandbox's JSON return value. */
69+
const executeJson = (session: McpSession, code: string) =>
70+
Effect.gen(function* () {
71+
let result = yield* session.call("execute", { code });
72+
if (result.text.includes("executionId:")) {
73+
result = yield* session.approvePaused(result.text);
74+
}
75+
expect(result.ok, `execute completed (got: ${result.text.slice(0, 400)})`).toBe(true);
76+
return JSON.parse(result.text) as Record<string, unknown>;
77+
});
78+
79+
const mintEmulatorApiKey = Effect.promise(async () => {
80+
const response = await fetch(`${EMULATOR_BASE}/_emulate/credentials`, {
81+
method: "POST",
82+
headers: { "content-type": "application/json" },
83+
body: JSON.stringify({ type: "api-key" }),
84+
});
85+
const body = (await response.json()) as { credential?: { token?: string } };
86+
const token = body.credential?.token;
87+
if (!token) throw new Error(`emulator credential mint failed: ${JSON.stringify(body)}`);
88+
return token;
89+
});
90+
91+
scenario(
92+
"Connect · developer session: agent chat → handoff link → paste key → verified send",
93+
{ timeout: 240_000 },
94+
Effect.scoped(
95+
Effect.gen(function* () {
96+
const target = yield* Target;
97+
const mcp = yield* Mcp;
98+
const browser = yield* Browser;
99+
const cli = yield* Cli;
100+
const runDir = yield* RunDir;
101+
102+
const integration = unique("resendsesh");
103+
const emailSubject = unique("dev-session");
104+
const apiKey = yield* mintEmulatorApiKey;
105+
const identity = yield* target.newIdentity();
106+
const session = mcp.session(identity);
107+
108+
yield* withChatTheater(
109+
cli,
110+
{ title: "executor agent — connect Resend", record: join(runDir, "terminal.cast") },
111+
(chat) =>
112+
Effect.gen(function* () {
113+
// Real MCP OAuth + tool discovery happens behind this call.
114+
yield* chat.tool(
115+
{ name: "executor (mcp)", result: (tools) => `${tools.length} tools available` },
116+
session.listTools(),
117+
);
118+
119+
yield* chat.user(
120+
"Add the Resend API to my executor and give me a link to connect my account",
121+
);
122+
yield* chat.assistant("I'll register the Resend API in your Executor now.");
123+
const added = yield* chat.tool(
124+
{ name: "execute", input: addSpecCode(integration) },
125+
executeJson(session, addSpecCode(integration)),
126+
);
127+
expect(added.ok, `addSpec succeeded: ${JSON.stringify(added)}`).toBe(true);
128+
129+
yield* chat.assistant("Registered. Creating your connect link…");
130+
const handoff = yield* chat.tool(
131+
{ name: "execute", input: createHandoffCode(integration) },
132+
executeJson(session, createHandoffCode(integration)),
133+
);
134+
expect(handoff.ok, `createHandoff succeeded: ${JSON.stringify(handoff)}`).toBe(true);
135+
const handoffUrl = String(handoff.url);
136+
expect(new URL(handoffUrl).origin, "handoff targets this deployment").toBe(
137+
new URL(target.baseUrl).origin,
138+
);
139+
140+
yield* chat.assistant(
141+
`Open this link to connect your Resend account:\n\n${handoffUrl}\n\nTell me once you've pasted your API key.`,
142+
);
143+
144+
// The browser hop — the terminal session stays open while the
145+
// "user" pastes the key; the paste is the real add-account UI.
146+
yield* chat.status("you, in the browser: opening the link and pasting the API key…");
147+
yield* browser.session(identity, async ({ page, step }) => {
148+
await step("Open the connect link from the chat", async () => {
149+
await page.goto(handoffUrl, { waitUntil: "networkidle" });
150+
await page
151+
.getByRole("heading", { name: /Add connection/ })
152+
.waitFor({ timeout: 15_000 });
153+
});
154+
await step("Paste the Resend API key and connect", async () => {
155+
const credential = page.getByPlaceholder(/paste the value \/ token/i);
156+
await credential.waitFor({ timeout: 15_000 });
157+
await credential.fill(apiKey);
158+
await page.getByRole("button", { name: "Add connection", exact: true }).click();
159+
await page
160+
.getByRole("heading", { name: /Add connection/ })
161+
.waitFor({ state: "hidden", timeout: 20_000 });
162+
});
163+
});
164+
165+
yield* chat.user("Connected, now send a test email to prove it works");
166+
yield* chat.assistant("Sending a test email through your new connection…");
167+
const sent = yield* chat.tool(
168+
{ name: "execute", input: sendEmailCode(integration, emailSubject) },
169+
executeJson(session, sendEmailCode(integration, emailSubject)),
170+
);
171+
expect(sent.ok, `email sent through the connection: ${JSON.stringify(sent)}`).toBe(
172+
true,
173+
);
174+
175+
yield* chat.assistant("Test email sent - your Resend connection works.");
176+
}),
177+
);
178+
179+
// Final evidence: the emulator's ledger saw the send from Executor.
180+
const ledger = yield* Effect.promise(async () =>
181+
(await fetch(`${EMULATOR_BASE}/_emulate/ledger`)).text(),
182+
);
183+
expect(
184+
ledger.includes(emailSubject),
185+
"the emulator request ledger recorded the test email",
186+
).toBe(true);
187+
}),
188+
),
189+
);

0 commit comments

Comments
 (0)