Developer-session e2e: chat-theater recordings, focus timeline, one-mp4 films

e2e/AGENTS.md

@@ -133,3 +133,32 @@ When handing results to the user, follow the evidence contract in the root
   values, not booleans — `expect(list).toContain(x)`, never
   `expect(list.includes(x)).toBe(true)` — so failures show the data.
 - Keep it deterministic: no sleeps; wait on conditions.
+
+## Developer-session recordings (chat theater + desk)
+
+Some scenarios are meant to be WATCHED — they show the product the way a
+developer actually uses it. Three tiers, pick deliberately:
+
+1. **Chat theater** (`src/clients/chat-theater.ts`): the default for
+   product-flow recordings. The "agent" is a chat renderer in a recorded
+   PTY; every tool spinner brackets a REAL mcporter MCP call (OAuth,
+   execute, approval resume). No inference, no third-party binary.
+   Exemplar: `scenarios/connect-handoff-session.test.ts`. Artifacts:
+   `terminal.cast` (the chat) + `session.mp4` (browser hops); the viewer
+   plays them in story order.
+2. **Replay brain + real client** (`src/clients/replay-brain.ts`): when the
+   third-party CLIENT's behavior is under test (OpenCode/Claude Code
+   protocol handling). A scripted OpenAI-wire server plays the LLM; the
+   real client does everything else. Script by transcript inspection, never
+   turn counting.
+3. **Real-inference evals**: a different axis (performance distributions,
+   not pass/fail). Not in this suite.
+
+**The Desk** (`desk/`): films a scenario on one virtual Linux desktop — the
+chat renderer in a visible xterm, the browser as a real headed window, one
+ffmpeg x11grab. The film replaces session.mp4 in the run dir; the scenario
+file is unchanged (chat-theater switches transports on `E2E_DESK=1`).
+
+```
+e2e/desk/run.sh [scenario] [project]   # docker; first run builds + installs
+```

e2e/desk/Dockerfile

@@ -0,0 +1,27 @@
+# The Desk: a virtual Linux desktop that e2e scenarios are FILMED on — one
+# Xvfb display showing a real terminal window (the chat) and a real headed
+# Chromium (the browser hop), captured by a single ffmpeg x11grab into one
+# session.mp4. No splicing: the film is the screen.
+#
+# The repo is bind-mounted at /repo at runtime; node_modules and caches live
+# in named volumes (host artifacts are macOS-native and can't be reused).
+FROM node:22-bookworm
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    xvfb openbox xterm ffmpeg xdotool x11-utils x11-xserver-utils \
+    fonts-dejavu-core fonts-noto-color-emoji \
+    curl unzip ca-certificates git \
+  && rm -rf /var/lib/apt/lists/*
+
+# Bun — the repo's package manager and script runner.
+RUN curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.13" \
+  && ln -s /root/.bun/bin/bun /usr/local/bin/bun \
+  && ln -s /root/.bun/bin/bun /usr/local/bin/bunx
+
+# Chromium + system deps baked into the image, pinned to the repo's
+# playwright version so the runtime install check is a no-op.
+ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
+RUN npx -y playwright@1.60.0 install --with-deps chromium && rm -rf /root/.npm
+
+WORKDIR /repo
+ENTRYPOINT ["bash", "e2e/desk/entry.sh"]

e2e/desk/entry.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# Desk entrypoint (runs inside the desk image): boot a display, roll one
+# camera on it, run the scenario with E2E_DESK=1 so its windows land on the
+# display, then drop the film into the scenario's run dir as session.mp4.
+set -euo pipefail
+
+SCENARIO="${DESK_SCENARIO:-scenarios/connect-handoff-session.test.ts}"
+PROJECT="${DESK_PROJECT:-cloud}"
+SIZE="${DESK_SIZE:-1440x900}"
+
+echo "[desk] installing dependencies (cached in a volume after first run)…"
+bun install --frozen-lockfile
+
+echo "[desk] display :99 at ${SIZE}"
+Xvfb :99 -screen 0 "${SIZE}x24" -nolisten tcp &
+export DISPLAY=:99
+sleep 0.5
+openbox &
+xsetroot -solid "#0b0b10" || true
+
+mkdir -p /desk-out
+echo "[desk] camera rolling"
+ffmpeg -loglevel error -f x11grab -framerate 24 -video_size "$SIZE" -i :99 \
+  -c:v libx264 -preset veryfast -crf 24 -pix_fmt yuv420p -y /desk-out/desk.mp4 &
+FFMPEG_PID=$!
+
+set +e
+(cd e2e && E2E_DESK=1 npx vitest run --project "$PROJECT" "$SCENARIO")
+STATUS=$?
+set -e
+
+sleep 1
+kill -INT "$FFMPEG_PID" 2>/dev/null || true
+wait "$FFMPEG_PID" 2>/dev/null || true
+
+# The desk film IS the session recording — it replaces the browser-only
+# video the surface recorded inside the run.
+RUN_DIR=$(ls -dt e2e/runs/"$PROJECT"/*/ 2>/dev/null | head -1)
+if [ -n "$RUN_DIR" ] && [ -f /desk-out/desk.mp4 ]; then
+  cp /desk-out/desk.mp4 "${RUN_DIR}session.mp4"
+  echo "[desk] film → ${RUN_DIR}session.mp4"
+fi
+exit "$STATUS"

e2e/desk/run.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Film a scenario on the Desk (one virtual desktop, one recording).
+#   e2e/desk/run.sh [scenario-path] [project]
+# First run builds the image and fills the node_modules volume; later runs
+# reuse both.
+set -euo pipefail
+cd "$(dirname "$0")/../.."
+
+docker build -t executor-e2e-desk e2e/desk
+
+docker run --rm \
+  -v "$PWD":/repo \
+  -v executor-desk-node-modules:/repo/node_modules \
+  -v executor-desk-bun-cache:/root/.bun/install/cache \
+  -e DESK_SCENARIO="${1:-scenarios/connect-handoff-session.test.ts}" \
+  -e DESK_PROJECT="${2:-cloud}" \
+  executor-e2e-desk

e2e/scenarios/connect-handoff-session.test.ts

@@ -0,0 +1,189 @@
+// The connect handoff as a DEVELOPER SESSION — the way a human actually
+// tests this: an agent chat in a real terminal where the agent wires up the
+// API over MCP and drops a connect link, a browser hop to paste the key,
+// then back to the chat to prove the connection works with a live send.
+//
+// No inference, no third-party agent binary: the "agent" is the chat
+// theater (src/clients/chat-theater.ts) presenting REAL mcporter MCP calls
+// — OAuth, execute, approval pause/resume all genuine, every tool spinner
+// on screen bracketing the actual call it narrates. The provider on the
+// other side is real too (resend.emulators.dev); its request ledger is the
+// final evidence.
+import { randomBytes } from "node:crypto";
+import { join } from "node:path";
+
+import { expect } from "@effect/vitest";
+import { Effect } from "effect";
+
+import { scenario } from "../src/scenario";
+import { Browser, Cli, Mcp, RunDir, Target } from "../src/services";
+import { withChatTheater } from "../src/clients/chat-theater";
+import type { McpSession } from "../src/surfaces/mcp";
+
+const EMULATOR_BASE = "https://resend.emulators.dev";
+
+const unique = (prefix: string) => `${prefix}_${randomBytes(4).toString("hex")}`;
+
+// The emulator serves its own OpenAPI document (bearer auth, base URL in
+// `servers`) — adding by URL with nothing else is exactly what an agent
+// does, and the platform derives the paste-a-token auth method from the
+// spec's security scheme.
+const EMULATOR_SPEC_URL = `${EMULATOR_BASE}/openapi.json`;
+
+const addSpecCode = (slug: string) => `
+const added = await tools.executor.openapi.addSpec({
+  spec: { kind: "url", url: ${JSON.stringify(EMULATOR_SPEC_URL)} },
+  slug: ${JSON.stringify(slug)},
+});
+return added.ok ? { ok: true, slug: added.data.slug, toolCount: added.data.toolCount } : { ok: false, error: added.error };
+`;
+
+const createHandoffCode = (slug: string) => `
+const handoff = await tools.executor.coreTools.connections.createHandoff({
+  integration: ${JSON.stringify(slug)},
+  owner: "org",
+  label: "Resend",
+});
+return handoff.ok ? { ok: true, url: handoff.data.url } : { ok: false, error: handoff.error };
+`;
+
+const sendEmailCode = (slug: string, subject: string) => `
+const found = await tools.search({ namespace: ${JSON.stringify(slug)}, query: "send email", limit: 5 });
+const path = found.items[0]?.path;
+if (!path) return { ok: false, error: "no send tool found" };
+let t = tools;
+for (const seg of path.split(".")) t = t[seg];
+const sent = await t({
+  body: {
+    from: "onboarding@example.com",
+    to: "dev-session@example.com",
+    subject: ${JSON.stringify(subject)},
+    html: "<p>connect-handoff developer session</p>",
+  },
+});
+return { ok: sent.ok, path, result: sent.ok ? sent.data : sent.error };
+`;
+
+/** Run `execute`, auto-approving a paused execution (policy elicitation)
+ *  once, and parse the sandbox's JSON return value. */
+const executeJson = (session: McpSession, code: string) =>
+  Effect.gen(function* () {
+    let result = yield* session.call("execute", { code });
+    if (result.text.includes("executionId:")) {
+      result = yield* session.approvePaused(result.text);
+    }
+    expect(result.ok, `execute completed (got: ${result.text.slice(0, 400)})`).toBe(true);
+    return JSON.parse(result.text) as Record<string, unknown>;
+  });
+
+const mintEmulatorApiKey = Effect.promise(async () => {
+  const response = await fetch(`${EMULATOR_BASE}/_emulate/credentials`, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify({ type: "api-key" }),
+  });
+  const body = (await response.json()) as { credential?: { token?: string } };
+  const token = body.credential?.token;
+  if (!token) throw new Error(`emulator credential mint failed: ${JSON.stringify(body)}`);
+  return token;
+});
+
+scenario(
+  "Connect · developer session: agent chat → handoff link → paste key → verified send",
+  { timeout: 240_000 },
+  Effect.scoped(
+    Effect.gen(function* () {
+      const target = yield* Target;
+      const mcp = yield* Mcp;
+      const browser = yield* Browser;
+      const cli = yield* Cli;
+      const runDir = yield* RunDir;
+
+      const integration = unique("resendsesh");
+      const emailSubject = unique("dev-session");
+      const apiKey = yield* mintEmulatorApiKey;
+      const identity = yield* target.newIdentity();
+      const session = mcp.session(identity);
+
+      yield* withChatTheater(
+        cli,
+        { title: "executor agent — connect Resend", record: join(runDir, "terminal.cast") },
+        (chat) =>
+          Effect.gen(function* () {
+            // Real MCP OAuth + tool discovery happens behind this call.
+            yield* chat.tool(
+              { name: "executor (mcp)", result: (tools) => `${tools.length} tools available` },
+              session.listTools(),
+            );
+
+            yield* chat.user(
+              "Add the Resend API to my executor and give me a link to connect my account",
+            );
+            yield* chat.assistant("I'll register the Resend API in your Executor now.");
+            const added = yield* chat.tool(
+              { name: "execute", input: addSpecCode(integration) },
+              executeJson(session, addSpecCode(integration)),
+            );
+            expect(added.ok, `addSpec succeeded: ${JSON.stringify(added)}`).toBe(true);
+
+            yield* chat.assistant("Registered. Creating your connect link…");
+            const handoff = yield* chat.tool(
+              { name: "execute", input: createHandoffCode(integration) },
+              executeJson(session, createHandoffCode(integration)),
+            );
+            expect(handoff.ok, `createHandoff succeeded: ${JSON.stringify(handoff)}`).toBe(true);
+            const handoffUrl = String(handoff.url);
+            expect(new URL(handoffUrl).origin, "handoff targets this deployment").toBe(
+              new URL(target.baseUrl).origin,
+            );
+
+            yield* chat.assistant(
+              `Open this link to connect your Resend account:\n\n${handoffUrl}\n\nTell me once you've pasted your API key.`,
+            );
+
+            // The browser hop — the terminal session stays open while the
+            // "user" pastes the key; the paste is the real add-account UI.
+            yield* chat.status("you, in the browser: opening the link and pasting the API key…");
+            yield* browser.session(identity, async ({ page, step }) => {
+              await step("Open the connect link from the chat", async () => {
+                await page.goto(handoffUrl, { waitUntil: "networkidle" });
+                await page
+                  .getByRole("heading", { name: /Add connection/ })
+                  .waitFor({ timeout: 15_000 });
+              });
+              await step("Paste the Resend API key and connect", async () => {
+                const credential = page.getByPlaceholder(/paste the value \/ token/i);
+                await credential.waitFor({ timeout: 15_000 });
+                await credential.fill(apiKey);
+                await page.getByRole("button", { name: "Add connection", exact: true }).click();
+                await page
+                  .getByRole("heading", { name: /Add connection/ })
+                  .waitFor({ state: "hidden", timeout: 20_000 });
+              });
+            });
+
+            yield* chat.user("Connected, now send a test email to prove it works");
+            yield* chat.assistant("Sending a test email through your new connection…");
+            const sent = yield* chat.tool(
+              { name: "execute", input: sendEmailCode(integration, emailSubject) },
+              executeJson(session, sendEmailCode(integration, emailSubject)),
+            );
+            expect(sent.ok, `email sent through the connection: ${JSON.stringify(sent)}`).toBe(
+              true,
+            );
+
+            yield* chat.assistant("Test email sent - your Resend connection works.");
+          }),
+      );
+
+      // Final evidence: the emulator's ledger saw the send from Executor.
+      const ledger = yield* Effect.promise(async () =>
+        (await fetch(`${EMULATOR_BASE}/_emulate/ledger`)).text(),
+      );
+      expect(
+        ledger.includes(emailSubject),
+        "the emulator request ledger recorded the test email",
+      ).toBe(true);
+    }),
+  ),
+);