Skip to content

Developer-session e2e: chat-theater recordings, focus timeline, one-mp4 films#962

Merged
RhysSullivan merged 11 commits into
mainfrom
claude/e2e-developer-session
Jun 12, 2026
Merged

Developer-session e2e: chat-theater recordings, focus timeline, one-mp4 films#962
RhysSullivan merged 11 commits into
mainfrom
claude/e2e-developer-session

Conversation

@RhysSullivan

@RhysSullivan RhysSullivan commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Watchable e2e runs that read like a developer actually using the product: chat with an agent in a terminal, get a connect link, paste a key in the browser, come back and verify — all real calls, recorded.

Connect · developer session: agent chat → handoff link → paste key → verified send (cloud)

What's in here

  • Chat theater (e2e/src/clients/chat-theater.ts + agent-chat-tui.ts): the default for product-flow recordings. A chat renderer runs in the recorded PTY; every tool call on screen brackets a REAL mcporter MCP call (OAuth, execute, approval resume). Tool calls render like an agent TUI — tool name, the call's real input block, live spinner for the call's real duration, the real result line. No inference, no third-party binary, ~28s per run.
  • Replay brain (e2e/src/clients/replay-brain.ts): a scripted OpenAI-wire chat-completions server for the distinct tier where a third-party CLIENT's behavior is under test (the real OpenCode binary chats through it; scripted by transcript inspection, not turn counting).
  • Focus timeline (e2e/src/timeline.ts): acting on a window IS focusing it — Playwright steps mark the browser focused, chat events mark the terminal focused. Surfaces emit timeline.json as a side effect of normal driving.
  • e2e/scripts/film.ts: cuts the run's two real recordings into ONE mp4 that plays like a screen capture of tabbing between full-screen windows, using the focus timeline as the edit decision list. No display server needed (agg + ffmpeg).
  • The Desk (e2e/desk/): optional Docker stage that films a scenario on a real virtual desktop (visible xterm + headed Chromium, single x11grab) for when actual window layering matters. Scenario files are identical across all presentations.
  • Viewer: runs with both a cast and a video play in story order; film.mp4 plays as the session when present.
  • Scenario (e2e/scenarios/connect-handoff-session.test.ts): the connect-handoff developer session — adds the emulated Resend API by its served spec URL only (emulators.dev now serves OpenAPI documents from every REST emulator), pastes an emulator-minted key through the real add-account UI, and proves the connection with a send confirmed by the emulator's request ledger.

Relationship to #957

Was stacked on #957 (addSpec auth derivation), which has merged — this branch is now based directly on main. The connect-handoff developer-session scenario is the end-to-end regression gate for that fix.

Split into a stack

This PR was split — it now carries only the recording framework above. The rest of the original work follows in:

  • e2e distributed tracing (suite-owned motel, browser OTLP, trace join)
  • the live session player + trace rail in the runs viewer
  • production browser span export → Axiom

Stack

  1. Developer-session e2e: chat-theater recordings, focus timeline, one-mp4 films #962 👈 current
  2. E2e distributed tracing: suite-owned motel, browser OTLP, end-to-end trace join #979
  3. Runs viewer: live session player with URL bar, trace rail, self-hosted trace viewer #980
  4. Production browser span export: same-origin OTLP forwarded to Axiom #981

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
executor-marketing 0c1d824 Commit Preview URL

Branch Preview URL
Jun 12 2026, 05:50 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
executor-cloud 0c1d824 Jun 12 2026, 05:50 PM

@pkg-pr-new

pkg-pr-new Bot commented Jun 12, 2026

Copy link
Copy Markdown

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@962

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@962

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@962

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@962

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@962

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@962

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@962

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@962

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@962

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@962

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@962

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@962

executor

npm i https://pkg.pr.new/executor@962

commit: 0c1d824

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Torn down — the PR is closed.

@RhysSullivan RhysSullivan force-pushed the claude/e2e-developer-session branch from aa3e919 to 9486dc2 Compare June 12, 2026 17:39
A scenario now runs the way a developer actually tests the product: chat
with the real OpenCode TUI in a recorded PTY, the agent registers an API
and drops a connect link in the chat (real MCP tool execution), a browser
opens the link and pastes a real emulator-minted key while the terminal
session stays open, and the chat resumes to verify the connection with a
live send confirmed by the provider emulator's request ledger.

- e2e/src/clients/replay-brain.ts: local OpenAI-wire chat-completions
  server (SSE + tool calls) scripted per scenario by transcript
  inspection, so approval pauses and OpenCode side requests don't derail
  it. Tool names resolve by suffix against whatever the agent offers.
- clients/opencode.ts: makeOpenCodeHome optionally wires a 'replay'
  provider (openai-compatible) at the brain and pre-allows permissions.
- scenarios/connect-handoff-session.test.ts: one continuous terminal.cast
  timeline with the browser hop forked concurrently; chat-brain.json
  transcript written as a run artifact even on failure.
…rence

The developer-session recording no longer needs a third-party agent binary
or a faked LLM: the 'agent' is a chat presentation (agent-chat-tui.ts, a
renderer in the recorded PTY) driven by chat-theater.ts while the scenario
makes REAL mcporter MCP calls — OAuth, execute, approval pause/resume all
genuine, each on-screen tool spinner bracketing the actual call it
narrates. Runs anywhere bun runs (no opencode capability), ~28s on cloud.

replay-brain.ts stays for the distinct tier where the third-party client's
own protocol behavior is under test (OpenCode/Claude Code); real-inference
evals remain a separate axis (performance distributions, not pass/fail
scenarios).
Runs from chat-theater scenarios produce a terminal.cast (the chat) AND a
session.mp4 (the browser hop). The run page previously picked one
(video ?? cast), hiding the chat that led to the browser. Now both render
under a single 'session' tab in story order — 1 · the chat, in the
terminal; 2 · the browser hop, mid-chat — with autoplay reserved for
video-only runs.
e2e/desk/run.sh builds a Linux image (Xvfb + openbox + xterm + headed
Chromium + ffmpeg) and runs a scenario inside it with E2E_DESK=1: the chat
renderer opens in a visible UTF-8 xterm (events over a FIFO instead of a
PTY), the browser hop is a real headed window raised over it, and a single
x11grab films the whole screen for the whole run — no splicing. The film
replaces session.mp4 in the scenario's run dir; the scenario file itself
is unchanged (chat-theater switches transports on E2E_DESK).

First run builds the image and fills a node_modules volume; subsequent
films of the connect-handoff session take ~2 minutes end to end.
bun scripts/film.ts runs/<target>/<slug> turns a chat-theater run's two
real recordings into one video that plays like a screen capture of a
developer tabbing between full-screen windows: the cast (rendered without
idle compression so cast time == video time) is cut at the browser hop —
located by the scenario's narrator line, falling back to the largest
output gap — and the browser recording fills the gap. Both segments are
the genuine recordings; only the cut is editorial, exactly like tabbing.

The film registers itself in result.json and the viewer plays film.mp4 as
the session when present. Runs anywhere (agg + ffmpeg) — no Docker, no
display server; the Desk remains for when actual window layering matters.
Acting on a window IS focusing it: a Playwright step marks the browser
focused, a chat-theater event marks the terminal focused. Surfaces record
these transitions (plus each recording's start anchor) into the run's
timeline.json as a side effect of normal driving — no scenario changes,
no declared focus anywhere.

film.ts now builds its cut list from that timeline (any number of hops,
each act mapped into its recording's own clock via the anchors), keeping
the narrator-line/largest-gap heuristic only as a fallback for runs
without one. The film is the operations, edited by where they happened.
The emulator serves its own OpenAPI document, so the scenario stops
hand-writing a Resend subset and an explicit auth template — addSpec gets
just the spec URL, exactly what an agent would send, and the platform
derives base URL and the paste-a-token method from the spec itself.
chat.tool now takes the call itself — tool name, the REAL input (the
sandbox code the scenario executes, shown as the indented preview block
agent TUIs draw), and a result-line renderer — instead of a prose label.
The renderer paints name → input block → live spinner → ✓ with the real
duration and the real result's first line. The recording reads like an
actual agent session: what was called, with what, how long it took, what
came back.
film.mp4 was a manual post-step, so any re-run left the viewer showing
the parts (cast + browser video) instead of the spliced session. The
runner now invokes scripts/film.ts after every successful run that has
both terminal.cast and session.mp4 — best-effort, so missing agg/ffmpeg
degrades back to the parts rather than failing the run. Every developer-
session run's page now plays the single film by default.
Every REST emulator now serves its own OpenAPI document; the connect
scenarios register integrations from those live spec URLs instead of
hand-written specs.
@RhysSullivan RhysSullivan force-pushed the claude/e2e-developer-session branch from 9486dc2 to 0c1d824 Compare June 12, 2026 17:40
@RhysSullivan RhysSullivan merged commit fe5181b into main Jun 12, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant