Turnkey fresh-worktree dev env: atomic per-checkout e2e ports, bootstrap, leak reaper (#973)

* Make fresh-worktree e2e turnkey: per-checkout ports, port-squat fail-fast, bootstrap script

- e2e ports are now derived from a hash of the checkout root (42000-45999
  blocks) so concurrent worktrees never collide; E2E_*_PORT/URL still override.
- globalsetups fail fast with the squatting PID when a port is taken instead
  of silently attaching to a foreign checkout's dev server (the cause of
  31-scenario auth-error cascades).
- scripts/bootstrap.ts: one idempotent command for submodules + install +
  playwright; scripts/reap-dev-servers.ts kills orphaned dev stacks.
- e2e: bun run ports prints this checkout's derived ports.
- sources-api scenario: align with payload-first OpenAPI results (PR #956
  changed the product, the cloud-only scenario was not updated).
- AGENTS.md: fresh-worktree setup + environment gotchas (fish shell, /tmp
  scripts, bun.lock conflicts, per-checkout ports).

* Add e2e run summary script + agent notes; format

* Make e2e port claiming atomic: lock port per block, walk past collisions

The hash-derived block was only collision-unlikely (28 checkouts over 400
blocks is birthday-paradox territory), and probe-then-bind raced. Each block
now reserves its last port as a lock held for the suite's lifetime: claimPorts
binds the lock (atomic — two racing suites can't both win), probes the
remaining ports for squatters, and walks forward block-by-block until it owns
a fully free block, publishing the claimed ports via E2E_*_PORT env so test
workers agree. Explicit env pins skip claiming entirely. Verified by squatting
the preferred block's lock+selfhost ports and watching the suite relocate one
block over and pass 17/17.

* bootstrap: vendor submodules opt-in via --forks

Nothing imports from vendor/ at runtime — the forks are consumed as
published npm packages (vendor/README.md is explicit). Forcing the
submodule init in bootstrap taught every fresh worktree a false
dependency. --forks remains for deliberately developing a fork.

Author: RhysSullivan

.gitignore

@@ -102,6 +102,9 @@ executor.jsonc
 MISTAKES.md
 DESIRES.md
 LEARNINGS.md
+
+# Agent scratch scripts (workspace-resolvable alternative to /tmp)
+scratch/
 .alchemy/
 
 # Pi coding agent

AGENTS.md

@@ -1,5 +1,30 @@
 # AGENTS.md
 
+## Fresh Checkout / Worktree Setup
+
+Run `bun run bootstrap` first in any fresh checkout or worktree. It is
+idempotent: runs `bun install` (whose prepare hook builds the internal
+packages dev servers fail without) and installs Playwright chromium.
+Skipping it is why fresh worktrees die with "Failed to resolve entry for
+package '@executor-js/vite-plugin'". The `vendor/` submodules are NOT
+needed — nothing imports from `vendor/` at runtime; those forks are consumed
+from npm (see `vendor/README.md`). `bun run bootstrap --forks` inits them
+only when you're deliberately developing a fork.
+
+## Environment Gotchas (learned the hard way)
+
+- The shell is fish, and the working directory resets between Bash calls. Use
+  absolute paths rooted at THIS worktree (check `pwd`), never
+  `/Users/rhys/src/executor` from memory, and don't rely on a prior `cd`.
+- Don't write probe scripts to `/tmp` — they can't resolve workspace packages
+  (`effect`, `playwright`, ...). Put scratch scripts under the repo root
+  (`scratch/` is gitignored) so bun resolves the workspace.
+- `bun.lock` conflicts on rebase/merge: take either side, then re-run
+  `bun install` to regenerate it — never hand-merge the lockfile.
+- e2e dev-server ports are derived per checkout (`cd e2e && bun run ports`).
+  If a boot reports a squatted port, an old dev server leaked — kill it by
+  PID from the error message; don't move your own ports to dodge it.
+
 ## Task Completion Requirements
 
 - Use Effect Vitest for tests.

e2e/AGENTS.md

@@ -83,18 +83,28 @@ const r = yield * session.call("execute", { code: "return 1 + 1;" });
 cd e2e
 bun run test               # boots both dev servers, runs everything
 bun run test:cloud         # one target
-# attach to an already-running server while iterating:
-E2E_CLOUD_URL=http://127.0.0.1:4798 ../node_modules/.bin/vitest run --project cloud <file>
-E2E_SELFHOST_URL=http://localhost:4799 ../node_modules/.bin/vitest run --project selfhost <file>
+bun run ports              # print THIS checkout's derived ports
+# attach to an already-running server while iterating (use `bun run ports` URLs):
+E2E_CLOUD_URL=http://127.0.0.1:<port> ../node_modules/.bin/vitest run --project cloud <file>
+E2E_SELFHOST_URL=http://localhost:<port> ../node_modules/.bin/vitest run --project selfhost <file>
 ```
 
+Ports are claimed at boot (see `src/ports.ts`): each checkout hashes its repo
+root to a preferred block, atomically locks it (a held lock port makes races
+impossible), and walks to the next free block if it's locked or squatted — so
+concurrent suites in different worktrees can never collide or attach to each
+other's servers. `bun run ports` shows the preferred block; the boot log says
+if a suite moved. `E2E_*_PORT` env vars pin ports explicitly (no probing) and
+`E2E_<TARGET>_URL` attaches to a running instance.
+
 Each run writes `runs/<target>/<slug>/result.json` plus any browser artifacts
 (trace.zip / session.mp4 / screenshots). `bun run serve` hosts the scenario ×
 target matrix; a run page links the trace into Playwright's trace viewer.
 
 ## Discovering endpoints
 
-- The full OpenAPI spec: `curl http://127.0.0.1:4798/api/openapi.json` (cloud).
+- The full OpenAPI spec: `curl http://127.0.0.1:<cloud port>/api/openapi.json`
+  (cloud; port from `bun run ports`).
 - The typed client mirrors it: `client.<group>.<endpoint>(...)` with groups
   tools/integrations/connections/providers/executions/oauth/policies.
 - To see payload shapes, read the API definitions under

e2e/cloud/sources-api.test.ts

@@ -184,13 +184,12 @@ scenario(
         }),
       );
       expect(execution.isError, "the execution succeeded").toBe(false);
+      // Payload-first: `data` IS the upstream body; transport facts (status,
+      // headers) ride in the optional `http` side channel.
       expect(execution.structured, "the tool returned the upstream's echo").toMatchObject({
         result: {
           ok: true,
-          data: {
-            status: 200,
-            data: { message: "hello", suffix: "world", path: "/echo/hello" },
-          },
+          data: { message: "hello", suffix: "world", path: "/echo/hello" },
         },
       });
 

e2e/package.json

@@ -8,6 +8,8 @@
     "test:cloud": "vitest run --project cloud",
     "test:selfhost": "vitest run --project selfhost",
     "test:watch": "vitest",
+    "ports": "bun scripts/ports.ts",
+    "summary": "bun scripts/summary.ts",
     "viewer:build": "bun scripts/rebuild-viewer.ts",
     "serve": "bun scripts/rebuild-viewer.ts && bun scripts/serve.ts",
     "typecheck": "tsc --noEmit",

e2e/scripts/ports.ts

@@ -0,0 +1,20 @@
+// Print this checkout's PREFERRED e2e ports (see src/ports.ts). These are
+// where a suite normally boots; if the block is locked or squatted at boot
+// time, claimPorts walks to the next free block and the suite logs the move.
+// When attaching mid-run, the booted vite's actual port is authoritative —
+// check the suite's log line or `ps | grep 'vite dev'`.
+import {
+  AUTUMN_EMULATOR_PORT,
+  CLOUD_DB_PORT,
+  CLOUD_PORT,
+  WORKOS_EMULATOR_PORT,
+} from "../targets/cloud";
+import { SELFHOST_PORT } from "../targets/selfhost";
+import { repoRoot } from "../src/ports";
+
+console.log(`preferred e2e ports for ${repoRoot}`);
+console.log(`  cloud           http://127.0.0.1:${CLOUD_PORT}`);
+console.log(`  cloud dev-db    ${CLOUD_DB_PORT}`);
+console.log(`  workos emulator ${WORKOS_EMULATOR_PORT}`);
+console.log(`  autumn emulator ${AUTUMN_EMULATOR_PORT}`);
+console.log(`  selfhost        http://localhost:${SELFHOST_PORT}`);

e2e/scripts/summary.ts

@@ -0,0 +1,39 @@
+// One-line-per-failure digest of the last e2e run: reads
+// runs/<target>/<slug>/result.json and prints pass/fail counts plus each
+// failure's scenario name. `bun run summary [target...]` (default: all).
+import { readdirSync, readFileSync } from "node:fs";
+import { join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const runsDir = fileURLToPath(new URL("../runs/", import.meta.url));
+
+const targets =
+  process.argv.length > 2
+    ? process.argv.slice(2)
+    : readdirSync(runsDir, { withFileTypes: true })
+        .filter((entry) => entry.isDirectory())
+        .map((entry) => entry.name);
+
+for (const target of targets) {
+  const targetDir = join(runsDir, target);
+  let slugs: string[];
+  try {
+    slugs = readdirSync(targetDir);
+  } catch {
+    console.log(`${target}: no runs`);
+    continue;
+  }
+  const failures: { scenario: string; endedAt: string }[] = [];
+  let passed = 0;
+  for (const slug of slugs) {
+    try {
+      const result = JSON.parse(readFileSync(join(targetDir, slug, "result.json"), "utf8"));
+      if (result.ok) passed++;
+      else failures.push({ scenario: result.scenario ?? slug, endedAt: result.endedAt ?? "?" });
+    } catch {
+      // No result.json (partial run dir) — not a verdict either way.
+    }
+  }
+  console.log(`${target}: ${passed} passed, ${failures.length} failed`);
+  for (const failure of failures) console.log(`  FAIL ${failure.scenario} (${failure.endedAt})`);
+}

e2e/setup/cloud.globalsetup.ts

@@ -10,16 +10,9 @@ import { fileURLToPath } from "node:url";
 // Vendored fork import (same pattern as mcporter).
 import { createEmulator } from "@executor-js/emulate";
 
+import { claimPorts } from "../src/ports";
+import { E2E_COOKIE_PASSWORD, E2E_WORKOS_CLIENT_ID } from "../targets/cloud";
 import { bootProcesses, waitForHttp } from "./boot";
-import {
-  CLOUD_BASE_URL,
-  CLOUD_DB_PORT,
-  CLOUD_PORT,
-  WORKOS_EMULATOR_PORT,
-  AUTUMN_EMULATOR_PORT,
-  E2E_WORKOS_CLIENT_ID,
-  E2E_COOKIE_PASSWORD,
-} from "../targets/cloud";
 
 const cloudDir = fileURLToPath(new URL("../../apps/cloud/", import.meta.url));
 
@@ -29,6 +22,20 @@ export default async function setup(): Promise<(() => Promise<void>) | void> {
     return;
   }
 
+  // Claim a free port block (preferred block first, walk forward past
+  // squatters/colliding checkouts) and publish via env so the test workers —
+  // spawned after this — derive the same URLs. The imported targets/cloud
+  // constants were computed BEFORE the claim, so use the claimed values here.
+  const { ports, release } = await claimPorts([
+    { envVar: "E2E_CLOUD_PORT", offset: 0, label: "cloud vite dev" },
+    { envVar: "E2E_CLOUD_DB_PORT", offset: 1, label: "cloud dev-db (PGlite)" },
+    { envVar: "E2E_WORKOS_EMULATOR_PORT", offset: 2, label: "WorkOS emulator" },
+    { envVar: "E2E_AUTUMN_EMULATOR_PORT", offset: 3, label: "Autumn emulator" },
+  ]);
+  const cloudPort = ports.E2E_CLOUD_PORT!;
+  const dbPort = ports.E2E_CLOUD_DB_PORT!;
+  const baseUrl = `http://127.0.0.1:${cloudPort}`;
+
   // Fresh dev DB per suite run — hermetic, like the selfhost data dir. The
   // WorkOS emulator mints org ids from a per-process counter, so a persisted
   // DB from a previous invocation collides with the new boot's ids (identities
@@ -39,8 +46,8 @@ export default async function setup(): Promise<(() => Promise<void>) | void> {
   // MCP access tokens minted by the emulator's OAuth server must carry the
   // app's client id as audience (what the resource server verifies).
   process.env.EMULATE_WORKOS_AUDIENCE = E2E_WORKOS_CLIENT_ID;
-  const workos = await createEmulator({ service: "workos", port: WORKOS_EMULATOR_PORT });
-  const autumn = await createEmulator({ service: "autumn", port: AUTUMN_EMULATOR_PORT });
+  const workos = await createEmulator({ service: "workos", port: ports.E2E_WORKOS_EMULATOR_PORT! });
+  const autumn = await createEmulator({ service: "autumn", port: ports.E2E_AUTUMN_EMULATOR_PORT! });
 
   const env = {
     // Real client, emulated service.
@@ -51,16 +58,16 @@ export default async function setup(): Promise<(() => Promise<void>) | void> {
     WORKOS_COOKIE_PASSWORD: E2E_COOKIE_PASSWORD,
     AUTUMN_SECRET_KEY: "am_test_emulate",
     ENCRYPTION_KEY: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
-    DATABASE_URL: `postgresql://postgres:postgres@127.0.0.1:${CLOUD_DB_PORT}/postgres`,
+    DATABASE_URL: `postgresql://postgres:postgres@127.0.0.1:${dbPort}/postgres`,
     EXECUTOR_DIRECT_DATABASE_URL: "true",
     CLOUDFLARE_INCLUDE_PROCESS_ENV: "true",
-    VITE_PUBLIC_SITE_URL: CLOUD_BASE_URL,
+    VITE_PUBLIC_SITE_URL: baseUrl,
     // The AuthKit domain (MCP OAuth metadata + JWKS) is the emulator too.
     MCP_AUTHKIT_DOMAIN: workos.url,
-    MCP_RESOURCE_ORIGIN: CLOUD_BASE_URL,
+    MCP_RESOURCE_ORIGIN: baseUrl,
     ALLOW_LOCAL_NETWORK: "true",
     // Throwaway PGlite on its own port + dir so it never fights `bun dev`.
-    DEV_DB_PORT: String(CLOUD_DB_PORT),
+    DEV_DB_PORT: String(dbPort),
     DEV_DB_PATH: dbPath,
   };
 
@@ -69,7 +76,7 @@ export default async function setup(): Promise<(() => Promise<void>) | void> {
       { cmd: "bun", args: ["run", "scripts/dev-db.ts"], cwd: cloudDir, env },
       {
         cmd: "bunx",
-        args: ["vite", "dev", "--port", String(CLOUD_PORT), "--strictPort", "--host", "127.0.0.1"],
+        args: ["vite", "dev", "--port", String(cloudPort), "--strictPort", "--host", "127.0.0.1"],
         cwd: cloudDir,
         env,
       },
@@ -78,18 +85,20 @@ export default async function setup(): Promise<(() => Promise<void>) | void> {
   );
 
   try {
-    await waitForHttp(CLOUD_BASE_URL);
+    await waitForHttp(baseUrl);
     // The API plane is ready when login actually redirects to AuthKit.
-    await waitForHttp(`${CLOUD_BASE_URL}/api/auth/login`, { expectRedirect: true });
+    await waitForHttp(`${baseUrl}/api/auth/login`, { expectRedirect: true });
   } catch (error) {
     await procs.teardown();
     await workos.close();
     await autumn.close();
+    await release();
     throw error;
   }
   return async () => {
     await procs.teardown();
     await workos.close();
     await autumn.close();
+    await release();
   };
 }

e2e/setup/selfhost.globalsetup.ts

@@ -6,8 +6,9 @@ import { rmSync } from "node:fs";
 import { resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 
+import { claimPorts } from "../src/ports";
+import { SELFHOST_ADMIN } from "../targets/selfhost";
 import { bootProcesses, waitForHttp } from "./boot";
-import { SELFHOST_ADMIN, SELFHOST_BASE_URL, SELFHOST_PORT } from "../targets/selfhost";
 
 const selfhostDir = fileURLToPath(new URL("../../apps/host-selfhost/", import.meta.url));
 
@@ -17,6 +18,16 @@ export default async function setup(): Promise<(() => Promise<void>) | void> {
     return;
   }
 
+  // Claim a free port (preferred block first, walk forward past squatters)
+  // and publish via env so the test workers derive the same URL. The imported
+  // targets/selfhost constants were computed BEFORE the claim — don't use them
+  // for ports/URLs here.
+  const { ports, release } = await claimPorts([
+    { envVar: "E2E_SELFHOST_PORT", offset: 4, label: "selfhost vite dev" },
+  ]);
+  const port = ports.E2E_SELFHOST_PORT!;
+  const baseUrl = `http://localhost:${port}`;
+
   // Fresh data dir per suite run — hermetic; in-suite isolation comes from
   // fresh identities, not resets.
   const dataDir = resolve(selfhostDir, ".e2e-data");
@@ -26,14 +37,14 @@ export default async function setup(): Promise<(() => Promise<void>) | void> {
     [
       {
         cmd: "bunx",
-        args: ["--bun", "vite", "dev", "--port", String(SELFHOST_PORT), "--strictPort"],
+        args: ["--bun", "vite", "dev", "--port", String(port), "--strictPort"],
         cwd: selfhostDir,
         env: {
           EXECUTOR_DATA_DIR: dataDir,
           BETTER_AUTH_SECRET: "executor-selfhost-e2e-secret-0123456789",
           EXECUTOR_BOOTSTRAP_ADMIN_EMAIL: SELFHOST_ADMIN.email,
           EXECUTOR_BOOTSTRAP_ADMIN_PASSWORD: SELFHOST_ADMIN.password,
-          EXECUTOR_WEB_BASE_URL: SELFHOST_BASE_URL,
+          EXECUTOR_WEB_BASE_URL: baseUrl,
           // The harness boots loopback MCP/OAuth test servers and points the
           // instance at them; the hosted SSRF guard would otherwise block
           // outbound probes/dials to localhost. Hermetic test instance only.
@@ -45,10 +56,14 @@ export default async function setup(): Promise<(() => Promise<void>) | void> {
   );
 
   try {
-    await waitForHttp(SELFHOST_BASE_URL);
+    await waitForHttp(baseUrl);
   } catch (error) {
     await procs.teardown();
+    await release();
     throw error;
   }
-  return procs.teardown;
+  return async () => {
+    await procs.teardown();
+    await release();
+  };
 }