feat(agent): give cloud tasks isolated computer use

Generated-By: PostHog Code
Task-Id: c6d78416-85c8-4304-bd69-e309562468c4

Author: richardsolomou

docs/COMPUTER-USE.md

@@ -2,39 +2,37 @@
 
 # Computer use
 
-PostHog Code can give local agent sessions and cloud tasks tools to see and control the macOS desktop across supported agent adapters.
+PostHog Code can give local agent sessions tools to control the Mac and cloud tasks tools to control an isolated Linux desktop in their sandbox.
 
 ## Enable it
 
 1. Open **Settings → Advanced**.
 2. Enable **Computer use**.
 3. Start a new local session or cloud task.
-4. Grant Screen Recording and Accessibility access when macOS requests it.
+4. For local sessions, grant Screen Recording and Accessibility access when macOS requests it.
 
 The setting applies when a session or cloud task starts. Existing runs are unchanged.
 
 ## Behavior
 
 - Computer use is opt-in and disabled by default.
-- It is available to local sessions on macOS and cloud tasks created from the desktop app.
+- Local sessions control the user's Mac. Cloud tasks control a virtual Linux desktop inside their own sandbox.
 - Agents can capture the desktop, list visible applications, open or focus applications, click coordinates, type text, and press keys.
 - Claude, Codex, and future adapters receive the same PostHog-owned MCP tools rather than adapter-specific computer-control implementations.
-- Cloud tasks relay computer actions back to the desktop app, which must remain open and connected for the run to keep using them.
 - Unsupported operating systems do not receive the tools.
 - Tool calls use the existing MCP tool-call and approval pipeline.
 
 ## Safety
 
 - Review the screen before approving an action and verify the result after it runs.
 - Do not ask an agent to enter passwords, tokens, recovery codes, or other secrets.
-- Cloud computer actions require approval on the connected desktop unless the action was allowed for the rest of that run.
-- Disable the setting before starting a new session or cloud task to omit the tools. Closing the desktop app disconnects computer use from active cloud tasks.
+- Disable the setting before starting a new session or cloud task to omit the tools.
 - Revoke access in **System Settings → Privacy & Security → Screen Recording** or **Accessibility** to prevent native control.
 
 ## Scope
 
-The implementation uses macOS system utilities for screenshots, application launching, mouse input, and keyboard input. Cloud workers do not access the computer directly: PostHog Code designates a built-in MCP server for the cloud run, relays each request through the existing task command channel, and executes approved actions on the connected desktop. Relay designations are held in memory, so restarting the app disconnects active runs rather than silently reconnecting them.
+Local sessions use macOS system utilities for screenshots, application launching, mouse input, and keyboard input. Cloud tasks start a virtual X display and lightweight window manager in the sandbox, then use Linux desktop utilities for the same actions. The cloud desktop contains only the task sandbox and cannot access the user's computer.
 
 ## Implementation
 
-Computer actions are registered in the shared local-tools MCP registry. The Claude adapter exposes that registry through its in-process MCP server, while the Codex adapter and cloud relay expose the same registry through the packaged stdio MCP server. Local session metadata and cloud run relay designations gate the tools by operating system and the user's opt-in setting.
+Computer actions are registered in the shared local-tools MCP registry. The Claude adapter exposes that registry through its in-process MCP server, while the Codex adapter exposes the same registry through the packaged stdio MCP server. Session metadata gates the tools by environment, operating system, and the user's opt-in setting.

docs/cloud-mcp-relay.md

@@ -1,5 +1,3 @@
-<!-- markdownlint-disable MD013 -->
-
 # Design: relaying local MCP servers into cloud task runs
 
 Status: **implemented** (same PR as the import work), behind the same
@@ -18,14 +16,6 @@ them, and relayed servers stop working for in-flight runs (they 503 after the
 liveness window) rather than surviving a handoff. Tool allowlisting for
 relayed servers persists for the session lifetime, not to the backend.
 
-The built-in `posthog-code-computer-use` designation resolves to the packaged
-PostHog local-tools MCP server instead of `~/.claude.json`. It is added to new
-cloud runs when the desktop **Computer use** setting is enabled and exposes
-only the macOS computer tools. The same relay approval policy applies, so tool
-calls execute only while the creating desktop remains connected. Runs with
-imported or relayed MCP servers skip pre-warmed sandbox reuse because the
-server list is not known when the warm run starts.
-
 ## Problem
 
 Two classes of local MCP servers cannot be imported into a cloud sandbox by
@@ -63,7 +53,7 @@ The MCP relay mirrors this with a new event/command pair.
 
 ## Architecture
 
-```text
+```
 sandbox                                          desktop
 ┌──────────────────────────────┐                 ┌──────────────────────────────┐
 │ Claude/Codex adapter          │                 │ McpRelayService              │

packages/agent/package.json

@@ -72,10 +72,6 @@
       "types": "./dist/adapters/claude/mcp/tool-metadata.d.ts",
       "import": "./dist/adapters/claude/mcp/tool-metadata.js"
     },
-    "./adapters/local-tools/stdio-server": {
-      "types": "./dist/adapters/local-tools/stdio-server.d.ts",
-      "import": "./dist/adapters/local-tools/stdio-server.js"
-    },
     "./execution-mode": {
       "types": "./dist/execution-mode.d.ts",
       "import": "./dist/execution-mode.js"

packages/agent/src/adapters/codex-app-server/local-tools-mcp.test.ts

@@ -1,41 +1,20 @@
 import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
 import { LOCAL_TOOLS_MCP_NAME } from "../local-tools";
-import { buildComputerUseStdioServer } from "../local-tools/stdio-server";
 import { buildLocalToolsServer } from "./local-tools-mcp";
 
-// Isolate bundled-script resolution and the sandbox env file; the test only
-// inspects the generated stdio config.
+// The dist asset isn't on the walk-up path in unit tests, so make existsSync
+// succeed; nothing spawns the script — we only inspect the path.
 vi.mock("node:fs", async (importActual) => {
   const actual = await importActual<typeof import("node:fs")>();
-  return {
-    ...actual,
-    existsSync: vi.fn().mockReturnValue(true),
-    readFileSync: vi.fn(() => {
-      throw new Error("sandbox env file unavailable");
-    }),
-  };
+  return { ...actual, existsSync: vi.fn().mockReturnValue(true) };
 });
 
 describe("buildLocalToolsServer", () => {
-  it("builds the computer-only server for a macOS desktop relay", () => {
-    const server = buildComputerUseStdioServer("/repo", "darwin");
-
-    expect(server?.env.POSTHOG_LOCAL_TOOLS_ENABLED).toBe(
-      [
-        "computer_screenshot",
-        "computer_list_applications",
-        "computer_open_application",
-        "computer_click",
-        "computer_type",
-        "computer_key",
-      ].join(","),
-    );
-  });
-
   const saved = {
     sandbox: process.env.IS_SANDBOX,
     ghToken: process.env.GH_TOKEN,
     githubToken: process.env.GITHUB_TOKEN,
+    display: process.env.DISPLAY,
   };
 
   beforeEach(() => {
@@ -44,12 +23,14 @@ describe("buildLocalToolsServer", () => {
     delete process.env.IS_SANDBOX;
     delete process.env.GH_TOKEN;
     delete process.env.GITHUB_TOKEN;
+    delete process.env.DISPLAY;
   });
 
   afterEach(() => {
     restore("IS_SANDBOX", saved.sandbox);
     restore("GH_TOKEN", saved.ghToken);
     restore("GITHUB_TOKEN", saved.githubToken);
+    restore("DISPLAY", saved.display);
   });
 
   function restore(key: string, value: string | undefined): void {
@@ -174,6 +155,20 @@ describe("buildLocalToolsServer", () => {
     );
   });
 
+  it("exposes computer tools for opted-in cloud Linux sessions", () => {
+    process.env.DISPLAY = ":99";
+    const server = buildLocalToolsServer(
+      { cwd: "/repo", platform: "linux" },
+      { environment: "cloud", computerUse: true },
+    );
+
+    const enabled =
+      server?.env.find((entry) => entry.name === "POSTHOG_LOCAL_TOOLS_ENABLED")
+        ?.value ?? "";
+    expect(enabled.split(",")).toContain("computer_screenshot");
+    expect(server?.env).toContainEqual({ name: "DISPLAY", value: ":99" });
+  });
+
   it.each([
     {
       platform: "linux" as const,

packages/agent/src/adapters/codex-app-server/local-tools-mcp.ts

@@ -6,13 +6,15 @@
  */
 
 import type { McpServerStdio } from "@agentclientprotocol/sdk";
+import { ghTokenEnv } from "@posthog/git/signed-commit";
 import { resolveGithubToken } from "../../utils/github-token";
+import { resolveBundledMcpScript } from "../../utils/resolve-bundled-script";
 import {
   enabledLocalTools,
+  LOCAL_TOOLS_MCP_NAME,
   type LocalToolCtx,
   type LocalToolGateMeta,
 } from "../local-tools";
-import { buildLocalToolsStdioServer } from "../local-tools/stdio-server";
 import { resolveTaskId } from "../session-meta";
 
 /**
@@ -27,6 +29,42 @@ export interface LocalToolsMeta extends LocalToolGateMeta {
   baseBranch?: string;
 }
 
+function toMcpServerStdio(
+  ctx: LocalToolCtx,
+  enabledNames: string[],
+): McpServerStdio {
+  const scriptPath = resolveBundledMcpScript(
+    "adapters/codex-app-server/local-tools-mcp-server.js",
+  );
+  const ctxBase64 = Buffer.from(JSON.stringify(ctx)).toString("base64");
+  const env = [
+    { name: "POSTHOG_LOCAL_TOOLS_CTX", value: ctxBase64 },
+    { name: "POSTHOG_LOCAL_TOOLS_ENABLED", value: enabledNames.join(",") },
+    // Codex spawns this command with ELECTRON_RUN_AS_NODE removed from its own
+    // env (spawn.ts). In packaged desktop installs process.execPath is the app
+    // binary, which boots the full app without this var. Inert on real node.
+    { name: "ELECTRON_RUN_AS_NODE", value: "1" },
+  ];
+  if (process.env.DISPLAY) {
+    env.push({ name: "DISPLAY", value: process.env.DISPLAY });
+  }
+  if (ctx.token) {
+    // Token also on the child env so its own git remote ops authenticate.
+    env.push(
+      ...Object.entries(ghTokenEnv(ctx.token)).map(([name, value]) => ({
+        name,
+        value,
+      })),
+    );
+  }
+  return {
+    name: LOCAL_TOOLS_MCP_NAME,
+    command: process.execPath,
+    args: [scriptPath],
+    env,
+  };
+}
+
 /**
  * Returns the local-tools stdio server config to inject, or null when no tool's
  * gate passes (e.g. local/desktop run with no GH token). Tools self-gate via the
@@ -52,12 +90,8 @@ export function buildLocalToolsServer(
   if (tools.length === 0) {
     return null;
   }
-  const server = buildLocalToolsStdioServer(
+  return toMcpServerStdio(
     toolCtx,
     tools.map((t) => t.name),
   );
-  return {
-    ...server,
-    env: Object.entries(server.env).map(([name, value]) => ({ name, value })),
-  };
 }

packages/agent/src/adapters/local-tools/registry.ts

@@ -31,7 +31,7 @@ export interface LocalToolGateMeta {
   channelMode?: boolean;
   /** Spoken narration is on for this session: enables the speak tool. */
   spokenNarration?: boolean;
-  /** Desktop computer control is enabled for this local session. */
+  /** Computer control is enabled for this session. */
   computerUse?: boolean;
 }
 

packages/agent/src/adapters/local-tools/stdio-server.ts

@@ -1,4 +1,4 @@
-import { execFile } from "node:child_process";
+import { execFile, spawn } from "node:child_process";
 import { readFile, unlink } from "node:fs/promises";
 import { beforeEach, describe, expect, it, vi } from "vitest";
 import {
@@ -10,18 +10,21 @@ import {
   computerUseTools,
 } from "./computer-use";
 
-vi.mock("node:child_process", () => ({ execFile: vi.fn() }));
+vi.mock("node:child_process", () => ({ execFile: vi.fn(), spawn: vi.fn() }));
 vi.mock("node:fs/promises", () => ({
   readFile: vi.fn(),
   unlink: vi.fn().mockResolvedValue(undefined),
 }));
 
 const mockedExecFile = vi.mocked(execFile);
+const mockedSpawn = vi.mocked(spawn);
 const mockedReadFile = vi.mocked(readFile);
 const mockedUnlink = vi.mocked(unlink);
 
 const context = { cwd: "/repo", platform: "darwin" as const };
 const enabledMeta = { environment: "local" as const, computerUse: true };
+const cloudContext = { cwd: "/repo", platform: "linux" as const };
+const cloudMeta = { environment: "cloud" as const, computerUse: true };
 
 describe("computer use tools", () => {
   beforeEach(() => {
@@ -33,6 +36,7 @@ describe("computer use tools", () => {
       }
       return undefined;
     }) as unknown as typeof execFile);
+    mockedSpawn.mockReturnValue({ unref: vi.fn() } as never);
     mockedReadFile.mockResolvedValue(Buffer.from("png"));
     mockedUnlink.mockResolvedValue(undefined);
   });
@@ -71,6 +75,12 @@ describe("computer use tools", () => {
     expect(computerScreenshotTool.isEnabled(context, enabledMeta)).toBe(true);
   });
 
+  it("is enabled for opted-in cloud Linux sessions", () => {
+    expect(computerScreenshotTool.isEnabled(cloudContext, cloudMeta)).toBe(
+      true,
+    );
+  });
+
   it("returns a screenshot image and removes the temporary file", async () => {
     const result = await computerScreenshotTool.handler(context, {});
 
@@ -130,6 +140,33 @@ describe("computer use tools", () => {
     );
   });
 
+  it("opens the cloud browser without invoking a shell", async () => {
+    const unref = vi.fn();
+    mockedSpawn.mockReturnValueOnce({ unref } as never);
+
+    await computerOpenApplicationTool.handler(cloudContext, {
+      application: "Browser",
+    });
+
+    expect(mockedSpawn).toHaveBeenCalledWith(
+      "/usr/bin/epiphany",
+      ["--new-window"],
+      { detached: true, stdio: "ignore" },
+    );
+    expect(unref).toHaveBeenCalledOnce();
+  });
+
+  it("uses Linux desktop utilities in cloud sessions", async () => {
+    await computerClickTool.handler(cloudContext, { x: 100, y: 200 });
+
+    expect(mockedExecFile).toHaveBeenCalledWith(
+      "/usr/bin/xdotool",
+      ["mousemove", "100", "200", "click", "1"],
+      { encoding: "utf8" },
+      expect.any(Function),
+    );
+  });
+
   it.each([
     [computerClickTool, { x: 100, y: 200 }],
     [computerTypeTool, { text: "hello" }],