feat(agent): relay computer use to cloud tasks

Generated-By: PostHog Code
Task-Id: c424306e-3221-4201-bbce-83723deb6a13

Author: richardsolomou

docs/COMPUTER-USE.md

@@ -2,37 +2,39 @@
 
 # Computer use
 
-PostHog Code can give local agent sessions tools to see and control the macOS desktop across supported agent adapters.
+PostHog Code can give local agent sessions and cloud tasks tools to see and control the macOS desktop across supported agent adapters.
 
 ## Enable it
 
 1. Open **Settings → Advanced**.
 2. Enable **Computer use**.
-3. Start a new local session.
+3. Start a new local session or cloud task.
 4. Grant Screen Recording and Accessibility access when macOS requests it.
 
-The setting applies when a session starts. Existing sessions are unchanged.
+The setting applies when a session or cloud task starts. Existing runs are unchanged.
 
 ## Behavior
 
 - Computer use is opt-in and disabled by default.
-- It is available only to local sessions on macOS.
+- It is available to local sessions on macOS and cloud tasks created from the desktop app.
 - Agents can capture the desktop, list visible applications, open or focus applications, click coordinates, type text, and press keys.
 - Claude, Codex, and future adapters receive the same PostHog-owned MCP tools rather than adapter-specific computer-control implementations.
-- Cloud sessions and unsupported operating systems do not receive the tools.
+- Cloud tasks relay computer actions back to the desktop app, which must remain open and connected for the run to keep using them.
+- Unsupported operating systems do not receive the tools.
 - Tool calls use the existing MCP tool-call and approval pipeline.
 
 ## Safety
 
 - Review the screen before approving an action and verify the result after it runs.
 - Do not ask an agent to enter passwords, tokens, recovery codes, or other secrets.
-- Disable the setting and start a new session to remove the tools.
+- Cloud computer actions require approval on the connected desktop unless the action was allowed for the rest of that run.
+- Disable the setting before starting a new session or cloud task to omit the tools. Closing the desktop app disconnects computer use from active cloud tasks.
 - Revoke access in **System Settings → Privacy & Security → Screen Recording** or **Accessibility** to prevent native control.
 
 ## Scope
 
-The initial implementation uses macOS system utilities for screenshots, application launching, mouse input, and keyboard input. It does not provide computer control to cloud tasks because those tasks do not run on the user's computer. Remote computer use would require an explicit device connection, user-presence controls, and a separate security design.
+The implementation uses macOS system utilities for screenshots, application launching, mouse input, and keyboard input. Cloud workers do not access the computer directly: PostHog Code designates a built-in MCP server for the cloud run, relays each request through the existing task command channel, and executes approved actions on the connected desktop. Relay designations are held in memory, so restarting the app disconnects active runs rather than silently reconnecting them.
 
 ## Implementation
 
-Computer actions are registered in the shared local-tools MCP registry. The Claude adapter exposes that registry through its in-process MCP server, while the Codex adapter exposes the same registry through its stdio MCP server. Session metadata gates the tools by environment, operating system, and the user's opt-in setting.
+Computer actions are registered in the shared local-tools MCP registry. The Claude adapter exposes that registry through its in-process MCP server, while the Codex adapter and cloud relay expose the same registry through the packaged stdio MCP server. Local session metadata and cloud run relay designations gate the tools by operating system and the user's opt-in setting.

docs/cloud-mcp-relay.md

@@ -1,3 +1,5 @@
+<!-- markdownlint-disable MD013 -->
+
 # Design: relaying local MCP servers into cloud task runs
 
 Status: **implemented** (same PR as the import work), behind the same
@@ -16,6 +18,12 @@ them, and relayed servers stop working for in-flight runs (they 503 after the
 liveness window) rather than surviving a handoff. Tool allowlisting for
 relayed servers persists for the session lifetime, not to the backend.
 
+The built-in `posthog-code-computer-use` designation resolves to the packaged
+PostHog local-tools MCP server instead of `~/.claude.json`. It is added to new
+cloud runs when the desktop **Computer use** setting is enabled and exposes
+only the macOS computer tools. The same relay approval policy applies, so tool
+calls execute only while the creating desktop remains connected.
+
 ## Problem
 
 Two classes of local MCP servers cannot be imported into a cloud sandbox by
@@ -53,7 +61,7 @@ The MCP relay mirrors this with a new event/command pair.
 
 ## Architecture
 
-```
+```text
 sandbox                                          desktop
 ┌──────────────────────────────┐                 ┌──────────────────────────────┐
 │ Claude/Codex adapter          │                 │ McpRelayService              │

packages/agent/package.json

@@ -72,6 +72,10 @@
       "types": "./dist/adapters/claude/mcp/tool-metadata.d.ts",
       "import": "./dist/adapters/claude/mcp/tool-metadata.js"
     },
+    "./adapters/local-tools/stdio-server": {
+      "types": "./dist/adapters/local-tools/stdio-server.d.ts",
+      "import": "./dist/adapters/local-tools/stdio-server.js"
+    },
     "./execution-mode": {
       "types": "./dist/execution-mode.d.ts",
       "import": "./dist/execution-mode.js"

packages/agent/src/adapters/codex-app-server/local-tools-mcp.test.ts

@@ -1,5 +1,6 @@
 import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
 import { LOCAL_TOOLS_MCP_NAME } from "../local-tools";
+import { buildComputerUseStdioServer } from "../local-tools/stdio-server";
 import { buildLocalToolsServer } from "./local-tools-mcp";
 
 // The dist asset isn't on the walk-up path in unit tests, so make existsSync
@@ -10,6 +11,21 @@ vi.mock("node:fs", async (importActual) => {
 });
 
 describe("buildLocalToolsServer", () => {
+  it("builds the computer-only server for a macOS desktop relay", () => {
+    const server = buildComputerUseStdioServer("/repo", "darwin");
+
+    expect(server?.env.POSTHOG_LOCAL_TOOLS_ENABLED).toBe(
+      [
+        "computer_screenshot",
+        "computer_list_applications",
+        "computer_open_application",
+        "computer_click",
+        "computer_type",
+        "computer_key",
+      ].join(","),
+    );
+  });
+
   const saved = {
     sandbox: process.env.IS_SANDBOX,
     ghToken: process.env.GH_TOKEN,

packages/agent/src/adapters/codex-app-server/local-tools-mcp.ts

@@ -6,15 +6,13 @@
  */
 
 import type { McpServerStdio } from "@agentclientprotocol/sdk";
-import { ghTokenEnv } from "@posthog/git/signed-commit";
 import { resolveGithubToken } from "../../utils/github-token";
-import { resolveBundledMcpScript } from "../../utils/resolve-bundled-script";
 import {
   enabledLocalTools,
-  LOCAL_TOOLS_MCP_NAME,
   type LocalToolCtx,
   type LocalToolGateMeta,
 } from "../local-tools";
+import { buildLocalToolsStdioServer } from "../local-tools/stdio-server";
 import { resolveTaskId } from "../session-meta";
 
 /**
@@ -29,39 +27,6 @@ export interface LocalToolsMeta extends LocalToolGateMeta {
   baseBranch?: string;
 }
 
-function toMcpServerStdio(
-  ctx: LocalToolCtx,
-  enabledNames: string[],
-): McpServerStdio {
-  const scriptPath = resolveBundledMcpScript(
-    "adapters/codex-app-server/local-tools-mcp-server.js",
-  );
-  const ctxBase64 = Buffer.from(JSON.stringify(ctx)).toString("base64");
-  const env = [
-    { name: "POSTHOG_LOCAL_TOOLS_CTX", value: ctxBase64 },
-    { name: "POSTHOG_LOCAL_TOOLS_ENABLED", value: enabledNames.join(",") },
-    // Codex spawns this command with ELECTRON_RUN_AS_NODE removed from its own
-    // env (spawn.ts). In packaged desktop installs process.execPath is the app
-    // binary, which boots the full app without this var. Inert on real node.
-    { name: "ELECTRON_RUN_AS_NODE", value: "1" },
-  ];
-  if (ctx.token) {
-    // Token also on the child env so its own git remote ops authenticate.
-    env.push(
-      ...Object.entries(ghTokenEnv(ctx.token)).map(([name, value]) => ({
-        name,
-        value,
-      })),
-    );
-  }
-  return {
-    name: LOCAL_TOOLS_MCP_NAME,
-    command: process.execPath,
-    args: [scriptPath],
-    env,
-  };
-}
-
 /**
  * Returns the local-tools stdio server config to inject, or null when no tool's
  * gate passes (e.g. local/desktop run with no GH token). Tools self-gate via the
@@ -87,8 +52,12 @@ export function buildLocalToolsServer(
   if (tools.length === 0) {
     return null;
   }
-  return toMcpServerStdio(
+  const server = buildLocalToolsStdioServer(
     toolCtx,
     tools.map((t) => t.name),
   );
+  return {
+    ...server,
+    env: Object.entries(server.env).map(([name, value]) => ({ name, value })),
+  };
 }

packages/agent/src/adapters/local-tools/stdio-server.ts

@@ -0,0 +1,57 @@
+import { ghTokenEnv } from "@posthog/git/signed-commit";
+import { resolveBundledMcpScript } from "../../utils/resolve-bundled-script";
+import {
+  LOCAL_TOOLS_MCP_NAME,
+  type LocalToolCtx,
+  type LocalToolGateMeta,
+} from "./registry";
+import { computerUseTools } from "./tools/computer-use";
+
+export interface LocalToolsStdioServer {
+  name: string;
+  command: string;
+  args: string[];
+  env: Record<string, string>;
+}
+
+export function buildLocalToolsStdioServer(
+  ctx: LocalToolCtx,
+  enabledNames: string[],
+): LocalToolsStdioServer {
+  const scriptPath = resolveBundledMcpScript(
+    "adapters/codex-app-server/local-tools-mcp-server.js",
+  );
+  const env: Record<string, string> = {
+    POSTHOG_LOCAL_TOOLS_CTX: Buffer.from(JSON.stringify(ctx)).toString(
+      "base64",
+    ),
+    POSTHOG_LOCAL_TOOLS_ENABLED: enabledNames.join(","),
+    ELECTRON_RUN_AS_NODE: "1",
+  };
+  if (ctx.token) {
+    Object.assign(env, ghTokenEnv(ctx.token));
+  }
+  return {
+    name: LOCAL_TOOLS_MCP_NAME,
+    command: process.execPath,
+    args: [scriptPath],
+    env,
+  };
+}
+
+export function buildComputerUseStdioServer(
+  cwd: string,
+  platform: NodeJS.Platform = process.platform,
+): LocalToolsStdioServer | null {
+  const ctx: LocalToolCtx = { cwd, platform };
+  const meta: LocalToolGateMeta = {
+    environment: "local",
+    computerUse: true,
+  };
+  const tools = computerUseTools.filter((tool) => tool.isEnabled(ctx, meta));
+  if (tools.length === 0) return null;
+  return buildLocalToolsStdioServer(
+    ctx,
+    tools.map((tool) => tool.name),
+  );
+}

packages/agent/tsup.config.ts

@@ -124,6 +124,7 @@ export default defineConfig([
       "src/adapters/codex-app-server/models.ts",
       "src/adapters/codex-app-server/local-tools-mcp-server.ts",
       "src/adapters/claude/mcp/tool-metadata.ts",
+      "src/adapters/local-tools/stdio-server.ts",
       "src/adapters/reasoning-effort.ts",
       "src/execution-mode.ts",
       "src/server/schemas.ts",

packages/core/src/local-mcp/localMcpImport.test.ts

@@ -1,6 +1,10 @@
-import type { LocalMcpServerDescriptor } from "@posthog/shared";
+import {
+  CLOUD_COMPUTER_USE_MCP_NAME,
+  type LocalMcpServerDescriptor,
+} from "@posthog/shared";
 import { describe, expect, it } from "vitest";
 import {
+  addCloudComputerUseRelay,
   classifyLocalMcpServer,
   isPrivateHostname,
   type LocalMcpCloudClassification,
@@ -199,6 +203,21 @@ describe("LocalMcpImportService", () => {
 });
 
 describe("partitionLocalMcpServersForRun", () => {
+  it("puts built-in computer use first in the relay designation", () => {
+    const result = addCloudComputerUseRelay([], true);
+
+    expect(result).toEqual([{ name: CLOUD_COMPUTER_USE_MCP_NAME }]);
+  });
+
+  it("does not duplicate the built-in computer use relay name", () => {
+    const result = addCloudComputerUseRelay(
+      [{ name: CLOUD_COMPUTER_USE_MCP_NAME }],
+      true,
+    );
+
+    expect(result).toEqual([{ name: CLOUD_COMPUTER_USE_MCP_NAME }]);
+  });
+
   const importable = (name: string): LocalMcpCloudClassification => ({
     name,
     availability: "importable",

packages/core/src/local-mcp/localMcpImport.ts

@@ -4,7 +4,11 @@ import type {
   CloudMcpServerRelayDesignation,
   LocalMcpServerDescriptor,
 } from "@posthog/shared";
-import { isPrivateIpv4Octets, isPrivateIpv6Literal } from "@posthog/shared";
+import {
+  CLOUD_COMPUTER_USE_MCP_NAME,
+  isPrivateIpv4Octets,
+  isPrivateIpv6Literal,
+} from "@posthog/shared";
 import { inject, injectable } from "inversify";
 import { LOCAL_MCP_WORKSPACE_CLIENT } from "./identifiers";
 
@@ -151,6 +155,17 @@ export interface LocalMcpServersForRun {
   relayed: CloudMcpServerRelayDesignation[];
 }
 
+export function addCloudComputerUseRelay(
+  servers: CloudMcpServerRelayDesignation[],
+  enabled: boolean,
+): CloudMcpServerRelayDesignation[] {
+  if (!enabled) return servers;
+  return [
+    { name: CLOUD_COMPUTER_USE_MCP_NAME },
+    ...servers.filter((server) => server.name !== CLOUD_COMPUTER_USE_MCP_NAME),
+  ].slice(0, MAX_RELAYED_MCP_SERVERS);
+}
+
 /**
  * Split classified local servers into the run-creation payload's imported and
  * relayed lists for the run's adapter.
@@ -170,7 +185,7 @@ export function partitionLocalMcpServersForRun(
   const imported = relayImportable
     ? []
     : servers.flatMap((server) => (server.remote ? [server.remote] : []));
-  const relayed = servers
+  const relayedServers = servers
     .filter(
       (server) =>
         server.availability === "requires_desktop" ||
@@ -185,8 +200,8 @@ export function partitionLocalMcpServersForRun(
           ? -1
           : 1,
     )
-    .slice(0, MAX_RELAYED_MCP_SERVERS)
     .map((server) => ({ name: server.name }));
+  const relayed = relayedServers.slice(0, MAX_RELAYED_MCP_SERVERS);
   return { imported, relayed };
 }
 

packages/core/src/task-detail/taskCreationHost.ts

@@ -76,6 +76,7 @@ export interface RecordClaudeCliImportArgs {
 export interface ITaskCreationHost {
   getAuthenticatedClient(): Promise<TaskCreationApiClient | null>;
   assertCloudUsageAvailable(): Promise<void>;
+  isComputerUseEnabled(): boolean;
   getTaskDirectory(taskId: string, repoKey?: string): Promise<string | null>;
   /**
    * Ensure a per-task scratch working directory exists for a repo-less channel