Skip to content

Commit 44fff2f

Browse files
authored
Auto-connect stdio MCP servers; store env as connection secrets (#1129)
* Auto-connect stdio MCP servers; store env as connection secrets Adding a stdio MCP server registered an integration but no connection, so the v1.5 per-connection tool model produced zero tools on a fresh install. Auto-create the default connection on add (no-auth, or one-shot env values), declare secret env vars as a stdio_env auth method whose values live on the connection's secret store, add a boot-time reconcile for pre-existing stdio integrations, and give the add form a TagInput for declaring env var names. * Derive stdio MCP display name from the package, not the runner The stdio add form fell back to the bare command (npx, uvx, ...) for both the display name and namespace, so every npm-launched server landed as "npx". Derive a name from the package/module being run instead: skip the runner and its flags/subcommands, drop the npm scope, version, path, and the usual MCP affixes (mcp-server-, server-, -mcp), then title-case. So "npx -y @modelcontextprotocol/server-github" becomes "Github MCP" and "uvx mcp-server-time" becomes "Time MCP", falling back to the raw command when nothing meaningful can be extracted. * Run the local e2e project in CI The local vitest project (e2e/local/**, including the stdio MCP auto-connect coverage) was excluded from both the default e2e chain and root `bun run test`, so it never ran in CI. Add a test:local script and a dedicated CI job that boots the scenarios with Node 22 and Chromium installed, since each one launches a real executor web server and some drive a browser. * Fix ci.yml: restore desktop-smoke job header The previous commit's edit consumed the desktop-smoke job header when inserting e2e-local, leaving its body orphaned with a duplicate steps key. GitHub rejected the workflow on startup. Restore the header. * Make local e2e reliable in CI: serialize boots, add headless shell The local project booted its executor web servers with file parallelism on, the only server-booting project that did. On a CI runner several cold vite boots at once thrashed past the token-URL wait, timing out stdio-mcp and auth. Run the files serially like every other server-booting project so the first boot warms the vite cache. Also install chromium-headless-shell, which the browser scenarios launch and which does not come with the chromium download. * Install Playwright browsers via e2e's pinned version Running the install from the repo root let bunx float to the latest playwright, which fetched a browser build the test runtime (playwright 1.60.0, pinned in e2e) never looks for, so the browser scenarios failed to launch. Install from e2e so the resolved playwright matches the one the scenarios use. * Scope the e2e CI job to the stdio MCP scenario Running the whole local project in one CI job stacked several executor web + vite boots on the runner; a later boot would intermittently stall past the token-URL wait, and the auth scenario carries its own pre-existing browser flakiness. Run just the stdio MCP scenario, the auto-connect / env-as-secret regression guard, on its own so the job is a reliable gate. Also give the cold-boot wait headroom (120s to 180s) for CI runners. Expanding to the full local suite is a follow-up once the rest is stabilized.
1 parent add2e40 commit 44fff2f

23 files changed

Lines changed: 914 additions & 102 deletions

.github/workflows/ci.yml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,44 @@ jobs:
7474

7575
- run: bun run test
7676

77+
e2e-local:
78+
name: E2E (stdio MCP)
79+
runs-on: ubuntu-latest
80+
steps:
81+
- uses: actions/checkout@v4
82+
83+
- uses: oven-sh/setup-bun@v2
84+
with:
85+
bun-version: 1.3.11
86+
87+
# The local scenarios boot a real `executor web` (which spawns a Node
88+
# sidecar) and some drive a browser, so pin Node 22 and install Chromium.
89+
- uses: actions/setup-node@v4
90+
with:
91+
node-version: 22
92+
93+
- run: bun install --frozen-lockfile
94+
95+
# `chromium` and the new `chromium-headless-shell` ship as separate
96+
# downloads; the browser-driven scenarios launch the headless shell.
97+
# Install from e2e so bunx resolves ITS pinned playwright (the version the
98+
# tests run against) rather than floating to the latest, which would fetch
99+
# a browser build the test runtime does not look for.
100+
- name: Install Playwright Chromium
101+
run: bunx playwright install --with-deps chromium chromium-headless-shell
102+
working-directory: e2e
103+
104+
# The `local` project is excluded from the default `test` chain (each
105+
# scenario boots its own `executor web`). Run just the stdio MCP scenario
106+
# here: it is the auto-connect / env-as-secret regression guard, and
107+
# running it alone avoids the boot-resource accumulation and the
108+
# pre-existing browser flakiness of the rest of the local suite. Expanding
109+
# to the full `local` project (bun run test:local) is a follow-up once
110+
# those are stabilized.
111+
- name: Run the stdio MCP scenario
112+
run: bunx vitest run --project local local/stdio-mcp.test.ts
113+
working-directory: e2e
114+
77115
desktop-smoke:
78116
name: Desktop smoke build
79117
runs-on: ubuntu-latest

apps/local/src/executor.ts

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import {
1414
} from "@executor-js/sdk";
1515
import { collectTables } from "@executor-js/api/server";
1616
import { loadPluginsFromJsonc } from "@executor-js/config";
17+
import type { McpPluginExtension } from "@executor-js/plugin-mcp";
1718

1819
import executorConfig from "../executor.config";
1920
import { localDataMigrations } from "./db/data-migrations";
@@ -229,6 +230,26 @@ const createLocalExecutorLayer = (options: LocalExecutorOptions = {}) => {
229230
}
230231
}
231232

233+
// Heal stdio MCP integrations added before auto-connect existed (they
234+
// landed with zero connections ⇒ zero tools) and move any legacy inline
235+
// env into the secret store. No-op on a fresh install; never fails boot.
236+
// Local is the only app that enables stdio, so this only runs here.
237+
// oxlint-disable-next-line executor/no-double-cast -- typed boundary: the executor IS its own plugin-extension map (executor[pluginId]) but LocalExecutor doesn't surface per-plugin extensions statically
238+
const mcpExtension = (executor as unknown as { readonly mcp?: McpPluginExtension }).mcp;
239+
if (mcpExtension) {
240+
yield* mcpExtension
241+
.reconcileStdioConnections()
242+
.pipe(
243+
Effect.catch(() =>
244+
Effect.sync(() =>
245+
console.warn(
246+
"[executor] stdio connection reconcile failed; existing stdio servers may show no tools until re-added",
247+
),
248+
),
249+
),
250+
);
251+
}
252+
232253
return { executor, plugins };
233254
}),
234255
);
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
// A zero-dependency MCP server over stdio, for the `local` e2e project.
2+
//
3+
// The real `@modelcontextprotocol/sdk` stdio server pulls in the whole SDK and
4+
// is awkward to resolve from an arbitrary spawn cwd under bun's node_modules
5+
// layout. The MCP stdio framing is just newline-delimited JSON-RPC, so we hand-
6+
// roll the three methods a tool-discovery + invoke round-trip needs:
7+
// `initialize`, `tools/list`, `tools/call` (plus `ping`). This keeps the
8+
// fixture a single self-contained file the executor server can launch as
9+
// `node <thisfile>` with nothing to install.
10+
//
11+
// It exposes one tool, `echo_tool`, and (when EXECUTOR_E2E_SECRET is set in the
12+
// child env) a second `whoami` tool that returns that env value — so a scenario
13+
// can prove a per-connection secret env var actually reached the subprocess.
14+
15+
import { createInterface } from "node:readline";
16+
17+
const send = (message) => {
18+
process.stdout.write(`${JSON.stringify(message)}\n`);
19+
};
20+
21+
const TOOLS = [
22+
{
23+
name: "echo_tool",
24+
description: "Echoes the provided text back",
25+
inputSchema: {
26+
type: "object",
27+
properties: { text: { type: "string" } },
28+
required: ["text"],
29+
},
30+
},
31+
];
32+
33+
if (process.env.EXECUTOR_E2E_SECRET) {
34+
TOOLS.push({
35+
name: "whoami",
36+
description: "Returns the secret env value the server was launched with",
37+
inputSchema: { type: "object", properties: {} },
38+
});
39+
}
40+
41+
const handle = (msg) => {
42+
if (msg.method === "initialize") {
43+
send({
44+
jsonrpc: "2.0",
45+
id: msg.id,
46+
result: {
47+
// Echo the client's protocol version so we never fail version
48+
// negotiation against whatever SDK build is on the other end.
49+
protocolVersion: msg.params?.protocolVersion ?? "2025-06-18",
50+
capabilities: { tools: {} },
51+
serverInfo: { name: "executor-e2e-stdio", version: "1.0.0" },
52+
},
53+
});
54+
return;
55+
}
56+
57+
// Notifications carry no id and expect no response.
58+
if (msg.id === undefined || msg.id === null) return;
59+
60+
if (msg.method === "ping") {
61+
send({ jsonrpc: "2.0", id: msg.id, result: {} });
62+
return;
63+
}
64+
65+
if (msg.method === "tools/list") {
66+
send({ jsonrpc: "2.0", id: msg.id, result: { tools: TOOLS } });
67+
return;
68+
}
69+
70+
if (msg.method === "tools/call") {
71+
const name = msg.params?.name;
72+
const text =
73+
name === "whoami"
74+
? (process.env.EXECUTOR_E2E_SECRET ?? "")
75+
: String(msg.params?.arguments?.text ?? "");
76+
send({
77+
jsonrpc: "2.0",
78+
id: msg.id,
79+
result: { content: [{ type: "text", text }] },
80+
});
81+
return;
82+
}
83+
84+
send({
85+
jsonrpc: "2.0",
86+
id: msg.id,
87+
error: { code: -32601, message: `Method not found: ${msg.method}` },
88+
});
89+
};
90+
91+
const rl = createInterface({ input: process.stdin });
92+
rl.on("line", (line) => {
93+
const trimmed = line.trim();
94+
if (!trimmed) return;
95+
let msg;
96+
// oxlint-disable-next-line executor/no-try-catch-or-throw -- standalone zero-dep fixture: hand-rolled JSON-RPC framing, not product code
97+
try {
98+
// oxlint-disable-next-line executor/no-json-parse -- standalone zero-dep fixture: hand-rolled JSON-RPC framing, not product code
99+
msg = JSON.parse(trimmed);
100+
} catch {
101+
return;
102+
}
103+
handle(msg);
104+
});

e2e/local/local-server.ts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,9 @@ export const withLocalServer = (
6161
markFocus(runDir, "terminal");
6262
const snapshot = await term.screen.waitUntil(
6363
(current) => TOKEN_URL.test(current.text),
64-
{ timeoutMs: 120_000 },
64+
// A cold `executor web` runs vite's optimizeDeps before it prints
65+
// the URL; give CI runners headroom over the warm ~3s boot.
66+
{ timeoutMs: 180_000 },
6567
);
6668
const url = TOKEN_URL.exec(snapshot.text)?.[0];
6769
if (!url) {

e2e/local/stdio-mcp.test.ts

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
// Repro + regression guard for the user report: "On a totally fresh install
2+
// (no existing data dir) on macOS, Executor does not detect any tools for a
3+
// STDIO MCP server."
4+
//
5+
// `withLocalServer` boots a real `executor web` on a THROWAWAY data dir (the
6+
// fresh-install condition) and the `local` app is the only surface that enables
7+
// stdio MCP (`dangerouslyAllowStdioMCP: true`). We add a stdio MCP server over
8+
// the bearer-authed API and assert its tools are discoverable — and that the
9+
// secret env it needs is stored on the connection (the secret store), not in
10+
// the integration's config blob.
11+
//
12+
// The original bug: `mcp.addServer` only registered an INTEGRATION. Per the
13+
// v1.5 integrations/connections split, tools are produced per-CONNECTION, and a
14+
// stdio add never created one, so the integration landed with zero connections
15+
// and zero tools. The fix auto-creates the default connection on add and routes
16+
// the env values into the connection's secret store.
17+
import { fileURLToPath } from "node:url";
18+
19+
import { expect } from "@effect/vitest";
20+
import { Effect } from "effect";
21+
import { HttpApiClient } from "effect/unstable/httpapi";
22+
import { FetchHttpClient, HttpClient, HttpClientRequest } from "effect/unstable/http";
23+
import { composePluginApi } from "@executor-js/api/server";
24+
import { mcpHttpPlugin } from "@executor-js/plugin-mcp/api";
25+
import { AuthTemplateSlug, ConnectionName, IntegrationSlug } from "@executor-js/sdk/shared";
26+
27+
import { scenario } from "../src/scenario";
28+
import { Cli, RunDir } from "../src/services";
29+
import { withLocalServer } from "./local-server";
30+
31+
const api = composePluginApi([mcpHttpPlugin()] as const);
32+
33+
const FIXTURE = fileURLToPath(new URL("./fixtures/stdio-mcp-server.mjs", import.meta.url));
34+
35+
// The fixture exposes `whoami` ONLY when EXECUTOR_E2E_SECRET is present in its
36+
// process env. So `whoami` showing up in the discovered tools is direct proof
37+
// the connection's secret env reached the spawned subprocess.
38+
const SECRET = "s3cr3t-from-the-vault";
39+
40+
scenario(
41+
"Local · a stdio MCP server's tools are detected on a fresh install, with env stored as a secret",
42+
{ timeout: 180_000 },
43+
Effect.gen(function* () {
44+
const cli = yield* Cli;
45+
const runDir = yield* RunDir;
46+
47+
yield* withLocalServer(cli, runDir, (server) =>
48+
Effect.gen(function* () {
49+
const client = yield* HttpApiClient.make(api, {
50+
baseUrl: new URL("/api", server.origin).toString(),
51+
transformClient: HttpClient.mapRequest((request) =>
52+
HttpClientRequest.setHeader(request, "authorization", `Bearer ${server.token}`),
53+
),
54+
}).pipe(Effect.provide(FetchHttpClient.layer));
55+
56+
const slug = "e2e-stdio";
57+
58+
// Add the stdio server exactly as the desktop/local "Add MCP" flow does,
59+
// including a secret env var the server needs.
60+
yield* client.mcp.addServer({
61+
payload: {
62+
transport: "stdio",
63+
name: "E2E Stdio",
64+
command: "node",
65+
args: [FIXTURE],
66+
env: { EXECUTOR_E2E_SECRET: SECRET },
67+
slug,
68+
},
69+
});
70+
71+
// The integration lands in the catalog — the add itself works.
72+
const integrations = yield* client.integrations.list();
73+
expect(
74+
integrations.map((i) => String(i.slug)),
75+
"the stdio MCP integration is registered",
76+
).toContain(slug);
77+
78+
// The add auto-creates the default connection (the v1.5 split makes this
79+
// the thing that drives tool discovery). Pre-fix there were zero.
80+
const connections = yield* client.connections.list({ query: { integration: slug } });
81+
expect(
82+
connections.map((c) => String(c.name)),
83+
"a default connection was auto-created for the stdio server",
84+
).toContain("default");
85+
86+
// THE SYMPTOM, fixed: the stdio server's tools are detected. `whoami`
87+
// appearing proves the connection's secret env reached the subprocess.
88+
const tools = yield* client.tools.list({ query: { integration: slug } });
89+
const names = tools.map((t) => t.name);
90+
expect(names, "the stdio server's base tool is detected").toContain("echo_tool");
91+
expect(
92+
names,
93+
"the secret env var reached the spawned subprocess (whoami is gated on it)",
94+
).toContain("whoami");
95+
96+
// "Properly store auth": the secret value is NOT in the integration's
97+
// config blob — only the var NAME is declared there; the value lives on
98+
// the connection (the secret store).
99+
const stored = yield* client.mcp.getServer({ params: { slug } });
100+
expect(
101+
JSON.stringify(stored?.config ?? {}),
102+
"the secret value is not persisted in the integration config",
103+
).not.toContain(SECRET);
104+
105+
// --- The UI path: DECLARE env var names, then provide the secret value
106+
// as a connection credential (what the add form now does). ---
107+
const declSlug = "e2e-stdio-decl";
108+
yield* client.mcp.addServer({
109+
payload: {
110+
transport: "stdio",
111+
name: "E2E Stdio Declared",
112+
command: "node",
113+
args: [FIXTURE],
114+
envVars: ["EXECUTOR_E2E_SECRET"],
115+
slug: declSlug,
116+
},
117+
});
118+
119+
// Declaring a secret env var (no value) does NOT auto-connect: the
120+
// secret is still missing, so there are no tools until you connect.
121+
const beforeConns = yield* client.connections.list({ query: { integration: declSlug } });
122+
expect(beforeConns, "no connection until the secret is provided").toHaveLength(0);
123+
const beforeTools = yield* client.tools.list({ query: { integration: declSlug } });
124+
expect(beforeTools, "no tools until the secret is provided").toHaveLength(0);
125+
126+
// Provide the secret as the connection credential (the connect step).
127+
yield* client.connections.create({
128+
payload: {
129+
owner: "org",
130+
name: ConnectionName.make("default"),
131+
integration: IntegrationSlug.make(declSlug),
132+
template: AuthTemplateSlug.make("env"),
133+
values: { EXECUTOR_E2E_SECRET: SECRET },
134+
},
135+
});
136+
137+
const declTools = yield* client.tools.list({ query: { integration: declSlug } });
138+
expect(
139+
declTools.map((t) => t.name),
140+
"connecting with the secret discovers the env-gated tool",
141+
).toContain("whoami");
142+
}),
143+
);
144+
}),
145+
);

e2e/package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
"test:selfhost": "vitest run --project selfhost",
1111
"test:selfhost-docker": "vitest run --project selfhost-docker",
1212
"test:cloudflare": "vitest run --project cloudflare",
13+
"test:local": "vitest run --project local",
1314
"test:watch": "vitest",
1415
"ports": "bun scripts/ports.ts",
1516
"summary": "bun scripts/summary.ts",

e2e/vitest.config.ts

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,15 +79,18 @@ export default defineConfig({
7979
}),
8080
// The single-user local app. Each scenario launches its OWN `executor
8181
// web` via the CLI on a throwaway data dir + an OS-assigned port, so
82-
// there is no shared instance and scenarios are independent — file
83-
// parallelism is ON. No globalSetup (nothing shared to boot). Only
84-
// local/** scenarios. Not part of the default `npm run test` chain; run
85-
// with `vitest run --project local`.
82+
// there is no shared instance and scenarios are independent. Files run
83+
// SERIALLY (like every other server-booting project): a cold `executor
84+
// web` boot runs vite's optimizeDeps, and several booting at once on a
85+
// CI runner thrash hard enough to blow the token-URL wait. Serial lets
86+
// the first boot warm the shared vite cache so the rest come up fast. No
87+
// globalSetup (nothing shared to boot). Only local/** scenarios. Not part
88+
// of the default `npm run test` chain; run with `vitest run --project local`.
8689
project("local", {
8790
include: ["local/**/*.test.ts"],
8891
globalSetup: [],
89-
fileParallelism: true,
90-
testTimeout: 180_000,
92+
fileParallelism: false,
93+
testTimeout: 240_000,
9194
}),
9295
// The supervised CLI daemon inside a guest VM, one project per OS. The
9396
// globalsetup provisions a VM, `executor service install`s the daemon, and

packages/core/api/src/integrations/api.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ const IntegrationParams = { slug: IntegrationSlug };
3131
/** Where a credential value is carried — mirrors the SDK's
3232
* `AuthPlacementDescriptor`. */
3333
const PlacementDescriptor = Schema.Struct({
34-
carrier: Schema.Literals(["header", "query"]),
34+
carrier: Schema.Literals(["header", "query", "env"]),
3535
name: Schema.String,
3636
prefix: Schema.String,
3737
/** Input variable this placement renders from (absent ⇒ `token`). Without

packages/core/sdk/src/integration.ts

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,11 @@ export interface IntegrationDisplayDescriptor {
2424
readonly url?: string;
2525
}
2626

27-
/** Where a credential value is carried on the outbound request. Mirrors the
28-
* client's `Placement`. */
27+
/** Where a credential value is carried. `header`/`query` place it on an
28+
* outbound HTTP request (mirrors the client's `Placement`); `env` injects it
29+
* as an environment variable for a stdio (subprocess) integration. */
2930
export interface AuthPlacementDescriptor {
30-
readonly carrier: "header" | "query";
31+
readonly carrier: "header" | "query" | "env";
3132
readonly name: string;
3233
/** Literal prepended to the value (e.g. `"Bearer "`). Empty when bare. */
3334
readonly prefix: string;

0 commit comments

Comments
 (0)