Skip to content

Commit 31e4887

Browse files
feat(launcher): local launcher for HTTP/SSE upstreams (spec 046) (#452)
* docs(spec): plan 046 — local launcher for HTTP/SSE upstreams Adds the implementation plan for letting mcpproxy spawn a local command before connecting to its HTTP/SSE endpoint. Today the command field is silently ignored when protocol is http/sse/ streamable-http; this plan decouples launcher from transport so {command, url, protocol: http|sse|streamable-http} becomes a first-class config combo (spawn, wait for URL, connect, own lifecycle on disconnect/restart/shutdown). Plan-only commit — no code yet. Phase 0 of the plan is a no- behaviour-change refactor that lifts env/Docker/working-dir plumbing out of connection_stdio.go into a new internal/upstream/ launcher/ package; Phase 1 wires it into the HTTP/SSE connect path; Phase 2 lands tests + docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(launcher): add internal/upstream/launcher package (Phase 0) New self-contained package that mcpproxy will use to spawn the local upstream process behind HTTP/SSE/streamable-HTTP transports. Phase 0 of spec 046: the launcher is fully implemented and tested, but no Connect/Disconnect path uses it yet — that wiring lands in the next commit so reviewers can read the lifecycle module on its own. API surface: - Spec — caller-built *exec.Cmd plus LogSink/Name/StopGrace. - Handle — Stop/Wait/Done/Pid; Stop is idempotent and waits for the child to be reaped before returning so "Stop returned -> port is free" is reliable. - Spawn(ctx,spec,log) — owns child lifecycle from cmd.Start onward. Pumps stdout+stderr line-by-line into LogSink (one Write per line, serialized internally so an arbitrary io.Writer is safe). - WaitForURL(ctx,url,timeout) — TCP-dial polling, NOT http.Get (gotcha #2 in the spec: SSE GETs stream forever). Infers default ports for http/https/ws/wss. Process group handling is unix-only via applyProcAttrs (Setpgid + Pgid=0), so SIGTERM/SIGKILL reach grandchildren spawned by sh -c ... and docker run .... Windows gets best-effort stubs that match the process_windows.go TODO already in core/. Tests: 15 cases covering immediate/late/never-bound listeners, ctx cancel, bad-URL parse rejection, default-port inference, graceful SIGTERM exit, SIGKILL fallback after StopGrace, natural exit code capture, idempotent Stop under concurrent callers, and LogSink capture. Integration test exercises Spawn + WaitForURL together with a python listener subprocess (skipped if python3 missing). No behaviour change in this commit — the package is dead code until the next one. Refs spec: specs/046-local-launcher-for-http-sse/plan.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(upstream): launch local HTTP/SSE upstreams via launcher (Phase 1) Wires internal/upstream/launcher into Client.Connect for the http, sse, and streamable-http transports when ServerConfig.Command is also set. The launcher spawns the child process, blocks on launcher.WaitForURL until the listener accepts a TCP connection, then hands off to the existing transport-level connectHTTP / connectSSE. On Disconnect, the MCP client is closed first (so the child sees the network drop cleanly) and the launcher Handle is Stop()ed second (SIGTERM -> grace -> SIGKILL). Stdio is deliberately excluded: mcp-go's Stdio transport spawns its own child via its CommandFunc, and routing that through launcher.Spawn would require patching mcp-go. Instead, both the stdio CommandFunc path and the new HTTP/SSE launcher path call into the same set of *Client command-prep helpers (setupDockerIsolation, wrapWithUserShell, injectEnvVarsIntoDockerArgs, insertCidfileIntoShellDockerCommand), preserving the spec's "Docker isolation in one place" requirement without the riskier stdio refactor. Config / API: - ServerConfig.LauncherWaitTimeout (Duration, default 30s) caps the wait between spawn and listener-up. CopyServerConfig carries it. - Client gains launcherHandle / launcherCIDFile fields. Stdio Clients leave them nil so existing cleanup paths (killProcessGroup, processCmd.Process.Kill) keep their single owner. Behaviour back-compat: when Command is set together with a URL but no explicit protocol, Command still wins -> stdio, URL ignored (matches today). To opt into the launcher you must set protocol to "http", "sse", or "streamable-http" explicitly. Locking: Connect holds c.mu for its duration, so the failure-cleanup path inlines the launcher teardown and releases c.mu briefly around handle.Stop (which can block until the child is reaped). The connecting flag prevents a concurrent Connect from sneaking in during that window. DisconnectWithContext releases c.mu before its stopLauncher call, so the public method uses normal locking. Watch goroutine: on unexpected child exit while connected, watchLauncher invokes Disconnect so the existing reconnect loop in internal/upstream/manager takes over instead of waiting for the transport's keepalive to time out. Verified by go vet + the full internal/upstream/... and internal/config/... test suites (existing tests stay green; deadlock in connect-failure cleanup that surfaced in TestClient_Connect_SSE_ NotSupported was fixed before commit). Refs spec: specs/046-local-launcher-for-http-sse/plan.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(spec): cover local-launcher HTTP/SSE config + restart semantics (Phase 2) - docs/configuration.md: new "Locally-launched HTTP / SSE servers" section walks through the {command, url, protocol: http|sse| streamable-http} combo, a config example with launcher_wait_timeout, per-server log routing, and a back-compat behaviour matrix that makes the "command + url under auto -> stdio wins" footgun explicit. - docs/cli-management-commands.md: restart semantics note covering the launcher stop-then-start order and the 5s SIGKILL grace timeout. - specs/046-local-launcher-for-http-sse/execution_log.md: design decisions, deviations from the original plan (stdio refactor scoped down to shared command-prep helpers), bugs found and fixed during verification (connect deadlock, bytes.Buffer race, banner false- positive in SIGKILL test, ftp-with-port acceptance), and the exact verification commands. Refs spec: specs/046-local-launcher-for-http-sse/plan.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): add launcher-lifecycle gate driving a real HTTP MCP fixture Closes the gap unit tests can't: proves a built mcpproxy binary actually spawns a self-launched HTTP MCP server, completes the MCP handshake, serves tools/list through the proxy, and reaps the child cleanly on disable / restart / shutdown. What's new: - test/launcher-server/main.go — minimal HTTP MCP fixture (~220 LOC). Implements initialize, tools/list (returns a "ping" tool), tools/call, plus 405-on-GET so mcp-go's StreamableHTTP transport falls back to POST-only. Exits cleanly on SIGTERM (5s shutdown). Heartbeat to stdout proves the per-server log pump works. - test/e2e-config.template.json — new "launcher-test" upstream ({protocol: http, command: ./test/launcher-server/launcher-server, url: http://127.0.0.1:39933/mcp, launcher_wait_timeout: 10s}). - scripts/test-api-e2e.sh: * Prereq step now `go build`s the fixture before booting mcpproxy. * New wait_for_launcher_test_server() polls /servers until launcher-test reports connected. * New test_launcher_lifecycle() runs six sub-assertions: 1. tools/list returns the fixture's "ping" tool through the proxy (proves Spawn -> WaitForURL -> connectHTTP -> MCP initialize -> ListTools end-to-end). 2. pgrep finds the child process by argv. 3. POST /servers/launcher-test/restart yields a different PID. 4. POST /servers/launcher-test/disable reaps the child within 8s. 5. POST /servers/launcher-test/enable respawns it with a fresh PID. 6. /servers/launcher-test/logs?tail=200 contains the launcher banner or fixture stdout. * cleanup() pkill's any leaked launcher-server processes so a failure mid-test doesn't taint the next run. .gitignore: ignore the built fixture binary (/test/launcher-server/launcher-server). Source stays tracked. Refs spec: specs/046-local-launcher-for-http-sse/plan.md (Phase 2 e2e follow-up that was listed as outstanding in the PR test plan). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent a11d002 commit 31e4887

21 files changed

Lines changed: 2323 additions & 14 deletions

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
*.so
66
*.dylib
77
/mcpproxy
8+
/test/launcher-server/launcher-server
89
__debug_bin*
910

1011
# Playwright MCP artifacts

docs/cli-management-commands.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -352,6 +352,14 @@ mcpproxy upstream restart --all
352352

353353
**Note:** Restart does not require confirmation as it's non-destructive.
354354

355+
**Locally-launched HTTP/SSE upstreams:** when a server is configured with both
356+
`command` and an HTTP/SSE `url` (see [docs/configuration.md](configuration.md#locally-launched-http--sse-servers)),
357+
`restart` stops the spawned child (`SIGTERM` → grace → `SIGKILL`) before
358+
re-running Connect. The grace timeout is fixed at 5s today; the next start
359+
won't begin until the previous child is fully reaped, so you can rely on the
360+
port being free after the command returns. Stop ordering is: close MCP client
361+
→ stop launched child → release per-server state.
362+
355363
---
356364

357365
### `mcpproxy doctor`

docs/configuration.md

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -147,8 +147,9 @@ MCPProxy looks for configuration in these locations (in order):
147147
| `args` | array | No | Command arguments |
148148
| `url` | string | Yes* | Server URL (required for `http`/`sse`/`streamable-http` protocols) |
149149
| `headers` | object | No | HTTP headers for HTTP-based protocols |
150-
| `working_dir` | string | No | Working directory for stdio servers (default: current directory) |
151-
| `env` | object | No | Environment variables for stdio servers |
150+
| `working_dir` | string | No | Working directory for stdio servers, or for the locally-launched child of an HTTP/SSE server (default: current directory) |
151+
| `env` | object | No | Environment variables for stdio servers, or for the locally-launched child of an HTTP/SSE server |
152+
| `launcher_wait_timeout` | duration | No | When `command` is set together with an HTTP/SSE `url`, how long mcpproxy waits for that URL to become reachable after spawning the child (e.g. `"15s"`, default `"30s"`) |
152153
| `oauth` | object | No | OAuth configuration (see [OAuth Configuration](#oauth-configuration)) |
153154
| `isolation` | object | No | Per-server Docker isolation settings (see [Docker Isolation](#docker-isolation)) |
154155
| `enabled` | boolean | No | Enable/disable server (default: `true`) |
@@ -210,6 +211,54 @@ MCPProxy looks for configuration in these locations (in order):
210211
}
211212
```
212213

214+
### Locally-launched HTTP / SSE servers
215+
216+
By default `command` is only used for `stdio` servers. When you set `command`
217+
together with an HTTP/SSE `url` and an explicit `protocol` of `http`, `sse`,
218+
or `streamable-http`, mcpproxy will:
219+
220+
1. Spawn the command (with `args`, `env`, `working_dir`, and Docker isolation
221+
exactly like a stdio server).
222+
2. Wait up to `launcher_wait_timeout` (default 30s) for `url` to accept a TCP
223+
connection.
224+
3. Connect via the configured HTTP/SSE transport.
225+
4. Own the child's lifecycle — the process is stopped (`SIGTERM`, then
226+
`SIGKILL` after a grace period) on disconnect, restart, server-disable, or
227+
mcpproxy shutdown. Unexpected exits trigger an automatic disconnect, which
228+
the existing reconnect path picks up.
229+
230+
```json
231+
{
232+
"name": "local-http-mcp",
233+
"protocol": "http",
234+
"url": "http://127.0.0.1:9999/mcp",
235+
"command": "node",
236+
"args": ["./examples/echo-http-server.js", "--port", "9999"],
237+
"working_dir": "/path/to/repo",
238+
"launcher_wait_timeout": "15s",
239+
"enabled": true
240+
}
241+
```
242+
243+
`stdout` and `stderr` of the child are routed to the per-server log, so
244+
`mcpproxy upstream logs <name>` continues to work the same way it does for
245+
stdio servers.
246+
247+
#### Behaviour matrix when both `command` and `url` are set
248+
249+
| `protocol` | `command` | `url` | Behaviour |
250+
|---|---|---|---|
251+
| `stdio` (explicit) | set | any | Stdio transport, child via stdin/stdout — `url` ignored. |
252+
| `http` / `sse` / `streamable-http` (explicit) | set | set | **Locally-launched HTTP/SSE** — spawn child, wait for URL, connect via network. |
253+
| `http` / `sse` / `streamable-http` (explicit) | unset | set | Connect to remote URL — no spawn. |
254+
| `auto` or unset | set | any | Stdio (`command` wins over `url` for back-compat — set `protocol` explicitly to opt into the launcher). |
255+
| `auto` or unset | unset | set | HTTP/SSE remote — no spawn. |
256+
257+
The "command wins" rule under `auto` is intentional: it preserves backwards
258+
compatibility with configurations written before the launcher feature
259+
existed. To launch a local HTTP/SSE server you **must** set `protocol`
260+
explicitly to one of `http`, `sse`, or `streamable-http`.
261+
213262
### OAuth Configuration
214263

215264
```json

internal/config/config.go

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,13 @@ type ServerConfig struct {
224224
Updated time.Time `json:"updated,omitempty" mapstructure:"updated"`
225225
Isolation *IsolationConfig `json:"isolation,omitempty" mapstructure:"isolation"` // Per-server isolation settings
226226
ReconnectOnUse bool `json:"reconnect_on_use,omitempty" mapstructure:"reconnect-on-use"` // Attempt reconnection when a tool call targets a disconnected server
227+
228+
// LauncherWaitTimeout caps how long mcpproxy will wait for a locally-launched
229+
// HTTP/SSE upstream's URL to become reachable after Spawn(). Only consulted
230+
// when the server is configured with both Command and an HTTP/SSE URL — i.e.,
231+
// mcpproxy starts the process AND connects via network. Stdio servers ignore
232+
// this field. Zero or unset → 30s default.
233+
LauncherWaitTimeout Duration `json:"launcher_wait_timeout,omitempty" mapstructure:"launcher_wait_timeout" swaggertype:"string"`
227234
}
228235

229236
// OAuthConfig represents OAuth configuration for a server

internal/config/merge.go

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -528,17 +528,18 @@ func CopyServerConfig(src *ServerConfig) *ServerConfig {
528528
}
529529

530530
dst := &ServerConfig{
531-
Name: src.Name,
532-
URL: src.URL,
533-
Protocol: src.Protocol,
534-
Command: src.Command,
535-
WorkingDir: src.WorkingDir,
536-
Enabled: src.Enabled,
537-
Quarantined: src.Quarantined,
538-
SkipQuarantine: src.SkipQuarantine,
539-
Shared: src.Shared,
540-
Created: src.Created,
541-
Updated: src.Updated,
531+
Name: src.Name,
532+
URL: src.URL,
533+
Protocol: src.Protocol,
534+
Command: src.Command,
535+
WorkingDir: src.WorkingDir,
536+
Enabled: src.Enabled,
537+
Quarantined: src.Quarantined,
538+
SkipQuarantine: src.SkipQuarantine,
539+
Shared: src.Shared,
540+
Created: src.Created,
541+
Updated: src.Updated,
542+
LauncherWaitTimeout: src.LauncherWaitTimeout,
542543
}
543544

544545
// Copy slices

internal/upstream/core/client.go

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ import (
1919
"github.com/smart-mcp-proxy/mcpproxy-go/internal/secret"
2020
"github.com/smart-mcp-proxy/mcpproxy-go/internal/secureenv"
2121
"github.com/smart-mcp-proxy/mcpproxy-go/internal/storage"
22+
"github.com/smart-mcp-proxy/mcpproxy-go/internal/upstream/launcher"
2223
"github.com/smart-mcp-proxy/mcpproxy-go/internal/upstream/types"
2324

2425
"github.com/mark3labs/mcp-go/client"
@@ -98,6 +99,14 @@ type Client struct {
9899
containerName string // Store container name for cleanup via docker container commands
99100
isDockerCommand bool
100101

102+
// Local launcher tracking — only populated when this Client is using
103+
// HTTP/SSE/streamable-HTTP transport AND ServerConfig.Command is set.
104+
// In that mode mcpproxy spawns the upstream process before connecting,
105+
// and owns its lifecycle via the handle below. Stdio servers leave
106+
// these fields nil — they spawn through mcp-go's stdio transport.
107+
launcherHandle launcher.Handle
108+
launcherCIDFile string
109+
101110
// Notification callback for tools/list_changed
102111
onToolsChanged func(serverName string)
103112
}

internal/upstream/core/connection.go

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package core
33
import (
44
"context"
55
"fmt"
6+
"os"
67
"time"
78

89
"github.com/smart-mcp-proxy/mcpproxy-go/internal/transport"
@@ -118,6 +119,21 @@ func (c *Client) Connect(ctx context.Context) error {
118119

119120
// Create and connect client based on transport type
120121
var err error
122+
// Locally-launched HTTP/SSE upstreams: spawn the child process before
123+
// the transport-level connect, then wait for its URL to become
124+
// reachable. Stdio is excluded because the stdio transport spawns
125+
// through mcp-go itself; running the launcher here would double-spawn.
126+
switch c.transportType {
127+
case transportHTTP, transportHTTPStreamable, transportSSE:
128+
if c.config.Command != "" {
129+
c.logger.Debug("🚀 Launching local upstream before HTTP/SSE connect",
130+
zap.String("server", c.config.Name),
131+
zap.String("transport", c.transportType))
132+
if launchErr := c.connectWithLauncher(ctx); launchErr != nil {
133+
return fmt.Errorf("failed to launch local upstream: %w", launchErr)
134+
}
135+
}
136+
}
121137
switch c.transportType {
122138
case transportStdio:
123139
c.logger.Debug("📡 Using STDIO transport")
@@ -183,6 +199,36 @@ func (c *Client) Connect(ctx context.Context) error {
183199
c.processGroupID = 0
184200
}
185201

202+
// Stop any locally-launched upstream child the HTTP/SSE path
203+
// started — connectWithLauncher itself only stops it on
204+
// wait-for-url failure, not on subsequent transport-level
205+
// connect failure.
206+
//
207+
// IMPORTANT: c.mu is held for the duration of Connect (see
208+
// the c.mu.Lock at the top of this function), so we can read
209+
// the launcher fields directly. We release the lock briefly
210+
// around handle.Stop because Stop blocks until the child is
211+
// reaped and we don't want to hold c.mu that long; the
212+
// `connecting` flag already prevents a concurrent Connect.
213+
if c.launcherHandle != nil {
214+
handle := c.launcherHandle
215+
cidFile := c.launcherCIDFile
216+
c.launcherHandle = nil
217+
c.launcherCIDFile = ""
218+
c.mu.Unlock()
219+
stopCtx, stopCancel := context.WithTimeout(context.Background(), 10*time.Second)
220+
if stopErr := handle.Stop(stopCtx); stopErr != nil {
221+
c.logger.Warn("error stopping launcher during connect-failure cleanup",
222+
zap.String("server", c.config.Name),
223+
zap.Error(stopErr))
224+
}
225+
stopCancel()
226+
if cidFile != "" {
227+
_ = os.Remove(cidFile)
228+
}
229+
c.mu.Lock()
230+
}
231+
186232
return fmt.Errorf("failed to connect: %w", err)
187233
}
188234

0 commit comments

Comments
 (0)