Backend routing breaks after proxy runner restart with backendReplicas > 1

[jerm-dro]

Problem

When an MCPServer has backendReplicas > 1, the proxy runner stores the ClusterIP service URL (e.g. http://mcp-<name>:8080) as backend_url in session metadata. After a proxy runner restart, it recovers the session from Redis but routes via ClusterIP — kube-proxy may send the request to a backend pod that never handled the initialize for that session.

The backend pod returns HTTP 404 with JSON-RPC -32001 ("session not found") because it has no record of that session. The mcp-go client surfaces this as:

transport error: failed to send request: session terminated (404). need to re-initialize

The session is not actually terminated — it simply doesn't exist on the pod that received the request.

Parent issue: #4484

Failing acceptance test

PR #4574 adds an E2E test that demonstrates this bug. It fails on all 3 Kubernetes versions in CI:

• v1.33.7, v1.34.3, v1.35.1

Error from CI (v1.35.1):

[FAILED] Request 1/5 should succeed — session should route to the correct backend
Unexpected error:
    transport error: failed to send request: session terminated (404). need to re-initialize
In [It] at: mcpserver_scaling_test.go:382

Test flow

1. Deploy MCPServer with replicas=1, backendReplicas=2, Redis session storage, sessionAffinity=None
2. Initialize MCP session, call tools/list — succeeds
3. Delete the proxy runner pod (Deployment recreates it)
4. Send 5 tools/list requests with the same session ID — fails on request 1

With 2 backends and random routing, P(all 5 hit correct pod) ≈ 3%.

Root cause

1. StatefulSet serviceName mismatch: buildStatefulSetSpec sets serviceName to containerName, but the headless service is named mcp-<containerName>-headless. These must match for Kubernetes to create pod DNS records.

2. ClusterIP stored as backend_url: RoundTrip stores targetURI (ClusterIP) as backend_url. After restart, Rewrite routes via ClusterIP with sessionAffinity=None, hitting a random backend.

Proposed fix

Store a pod-specific headless DNS URL (e.g. myserver-0.mcp-myserver-headless.default.svc.cluster.local:8080) as backend_url instead of the ClusterIP:

• Fix StatefulSet serviceName to match headless service name
• Add HeadlessServiceConfig to ScalingConfig
• Pre-select a random StatefulSet pod on initialize, store its headless DNS as backend_url
• Operator populates config when backendReplicas > 1
• Backward compatible: when config is nil (single backend), behavior is unchanged

[JAORMX]

Heads up: while reviewing #4638, we found some fundamental issues with the headless service it builds on top of (wrong selector target, StatefulSet serviceName mismatch, random pod selection after initialize). Opened #4668 to track rethinking the headless service before we add more plumbing on top.

[yrobla]

Proposed fix: per-pod ClusterIP services

After reviewing the current approach (headless DNS routing) with the team, we identified a security concern: storing pod-specific headless DNS URLs in Redis leaks internal cluster topology and allows bypassing the proxy's auth middleware from within the cluster. Since the headless service is also going to be removed, here's the proposed replacement.

Approach

The operator creates one ClusterIP service per backend pod ordinal alongside the existing shared service:

mcp-<name>        → selects all backend pods (existing, unchanged)
mcp-<name>-pod-0  → selects only pod <name>-0  (new)
mcp-<name>-pod-1  → selects only pod <name>-1  (new)

Each per-pod service uses the label Kubernetes automatically assigns to StatefulSet pods:

selector:
  statefulset.kubernetes.io/pod-name: <name>-<ordinal>

The proxy runner computes the ordinal deterministically from the session ID:

ordinal = fnv32(sessionID) % backendReplicas
backend = "http://mcp-<name>-pod-{ordinal}:<port>"

Why this is better

• No topology leakage: nothing about pod addressing is stored in Redis — routing is computed on every request from the session ID alone
• No bypass risk: ClusterIP services route through kube-proxy, not raw pod IPs; the proxy's auth middleware cannot be circumvented
• Survives proxy restart: any proxy pod computes the same ordinal for a given session ID — no state recovery needed
• Standard Kubernetes pattern: per-pod services via statefulset.kubernetes.io/pod-name selector are well-supported and widely used
• Operator manages lifecycle: services are created/deleted as backendReplicas scales, same as today's shared service

Changes required

1. Operator (): add createPerPodServices that creates one ClusterIP service per ordinal up to backendReplicas; delete excess services on scale-down
2. Runner config: replace HeadlessServiceConfig with a BackendServiceConfig carrying the base service name (mcp-<name>-pod) and replica count
3. Proxy (): on each initialize request, compute fnv32(sessionID) % replicas and store nothing — just forward to the right per-pod service

[JAORMX]

So, I've been looking at this issue and the proposed fixes, and I think we need to take a step back here.

Both proposals (headless DNS routing and per-pod ClusterIP services) share a fundamental problem: they create infrastructure that exposes backend pods outside the proxy runner. The headless DNS approach puts pod-specific addresses in Redis and makes them resolvable from anywhere in the cluster. The per-pod services create addressable endpoints that bypass the proxy's auth/authz/audit middleware entirely. For stdio servers this is a non-starter since the proxy runner IS the only HTTP endpoint... but even for SSE/streamable-http, we should never bypass the proxy runner. That's the whole point of the architecture.

Also, the fnv32(sessionID) % replicas idea doesn't actually work for the initialize flow. At initialize time there IS no session ID yet. The backend creates it. Per the MCP spec (both 2025-03-26 and 2025-11-25):

A server using the Streamable HTTP transport MAY assign a session ID at initialization time, by including it in an Mcp-Session-Id header on the HTTP response containing the InitializeResult.

And:

...it MUST start a new session by sending a new InitializeRequest without a session ID attached.

So you can't deterministically route the initialize to a specific pod based on the session ID. It doesn't exist yet.

Note that the sticky session work we already landed (ClientIP affinity on the backend service via #3992, Redis session storage via #4525) covers most of this. The gap is specifically: when the proxy runner restarts, it gets a new pod IP, so the L4 ClientIP affinity mapping to the backend is lost. The session is in Redis, but backend_url points to the ClusterIP service URL, which doesn't help route to the right pod.

The fix should be way simpler than what's proposed here. The backend_url session metadata mechanism already works perfectly... it just stores the wrong value. Go's net/http/httptrace package gives us a GotConn callback that fires after kube-proxy DNAT. info.Conn.RemoteAddr() returns the actual pod IP that handled the connection. So:

1. On initialize, attach an httptrace.ClientTrace with GotConn to capture the resolved pod IP
2. Store http://<pod-ip>:<port> as backend_url instead of the ClusterIP URL
3. That's it. The existing Rewrite function already reads backend_url and overrides the outbound URL

All traffic still flows through the proxy runner. No new Kubernetes resources. No new bypass vectors. ~30 lines of code in transparent_proxy.go.

"But what if the backend pod restarts and gets a new IP?" Well... the backend's in-memory MCP session state is gone too. The client has to re-initialize regardless. The proxy should handle the stale IP gracefully (connection refused or 404 → clear session from Redis → client re-initializes → new pod IP captured). This isn't a new failure mode.

For environments with frequent backend pod churn, we can add an EndpointSlice watcher as a follow-up that maps pod IPs to stable StatefulSet ordinals. But that's an optimization, not the fix.