feat(operator): add five-container Bulwark worker pod [GMS 03/18]

[hutm]

Summary

Deploy the failure-isolated worker topology used by the GMS recovery design: the inference ranks, GMS sidecars, and Bulwark coordination run together in one pod.

Scope

Adds operator and deployment wiring for the five-container pod, including fast rank failure propagation. It does not add GMS memory semantics.

Stack

Position 3/18 in the GMS-managed KV review train. This PR is based on review/gms-v2-latehost-03-1-vllm-common-failover.

Parent: #10450
Tracking: #10454

Review migration

This is the current review entry replacing historical merged PR #10471. GitHub does not allow a merged PR to be retargeted; #10471 and all of its comments and reviews remain intact as the historical record.

The train is rebased on main at aeda23b483831df2f75b697b8ccf65f4ffd9f3d6. The reordered functionality is preserved while each PR boundary is independently buildable and lintable: compatibility call sites and the engine-facing placement adapter now appear at first use; missing type imports and test markers were added; repository-pinned formatting was applied; one unused constant was removed; and every commit carries a DCO sign-off. The complete range passes git diff --check and the full pre-commit suite.

Validation

Validated with go test ./internal/dynamo ./internal/webhook/validation and go build ./cmd/... at this stack boundary.

---

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[datadog-official]

⚠️ Warnings

🚦 2 Pipeline jobs failed

Docs link check | lychee     

Pre Merge | pre-merge-status-check     

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8696e23 | Docs | Give us feedback!}

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🟡 Engine configuration flag silently dropped for multi-node SGLang worker pods with failover enabled

The --disable-piecewise-cuda-graph flag is appended as a separate Args element (c.Args = append(c.Args, ...) at deploy/operator/internal/dynamo/failover_sglang.go:25), but when the container command has already been shell-wrapped by the SGLang multinode path, the flag lands outside the shell script and is silently ignored.

Impact: Multi-node SGLang worker pods with intra-pod failover start without the required CUDA graph flag, potentially causing engine crashes or incorrect behavior during failover.

Mechanism: shell wrapping makes bare append ineffective

When the SGLang backend processes a multinode worker pod, getMultinodeFlags (deploy/operator/internal/dynamo/backend_sglang.go:72-87) returns needsShell=true for the Grove deployer (via GroveMultinodeDeployer.GetNodeRank() at deploy/operator/internal/dynamo/grove.go:51). This causes injectFlagsIntoContainerCommand to wrap the command into Command: ["sh", "-c"], Args: ["exec python3 -m sglang ... --multinode-flags"].

Later, buildFailoverPod clones this shell-wrapped container into engine-0 and engine-1, then calls applySGLangFailoverOverrides. This function does:

c.Args = append(c.Args, sglangDisablePiecewiseCudaGraphFlag)

Result: Args: ["exec python3 ...", "--disable-piecewise-cuda-graph"]

Kubernetes runs: sh -c "exec python3 ..." --disable-piecewise-cuda-graph

With sh -c, only the first argument is the script; subsequent arguments become positional parameters ($0, $1) that the script never references, so the flag is silently discarded.

Compare with how the vLLM override (applyVLLMOverrides) uses env vars and staggerFlagValue (which does strings.Replace inside existing args), or TRT-LLM's retargetTRTLLMSshPort which also uses strings.ReplaceAll on existing args — both correctly handle shell-wrapped commands.

The fix should use injectFlagsIntoContainerCommand(c, sglangDisablePiecewiseCudaGraphFlag, false, "sglang") or equivalent logic that handles the shell-wrapped case.

Prompt for agents

In failover_sglang.go:25, applySGLangFailoverOverrides appends --disable-piecewise-cuda-graph as a bare Args element. This works only when the container command is a direct Python command (Command: [python3, -m, sglang], Args: [--model, ...]). When the SGLang backend has shell-wrapped the command for multinode workers (Command: [sh, -c], Args: [exec python3 -m sglang ... --multinode-flags]), the appended flag lands outside the shell script and is silently discarded by sh.

The fix should use injectFlagsIntoContainerCommand(c, sglangDisablePiecewiseCudaGraphFlag, false, "sglang") instead of c.Args = append(c.Args, ...). This function already handles both direct Python commands and shell-wrapped commands by using regex injection for the non-Python path.

Alternatively, the check containerCommandLineHasFlag could be paired with a shell-aware insertion that injects the flag into the shell script string when args[0] is a shell command.

Relevant files:
- deploy/operator/internal/dynamo/failover_sglang.go (the fix location)
- deploy/operator/internal/dynamo/utils.go (injectFlagsIntoContainerCommand helper)
- deploy/operator/internal/dynamo/backend_sglang.go:72-87 (the multinode path that shell-wraps)

---

Was this helpful? React with 👍 or 👎 to provide feedback.