feat(inference): unify single-/multi-node deployments behind one global router

[mikasenghaas]

Motivation

A deployment of N single-node replicas (num_infer_nodes=1, num_infer_replicas=N — our default RL shape) starts N per-replica routers and load-balances client-side in the orchestrator (least in-flight). That balancing is blind to prefix-cache locality across replicas: consecutive turns of the same rollout can land on different replicas and re-prefill the shared prefix — expensive for agentic/SWE workloads with 100k+ shared-prefix contexts.

Change

• One global vllm-router on the first inference node fronts every per-rank backend across all replicas (non-disaggregated path). Replicas keep their meaning for vLLM DP/EP grouping only.
• The orchestrator now always sees exactly one inference URL — single-node-replica and multi-node deployments have the same defaults and behave fundamentally the same.
• With the default consistent_hash policy (keyed on X-Session-ID = trajectory_id), a rollout's turns pin to one backend → pool-wide prefix-cache affinity + skew rebalancing.
• Disaggregated (P/D) keeps per-replica routers. llm-d endpoint discovery now spans all inference nodes.
• Admin URLs (weight updates, metrics) unchanged: still per-rank.

Benchmark (in progress)

3-way comparison on Nemotron-3 Super 120B SWE RL (131k ctx, 4 train + 2 infer H200 nodes, 5 steps, no evals, 256 inflight; inference stats averaged over steps 2–5):

1. baseline (main): 2 single-node replicas, per-replica routers + client-side LB
2. wide-EP (main): 1 replica × 2 nodes, enable_expert_parallel (ep=16)
3. unified (this PR): 2 single-node replicas behind one global router

Results will be posted here.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes core SLURM launch and routing for the default RL multi-replica path (single router SPOF on node 0, different URL shape for orchestrator); P/D and admin URLs are unchanged but inference grouping logic was adjusted for EP edge cases.

Overview
Non-disaggregated multi-replica RL inference now uses one global router on the first inference node instead of one router per replica. The orchestrator gets a single INFER_URL; worker URLs for routing are collected from every inference node and DP rank into one ALL_ROUTER_ARGS list (no more per-replica | segments or multiple client URLs).

Disaggregated (P/D) behavior is unchanged: still one router per replica and per-replica INFER_URLS.

For llm-d on the regular (non-P/D) path, endpoint discovery now spans num_infer_nodes (all inference nodes), not just nodes per replica.

Wide-EP / external-LB launch only wires vLLM into a multi-rank DP group when expert parallel is on and INFER_GROUP_DP > 1; otherwise ranks start as independent engines (avoids invalid external-LB DP args when data_parallel_size == 1 while still allowing EP within a single TP group).

Docs in inference.md are updated to describe the global router for multi-replica and multi-node RL, and to contrast with P/D’s per-replica routers.

^{Reviewed by Cursor Bugbot for commit 80e6f66. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mikasenghaas]

Benchmark results

Nemotron-3 Super 120B SWE RL (scaleswe-v1), 131k ctx, 4 train + 2 infer H200 nodes per run, 256 inflight rollouts, no evals; steady-state means over each run's serving window (first 10 min dropped; 50–75 min windows; inference/agg/* from the orchestrator's vLLM Prometheus collector).

metric	baseline: 2×(1-node replica), per-replica routers + client-side LB (main)	unified: same shape, one global router (this PR)	wide-EP: 1×2-node replica, ep=16 (main, enforce_eager)
token throughput (prompt+gen)	349.2k tok/s	400.3k tok/s (+14.6%)	91.0k tok/s
completed requests /s	10.67	12.44 (+16.6%)	3.74
prefix-cache hit rate	0.969	0.969	0.955
avg TTFT	0.34 s	0.76 s	0.43 s
avg TPOT	26 ms	28 ms	143 ms
avg queue time	9 ms	73 ms	3 ms
running requests	105	127	203
e2e latency	10.2 s	10.9 s	51.3 s

Takeaways

• The global router keeps the pool ~21% fuller (127 vs 105 running requests): work queues briefly at busy backends (73 ms) instead of capacity sitting idle behind the per-replica-URL client-side balancing. Net +15–17% throughput at equal prefix-cache hit rate; TTFT cost (+0.4 s) is negligible for RL rollout serving.
• Prefix-cache hit rate was already high in the baseline (rollouts pin to a client per trajectory), so the win is utilization, not cache affinity — but the global router preserves affinity while fixing utilization.
• Wide-EP caveat (not a fair number): cross-node enable_expert_parallel for NemotronH deadlocks deterministically during CUDA-graph capture on the non-DeepEP all2all path (NCCL _ALLGATHER_BASE SeqNum=7163 timeout, 2/2 attempts, both nodes) — the run above had to use enforce_eager, so its TPOT (143 ms) is dominated by eager MoE decode. Wide-EP for this model needs the DeepEP kernel install and a capture fix to be evaluated properly; filing separately.

Benchmarked runs: swe-bench-base, swe-bench-uni, swe-bench-ep16 in wandb primeintellect/int5-rl-swe-terminal.

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ad58793. Configure here.}