Skip to content

Commit 4bd7873

Browse files
ivaniumivanium
committed
feat(agentx): add DeepSeek-V4 GB200 Dynamo recipes
Add the aggregate TP8 and four disaggregated DP/EP topologies validated by PR #2260. Use current NVIDIA/srt-slurm main for upstream per-node DP, health-count, and Mooncake fixes while retaining the still-required multi-node TP port patch. Configure the TP8 path to use FlashInfer all-reduce. Limit this validation sweep to TP8 c16 and representative 1P1D DEP8/DEP8 c256. 中文：新增 DeepSeek-V4 GB200 Dynamo AgentX 聚合式 TP8 配置及四种分离式 DP/EP 拓扑，均已通过 PR #2260 验证。改用 NVIDIA/srt-slurm 最新 main 分支以复用已上游的单节点内 DP 启动、健康检查计数和 Mooncake 修复，仅保留多节点 TP 仍需的端口补丁；同时为 TP8 启用 FlashInfer all-reduce。本次验证扫描仅运行 TP8 c16 和代表性的 1P1D DEP8/DEP8 c256。
1 parent e97a67f commit 4bd7873

10 files changed

Lines changed: 1158 additions & 20 deletions

‎benchmarks/benchmark_lib.sh‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1797,7 +1797,11 @@ build_replay_cmd() {
17971797
# X-Correlation-ID is useful tracing metadata but does not establish that
17981798
# binding by itself. AIPerf emits nvext.session_control bind/close actions
17991799
# keyed by the stable conversation correlation ID when this flag is set.
1800-
if [[ "${FRAMEWORK:-}" == dynamo-* ]]; then
1800+
# Opt-out: recipes set AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 to skip this.
1801+
# aiperf's conv-aware routing emits nvext.session_control, a removed POC field
1802+
# (dynamo #9920 / v1.3.0-dev) that current dynamo builds reject with a 400
1803+
# (they moved to router/routing_constraints/agent_context). Default stays on.
1804+
if [[ "${FRAMEWORK:-}" == dynamo-* && "${AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING:-1}" != "0" ]]; then
18011805
REPLAY_CMD+=" --use-dynamo-conv-aware-routing"
18021806
# The upstream 300s affinity TTL is shorter than an overloaded
18031807
# high-concurrency agentic request. Keep bindings alive across long

‎benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/agg-gb200-tp8-agentic.yaml‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
name: "svf-vllm-agg-gb200-tp8-agentic"
2+
3+
# AgentX low-concurrency aggregate recipe for DeepSeek-V4-Pro on GB200. One
4+
# TP8 vLLM worker spans two four-GPU nodes and serves both prefill and decode.
5+
6+
model:
7+
path: "deepseek-v4-pro"
8+
container: "vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-c188b96"
9+
precision: "fp4"
10+
11+
dynamo:
12+
hash: "1f74ef8c204ed6e283c22374f2b8c5cc83bf7b52"
13+
install: true
14+
15+
setup_script: vllm-container-deps.sh
16+
17+
slurm:
18+
time_limit: "8:00:00"
19+
20+
health_check:
21+
max_attempts: 720
22+
interval_seconds: 10
23+
24+
resources:
25+
gpu_type: "gb200"
26+
gpus_per_node: 4
27+
agg_nodes: 2
28+
agg_workers: 1
29+
gpus_per_agg: 8
30+
31+
infra:
32+
etcd_nats_dedicated_node: false
33+
nats_max_payload_mb: 32
34+
35+
frontend:
36+
type: dynamo
37+
enable_multiple_frontends: false
38+
args:
39+
router-mode: "kv"
40+
router-temperature: 0.0
41+
router-queue-threshold: 65536
42+
active-decode-blocks-threshold: "None"
43+
active-prefill-tokens-threshold: "None"
44+
active-prefill-tokens-threshold-frac: "None"
45+
46+
backend:
47+
type: vllm
48+
connector: null
49+
mooncake_kv_store:
50+
store_config:
51+
metadata_server: "P2PHANDSHAKE"
52+
global_segment_size: "150GB"
53+
local_buffer_size: "4GB"
54+
protocol: "rdma"
55+
# Pin to the IB compute fabric. device_name: "" auto-discovers RoCE/link-local
56+
# NICs (mlx5_2/mlx5_5) that can not route cross-node and hang mooncake setup.
57+
device_name: "mlx5_0,mlx5_1,mlx5_3,mlx5_4"
58+
mode: "embedded"
59+
enable_offload: false
60+
aggregated_environment:
61+
DYN_REQUEST_PLANE: "tcp"
62+
ETCD_LEASE_TTL: "600"
63+
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
64+
VLLM_RPC_TIMEOUT: "600000"
65+
TILELANG_CLEANUP_TEMP_FILES: "1"
66+
# Off: the c188b96 container's CUDA-graph capture path enters vLLM's
67+
# nccl_symm_mem_context without set_graph_pool_id, so capture asserts
68+
# "graph_pool_id is not set under graph capture". Re-enable once this recipe
69+
# moves to a container built from a vLLM with the cuda_graph.py fix.
70+
VLLM_USE_NCCL_SYMM_MEM: "0"
71+
TORCH_SYMMMEM: "NVSHMEM"
72+
NCCL_CUMEM_ENABLE: "1"
73+
NCCL_MNNVL_ENABLE: "1"
74+
NCCL_NVLS_ENABLE: "1"
75+
VLLM_SERVER_DEV_MODE: "1"
76+
VLLM_USE_V2_MODEL_RUNNER: "1"
77+
VLLM_USE_RUST_FRONTEND: "1"
78+
VLLM_ALLREDUCE_USE_FLASHINFER: "1"
79+
VLLM_FLASHINFER_ALLREDUCE_BACKEND: "auto"
80+
VLLM_MOONCAKE_LOAD_RECV_THREADS: "4"
81+
VLLM_CONNECTOR_PREFETCH_DEPTH: "8"
82+
VLLM_CONNECTOR_PREFETCH_KV_CAP: "0.65"
83+
DG_JIT_CACHE_DIR: "/tmp/dg-cache-dsv4-agentx"
84+
VLLM_DSV4_MEGA_FP8_COMBINE: "1"
85+
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800"
86+
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "32768"
87+
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB: "1024"
88+
UCX_MEMTYPE_CACHE: "n"
89+
UCX_MEMTYPE_REG_WHOLE: "n"
90+
UCX_RCACHE_MAX_UNRELEASED: "1024"
91+
UCX_TLS: "cuda_copy,cuda_ipc,tcp"
92+
NCCL_P2P_LEVEL: NVL
93+
MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
94+
# MC_STORE_CLIENT_METRIC: "1"
95+
# MC_STORE_CLIENT_METRIC_INTERVAL: "5"
96+
# MC_TE_METRIC: "0"
97+
98+
vllm_config:
99+
aggregated:
100+
kv-transfer-config: '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"enable_offload":false}}'
101+
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
102+
kv-cache-dtype: "fp8"
103+
tensor-parallel-size: 8
104+
pipeline-parallel-size: 1
105+
disable-custom-all-reduce: true
106+
enable-cumem-allocator: true
107+
# numa-bind disabled: srt-slurm's cpus-per-task/cgroup already constrains
108+
# CPU affinity, so vLLM's auto-NUMA detection bails and --numa-bind hard-
109+
# errors ("could not detect the GPU-to-NUMA topology"). Re-enable only with
110+
# an explicit --numa-bind-nodes mapping for this cluster.
111+
# numa-bind: true
112+
attention-config: '{"backend": "FLASHINFER_MLA_SPARSE_DSV4", "use_prefill_query_quantization": true, "use_fp4_indexer_cache": true}'
113+
max-model-len: 1048576
114+
# AgentX session fan-out can exceed the configured live-session
115+
# concurrency; retain scheduler/graph headroom for subagent bursts.
116+
max-num-seqs: 32
117+
max-num-batched-tokens: 8192
118+
trust-remote-code: true
119+
no-enable-flashinfer-autotune: true
120+
block-size: 256
121+
compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'
122+
max-cudagraph-capture-size: 32
123+
gpu-memory-utilization: 0.95
124+
kv-cache-memory: 37580963840 # 35 GiB
125+
stream-interval: 10
126+
no-disable-hybrid-kv-cache-manager: true
127+
tokenizer-mode: deepseek_v4
128+
129+
sbatch_directives:
130+
cpus-per-task: "72"
131+
132+
srun_options:
133+
container-remap-root: ""
134+
135+
benchmark:
136+
type: custom
137+
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
138+
env:
139+
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
140+
RESULT_DIR: /logs/agentic
141+
PORT: "8000"
142+
# One aggregate worker owns all eight GPUs. Use the aggregate accounting
143+
# path instead of summing independent prefill and decode allocations.
144+
# Aggregate (IS_MULTINODE=false) uses the single-node topology labels below
145+
# directly (num_gpus = TP), so they are set here rather than CI-injected.
146+
IS_MULTINODE: "false"
147+
TP: "8"
148+
EP_SIZE: "1"
149+
DP_ATTENTION: "false"
150+
# Off: aiperf emits nvext.session_control, rejected by current dynamo (see benchmark_lib.sh).
151+
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
152+
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400"
153+
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
154+
HF_HUB_CACHE: "/hf_hub_cache"
155+
WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"

‎benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb200-1p1d-dep8-dep12-agentic.yaml‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
name: "svf-vllm-disagg-gb200-1p1d-dep8-dep12-agentic"
2+
3+
# One DEP8 prefill worker feeding one DEP12 decode worker, with attention /
4+
# expert parallelism matching each role and hybrid load balancing.
5+
6+
model:
7+
path: "deepseek-v4-pro"
8+
container: "vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-c188b96"
9+
precision: "fp4"
10+
11+
dynamo:
12+
hash: "1f74ef8c204ed6e283c22374f2b8c5cc83bf7b52"
13+
install: true
14+
15+
setup_script: vllm-container-deps.sh
16+
17+
slurm:
18+
time_limit: "8:00:00"
19+
20+
health_check:
21+
max_attempts: 720
22+
interval_seconds: 10
23+
24+
resources:
25+
gpu_type: "gb200"
26+
gpus_per_node: 4
27+
prefill_nodes: 2
28+
decode_nodes: 3
29+
prefill_workers: 1
30+
decode_workers: 1
31+
gpus_per_prefill: 8
32+
gpus_per_decode: 12
33+
34+
infra:
35+
etcd_nats_dedicated_node: false
36+
nats_max_payload_mb: 32
37+
38+
frontend:
39+
type: dynamo
40+
enable_multiple_frontends: false
41+
args:
42+
router-mode: "kv"
43+
router-temperature: 0.0
44+
router-queue-threshold: 65536
45+
active-decode-blocks-threshold: "None"
46+
active-prefill-tokens-threshold: "None"
47+
active-prefill-tokens-threshold-frac: "None"
48+
tokenizer: "fastokens"
49+
50+
backend:
51+
type: vllm
52+
dp_launch_mode: per_node
53+
connector: null
54+
kv_events_config:
55+
prefill: true
56+
mooncake_kv_store:
57+
store_config:
58+
metadata_server: "P2PHANDSHAKE"
59+
global_segment_size: "150GB"
60+
local_buffer_size: "4GB"
61+
protocol: "rdma"
62+
# Pin to the IB compute fabric. device_name: "" auto-discovers RoCE/link-local
63+
# NICs (mlx5_2/mlx5_5) that can not route cross-node and hang mooncake setup.
64+
device_name: "mlx5_0,mlx5_1,mlx5_3,mlx5_4"
65+
mode: "embedded"
66+
enable_offload: false
67+
prefill_environment:
68+
# dynamo's primary etcd lease defaults to a 10s TTL. A brief etcd stall
69+
# during the CPU/memory/IO-heavy cold start (~380s weight load, etcd
70+
# co-located with a worker) then revokes every worker's lease at once, so
71+
# registration fails with "requested lease not found" and workers never go
72+
# ready (health check times out at 2h). 600s rides through transient etcd
73+
# stalls; longer TTL only delays dead-worker cleanup, handled independently.
74+
ETCD_LEASE_TTL: "600"
75+
DG_JIT_CACHE_DIR: "/tmp/dg-cache-dsv4-agentx"
76+
TORCH_SYMMMEM: "NVSHMEM"
77+
VLLM_USE_NCCL_SYMM_MEM: "1"
78+
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
79+
VLLM_RPC_TIMEOUT: "600000"
80+
VLLM_LOG_STATS_INTERVAL: "1"
81+
VLLM_V2_WARMUP_MAX_NUM_SEQS: "20"
82+
VLLM_SERVER_DEV_MODE: "1"
83+
VLLM_USE_V2_MODEL_RUNNER: "1"
84+
VLLM_MOONCAKE_LOAD_RECV_THREADS: "20"
85+
VLLM_CONNECTOR_PREFETCH_DEPTH: "8"
86+
VLLM_CONNECTOR_PREFETCH_KV_CAP: "0.65"
87+
VLLM_DSV4_MEGA_FP8_COMBINE: "1"
88+
VLLM_ALLREDUCE_USE_SYMM_MEM: "0"
89+
TILELANG_CLEANUP_TEMP_FILES: "1"
90+
UCX_MEMTYPE_CACHE: "n"
91+
UCX_TLS: "rc,cuda_copy"
92+
NCCL_CUMEM_ENABLE: "1"
93+
NCCL_MNNVL_ENABLE: "1"
94+
NCCL_NVLS_ENABLE: "1"
95+
VLLM_RANDOMIZE_DP_DUMMY_INPUTS: "1"
96+
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "32768"
97+
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
98+
MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
99+
# MC_STORE_CLIENT_METRIC: "1"
100+
# MC_STORE_CLIENT_METRIC_INTERVAL: "5"
101+
# MC_TE_METRIC: "0"
102+
decode_environment:
103+
# See prefill_environment: raise dynamo's primary etcd lease TTL from the
104+
# 10s default so the lease survives the CPU-saturated cold start.
105+
ETCD_LEASE_TTL: "600"
106+
DG_JIT_CACHE_DIR: "/tmp/dg-cache-dsv4-agentx"
107+
TORCH_SYMMMEM: "NVSHMEM"
108+
VLLM_USE_NCCL_SYMM_MEM: "1"
109+
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
110+
VLLM_RPC_TIMEOUT: "600000"
111+
VLLM_LOG_STATS_INTERVAL: "1"
112+
VLLM_V2_WARMUP_MAX_NUM_SEQS: "20"
113+
VLLM_SERVER_DEV_MODE: "1"
114+
VLLM_USE_V2_MODEL_RUNNER: "1"
115+
VLLM_MOONCAKE_LOAD_RECV_THREADS: "20"
116+
VLLM_DSV4_MEGA_FP8_COMBINE: "1"
117+
VLLM_ALLREDUCE_USE_SYMM_MEM: "0"
118+
TILELANG_CLEANUP_TEMP_FILES: "1"
119+
UCX_MEMTYPE_CACHE: "n"
120+
UCX_TLS: "rc,cuda_copy"
121+
NCCL_CUMEM_ENABLE: "1"
122+
NCCL_MNNVL_ENABLE: "1"
123+
NCCL_NVLS_ENABLE: "1"
124+
VLLM_RANDOMIZE_DP_DUMMY_INPUTS: "1"
125+
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "32768"
126+
MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
127+
# MC_STORE_CLIENT_METRIC: "1"
128+
# MC_STORE_CLIENT_METRIC_INTERVAL: "5"
129+
# MC_TE_METRIC: "0"
130+
131+
vllm_config:
132+
prefill:
133+
kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"enable_cross_layers_blocks":false,"enable_offload":false}}]}}'
134+
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
135+
kv-cache-dtype: "fp8"
136+
tensor-parallel-size: 1
137+
pipeline-parallel-size: 1
138+
data-parallel-size: 8
139+
data-parallel-rpc-port: 13345
140+
enable-expert-parallel: true
141+
enable-ep-weight-filter: true
142+
enable-cumem-allocator: true
143+
moe-backend: "deep_gemm_amxf4_mega_moe"
144+
numa-bind: true
145+
attention-config: '{"backend": "FLASHINFER_MLA_SPARSE_DSV4", "use_prefill_query_quantization": true, "use_fp4_indexer_cache": true}'
146+
max-model-len: 1048576
147+
max-num-seqs: 256
148+
max-num-batched-tokens: 8192
149+
trust-remote-code: true
150+
no-enable-flashinfer-autotune: true
151+
block-size: 256
152+
gpu-memory-utilization: 0.9
153+
no-disable-hybrid-kv-cache-manager: true
154+
tokenizer-mode: deepseek_v4
155+
decode:
156+
kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"enable_cross_layers_blocks":false,"enable_offload":false}}]}}'
157+
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
158+
kv-cache-dtype: "fp8"
159+
tensor-parallel-size: 1
160+
pipeline-parallel-size: 1
161+
data-parallel-size: 12
162+
data-parallel-rpc-port: 13345
163+
enable-expert-parallel: true
164+
enable-ep-weight-filter: true
165+
enable-cumem-allocator: true
166+
moe-backend: "deep_gemm_amxf4_mega_moe"
167+
numa-bind: true
168+
attention-config: '{"backend": "FLASHINFER_MLA_SPARSE_DSV4", "use_prefill_query_quantization": true, "use_fp4_indexer_cache": true}'
169+
max-model-len: 1048576
170+
max-num-seqs: 128
171+
max-num-batched-tokens: 256
172+
trust-remote-code: true
173+
no-enable-flashinfer-autotune: true
174+
block-size: 256
175+
compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'
176+
max-cudagraph-capture-size: 128
177+
gpu-memory-utilization: 0.95
178+
no-disable-hybrid-kv-cache-manager: true
179+
tokenizer-mode: deepseek_v4
180+
181+
sbatch_directives:
182+
cpus-per-task: "72"
183+
184+
srun_options:
185+
container-remap-root: ""
186+
187+
benchmark:
188+
type: custom
189+
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
190+
env:
191+
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
192+
RESULT_DIR: /logs/agentic
193+
PORT: "8000"
194+
IS_MULTINODE: "true"
195+
# Off: aiperf emits nvext.session_control, rejected by current dynamo (see benchmark_lib.sh).
196+
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
197+
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400"
198+
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
199+
HF_HUB_CACHE: "/hf_hub_cache"
200+
WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"

0 commit comments

Comments
 (0)