Skip to content

Commit 6b6d4c8

Browse files
RohitNagrajAnkur-singh
RohitNagraj
and
Ankur-singh
authored
feat: add DSV4 FP4 B300 Dynamo-SGLang STP configuration / 新增 DSV4 B300 Dynamo-SGLang STP 配置 (#2362)
* feat: add DSV4 FP4 B300 Dynamo-SGLang STP config 中文：新增 DSV4 FP4 B300 Dynamo-SGLang STP 配置，并接入共享 srt-slurm 配方矩阵。 * chore: link perf changelog to #2362 中文：将性能变更日志链接更新为 #2362。 * fix: use B300 DSV4 NVFP4 model path 中文：将 B300 DSV4 Dynamo-SGLang 配置指向服务器上预置的 DeepSeek-V4-Pro-NVFP4 模型路径。 * fix: pin srt-slurm recipe revision 固定 srt-slurm 配方版本，确保 DSV4 B300 Dynamo-SGLang 运行使用可复现的配置。 * fix(dsv4): pin UCX-enabled srt-slurm revision 修复（dsv4）：固定包含 UCX CUDA 传输配置的 srt-slurm 版本。 * fix: use checked-in DSV4 STP recipes Pin NVIDIA/srt-slurm main, overlay the checked-in recipes, and keep STP chat-template formatting disabled. 修复：固定 NVIDIA/srt-slurm main 提交，覆盖仓库内置配方，并保持 STP 聊天模板格式化关闭。 --------- Co-authored-by: Ankur-singh <ankusingh@nvidia.com>
1 parent cd1cd4c commit 6b6d4c8

8 files changed

Lines changed: 823 additions & 4 deletions

File tree

‎benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-dep4-dep8.yaml‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
name: "disagg-b300-1p1d-dep4-dep8"
2+
3+
model:
4+
path: "deepseek-v4-pro"
5+
container: "lmsysorg/sglang:nightly-dev-cu13-20260721-8905cbd4"
6+
precision: "fp4"
7+
8+
dynamo:
9+
hash: "41882ae9b07232eed4850fb1daf8c958abb2556a"
10+
install: true
11+
12+
sbatch_directives:
13+
cpus-per-task: "144"
14+
mem: "0"
15+
16+
resources:
17+
gpu_type: "b300"
18+
gpus_per_node: 8
19+
prefill_nodes: 1
20+
prefill_workers: 1
21+
gpus_per_prefill: 4
22+
decode_nodes: 1
23+
decode_workers: 1
24+
gpus_per_decode: 8
25+
26+
frontend:
27+
type: dynamo
28+
enable_multiple_frontends: true
29+
num_additional_frontends: 8
30+
env:
31+
args:
32+
router-mode: "kv"
33+
router-kv-overlap-score-weight: 0
34+
router-queue-threshold: 64
35+
router-temperature: 0.5
36+
no-kv-events: true
37+
38+
backend:
39+
type: sglang
40+
41+
prefill_environment:
42+
PYTHONUNBUFFERED: "1"
43+
SGLANG_RADIX_FORCE_MISS: "1"
44+
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
45+
SGLANG_DEFAULT_THINKING: "1"
46+
SGLANG_DSV4_REASONING_EFFORT: "max"
47+
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1"
48+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "8192"
49+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS: "1"
50+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND: "1"
51+
SGLANG_OPT_USE_ONLINE_COMPRESS: "1"
52+
NCCL_MNNVL_ENABLE: "1"
53+
NCCL_CUMEM_ENABLE: "1"
54+
MC_FORCE_MNNVL: "1"
55+
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "100000"
56+
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "100000"
57+
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW: "1"
58+
UCX_TLS: "cuda_copy,rc"
59+
DYN_SKIP_SGLANG_LOG_FORMATTING: "1"
60+
SGLANG_LOG_FORWARD_ITERS: "1"
61+
SGLANG_LOG_MS: "1"
62+
SGLANG_REQUEST_STATE_WAIT_TIMEOUT: "60"
63+
64+
decode_environment:
65+
PYTHONUNBUFFERED: "1"
66+
SGLANG_RADIX_FORCE_MISS: "1"
67+
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
68+
SGLANG_DEFAULT_THINKING: "1"
69+
SGLANG_DSV4_REASONING_EFFORT: "max"
70+
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1"
71+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "1280"
72+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS: "1"
73+
SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND: "1"
74+
SGLANG_OPT_USE_ONLINE_COMPRESS: "1"
75+
NCCL_MNNVL_ENABLE: "1"
76+
NCCL_CUMEM_ENABLE: "1"
77+
SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION: "8"
78+
MC_FORCE_MNNVL: "1"
79+
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "100000"
80+
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "100000"
81+
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW: "1"
82+
UCX_TLS: "cuda_copy,rc"
83+
DYN_SKIP_SGLANG_LOG_FORMATTING: "1"
84+
SGLANG_LOG_FORWARD_ITERS: "1"
85+
SGLANG_LOG_MS: "1"
86+
SGLANG_REQUEST_STATE_WAIT_TIMEOUT: "60"
87+
88+
sglang_config:
89+
prefill:
90+
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
91+
trust-remote-code: true
92+
watchdog-timeout: 86400
93+
skip-tokenizer-init: true
94+
stream-interval: 60
95+
96+
tensor-parallel-size: 4
97+
data-parallel-size: 4
98+
expert-parallel-size: 4
99+
100+
enable-dp-attention: true
101+
moe-a2a-backend: "megamoe"
102+
moe-dense-tp-size: 1
103+
104+
disaggregation-mode: "prefill"
105+
disaggregation-transfer-backend: nixl
106+
enable-dp-lm-head: true
107+
108+
mem-fraction-static: 0.90
109+
max-running-requests: 512
110+
cuda-graph-max-bs: 512
111+
chunked-prefill-size: 32768
112+
113+
decode:
114+
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
115+
trust-remote-code: true
116+
watchdog-timeout: 86400
117+
skip-tokenizer-init: true
118+
stream-interval: 60
119+
120+
moe-a2a-backend: "megamoe"
121+
moe-dense-tp-size: 1
122+
123+
disaggregation-mode: "decode"
124+
disaggregation-transfer-backend: nixl
125+
disaggregation-decode-polling-interval: 8
126+
127+
mem-fraction-static: 0.94
128+
swa-full-tokens-ratio: 0.20
129+
context-length: 9216
130+
tensor-parallel-size: 8
131+
data-parallel-size: 8
132+
expert-parallel-size: 8
133+
enable-dp-attention: true
134+
enable-dp-lm-head: true
135+
max-running-requests: 18432
136+
cuda-graph-max-bs: 1280
137+
138+
benchmark:
139+
type: "sa-bench"
140+
isl: 8192
141+
osl: 1024
142+
concurrencies: "256"
143+
req_rate: "inf"
144+
use_chat_template: false
145+
custom_tokenizer: "sa_bench_tokenizers.sglang_deepseek_v4.SGLangDeepseekV4Tokenizer"

‎benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-tp4-tp4.yaml‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
name: "disagg-b300-1p1d-tp4-tp4"
2+
3+
model:
4+
path: "deepseek-v4-pro"
5+
container: "lmsysorg/sglang:nightly-dev-cu13-20260721-8905cbd4"
6+
precision: "fp4"
7+
8+
dynamo:
9+
hash: "41882ae9b07232eed4850fb1daf8c958abb2556a"
10+
install: true
11+
12+
sbatch_directives:
13+
cpus-per-task: "144"
14+
mem: "0"
15+
16+
resources:
17+
gpu_type: "b300"
18+
gpus_per_node: 8
19+
prefill_nodes: 1
20+
prefill_workers: 1
21+
gpus_per_prefill: 4
22+
decode_nodes: 1
23+
decode_workers: 1
24+
gpus_per_decode: 4
25+
26+
frontend:
27+
type: dynamo
28+
enable_multiple_frontends: true
29+
num_additional_frontends: 8
30+
31+
backend:
32+
type: sglang
33+
34+
prefill_environment:
35+
PYTHONUNBUFFERED: "1"
36+
SGLANG_RADIX_FORCE_MISS: "1"
37+
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
38+
SGLANG_DEFAULT_THINKING: "1"
39+
SGLANG_DSV4_REASONING_EFFORT: "max"
40+
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1"
41+
NCCL_MNNVL_ENABLE: "1"
42+
NCCL_CUMEM_ENABLE: "1"
43+
MC_FORCE_MNNVL: "1"
44+
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "100000"
45+
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "100000"
46+
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW: "1"
47+
UCX_TLS: "cuda_copy,rc"
48+
49+
decode_environment:
50+
PYTHONUNBUFFERED: "1"
51+
SGLANG_RADIX_FORCE_MISS: "1"
52+
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
53+
SGLANG_DEFAULT_THINKING: "1"
54+
SGLANG_DSV4_REASONING_EFFORT: "max"
55+
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1"
56+
NCCL_MNNVL_ENABLE: "1"
57+
NCCL_CUMEM_ENABLE: "1"
58+
MC_FORCE_MNNVL: "1"
59+
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "100000"
60+
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "100000"
61+
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW: "1"
62+
UCX_TLS: "cuda_copy,rc"
63+
64+
sglang_config:
65+
prefill:
66+
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
67+
model-path: "/model/"
68+
trust-remote-code: true
69+
disable-radix-cache: true
70+
71+
disaggregation-mode: "prefill"
72+
watchdog-timeout: 86400
73+
disaggregation-transfer-backend: nixl
74+
75+
tensor-parallel-size: 4
76+
data-parallel-size: 1
77+
expert-parallel-size: 1
78+
79+
moe-runner-backend: "flashinfer_mxfp4"
80+
disable-flashinfer-autotune: true
81+
82+
mem-fraction-static: 0.90
83+
max-running-requests: 32
84+
cuda-graph-max-bs: 32
85+
chunked-prefill-size: 32768
86+
87+
88+
decode:
89+
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
90+
model-path: "/model/"
91+
trust-remote-code: true
92+
disable-radix-cache: true
93+
94+
disaggregation-mode: "decode"
95+
watchdog-timeout: 86400
96+
disaggregation-transfer-backend: nixl
97+
98+
tensor-parallel-size: 4
99+
data-parallel-size: 1
100+
expert-parallel-size: 1
101+
102+
moe-runner-backend: "flashinfer_mxfp4"
103+
disable-flashinfer-autotune: true
104+
105+
mem-fraction-static: 0.9
106+
max-running-requests: 64
107+
cuda-graph-max-bs: 64
108+
swa-full-tokens-ratio: 0.1
109+
context-length: 16384
110+
111+
112+
benchmark:
113+
type: "sa-bench"
114+
isl: 8192
115+
osl: 1024
116+
random_range_ratio: 0.8
117+
concurrencies: "1x8x16x32"
118+
req_rate: "inf"
119+
use_chat_template: false
120+
custom_tokenizer: "sa_bench_tokenizers.sglang_deepseek_v4.SGLangDeepseekV4Tokenizer"

0 commit comments

Comments
 (0)