Skip to content

Commit 1438043

Browse files
xinli-swhshrivastava-droidadibarra
authored
[NV] Add GLM-5 NVFP4 GB200 disaggregated Dynamo TensorRT-LLM MTP benchmarks / [NV] 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TensorRT-LLM MTP 基准测试 (#1800)
* Add GLM-5 NVFP4 GB200 disaggregated TRT-LLM MTP benchmarks via Dynamo - Add glm5-fp4-gb200-dynamo-trt-mtp config entry (13 ISL1K + 10 ISL8K fixed-seq-len sweep points) using tensorrtllm-runtime:1.3.0-dev.1-cuda13. - Check in the corresponding gb200_nvfp4 MTP recipes locally so CONFIG_FILE=recipes/GLM5/... paths resolve at runtime via the overlay. - Wire runners/launch_gb200-nv.sh to route glm5+fp4 dynamo-trt to the GLM-5-NVFP4 model path and to clone srt-slurm + copy the local GLM5 recipes over the upstream recipes/GLM5 tree. - Append perf-changelog entry. * Update perf-changelog pr-link for #1800 * Point glm5-fp4-gb200-dynamo-trt-mtp MODEL_PATH at Lustre Prior /home/sa-shared path was not present on the watchtower login node (the launcher's model-path pre-check surfaced FileNotFoundError immediately). Move MODEL_PATH to /mnt/lustre01/slurm-shared/glm-model/GLM-5-NVFP4, which is on Lustre and cross-mounted to every watchtower compute node. Mirrors the same fix applied to the non-MTP config. * Pin srt-slurm to v1.0.17 + adopt recipes/trtllm/glm5 overlay layout - Rename benchmarks/multi_node/srt-slurm-recipes/GLM5/disagg/trtllm_dynamo/ gb200_nvfp4/**/MTP/** -> benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5/... . - launch_gb200-nv.sh dynamo-trt+glm5 branch: git checkout v1.0.17, mkdir -p recipes/trtllm/glm5, cp -rT src/dst updated. - configs/nvidia-master.yaml: 23 CONFIG_FILE= paths under glm5-fp4-gb200-dynamo-trt-mtp retargeted to recipes/trtllm/glm5/... ; matching doc-URL comments repointed at v1.0.17. - perf-changelog: PR 1800 description updated to v1.0.17 + new overlay path. Mirrors the same treatment applied to the non-MTP config. gb300 sibling entries (recipes/GLM5/.../gb300_nvfp4/**) intentionally untouched - separate PR. * update srt-slurm checkpoint * update kv p2p --------- Co-authored-by: hshrivastava-droid <hshrivastava@nvidia.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
1 parent c31c974 commit 1438043

25 files changed

Lines changed: 3143 additions & 0 deletions

‎benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5/disagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/MTP/ctx1dep4_gen1dep16_batch32_eplb0_mtp3.yaml‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep16_batch32_eplb0_mtp3
2+
model:
3+
path: nvidia/GLM-5-NVFP4
4+
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
5+
precision: fp4
6+
resources:
7+
gpu_type: gb200
8+
prefill_nodes: 1
9+
prefill_workers: 1
10+
gpus_per_prefill: 4
11+
decode_workers: 1
12+
decode_nodes: 4
13+
gpus_per_decode: 16
14+
gpus_per_node: 4
15+
backend:
16+
type: trtllm
17+
prefill_environment:
18+
ENROOT_ALLOW_DEV: 'yes'
19+
MIMALLOC_PURGE_DELAY: '0'
20+
NCCL_GRAPH_MIXING_SUPPORT: '0'
21+
TLLM_LOG_LEVEL: INFO
22+
TRTLLM_ENABLE_PDL: '1'
23+
TRTLLM_SERVER_DISABLE_GC: '1'
24+
TRTLLM_WORKER_DISABLE_GC: '1'
25+
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
26+
decode_environment:
27+
ENROOT_ALLOW_DEV: 'yes'
28+
MIMALLOC_PURGE_DELAY: '0'
29+
NCCL_GRAPH_MIXING_SUPPORT: '0'
30+
TLLM_LOG_LEVEL: INFO
31+
TRTLLM_ENABLE_PDL: '1'
32+
TRTLLM_SERVER_DISABLE_GC: '1'
33+
TRTLLM_WORKER_DISABLE_GC: '1'
34+
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
35+
trtllm_config:
36+
prefill:
37+
tensor_parallel_size: 4
38+
moe_expert_parallel_size: 4
39+
pipeline_parallel_size: 1
40+
enable_attention_dp: true
41+
disable_overlap_scheduler: true
42+
trust_remote_code: true
43+
max_batch_size: 16
44+
max_num_tokens: 16384
45+
max_seq_len: 1064
46+
print_iter_log: true
47+
cuda_graph_config: null
48+
moe_config:
49+
backend: CUTEDSL
50+
kv_cache_config:
51+
dtype: fp8
52+
enable_block_reuse: false
53+
free_gpu_memory_fraction: 0.6
54+
cache_transceiver_config:
55+
backend: UCX
56+
max_tokens_in_buffer: 16384
57+
speculative_config:
58+
decoding_type: MTP
59+
num_nextn_predict_layers: 3
60+
decode:
61+
tensor_parallel_size: 16
62+
moe_expert_parallel_size: 16
63+
pipeline_parallel_size: 1
64+
enable_attention_dp: true
65+
enable_lm_head_tp_in_adp: true
66+
trust_remote_code: true
67+
max_batch_size: 32
68+
max_num_tokens: 128
69+
max_seq_len: 2088
70+
print_iter_log: true
71+
stream_interval: 100
72+
num_postprocess_workers: 4
73+
cuda_graph_config:
74+
enable_padding: true
75+
batch_sizes:
76+
- 1
77+
- 2
78+
- 4
79+
- 8
80+
- 16
81+
- 24
82+
- 32
83+
moe_config:
84+
backend: CUTEDSL
85+
use_low_precision_moe_combine: true
86+
kv_cache_config:
87+
dtype: fp8
88+
enable_block_reuse: false
89+
free_gpu_memory_fraction: 0.7
90+
cache_transceiver_config:
91+
backend: UCX
92+
max_tokens_in_buffer: 16384
93+
nvfp4_gemm_config:
94+
allowed_backends:
95+
- cutlass
96+
- cublaslt
97+
- cutedsl
98+
- cuda_core
99+
speculative_config:
100+
decoding_type: MTP
101+
num_nextn_predict_layers: 3
102+
benchmark:
103+
type: sa-bench
104+
isl: 1024
105+
osl: 1024
106+
concurrencies: '615'
107+
req_rate: 'inf'
108+
num_prompts_mult: 16
109+
use_chat_template: true
110+
frontend:
111+
type: dynamo
112+
enable_multiple_frontends: true
113+
health_check:
114+
max_attempts: 360
115+
interval_seconds: 10
116+
dynamo:
117+
install: false

‎benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5/disagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/MTP/ctx1dep4_gen1dep16_batch64_eplb0_mtp1.yaml‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep16_batch64_eplb0_mtp1
2+
model:
3+
path: nvidia/GLM-5-NVFP4
4+
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
5+
precision: fp4
6+
resources:
7+
gpu_type: gb200
8+
prefill_nodes: 1
9+
prefill_workers: 1
10+
gpus_per_prefill: 4
11+
decode_workers: 1
12+
decode_nodes: 4
13+
gpus_per_decode: 16
14+
gpus_per_node: 4
15+
backend:
16+
type: trtllm
17+
prefill_environment:
18+
ENROOT_ALLOW_DEV: 'yes'
19+
MIMALLOC_PURGE_DELAY: '0'
20+
NCCL_GRAPH_MIXING_SUPPORT: '0'
21+
TLLM_LOG_LEVEL: INFO
22+
TRTLLM_ENABLE_PDL: '1'
23+
TRTLLM_SERVER_DISABLE_GC: '1'
24+
TRTLLM_WORKER_DISABLE_GC: '1'
25+
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
26+
decode_environment:
27+
ENROOT_ALLOW_DEV: 'yes'
28+
MIMALLOC_PURGE_DELAY: '0'
29+
NCCL_GRAPH_MIXING_SUPPORT: '0'
30+
TLLM_LOG_LEVEL: INFO
31+
TRTLLM_ENABLE_PDL: '1'
32+
TRTLLM_SERVER_DISABLE_GC: '1'
33+
TRTLLM_WORKER_DISABLE_GC: '1'
34+
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
35+
trtllm_config:
36+
prefill:
37+
tensor_parallel_size: 4
38+
moe_expert_parallel_size: 4
39+
pipeline_parallel_size: 1
40+
enable_attention_dp: true
41+
disable_overlap_scheduler: true
42+
trust_remote_code: true
43+
max_batch_size: 16
44+
max_num_tokens: 16384
45+
max_seq_len: 1064
46+
print_iter_log: true
47+
cuda_graph_config: null
48+
moe_config:
49+
backend: CUTEDSL
50+
kv_cache_config:
51+
dtype: fp8
52+
enable_block_reuse: false
53+
free_gpu_memory_fraction: 0.6
54+
cache_transceiver_config:
55+
backend: UCX
56+
max_tokens_in_buffer: 16384
57+
speculative_config:
58+
decoding_type: MTP
59+
num_nextn_predict_layers: 1
60+
decode:
61+
tensor_parallel_size: 16
62+
moe_expert_parallel_size: 16
63+
pipeline_parallel_size: 1
64+
enable_attention_dp: true
65+
enable_lm_head_tp_in_adp: true
66+
trust_remote_code: true
67+
max_batch_size: 64
68+
max_num_tokens: 128
69+
max_seq_len: 2088
70+
print_iter_log: true
71+
stream_interval: 100
72+
num_postprocess_workers: 4
73+
cuda_graph_config:
74+
enable_padding: true
75+
batch_sizes:
76+
- 1
77+
- 2
78+
- 4
79+
- 8
80+
- 16
81+
- 24
82+
- 32
83+
- 40
84+
- 48
85+
- 56
86+
- 64
87+
moe_config:
88+
backend: CUTEDSL
89+
use_low_precision_moe_combine: true
90+
kv_cache_config:
91+
dtype: fp8
92+
enable_block_reuse: false
93+
free_gpu_memory_fraction: 0.7
94+
cache_transceiver_config:
95+
backend: UCX
96+
max_tokens_in_buffer: 16384
97+
nvfp4_gemm_config:
98+
allowed_backends:
99+
- cutlass
100+
- cublaslt
101+
- cutedsl
102+
- cuda_core
103+
speculative_config:
104+
decoding_type: MTP
105+
num_nextn_predict_layers: 1
106+
benchmark:
107+
type: sa-bench
108+
isl: 1024
109+
osl: 1024
110+
concurrencies: '1229'
111+
req_rate: 'inf'
112+
num_prompts_mult: 16
113+
use_chat_template: true
114+
frontend:
115+
type: dynamo
116+
enable_multiple_frontends: true
117+
health_check:
118+
max_attempts: 360
119+
interval_seconds: 10
120+
dynamo:
121+
install: false

‎benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5/disagg/trtllm_dynamo/gb200_nvfp4/ISL1K_OSL1K/MTP/ctx1dep4_gen1dep32_batch16_eplb0_mtp3.yaml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
name: glm5_nvfp4_ISL1K_OSL1K_ctx1dep4_gen1dep32_batch16_eplb0_mtp3
2+
model:
3+
path: nvidia/GLM-5-NVFP4
4+
container: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13
5+
precision: fp4
6+
resources:
7+
gpu_type: gb200
8+
prefill_nodes: 1
9+
prefill_workers: 1
10+
gpus_per_prefill: 4
11+
decode_workers: 1
12+
decode_nodes: 8
13+
gpus_per_decode: 32
14+
gpus_per_node: 4
15+
backend:
16+
type: trtllm
17+
prefill_environment:
18+
ENROOT_ALLOW_DEV: 'yes'
19+
MIMALLOC_PURGE_DELAY: '0'
20+
NCCL_GRAPH_MIXING_SUPPORT: '0'
21+
TLLM_LOG_LEVEL: INFO
22+
TRTLLM_ENABLE_PDL: '1'
23+
TRTLLM_SERVER_DISABLE_GC: '1'
24+
TRTLLM_WORKER_DISABLE_GC: '1'
25+
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
26+
decode_environment:
27+
ENROOT_ALLOW_DEV: 'yes'
28+
MIMALLOC_PURGE_DELAY: '0'
29+
NCCL_GRAPH_MIXING_SUPPORT: '0'
30+
TLLM_LOG_LEVEL: INFO
31+
TRTLLM_ENABLE_PDL: '1'
32+
TRTLLM_SERVER_DISABLE_GC: '1'
33+
TRTLLM_WORKER_DISABLE_GC: '1'
34+
UCX_TLS: 'cuda_ipc,cuda_copy,sm,self,tcp'
35+
trtllm_config:
36+
prefill:
37+
tensor_parallel_size: 4
38+
moe_expert_parallel_size: 4
39+
pipeline_parallel_size: 1
40+
enable_attention_dp: true
41+
disable_overlap_scheduler: true
42+
trust_remote_code: true
43+
max_batch_size: 16
44+
max_num_tokens: 16384
45+
max_seq_len: 1064
46+
print_iter_log: true
47+
cuda_graph_config: null
48+
moe_config:
49+
backend: CUTEDSL
50+
kv_cache_config:
51+
dtype: fp8
52+
enable_block_reuse: false
53+
free_gpu_memory_fraction: 0.6
54+
cache_transceiver_config:
55+
backend: UCX
56+
max_tokens_in_buffer: 16384
57+
speculative_config:
58+
decoding_type: MTP
59+
num_nextn_predict_layers: 3
60+
decode:
61+
tensor_parallel_size: 32
62+
moe_expert_parallel_size: 32
63+
pipeline_parallel_size: 1
64+
enable_attention_dp: true
65+
enable_lm_head_tp_in_adp: true
66+
trust_remote_code: true
67+
max_batch_size: 16
68+
max_num_tokens: 64
69+
max_seq_len: 2088
70+
print_iter_log: true
71+
stream_interval: 100
72+
num_postprocess_workers: 4
73+
cuda_graph_config:
74+
enable_padding: true
75+
batch_sizes:
76+
- 1
77+
- 2
78+
- 4
79+
- 8
80+
- 16
81+
moe_config:
82+
backend: CUTEDSL
83+
use_low_precision_moe_combine: true
84+
kv_cache_config:
85+
dtype: fp8
86+
enable_block_reuse: false
87+
free_gpu_memory_fraction: 0.6
88+
cache_transceiver_config:
89+
backend: UCX
90+
max_tokens_in_buffer: 16384
91+
nvfp4_gemm_config:
92+
allowed_backends:
93+
- cutlass
94+
- cublaslt
95+
- cutedsl
96+
- cuda_core
97+
speculative_config:
98+
decoding_type: MTP
99+
num_nextn_predict_layers: 3
100+
benchmark:
101+
type: sa-bench
102+
isl: 1024
103+
osl: 1024
104+
concurrencies: '666'
105+
req_rate: 'inf'
106+
num_prompts_mult: 16
107+
use_chat_template: true
108+
frontend:
109+
type: dynamo
110+
enable_multiple_frontends: true
111+
health_check:
112+
max_attempts: 360
113+
interval_seconds: 10
114+
dynamo:
115+
install: false

0 commit comments

Comments
 (0)