Skip to content

Commit d527fc8

Browse files
RohitNagrajRohitNagraj
authored
Add GLM-5.1 NVFP4 GB300 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB300 分离式 dynamo-sglang MTP 配置 (#2114)
* Add glm5-fp4-gb300-dynamo-sglang-mtp config * Update perf-changelog pr-link for #2114 * Strip zip_override selector before name-override sed in launch_gb300-nv.sh * Point gb300 MTP config at GLM-5.1 NVFP4 weights * Use glm5.1 model-prefix for gb300 MTP config (match PR #1780 convention) * Skip srtctl preflight for gb300 glm5.1 (weights on compute-only /scratch)
1 parent bed23ea commit d527fc8

3 files changed

Lines changed: 305 additions & 10 deletions

File tree

‎configs/nvidia-master.yaml‎

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11690,6 +11690,274 @@ glm5-fp8-gb200-dynamo-sglang:
1169011690
ep: 1
1169111691
dp-attn: false
1169211692

11693+
glm5-fp4-gb300-dynamo-sglang-mtp:
11694+
image: lmsysorg/sglang:v0.5.13.post1-cu130
11695+
model: nvidia/GLM-5.1-NVFP4
11696+
model-prefix: glm5.1
11697+
runner: gb300-nv
11698+
precision: fp4
11699+
framework: dynamo-sglang
11700+
multinode: true
11701+
disagg: true
11702+
scenarios:
11703+
fixed-seq-len:
11704+
# ---------- 8k1k high-throughput (wide-EP TP=16 decode, MTP) ----------
11705+
- isl: 8192
11706+
osl: 1024
11707+
search-space:
11708+
# 4p1d dep16. 6 nodes (2P + 4D).
11709+
- spec-decoding: "mtp"
11710+
conc-list: [364]
11711+
prefill:
11712+
num-worker: 4
11713+
tp: 2
11714+
ep: 2
11715+
dp-attn: true
11716+
additional-settings:
11717+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_hightpt[0]"
11718+
decode:
11719+
num-worker: 1
11720+
tp: 16
11721+
ep: 16
11722+
dp-attn: true
11723+
# 8p1d dep16. 8 nodes (4P + 4D).
11724+
- spec-decoding: "mtp"
11725+
conc-list: [512]
11726+
prefill:
11727+
num-worker: 8
11728+
tp: 2
11729+
ep: 2
11730+
dp-attn: true
11731+
additional-settings:
11732+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_hightpt[1]"
11733+
decode:
11734+
num-worker: 1
11735+
tp: 16
11736+
ep: 16
11737+
dp-attn: true
11738+
# ---------- 8k1k low-latency (per-node TP=4 decode workers, MTP) ----------
11739+
- isl: 8192
11740+
osl: 1024
11741+
search-space:
11742+
# 1p2d tp4. 3 nodes.
11743+
- spec-decoding: "mtp"
11744+
conc-list: [66]
11745+
prefill:
11746+
num-worker: 1
11747+
tp: 2
11748+
ep: 2
11749+
dp-attn: true
11750+
additional-settings:
11751+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[0]"
11752+
decode:
11753+
num-worker: 2
11754+
tp: 4
11755+
ep: 1
11756+
dp-attn: false
11757+
# 1p2d tp4, conc16. 3 nodes.
11758+
- spec-decoding: "mtp"
11759+
conc-list: [41]
11760+
prefill:
11761+
num-worker: 1
11762+
tp: 2
11763+
ep: 2
11764+
dp-attn: true
11765+
additional-settings:
11766+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[1]"
11767+
decode:
11768+
num-worker: 2
11769+
tp: 4
11770+
ep: 1
11771+
dp-attn: false
11772+
# 1p4d tp4. 5 nodes.
11773+
- spec-decoding: "mtp"
11774+
conc-list: [55]
11775+
prefill:
11776+
num-worker: 1
11777+
tp: 2
11778+
ep: 2
11779+
dp-attn: true
11780+
additional-settings:
11781+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[2]"
11782+
decode:
11783+
num-worker: 4
11784+
tp: 4
11785+
ep: 1
11786+
dp-attn: false
11787+
# 1p4d tp4, conc4. 5 nodes.
11788+
- spec-decoding: "mtp"
11789+
conc-list: [23]
11790+
prefill:
11791+
num-worker: 1
11792+
tp: 2
11793+
ep: 2
11794+
dp-attn: true
11795+
additional-settings:
11796+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[3]"
11797+
decode:
11798+
num-worker: 4
11799+
tp: 4
11800+
ep: 1
11801+
dp-attn: false
11802+
# 1p8d tp4. 9 nodes.
11803+
- spec-decoding: "mtp"
11804+
conc-list: [44]
11805+
prefill:
11806+
num-worker: 1
11807+
tp: 2
11808+
ep: 2
11809+
dp-attn: true
11810+
additional-settings:
11811+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[4]"
11812+
decode:
11813+
num-worker: 8
11814+
tp: 4
11815+
ep: 1
11816+
dp-attn: false
11817+
# 1p8d tp4, conc1. 9 nodes.
11818+
- spec-decoding: "mtp"
11819+
conc-list: [12]
11820+
prefill:
11821+
num-worker: 1
11822+
tp: 2
11823+
ep: 2
11824+
dp-attn: true
11825+
additional-settings:
11826+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_8k1k_lowlat[5]"
11827+
decode:
11828+
num-worker: 8
11829+
tp: 4
11830+
ep: 1
11831+
dp-attn: false
11832+
# ---------- 1k1k high-throughput (wide-EP decode, MTP) ----------
11833+
- isl: 1024
11834+
osl: 1024
11835+
search-space:
11836+
# 1p1d dep16. 5 nodes.
11837+
- spec-decoding: "mtp"
11838+
conc-list: [773]
11839+
prefill:
11840+
num-worker: 1
11841+
tp: 2
11842+
ep: 2
11843+
dp-attn: true
11844+
additional-settings:
11845+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[0]"
11846+
decode:
11847+
num-worker: 1
11848+
tp: 16
11849+
ep: 16
11850+
dp-attn: true
11851+
# 1p1d dep16, conc512. 5 nodes.
11852+
- spec-decoding: "mtp"
11853+
conc-list: [578]
11854+
prefill:
11855+
num-worker: 1
11856+
tp: 2
11857+
ep: 2
11858+
dp-attn: true
11859+
additional-settings:
11860+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[1]"
11861+
decode:
11862+
num-worker: 1
11863+
tp: 16
11864+
ep: 16
11865+
dp-attn: true
11866+
# 1p1d dep32. 9 nodes.
11867+
- spec-decoding: "mtp"
11868+
conc-list: [677]
11869+
prefill:
11870+
num-worker: 1
11871+
tp: 2
11872+
ep: 2
11873+
dp-attn: true
11874+
additional-settings:
11875+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[2]"
11876+
decode:
11877+
num-worker: 1
11878+
tp: 32
11879+
ep: 32
11880+
dp-attn: true
11881+
# 2p1d dep16. 5 nodes.
11882+
- spec-decoding: "mtp"
11883+
conc-list: [2048]
11884+
prefill:
11885+
num-worker: 2
11886+
tp: 2
11887+
ep: 2
11888+
dp-attn: true
11889+
additional-settings:
11890+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[3]"
11891+
decode:
11892+
num-worker: 1
11893+
tp: 16
11894+
ep: 16
11895+
dp-attn: true
11896+
# 2p1d dep16, conc1024. 5 nodes.
11897+
- spec-decoding: "mtp"
11898+
conc-list: [1362]
11899+
prefill:
11900+
num-worker: 2
11901+
tp: 2
11902+
ep: 2
11903+
dp-attn: true
11904+
additional-settings:
11905+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_hightpt[4]"
11906+
decode:
11907+
num-worker: 1
11908+
tp: 16
11909+
ep: 16
11910+
dp-attn: true
11911+
# ---------- 1k1k low-latency (per-node TP=4 decode workers, MTP) ----------
11912+
- isl: 1024
11913+
osl: 1024
11914+
search-space:
11915+
# 1p8d tp4, conc1. 9 nodes.
11916+
- spec-decoding: "mtp"
11917+
conc-list: [13]
11918+
prefill:
11919+
num-worker: 1
11920+
tp: 2
11921+
ep: 2
11922+
dp-attn: true
11923+
additional-settings:
11924+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_lowlat[0]"
11925+
decode:
11926+
num-worker: 8
11927+
tp: 4
11928+
ep: 1
11929+
dp-attn: false
11930+
# 1p8d tp4, conc16. 9 nodes.
11931+
- spec-decoding: "mtp"
11932+
conc-list: [162]
11933+
prefill:
11934+
num-worker: 1
11935+
tp: 2
11936+
ep: 2
11937+
dp-attn: true
11938+
additional-settings:
11939+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_lowlat[1]"
11940+
decode:
11941+
num-worker: 8
11942+
tp: 4
11943+
ep: 1
11944+
dp-attn: false
11945+
# 1p8d tp4, conc4. 9 nodes.
11946+
- spec-decoding: "mtp"
11947+
conc-list: [44]
11948+
prefill:
11949+
num-worker: 1
11950+
tp: 2
11951+
ep: 2
11952+
dp-attn: true
11953+
additional-settings:
11954+
- "CONFIG_FILE=recipes/gb300-fp4/glm5-mtp.yaml:zip_override_mtp_1k1k_lowlat[2]"
11955+
decode:
11956+
num-worker: 8
11957+
tp: 4
11958+
ep: 1
11959+
dp-attn: false
11960+
1169311961
glm5-fp8-gb300-dynamo-sglang:
1169411962
image: lmsysorg/sglang:v0.5.11-cu130
1169511963
model: zai-org/GLM-5-FP8

‎perf-changelog.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4627,3 +4627,10 @@
46274627
- "Add GLM-5.1 NVFP4 GB200 disaggregated dynamo-sglang MTP config covering 8k1k and 1k1k, high-throughput and low-latency topologies"
46284628
- "Image: lmsysorg/sglang:v0.5.13.post1-cu130"
46294629
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2115
4630+
4631+
- config-keys:
4632+
- glm5-fp4-gb300-dynamo-sglang-mtp
4633+
description:
4634+
- "Add GLM-5.1 NVFP4 GB300 disaggregated dynamo-sglang MTP config covering 8k1k and 1k1k, high-throughput and low-latency topologies"
4635+
- "Image: lmsysorg/sglang:v0.5.13.post1-cu130"
4636+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2114

‎runners/launch_gb300-nv.sh‎

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,11 @@ elif [[ $MODEL_PREFIX == "glm5" && $PRECISION == "fp4" && $FRAMEWORK == "dynamo-
4646
export SERVED_MODEL_NAME="glm-5-nvfp4"
4747
export MODEL_PATH=/scratch/models/GLM-5-NVFP4
4848
export SRT_SLURM_MODEL_PREFIX="nvidia/GLM-5-NVFP4"
49+
elif [[ $MODEL_PREFIX == "glm5.1" && $PRECISION == "fp4" ]]; then
50+
# SRT_SLURM_MODEL_PREFIX matches the model.path alias ("glm-5-fp4")
51+
# in our GLM-5.1 sglang recipes.
52+
export MODEL_PATH=/scratch/models/GLM-5.1-NVFP4
53+
export SRT_SLURM_MODEL_PREFIX="glm-5-fp4"
4954
elif [[ $MODEL_PREFIX == "glm5" && $PRECISION == "fp4" ]]; then
5055
export MODEL_PATH=/scratch/models/GLM-5-NVFP4
5156
export SRT_SLURM_MODEL_PREFIX="glm-5-fp4"
@@ -164,6 +169,12 @@ elif [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "glm5" ]]; then
164169
mkdir -p recipes/sglang/glm5/gb300-fp4
165170
cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/glm5/gb300-fp4" recipes/sglang/glm5/gb300-fp4
166171
fi
172+
elif [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "glm5.1" ]]; then
173+
# GLM-5.1 MTP recipe (recipes/gb300-fp4/glm5-mtp.yaml) lives on
174+
# NVIDIA/srt-slurm:main — check it out; no in-repo overlay needed.
175+
git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR"
176+
cd "$SRT_REPO_DIR"
177+
git checkout main
167178
elif [[ $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX == "qwen3.5" ]]; then
168179
# Overlay our version-controlled Qwen3.5 recipes onto the submission branch.
169180
git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR"
@@ -255,6 +266,7 @@ containers:
255266
dynamo-trtllm: ${SQUASH_FILE}
256267
dynamo-sglang: ${SQUASH_FILE}
257268
v0.5.11: ${SQUASH_FILE}
269+
v0.5.13.post1: ${SQUASH_FILE}
258270
"${IMAGE}": ${SQUASH_FILE}
259271
nginx-sqsh: ${NGINX_SQUASH_FILE}
260272
use_segment_sbatch_directive: false
@@ -277,20 +289,28 @@ if [[ -z "$CONFIG_FILE" ]]; then
277289
exit 1
278290
fi
279291

280-
# Override the job name in the config file with the runner name
281-
sed -i "s/^name:.*/name: \"${RUNNER_NAME}\"/" "$CONFIG_FILE"
282-
283-
# --no-preflight is only safe on the agentic path, where the recipe
284-
# resolves model.path to /scratch (compute-node-only NVMe) and the
285-
# srtctl process running on the GHA runner pod can't see it. Fixed-
286-
# seq-len recipes still resolve model.path to an NFS-visible location
287-
# where the precheck is a useful sanity guard, so keep enforcement on
288-
# for them.
292+
# Override the job name in the config file with the runner name.
293+
# CONFIG_FILE may carry a ":zip_override_...[i]" selector suffix that only
294+
# `srtctl apply -f` parses; strip it to the real path for the sed. srtctl
295+
# below still receives the full CONFIG_FILE (with selector).
296+
CONFIG_PATH="${CONFIG_FILE%%:*}"
297+
sed -i "s/^name:.*/name: \"${RUNNER_NAME}\"/" "$CONFIG_PATH"
298+
299+
# --no-preflight skips srtctl's pre-submit model-path stat, which runs on
300+
# the GHA runner host (im-gb300-login-02, an x86 login node). It's required
301+
# whenever model.path resolves to the node-local /scratch NVMe that the login
302+
# node can't see:
303+
# - the agentic path (DSv4-Pro checkpoint), and
304+
# - glm5.1, whose GLM-5.1-NVFP4 weights are prestaged on the compute-node
305+
# /scratch/models.
306+
# The engine still fails loudly at runtime if the path is genuinely missing on
307+
# the compute node. Other fixed-seq-len recipes resolve model.path to a
308+
# login-visible location, so keep the precheck enforced for them.
289309
SRTCTL_APPLY_ARGS=(
290310
-f "$CONFIG_FILE"
291311
--tags "gb300,${MODEL_PREFIX},${PRECISION},${ISL}x${OSL},infmax-$(date +%Y%m%d)"
292312
)
293-
if [[ "$IS_AGENTIC" == "1" ]]; then
313+
if [[ "$IS_AGENTIC" == "1" || "$MODEL_PREFIX" == "glm5.1" ]]; then
294314
SRTCTL_APPLY_ARGS+=(--no-preflight)
295315
fi
296316
if [[ -n "$SRTCTL_SETUP_SCRIPT" ]]; then

0 commit comments

Comments
 (0)