Skip to content

Commit 648d857

Browse files
csahithixinli-swAnkur-singh
authored
[NV] Add GB300 AgentX Qwen3.5 recipes (#2121)
* feat(qwen3.5-gb300-agentx): add Qwen3.5-NVFP4 GB300 SGLang AgentX benchmarks - Add qwen3.5-fp4-gb300-dynamo-sglang-agentic-agg and -disagg configs - Add agg/disagg benchmark recipes and result files - Add GB300 AgentX launch script support - Add perf changelog entry for PR 2121 * qwen3.5 agentic: image 20260715-50d1edaa; prefill NEXTN MTP; fix agg GPU accounting - Bump the qwen3.5-fp4 GB300 agentic image (agg + disagg master-config keys and the changelog line) to lmsysorg/sglang:nightly-dev-cu13-20260715-50d1edaa. - Enable prefill-side NEXTN MTP (speculative-algorithm NEXTN, num-steps 3, eagle-topk 1, num-draft-tokens 4) in the 7 disagg recipes' prefill sglang_config. - Fix the agg GPU accounting: master-config decode num-worker 1 -> 0, and process_agentic_result.py zeroes the decode-side parallelism when num_decode_gpu <= 0, so TP/num_gpus reflect the single aggregated worker (4) instead of double-counting (8). Disagg configs are unaffected. * qwen3.5 agentic: bump image to 20260716-b0b2dfbd Bump the qwen3.5-fp4 GB300 agentic image (agg + disagg master-config keys and the changelog line) to lmsysorg/sglang:nightly-dev-cu13-20260716-b0b2dfbd. * runners: add gb300-nv hardware metadata (available-cpu-dram-mib, gpus-per-node) The agentic DRAM-offload matrix logic requires available-cpu-dram-mib for any runner used by a kv-offloading: dram (hicache) config. gb300-nv had no hardware entry, so generate_sweep_configs test-config crashed for the gb300 agentic keys (qwen3.5 here, and dsv4 already on main). GB300 NVL72 uses the same 2x Grace / 4-GPU-per-node topology and LPDDR5X capacity as GB200, so mirror gb200-nv's available host DRAM (860_160 MiB) and gpus-per-node (4). * qwen3.5 agentic: bump image to 20260724-433429b1 * run-sweep: drop unneeded agentic collect-results gate Agentic sweep results are ingested by the dedicated trigger-agentic-ingest job (repository dispatch to InferenceX-app, main-push only), not through collect-results, whose gate intentionally excludes the agentic sweeps. This reverts the qwen branch's addition so run-sweep.yml matches main. * qwen3.5 agentic: pin Dynamo to ai-dynamo/dynamo#12081 (read-only ServerArgs fix) SGLang 20260724 (>= #31811) makes ServerArgs read-only after resolution, which crashed the Dynamo worker at startup (bare server_args.incremental_streaming_output assignment). Pin Dynamo to the ai-dynamo/dynamo#12081 head commit (8e4bfa69), a source build that sets incremental_streaming_output on parsed_args *before* resolution, so no post-resolution mutation is attempted. Swap the 7 disagg recipes' dynamo wheel 1.3.0.dev1 -> hash 8e4bfa69 and bump the disagg router version to match. * qwen3.5 agentic: migrate disagg recipes to X-Dynamo-Session-ID session routing Switch the 7 disagg recipes from the legacy nvext conv-aware routing (which the newer Dynamo no longer honors) to header-based session affinity: - benchmark.env: AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=1 -> AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true - frontend: add nginx_session_affinity + nginx_session_affinity_header X-Dynamo-Session-ID - frontend args: router-reset-states -> router-session-affinity-ttl-secs 3600 * benchmark_lib: honor AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID Skip the legacy nvext conv-aware CLI routing when a recipe opts into the X-Dynamo-Session-ID header path (AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true); the existing AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING opt-out and default are unchanged. * qwen3.5 c128: raise SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK 256 -> 512 The c128 (1P1D DEP4/DEP4) decode server crashed on a DeepEP low-latency dispatch assertion (x.size(0) <= num_max_dispatch_tokens_per_rank, deep_ep.cpp:1262): under DP-attention load imbalance one DP rank's token count (amplified by MTP draft tokens) exceeded the 256 buffer. Double the DeepEP dispatch buffer to 512 (still well under sglang's 1024 cap) to give headroom for the imbalanced+MTP worst case. --------- Co-authored-by: Xin Li <xinli@nvidia.com> Co-authored-by: Ankur-singh <ankusingh@nvidia.com>
1 parent af285ca commit 648d857

14 files changed

Lines changed: 1965 additions & 2 deletions

‎benchmarks/benchmark_lib.sh‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1810,7 +1810,11 @@ build_replay_cmd() {
18101810
# aiperf's conv-aware routing emits nvext.session_control, a removed POC field
18111811
# (dynamo #9920 / v1.3.0-dev) that current dynamo builds reject with a 400
18121812
# (they moved to router/routing_constraints/agent_context). Default stays on.
1813-
if [[ "${FRAMEWORK:-}" == dynamo-* && "${AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING:-1}" != "0" ]]; then
1813+
# New recipes instead set AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true
1814+
# to route by X-Dynamo-Session-ID header, which needs no routing CLI flag.
1815+
if [[ "${FRAMEWORK:-}" == dynamo-* \
1816+
&& "${AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING:-1}" != "0" \
1817+
&& "${AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID:-false}" != "true" ]]; then
18141818
REPLAY_CMD+=" --use-dynamo-conv-aware-routing"
18151819
# The upstream 300s affinity TTL is shorter than an overloaded
18161820
# high-concurrency agentic request. Keep bindings alive across long

‎benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/agentic/agg-gb300-tp4-c1-mtp-hicache-jid2191933.yaml‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
name: "agg-gb300-tp4-c1-mtp-hicache-jid2191933"
2+
3+
# Agentic-coding SGLang aggregated recipe for Qwen3.5-397B-A17B-NVFP4 on GB300
4+
# (single aggregated worker, TP4, NEXTN MTP + hierarchical cache).
5+
#
6+
# Ported from the manually-run srtctl base:/zip_override_conc: starter to the
7+
# flat single-variant agentic schema. Pure sglang (no dynamo): frontend.type
8+
# sglang serves directly. Concurrency comes from the GHA matrix (exported as
9+
# CONC into agentic_srt.sh), not the recipe. Serving tuning preserved verbatim
10+
# from the source run; only CI scaffolding is added.
11+
12+
model:
13+
path: "qwen3.5-fp4"
14+
container: "dynamo-sglang"
15+
precision: "fp4"
16+
17+
slurm:
18+
time_limit: "8:00:00"
19+
20+
health_check:
21+
max_attempts: 1440
22+
interval_seconds: 10
23+
24+
resources:
25+
gpu_type: "gb300"
26+
gpus_per_node: 4
27+
agg_nodes: 1
28+
agg_workers: 1
29+
gpus_per_agg: 4
30+
31+
infra:
32+
nats_max_payload_mb: 8
33+
34+
frontend:
35+
type: sglang
36+
37+
backend:
38+
type: sglang
39+
aggregated_environment:
40+
SGLANG_SIMULATE_ACC_LEN: '3.39'
41+
SGLANG_SIMULATE_ACC_METHOD: match-expected
42+
SGLANG_SIMULATE_ACC_TOKEN_MODE: "real-draft-token"
43+
TORCH_CUDA_ARCH_LIST: '10.0'
44+
PYTHONNOUSERSITE: '1'
45+
NCCL_NVLS_ENABLE: '1'
46+
SGL_ENABLE_JIT_DEEPGEMM: 'false'
47+
SGLANG_ENABLE_FLASHINFER_GEMM: 'true'
48+
sglang_config:
49+
aggregated:
50+
served-model-name: nvidia/Qwen3.5-397B-A17B-NVFP4
51+
model-path: /model/
52+
trust-remote-code: true
53+
tensor-parallel-size: 4
54+
data-parallel-size: 1
55+
expert-parallel-size: 1
56+
enable-symm-mem: true
57+
quantization: modelopt_fp4
58+
kv-cache-dtype: fp8_e4m3
59+
mamba-ssm-dtype: bfloat16
60+
mamba-scheduler-strategy: extra_buffer
61+
mamba-track-interval: 8192
62+
attention-backend: trtllm_mha
63+
moe-runner-backend: flashinfer_trtllm
64+
speculative-algorithm: NEXTN
65+
speculative-num-steps: 3
66+
speculative-eagle-topk: 1
67+
speculative-num-draft-tokens: 4
68+
cuda-graph-max-bs: 256
69+
max-running-requests: 1024
70+
max-prefill-tokens: 16384
71+
chunked-prefill-size: 16384
72+
mem-fraction-static: 0.8
73+
max-mamba-cache-size: 1024
74+
allow-auto-truncate: true
75+
stream-interval: 50
76+
scheduler-recv-interval: 10
77+
tokenizer-worker-num: 6
78+
page-size: 64
79+
enable-hierarchical-cache: true
80+
hicache-ratio: 1.01
81+
hicache-io-backend: kernel
82+
hicache-mem-layout: page_first_direct
83+
hicache-write-policy: write_through
84+
85+
sbatch_directives:
86+
mem: "0"
87+
cpus-per-task: "144"
88+
89+
srun_options:
90+
mem: "0"
91+
# gb300-nv: pyxis maps the calling user into the container as a non-root uid;
92+
# agentic_srt.sh's apt-get install git step needs EUID 0. Remap to uid 0 inside
93+
# the container. srt-slurm renders empty-string values as flag-only srun args.
94+
container-remap-root: ""
95+
96+
benchmark:
97+
type: custom
98+
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
99+
env:
100+
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
101+
RESULT_DIR: /logs/agentic
102+
PORT: "8000"
103+
# Aggregated: one worker serves both prefill and decode, so GPU accounting is
104+
# the single-worker form num_gpus = TP (not the disagg prefill+decode sum).
105+
# Force the single-node post-proc path; TP must match sglang_config tp-size.
106+
IS_MULTINODE: "false"
107+
TP: "4"
108+
# Match the source run's dataset (256k cc-traces-with-subagents variant).
109+
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126_256k
110+
# Container-side path of the aiperf mmap dataset cache; host-side mount wired
111+
# via launch_gb300-nv.sh srtslurm.yaml default_mounts.
112+
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
113+
# Persistent HF hub cache (also via default_mounts).
114+
HF_HUB_CACHE: "/hf_hub_cache"

‎benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/agentic/agg-gb300-tp4-c96-mtp-hicache-jid2195211.yaml‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
name: "agg-gb300-tp4-c96-mtp-hicache-jid2195211"
2+
3+
# Agentic-coding SGLang aggregated recipe for Qwen3.5-397B-A17B-NVFP4 on GB300
4+
# (single aggregated worker, TP4, NEXTN MTP + hierarchical cache).
5+
#
6+
# Ported from the manually-run srtctl base:/zip_override_conc: starter to the
7+
# flat single-variant agentic schema. Pure sglang (no dynamo): frontend.type
8+
# sglang serves directly. Concurrency comes from the GHA matrix (exported as
9+
# CONC into agentic_srt.sh), not the recipe. Serving tuning preserved verbatim
10+
# from the source run; only CI scaffolding is added.
11+
12+
model:
13+
path: "qwen3.5-fp4"
14+
container: "dynamo-sglang"
15+
precision: "fp4"
16+
17+
slurm:
18+
time_limit: "8:00:00"
19+
20+
health_check:
21+
max_attempts: 1440
22+
interval_seconds: 10
23+
24+
resources:
25+
gpu_type: "gb300"
26+
gpus_per_node: 4
27+
agg_nodes: 1
28+
agg_workers: 1
29+
gpus_per_agg: 4
30+
31+
infra:
32+
nats_max_payload_mb: 8
33+
34+
frontend:
35+
type: sglang
36+
37+
backend:
38+
type: sglang
39+
aggregated_environment:
40+
SGLANG_SIMULATE_ACC_LEN: '3.39'
41+
SGLANG_SIMULATE_ACC_METHOD: match-expected
42+
SGLANG_SIMULATE_ACC_TOKEN_MODE: "real-draft-token"
43+
TORCH_CUDA_ARCH_LIST: '10.0'
44+
PYTHONNOUSERSITE: '1'
45+
NCCL_NVLS_ENABLE: '1'
46+
SGL_ENABLE_JIT_DEEPGEMM: 'false'
47+
SGLANG_ENABLE_FLASHINFER_GEMM: 'true'
48+
sglang_config:
49+
aggregated:
50+
served-model-name: nvidia/Qwen3.5-397B-A17B-NVFP4
51+
model-path: /model/
52+
trust-remote-code: true
53+
tensor-parallel-size: 4
54+
data-parallel-size: 1
55+
expert-parallel-size: 1
56+
enable-symm-mem: true
57+
quantization: modelopt_fp4
58+
kv-cache-dtype: fp8_e4m3
59+
mamba-ssm-dtype: bfloat16
60+
mamba-scheduler-strategy: extra_buffer
61+
mamba-track-interval: 8192
62+
attention-backend: trtllm_mha
63+
moe-runner-backend: flashinfer_trtllm
64+
speculative-algorithm: NEXTN
65+
speculative-num-steps: 3
66+
speculative-eagle-topk: 1
67+
speculative-num-draft-tokens: 4
68+
cuda-graph-max-bs: 256
69+
max-running-requests: 1024
70+
max-prefill-tokens: 16384
71+
chunked-prefill-size: 16384
72+
mem-fraction-static: 0.8
73+
max-mamba-cache-size: 1024
74+
allow-auto-truncate: true
75+
stream-interval: 50
76+
scheduler-recv-interval: 10
77+
tokenizer-worker-num: 6
78+
page-size: 64
79+
enable-hierarchical-cache: true
80+
hicache-ratio: 1.05
81+
hicache-io-backend: kernel
82+
hicache-mem-layout: page_first_direct
83+
hicache-write-policy: write_through
84+
85+
sbatch_directives:
86+
mem: "0"
87+
cpus-per-task: "144"
88+
89+
srun_options:
90+
mem: "0"
91+
# gb300-nv: pyxis maps the calling user into the container as a non-root uid;
92+
# agentic_srt.sh's apt-get install git step needs EUID 0. Remap to uid 0 inside
93+
# the container. srt-slurm renders empty-string values as flag-only srun args.
94+
container-remap-root: ""
95+
96+
benchmark:
97+
type: custom
98+
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
99+
env:
100+
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
101+
RESULT_DIR: /logs/agentic
102+
PORT: "8000"
103+
# Aggregated: one worker serves both prefill and decode, so GPU accounting is
104+
# the single-worker form num_gpus = TP (not the disagg prefill+decode sum).
105+
# Force the single-node post-proc path; TP must match sglang_config tp-size.
106+
IS_MULTINODE: "false"
107+
TP: "4"
108+
# Match the source run's dataset (256k cc-traces-with-subagents variant).
109+
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126_256k
110+
# Container-side path of the aiperf mmap dataset cache; host-side mount wired
111+
# via launch_gb300-nv.sh srtslurm.yaml default_mounts.
112+
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
113+
# Persistent HF hub cache (also via default_mounts).
114+
HF_HUB_CACHE: "/hf_hub_cache"

0 commit comments

Comments
 (0)