Skip to content

Commit 2003066

Browse files
Oseltamivirclaude
Oseltamivir
and
claude
authored
Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks (#2268)
* feat: add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks Add glm5.2-fp4-b300-sglang-agentic from the SGLang cookbook B300 NVFP4 single-node recipes (STP only, no spec decoding): - Low-latency arm: TP8 with fp8 KV cache and cutedsl bf16 GEMM backend, conc [1, 2, 4, 8, 16, 32] - High-throughput arm: TP8/DP8 attention-DP behind sglang-router consistent hashing for session affinity, conc [48-512] - Image: lmsysorg/sglang:v0.5.15.post1-cu130 (ships sglang-router 0.3.2) - benchmark_lib.sh: glm5.2* added to the unfiltered agentic corpus branch (GLM-5.2 is a 1M-context model) Both arms validated on b300-nv: LL conc16 8k1k 10,043 total tok/s (TPOT 12.1 ms); HT conc256 34,429 total tok/s (TPOT 48.9 ms). Weights pre-staged at /data/models/GLM-5.2-NVFP4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fill perf-changelog pr-link for glm5.2 b300 agentic entry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: outlast AIPerf per-session keep-alive in glm5.2 b300 agentic recipe One warmup request hit ECONNRESET (uvicorn closes idle connections after SGLANG_TIMEOUT_KEEP_ALIVE=5s while AIPerf reuses one pooled connection per session with a 300s client keep-alive across inter-turn gaps of up to 10s), and any terminal warmup failure aborts the AIPerf run by design. Set SGLANG_TIMEOUT_KEEP_ALIVE=900 so the server outlasts the client pool. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: widen DP-attention chunked prefill for agentic corpus in glm5.2 b300 recipe The cookbook HT cell's --chunked-prefill-size 8192 is a whole-engine budget: under dp8 each rank prefills 1,024 tokens/step, which starves prefill on the 1M-context agentic corpus. A conc-256 warmup timed out after AIPerf's 1800s drain grace period with 15 giant sessions still prefilling while KV usage sat at ~0.01 (prefill-rate-bound, not memory-bound). Use the cookbook's own dp8 lever from the B200 cells: 32768 total = ~4096 tokens/rank/step. The TP arm keeps 8192 (full budget per step, passed warmup fine). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: enable glm47 tool-call and glm45 reasoning parsers for SWE-bench evals The agentic SWE-bench eval drives mini-swe-agent through native tool calling. Without --tool-call-parser, GLM-5.2's tool-call tokens stay as raw text in message.content, every instance dies with RepeatedFormatError ("No tool calls found in the response"), 287/300 patches come back empty, and swebench_lite scores 0.0000 against the 0.50 default threshold. Per the GLM-5.2 cookbook: the model emits the GLM-4.7-style <tool_call>/<arg_key>/<arg_value> format, so it needs the glm47 parser (glm45 does not parse it); the glm45 reasoning parser separates hybrid thinking into reasoning_content. Neither flag affects trace-replay throughput (pre-canned replay discards live responses). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: overridable agentic warmup grace; 3600s for glm5.2 b300 DPA arm The DPA conc-512 warmup drain converges healthily (~0.45 req/s, zero errors) but needs ~2500s end to end - the shared 1800s grace cuts it off with ~300 requests in flight. Make the grace period overridable via AGENTIC_WARMUP_GRACE_PERIOD (default unchanged at 1800) and set 3600 for the glm5.2 DP-attention arm. Grace is a maximum wait, not a fixed sleep, so lower-conc points are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: double SWE-bench step budget for glm5.2 b300 agentic evals With the glm47 parser fix in, the agentic SWE-bench eval produces real work: 10/11 submitted trajectories resolved (0.91 precision), but 12/23 exited LimitsExceeded before submitting - GLM-5.2's chat template defaults to reasoning_effort=Max when the client passes no chat_template_kwargs, and the heavy thinking exhausts the default 75-step budget. Score landed at 0.4348 vs the 0.50 floor. Export SWEBENCH_AGENT_STEP_LIMIT=150 in the recipe's eval path only; the shared default stays 75. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 69f9114 commit 2003066

4 files changed

Lines changed: 223 additions & 2 deletions

File tree

‎benchmarks/benchmark_lib.sh‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1660,7 +1660,7 @@ resolve_trace_source() {
16601660
# WEKA_LOADER_OVERRIDE.
16611661
local default_loader
16621662
case "${MODEL_PREFIX:-}" in
1663-
dsv4*|minimaxm3*)
1663+
dsv4*|glm5.2*|minimaxm3*)
16641664
default_loader="semianalysis_cc_traces_weka_062126"
16651665
;;
16661666
*)
@@ -1780,8 +1780,11 @@ build_replay_cmd() {
17801780
# only after those requests drain and resumes from the resulting live state.
17811781
REPLAY_CMD+=" --agentic-cache-warmup-duration 600"
17821782
# Give long-context warmup requests up to 30 minutes to drain before
1783+
# declaring warmup failed. Recipes whose saturation arms carry a larger
1784+
# in-flight working set may override via AGENTIC_WARMUP_GRACE_PERIOD
1785+
# (grace is a maximum wait, not a fixed sleep — drain exits when done).
17831786
# cancelling any remaining requests and starting profiling.
1784-
REPLAY_CMD+=" --warmup-grace-period 1800"
1787+
REPLAY_CMD+=" --warmup-grace-period ${AGENTIC_WARMUP_GRACE_PERIOD:-1800}"
17851788
# Use server-reported usage fields (prompt_tokens / completion_tokens) for
17861789
# ISL/OSL instead of client-side tokenizer.encode(). Auto-enables
17871790
# stream_options.include_usage on the OpenAI chat endpoint. Skips the

‎benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
set -x
4+
5+
# Agentic trace replay benchmark for GLM-5.2 NVFP4 on B300 using SGLang.
6+
#
7+
# Server flags follow the SGLang cookbook B300 NVFP4 single-node recipes
8+
# (https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2), STP only:
9+
# the cookbook's EAGLE MTP variants are intentionally not wired up yet.
10+
# DP_ATTENTION=false -> low-latency arm (TP8, fp8 KV, cutedsl bf16 GEMM)
11+
# DP_ATTENTION=true -> high-throughput arm (TP8 + DP8 attention-DP)
12+
#
13+
# Required env vars:
14+
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION,
15+
# EP_SIZE, DP_ATTENTION
16+
17+
source "$(dirname "$0")/../../benchmark_lib.sh"
18+
19+
check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION
20+
21+
if [[ "$KV_OFFLOADING" != "none" ]]; then
22+
echo "Error: KV_OFFLOADING=$KV_OFFLOADING is not supported by this recipe" >&2
23+
exit 1
24+
fi
25+
26+
if [[ -n "${SLURM_JOB_ID:-}" ]]; then
27+
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
28+
fi
29+
30+
# `hf download` creates the target dir if missing and is itself idempotent.
31+
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE.
32+
# Either way, MODEL_PATH is what the server is launched with.
33+
if [[ -n "${MODEL_PATH:-}" ]]; then
34+
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
35+
hf download "$MODEL" --local-dir "$MODEL_PATH"
36+
fi
37+
else
38+
hf download "$MODEL"
39+
export MODEL_PATH="$MODEL"
40+
fi
41+
nvidia-smi
42+
43+
resolve_trace_source
44+
install_agentic_deps
45+
46+
SERVER_LOG="$RESULT_DIR/server.log"
47+
mkdir -p "$RESULT_DIR"
48+
49+
# With attention-DP, front the DP ranks with sglang-router using consistent
50+
# hashing on the AIPerf correlation id so multi-turn sessions stay on the DP
51+
# rank that holds their radix-cache prefix.
52+
USE_SGLANG_ROUTER=false
53+
SGLANG_BACKEND_PORT="$PORT"
54+
ROUTER_LOG="$RESULT_DIR/router.log"
55+
if [ "$DP_ATTENTION" = "true" ]; then
56+
USE_SGLANG_ROUTER=true
57+
export AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=true
58+
SGLANG_BACKEND_PORT=$((PORT + 1))
59+
SGLANG_ROUTER_METRICS_PORT=$((PORT + 10000))
60+
fi
61+
62+
PARALLEL_ARGS=(--tp "$TP" --ep-size "$EP_SIZE")
63+
CHUNKED_PREFILL_SIZE=8192
64+
if [ "$DP_ATTENTION" = "true" ]; then
65+
# chunked-prefill-size is a whole-engine budget split across DP ranks:
66+
# the cookbook HT cell's 8192 becomes 1,024 tokens/rank/step under dp8,
67+
# which starves prefill on the 1M-context agentic corpus (observed: a
68+
# conc-256 warmup could not drain within AIPerf's 1800s grace period
69+
# while KV usage sat at ~0.01). Use the cookbook's own dp8 lever from
70+
# the B200 cells (32768 = ~4096/rank).
71+
CHUNKED_PREFILL_SIZE=32768
72+
# At conc 512 the saturation working set outlives the default 1800s
73+
# warmup drain grace: the drain converges healthily (~0.45 req/s, zero
74+
# errors) but needs ~2500s end to end. 3600 is a maximum wait, not a
75+
# fixed sleep — lower-conc DPA points still finish as fast as they drain.
76+
export AGENTIC_WARMUP_GRACE_PERIOD=3600
77+
PARALLEL_ARGS+=(
78+
--dp "$TP"
79+
--enable-dp-attention
80+
--tokenizer-worker-num "$TP"
81+
--dist-init-addr "127.0.0.1:$((PORT + 2000))"
82+
)
83+
else
84+
# Cookbook low-latency levers; the DP-attention cell omits them.
85+
PARALLEL_ARGS+=(
86+
--kv-cache-dtype fp8_e4m3
87+
--bf16-gemm-backend cutedsl
88+
--max-prefill-tokens 8192
89+
)
90+
fi
91+
92+
# AgentX concurrency counts live session trees, not individual requests.
93+
# Allow subagent fan-out to exceed CONC without clipping request bursts.
94+
MAX_RUNNING_REQUESTS=$((2 * CONC))
95+
GRAPH_ARGS=()
96+
if [ "$DP_ATTENTION" != "true" ]; then
97+
# Cookbook low-latency captures graphs up to its request cap; the
98+
# DP-attention cell leaves the CUDA-graph batch list at SGLang defaults.
99+
CUDA_GRAPH_MAX_BS=$MAX_RUNNING_REQUESTS
100+
[ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64
101+
GRAPH_ARGS=(--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS")
102+
fi
103+
104+
export PYTHONNOUSERSITE=1
105+
export TORCH_CUDA_ARCH_LIST=10.0
106+
# Agentic warmup dispatches hundreds of large prompts at once; allow up to
107+
# 15 minutes of TCP progress before AIPerf declares a connection dead.
108+
export AIPERF_HTTP_TCP_USER_TIMEOUT=900000
109+
# AIPerf pins one pooled keep-alive connection per session (client-side
110+
# keep-alive 300s) while uvicorn's default SGLANG_TIMEOUT_KEEP_ALIVE is 5s;
111+
# inter-turn idle gaps (capped at 10s) can reuse a socket exactly as the
112+
# server closes it -> ECONNRESET -> terminal warmup failure. Outlast the
113+
# client pool so the race cannot occur.
114+
export SGLANG_TIMEOUT_KEEP_ALIVE=900
115+
116+
SGLANG_CMD=(
117+
python3 -m sglang.launch_server
118+
--model-path "$MODEL_PATH"
119+
--served-model-name "$MODEL"
120+
--host 0.0.0.0
121+
--port "$SGLANG_BACKEND_PORT"
122+
--trust-remote-code
123+
"${PARALLEL_ARGS[@]}"
124+
--quantization modelopt_fp4
125+
# GLM-5.2 emits the GLM-4.7-style <tool_call>/<arg_key>/<arg_value> format;
126+
# the glm47 parser is required for structured message.tool_calls (glm45
127+
# leaves calls as raw text). Without it the SWE-bench mini-swe-agent eval
128+
# dies with RepeatedFormatError ("No tool calls found in the response") on
129+
# every instance and scores 0. Reasoning parser keeps hybrid-thinking
130+
# output in reasoning_content instead of polluting content. Neither flag
131+
# affects trace-replay throughput (pre-canned replay discards live
132+
# responses).
133+
--tool-call-parser glm47
134+
--reasoning-parser glm45
135+
--chunked-prefill-size "$CHUNKED_PREFILL_SIZE"
136+
--mem-fraction-static 0.85
137+
--max-running-requests "$MAX_RUNNING_REQUESTS"
138+
"${GRAPH_ARGS[@]}"
139+
--watchdog-timeout 1800
140+
--enable-metrics
141+
)
142+
143+
printf '%q ' "${SGLANG_CMD[@]}" | tee "$RESULT_DIR/sglang_command.txt"
144+
printf '\n' | tee -a "$RESULT_DIR/sglang_command.txt"
145+
146+
echo "Starting SGLang server for B300..."
147+
"${SGLANG_CMD[@]}" > "$SERVER_LOG" 2>&1 &
148+
SERVER_PID=$!
149+
echo "Server PID: $SERVER_PID"
150+
151+
wait_for_server_ready --port "$SGLANG_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"
152+
153+
if [ "$USE_SGLANG_ROUTER" = "true" ]; then
154+
echo "Starting SGLang router on port $PORT for $TP DP ranks..."
155+
python3 -m sglang_router.launch_router \
156+
--worker-urls "http://localhost:$SGLANG_BACKEND_PORT" \
157+
--policy consistent_hashing \
158+
--request-id-headers x-correlation-id \
159+
--dp-aware \
160+
--host 0.0.0.0 \
161+
--port "$PORT" \
162+
--prometheus-host 127.0.0.1 \
163+
--prometheus-port "$SGLANG_ROUTER_METRICS_PORT" \
164+
--connect-timeout-secs 900 \
165+
--request-timeout-secs 14400 \
166+
--disable-health-check \
167+
--disable-retries > "$ROUTER_LOG" 2>&1 &
168+
ROUTER_PID=$!
169+
echo "Router PID: $ROUTER_PID"
170+
wait_for_server_ready --port "$PORT" --server-log "$ROUTER_LOG" --server-pid "$ROUTER_PID"
171+
fi
172+
173+
if [ "${EVAL_ONLY}" = "true" ]; then
174+
# GLM-5.2's chat template defaults to reasoning_effort=Max when the
175+
# client passes no chat_template_kwargs (mini-swe-agent doesn't), and the
176+
# heavy thinking burns the default 75-step budget: on the 23-instance
177+
# slice, 12/23 trajectories exited LimitsExceeded unsubmitted while 10 of
178+
# the 11 that submitted resolved. Double the step budget for this recipe;
179+
# other recipes keep the shared 75 default.
180+
export SWEBENCH_AGENT_STEP_LIMIT=150
181+
run_eval --port "$PORT"
182+
else
183+
build_replay_cmd "$RESULT_DIR"
184+
REPLAY_CMD+=" --server-metrics http://localhost:$SGLANG_BACKEND_PORT/metrics"
185+
run_agentic_replay_and_write_outputs "$RESULT_DIR"
186+
fi

‎configs/nvidia-master.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7853,3 +7853,25 @@ qwen3.5-fp8-gb300-dynamo-sglang:
78537853
tp: 16
78547854
ep: 16
78557855
dp-attn: true
7856+
7857+
# GLM-5.2 B300 NVFP4 AgentX frontier from the SGLang cookbook B300 NVFP4
7858+
# single-node recipes (https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2),
7859+
# STP only (the cookbook's EAGLE MTP variants are deferred). The TP8 arm is the
7860+
# cookbook low-latency recipe (fp8 KV, cutedsl bf16 GEMM) and covers the
7861+
# interactivity end; the TP8/DP8 attention-DP arm is the cookbook
7862+
# high-throughput recipe behind sglang-router consistent hashing for session
7863+
# affinity. Conc lists are disjoint between arms so exp-names stay unique.
7864+
glm5.2-fp4-b300-sglang-agentic:
7865+
image: lmsysorg/sglang:v0.5.15.post1-cu130
7866+
model: nvidia/GLM-5.2-NVFP4
7867+
model-prefix: glm5.2
7868+
runner: cluster:b300-nv
7869+
precision: fp4
7870+
framework: sglang
7871+
multinode: false
7872+
scenarios:
7873+
agentic-coding:
7874+
- dram-utilization: 0.80
7875+
search-space:
7876+
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32] }
7877+
- { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [48, 64, 96, 128, 192, 256, 512], router: { name: sglang-router, version: "0.3.2" } }

‎perf-changelog.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4934,6 +4934,16 @@
49344934
- "Add MiniMax M3 NVFP4 B300 Dynamo-vLLM disaggregated EAGLE3 recipes"
49354935
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2182
49364936

4937+
- config-keys:
4938+
- glm5.2-fp4-b300-sglang-agentic
4939+
description:
4940+
- "Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks from the SGLang cookbook B300 NVFP4 recipes (STP only, no spec decoding)"
4941+
- "Image: lmsysorg/sglang:v0.5.15.post1-cu130; model nvidia/GLM-5.2-NVFP4"
4942+
- "Low-latency arm: TP8 (fp8 KV cache, cutedsl bf16 GEMM backend) at conc [1, 2, 4, 8, 16, 32]"
4943+
- "High-throughput arm: TP8/DP8 attention-DP behind sglang-router consistent hashing at conc [48, 64, 96, 128, 192, 256, 512]"
4944+
- "New 1M-context model prefix glm5.2 added to the unfiltered agentic corpus branch in benchmark_lib.sh"
4945+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2268
4946+
49374947
- config-keys:
49384948
- kimik2.5-int4-b200-vllm
49394949
- kimik2.5-int4-b300-vllm

0 commit comments

Comments
 (0)