[NV] Update H100 Qwen3.5 SGLang agg config

[anish-shanbhag]

Updates Qwen3.5-397B-A17B-FP8 H100 SGLang agg recipes with tuned configs for 1k/1k and 8k/1k.

Performance comparison

Source rows: results_bmk artifacts from baseline 2026-05-18 CI run vs updated 2026-06-04 CI run. tok/s/user is median_intvty.

Apples-to-apples note: deltas compare non-MTP Qwen3.5 FP8 H100 SGLang rows only. The May artifact has non-MTP TP8/EP8 rows at conc 4, 8, 16, and 32 for both workloads; new-only concurrencies are marked n/a on the May side.

ISL/OSL	Conc	May non-MTP tok/s/gpu	New non-MTP tok/s/gpu	Delta tok/s/gpu	May non-MTP tok/s/user	New non-MTP tok/s/user	Delta tok/s/user
1k/1k	1	n/a	38.9	n/a	n/a	165.7	n/a
1k/1k	2	n/a	68.3	n/a	n/a	144.7	n/a
1k/1k	4	100.3	113.0	+12.6%	106.3	119.9	+12.7%
1k/1k	8	159.4	200.0	+25.5%	83.0	108.1	+30.3%
1k/1k	16	245.2	255.8	+4.4%	62.7	66.5	+6.0%
1k/1k	32	352.3	453.4	+28.7%	44.9	64.6	+43.8%
1k/1k	64	n/a	734.4	n/a	n/a	52.2	n/a
1k/1k	128	n/a	1055.5	n/a	n/a	39.2	n/a
1k/1k	256	n/a	1316.4	n/a	n/a	28.3	n/a
8k/1k	1	n/a	170.2	n/a	n/a	163.2	n/a
8k/1k	2	n/a	297.5	n/a	n/a	142.1	n/a
8k/1k	4	440.2	484.5	+10.1%	104.2	114.7	+10.1%
8k/1k	8	676.7	737.5	+9.0%	80.6	89.4	+10.9%
8k/1k	16	978.7	1030.5	+5.3%	56.5	60.4	+6.9%
8k/1k	32	1338.6	1526.3	+14.0%	38.6	54.9	+42.1%
8k/1k	64	n/a	2065.1	n/a	n/a	38.7	n/a
8k/1k	128	n/a	2338.1	n/a	n/a	23.6	n/a
8k/1k	256	n/a	2506.1	n/a	n/a	10.5	n/a

1k/1k matched geomean: +17.4% tok/s/gpu, +22.3% tok/s/user; 8k/1k matched geomean: +9.5% tok/s/gpu, +16.7% tok/s/user.

---

Note

Low Risk
Benchmark and CI YAML tuning only; no production serving, auth, or application runtime paths.

Overview
Retunes the Qwen3.5-397B-A17B-FP8 aggregate SGLang on H100 benchmark for 1k/1k and 8k/1k fixed-seq sweeps.

CI/config (qwen3.5-fp8-h100-sglang): replaces a single TP8/EP8, conc 4–32 grid with TP8/EP1, conc 1–8 and TP8/EP8, conc 16–256 for both scenarios.

Launch recipe (qwen3.5_fp8_h100.sh): drops H100-vs-H200 memory caveats; applies expert parallel only when EP_SIZE > 1; maps conc to --scheduler-recv-interval (with hard fail on unsupported conc); raises max-running-requests (64→256), chunked-prefill-size (8192→16384), mem-fraction-static (0.75→0.8); adds --enable-symm-mem and passes scheduler args into sglang.launch_server.

Documents the change in perf-changelog.yaml.

^{Reviewed by Cursor Bugbot for commit 50bf385. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

[claude]

Claude finished @anish-shanbhag's task in 3m 54s —— View job

---

Review of PR #1544

• Read changed files and gather context
• Validate master config and perf-changelog
• Review benchmark script changes
• Post review summary

LGTM - no blocking issues found

The PR cleanly restructures the H100 Qwen3.5 SGLang aggregate recipe to sweep across three parallelism tiers (TP8/EP1 for low concurrency, TP8/EP8 for mid, TP8/EP8+DP-attention for high). Verified:

• nvidia-master.yaml modified and perf-changelog.yaml updated with entry appended at the end ✅
• EP is conditionally enabled via EP_SIZE > 1 (correct SGLang pattern) ✅
• DP-attention handling (--dp-size "$TP" --enable-dp-attention) is consistent with other scripts (e.g., dsv4_fp4_b300_sglang_mtp.sh) ✅
• Server launch command properly formatted with line continuations ✅
• Container image lmsysorg/sglang:v0.5.12-cu130 is publicly accessible on Docker Hub ✅
• case statement for scheduler-recv-interval covers all concurrency values reachable from the config search-space, with a safe * fallback that exits on unexpected values ✅

[claude]

LGTM, straightforward benchmark recipe tuning.

Extended reasoning...

Overview

This PR tunes the Qwen3.5-397B-A17B-FP8 H100 SGLang aggregate recipe for both 1k/1k and 8k/1k sweeps. It replaces a single search-space row with three tiered rows (TP8/EP1 conc 1-8, TP8/EP8 conc 16-64, TP8/EP8 + DP-attention conc 128-256), updates the benchmark shell script to thread DP_ATTENTION/EP_SIZE through optional CLI args, adds per-concurrency --scheduler-recv-interval tuning, and bumps max-running-requests/chunked-prefill-size/mem-fraction-static back up from the conservative H200-derived defaults. A changelog entry is included.

Security risks

None. This is a benchmark configuration and shell script change in a non-production, single-node benchmark recipe. No auth, crypto, networking, or input-handling surface area is touched.

Level of scrutiny

Low. This is benchmark recipe tuning isolated to one model/hardware combo. The script changes follow the established DP_ATTENTION env-var pattern used by other recipes (e.g. dsv4_fp4_b200_vllm.sh), the dp-attn search-space key is already supported by utils/matrix_logic/generate_sweep_configs.py, and the YAML edit is local to the qwen3.5-fp8-h100-sglang block.

Other factors

The new case "$CONC" covers every concurrency value the search-space will generate for the non-DP branches (1, 2, 4, 8, 16, 32, 64) and explicitly errors on unsupported values, so silent misconfiguration is unlikely. No bugs were flagged by the bug hunting system, and the prior commit 394e886 on main is on this same tuning track.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26198417664
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26198417664

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26610973668
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26610973668

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26611114296
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26611114296

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26611114296
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26611114296

[anish-shanbhag]

/sweep test-config --config-files .github/configs/nvidia-master.yaml --runner-node-filter h100-cw --config-keys qwen3.5-fp8-h100-sglang --conc 4 --seq-lens 8k1k --no-evals

[github-actions]

@anish-shanbhag Kicking off a sweep.

Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26614204211
Command: test-config --config-files .github/configs/nvidia-master.yaml --runner-node-filter h100-cw --config-keys qwen3.5-fp8-h100-sglang --conc 4 --seq-lens 8k1k --no-evals
Pinned ref: 213a1d2
Approval: not required (trusted collaborator).

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26611114296
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26611114296

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26611114296
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26611114296

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26611114296
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26611114296

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26611114296
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26611114296

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26916564786
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26916564786

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26918589379
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26918589379

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26918589379
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26918589379

[functionstackx]

thanks for the contribution! lgtm except for this isnt in sglang docs yet

[Ankur-singh]

@functionstackx
here is the PR sgl-project/sglang#27296

[functionstackx]

/reuse-sweep-run

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26975382615
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26975382615