Skip to content

Commit c478834

Browse files
committed
[AMD][MI35X] Qwen3.5-fp4 add ROCM_QUICK_REDUCE_QUANTIZATION=INT8, drop aiter-allreduce-fusion
Add ROCM_QUICK_REDUCE_QUANTIZATION=INT8 and remove --enable-aiter-allreduce-fusion for Qwen3.5-397B-A17B-MXFP4 mi355x single-node benchmarks (non-MTP and MTP).
1 parent 0094291 commit c478834

3 files changed

Lines changed: 13 additions & 3 deletions

File tree

benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_mi355x.sh

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ export SGLANG_USE_AITER=1
2121
export SGLANG_USE_AITER_UNIFIED_ATTN=1
2222
export AITER_FLYDSL_FORCE=1
2323
export SGLANG_MAMBA_SSM_DTYPE=bfloat16
24+
export ROCM_QUICK_REDUCE_QUANTIZATION=INT8
2425

2526
SERVER_LOG=/workspace/server.log
2627
MEM_FRAC_STATIC=${MEM_FRAC_STATIC:-0.8}
@@ -41,7 +42,7 @@ python3 -m sglang.launch_server --model-path=$MODEL --trust-remote-code \
4142
--model-loader-extra-config '{"enable_multithread_load": true}' \
4243
--watchdog-timeout 1200 \
4344
--disable-radix-cache \
44-
--enable-aiter-allreduce-fusion --max-running-requests $CONC \
45+
--max-running-requests $CONC \
4546
--page-size 16 \
4647
> $SERVER_LOG 2>&1 &
4748

benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_mi355x_mtp.sh

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ export SGLANG_USE_AITER=1
2121
export SGLANG_USE_AITER_UNIFIED_ATTN=1
2222
export AITER_FLYDSL_FORCE=1
2323
export SGLANG_MAMBA_SSM_DTYPE=bfloat16
24+
export ROCM_QUICK_REDUCE_QUANTIZATION=INT8
2425

2526
SERVER_LOG=/workspace/server.log
2627
MEM_FRAC_STATIC=${MEM_FRAC_STATIC:-0.8}
@@ -41,7 +42,7 @@ python3 -m sglang.launch_server --model-path=$MODEL --trust-remote-code \
4142
--model-loader-extra-config '{"enable_multithread_load": true}' \
4243
--watchdog-timeout 1200 \
4344
--disable-radix-cache \
44-
--enable-aiter-allreduce-fusion --max-running-requests $CONC \
45+
--max-running-requests $CONC \
4546
--page-size 16 \
4647
--speculative-algorithm EAGLE \
4748
--speculative-num-steps 3 \

perf-changelog.yaml

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4885,7 +4885,15 @@
48854885
description:
48864886
- "Bump image to lmsysorg/sglang-rocm:v0.5.15-rocm720-mi35x-20260713"
48874887
- "Add SGLANG_MAMBA_SSM_DTYPE=bfloat16 in both non-MTP and MTP benchmark scripts"
4888-
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2201
4888+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2201
4889+
4890+
- config-keys:
4891+
- qwen3.5-fp4-mi355x-sglang
4892+
- qwen3.5-fp4-mi355x-sglang-mtp
4893+
description:
4894+
- "Add ROCM_QUICK_REDUCE_QUANTIZATION=INT8 in both non-MTP and MTP benchmark scripts"
4895+
- "Remove --enable-aiter-allreduce-fusion from both non-MTP and MTP launch commands"
4896+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2202
48894897

48904898
- config-keys:
48914899
- dsv4-fp4-b200-vllm-agentic

0 commit comments

Comments
 (0)