[AMD] MiniMax-M3 MXFP8 MI355X vLLM: gate AITER sparse PA to 8k1k high-conc + native linear default / MiniMax-M3 MXFP8 MI355X vLLM：AITER 稀疏分页注意力门控至 8k1k 高并发 + 默认原生 linear (#2187)

hongxiayang · cursoragent · seungrokj · web-flow · commit b38bf415003c · 2026-07-22T15:55:19.000-05:00
* [AMD] MiniMax-M3 MXFP8 MI355X vLLM: enable AITER sparse PA + drop emulation linear Bump the minimaxm3-fp8-mi355x-vllm image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287). Enable it via VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4, num_kv_heads == 1 per rank) plus the recipe's quick all-reduce knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256. Drop --linear-backend emulation now that the Triton MXFP8 dense-linear GEMM is optimized. Deliberately do NOT add the #47269 --hf-overrides use_index_cache/index_topk_freq indexer skip: it reduces model-architecture FLOPs, disallowed by docs/PR_REVIEW_CHECKLIST.md. Sparse PA is a kernel-level speedup only. Verified locally on MI355X (gfx950) TP4 8k1k: conc1 10/10 and conc512 5120/5120 completed with zero failures (conc512 ~28.6k tok/s total, ~7.1k tok/s/GPU). 中文：将 minimaxm3-fp8-mi355x-vllm 镜像升级到 nightly-9e57de7197f234f9d9187715d96e07e007048c0f，该镜像已包含合并进主分支的 AITER page-16 稀疏分页注意力(sparse paged attention, vllm-project/vllm#47287)。 通过 VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1（配合 VLLM_ROCM_USE_AITER=1、fp8 KV cache 及 TP4，每个 rank num_kv_heads == 1）启用，并补上 recipe 的 quick all-reduce 调优开关 VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 与 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256。由于 Triton MXFP8 稠密 linear GEMM 已优化，移除 --linear-backend emulation。刻意不添加 #47269 的 --hf-overrides use_index_cache/index_topk_freq indexer 跳层开关：它会减少模型 架构 FLOPs，违反 docs/PR_REVIEW_CHECKLIST.md。sparse PA 仅为 kernel 级加速。 已在 MI355X(gfx950) TP4 8k1k 本地验证：conc1 10/10、conc512 5120/5120 全部成功 （conc512 约 28.6k tok/s 总吞吐，约 7.1k tok/s/GPU）。 Co-authored-by: Cursor <cursoragent@cursor.com> * [AMD] MiniMax-M3 MXFP8 MI355X vLLM: concurrency-gate emulation linear for 8k1k high-conc On the sparse-PA nightly the dense-linear backend crossover reversed vs #2003: the native Triton MXFP8 GEMM now wins in the memory-bound low-concurrency regime, while --linear-backend emulation (bf16 hipBLASLT) still wins in the compute-bound high-concurrency regime. Measured on gfx950 MXFP8 + sparse PA: emulation ~+3-5% at 8k1k conc 64-512 but ~-3% at conc 1-8. Gate emulation to isl>=8192 && conc>=64; native everywhere else (all 1k1k, 8k1k conc<64). Overridable via LINEAR_BACKEND. 中文：在 sparse-PA nightly 上，稠密 linear 后端的性能拐点相比 #2003 发生反转： 原生 Triton MXFP8 GEMM 现在在访存受限的低并发区间更快，而 --linear-backend emulation（bf16 hipBLASLT）仍在计算受限的高并发区间更快。gfx950 MXFP8 + sparse PA 实测：emulation 在 8k1k conc 64-512 约 +3-5%，在 conc 1-8 约 -3%。因此将 emulation 限定在 isl>=8192 且 conc>=64，其余（所有 1k1k、8k1k conc<64）使用原生 路径。可通过 LINEAR_BACKEND 覆盖。 Co-authored-by: Cursor <cursoragent@cursor.com> * [AMD] MiniMax-M3 MXFP8 MI355X vLLM: gate sparse-PA fast path to 8k1k high-conc AITER page-16 sparse PA is a long-context/high-batch optimization: it wins at 8k1k conc>=64 but adds overhead at short context (1k1k) and low concurrency. Gate both VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT (sparse PA) and --linear-backend emulation behind a single MM3_HIGH_CONC_FASTPATH condition (isl>=8192 && conc>=64); everything else falls back to the #2003 path (non-shuffled Triton attention + native linear) so there is no regression outside the target regime. Net vs #2003: 8k1k tput/gpu conc128 +3.3%, conc256 +2.5%, conc512 +4.7% (peak), neutral elsewhere. Overridable via MM3_HIGH_CONC_FASTPATH / LINEAR_BACKEND. 中文：AITER page-16 稀疏分页注意力(sparse PA)是长上下文/大批量优化：在 8k1k conc>=64 有收益，但在短上下文(1k1k)和低并发下带来额外开销。将 VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT（sparse PA）与 --linear-backend emulation 统一 用 MM3_HIGH_CONC_FASTPATH 条件（isl>=8192 且 conc>=64）门控；其余情况回退到 #2003 路径（非 shuffle 的 Triton 注意力 + 原生 linear），从而在目标区间之外不产生回退。 相对 #2003：8k1k tput/gpu conc128 +3.3%、conc256 +2.5%、conc512 +4.7%（峰值）， 其余基本持平。可通过 MM3_HIGH_CONC_FASTPATH / LINEAR_BACKEND 覆盖。 Co-authored-by: Cursor <cursoragent@cursor.com> * chore: set perf-changelog pr-link to #2187 中文：将 perf-changelog 的 pr-link 设置为 #2187。 Co-authored-by: Cursor <cursoragent@cursor.com> * minor tuning * [AMD] MiniMax-M3 MXFP8 MI355X vLLM: use INT4 quick all-reduce for all concurrencies Switch VLLM_ROCM_QUICK_REDUCE_QUANTIZATION from INT6 to INT4 unconditionally (#2003 used INT6); accuracy is guarded by the 8k1k evals. The AITER quick all-reduce tuning knobs (CAST_BF16_TO_FP16=0, QUANTIZATION_MIN_SIZE_KB=256) remain in the 8k1k high-conc fast path. Aimed at lifting the low/mid-conc points (e.g. the 8k1k conc8 dip) via cheaper all-reduces. 中文：将 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION 从 INT6 无条件改为 INT4（#2003 用的是 INT6），准确性由 8k1k evals 保障。AITER quick all-reduce 调优开关 (CAST_BF16_TO_FP16=0、QUANTIZATION_MIN_SIZE_KB=256) 仍保留在 8k1k 高并发快速路径中。 目的是通过更廉价的 all-reduce 提升低/中并发点（例如 8k1k conc8 的回退）。 Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: Alec Ibarra <93070681+adibarra@users.noreply.github.com>
diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh
@@ -3,6 +3,17 @@
 # MiniMax-M3 MXFP8 MI355X (gfx950) single-node vLLM recipe.
 # https://github.com/vllm-project/recipes/commit/2a3728ed9892debfd767a72a58ebc90b33f186e5
 # The recipe recommends MXFP8 from TP=4 on gfx950 and requires block size 128.
+#
+# AITER page-16 sparse paged-attention fast path (vllm-project/vllm#47287,
+# merged into the pinned nightly): maps MiniMax-M3's top-k 128-token sparse
+# blocks onto AITER page-16 block tables and runs AITER Gluon paged attention
+# over only the selected KV pages. This is a kernel-level speedup of the same
+# sparse-attention computation (no FLOP reduction), enabled via
+# VLLM_ROCM_USE_AITER=1 + VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 with fp8 KV cache
+# on a TP where each rank has num_kv_heads == 1 (TP4). We deliberately do NOT
+# pass the #47269 --hf-overrides use_index_cache/index_topk_freq cross-layer
+# indexer-skip override: it reduces model-architecture FLOPs, which is
+# disallowed by docs/PR_REVIEW_CHECKLIST.md.
 
 source "$(dirname "$0")/../../benchmark_lib.sh"
 
@@ -31,10 +42,39 @@ fi
 SERVER_LOG=/workspace/server.log
 export VLLM_ENGINE_READY_TIMEOUT_S=3600
 export VLLM_USE_BREAKABLE_CUDAGRAPH=0
-# MI355X mxfp8 recipe (vllm-project/recipes#581): INT6 quick all-reduce plus
-# the router-append shared-experts MoE fusion (vllm-project/vllm#46545). 
+# MI355X mxfp8 recipe (vllm-project/recipes#581): INT4 quick all-reduce plus
+# the router-append shared-experts MoE fusion (vllm-project/vllm#46545). INT4
+# quick all-reduce is applied at all concurrencies (accuracy is guarded by the
+# 8k1k evals); #2003 used INT6.
 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
-export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6
+export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
+
+# AITER page-16 sparse PA (vllm-project/vllm#47287) is a long-context,
+# high-concurrency optimization: it maps MiniMax-M3's top-k 128-token sparse
+# blocks onto AITER page-16 block tables. Measured on gfx950 MXFP8, it only wins
+# in the 8k1k high-concurrency tail and adds overhead at short context (1k1k) or
+# low batch. So enable the "high-conc fast path" (shuffled KV-cache layout for
+# sparse PA + the emulation dense-linear backend, see below) only for
+# isl>=8192 && conc>=64; everywhere else fall back to the #2003 path
+# (non-shuffled Triton attention + native linear). Overridable via
+# MM3_HIGH_CONC_FASTPATH=0/1.
+if [ -z "${MM3_HIGH_CONC_FASTPATH:-}" ]; then
+    if [ "$ISL" -ge 8192 ] && [ "$CONC" -ge 64 ]; then
+        MM3_HIGH_CONC_FASTPATH=1
+    else
+        MM3_HIGH_CONC_FASTPATH=0
+    fi
+fi
+
+if [ "$MM3_HIGH_CONC_FASTPATH" = "1" ]; then
+    export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1
+    # Quick all-reduce tuning from the MiniMax-M3 AITER recipe (vllm-project/vllm#47287):
+    # keep the bf16 accumulation and only quantize all-reduces above 256 KB.
+    export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0
+    export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256
+else
+    export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0
+fi
 
 if [ "${EVAL_ONLY}" = "true" ]; then
     setup_eval_context
@@ -61,6 +101,18 @@ export VLLM_ROCM_USE_AITER=1
 # concurrency. Overridable via env.
 MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-32768}"
 
+# Dense-linear backend, gated on the same high-conc fast path as sparse PA. On
+# this nightly the native Triton MXFP8 linear GEMM wins in the memory-bound
+# low-concurrency regime, while --linear-backend emulation (bf16 hipBLASLT) wins
+# in the compute-bound high-concurrency regime (~+3-5% at 8k1k conc>=64).
+# LINEAR_BACKEND overrides (a backend name to force it, or "native" to disable).
+LINEAR_ARGS=()
+if [ -n "${LINEAR_BACKEND:-}" ]; then
+    [ "$LINEAR_BACKEND" != "native" ] && LINEAR_ARGS=(--linear-backend "$LINEAR_BACKEND")
+elif [ "$MM3_HIGH_CONC_FASTPATH" = "1" ]; then
+    LINEAR_ARGS=(--linear-backend emulation)
+fi
+
 start_gpu_monitor
 
 set -x
@@ -74,7 +126,7 @@ vllm serve "$MODEL" --port "$PORT" \
     --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \
     --kv-cache-dtype fp8 \
     --attention-backend TRITON_ATTN \
-    --linear-backend emulation \
+    "${LINEAR_ARGS[@]}" \
     --tool-call-parser minimax_m3 \
     --reasoning-parser minimax_m3 \
     --enable-auto-tool-choice > "$SERVER_LOG" 2>&1 &
diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml
@@ -1621,7 +1621,7 @@ dsv4-fp4-mi355x-atom-disagg:
           - "DECODE_NODES=1"
       # 1P1D TP8
 minimaxm3-fp8-mi355x-vllm:
-  image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa
+  image: vllm/vllm-openai-rocm:nightly-9e57de7197f234f9d9187715d96e07e007048c0f
   model: MiniMaxAI/MiniMax-M3-MXFP8
   model-prefix: minimaxm3
   runner: mi355x
diff --git a/perf-changelog.yaml b/perf-changelog.yaml
@@ -4751,6 +4751,17 @@
     - "6 topologies across 1k/1k and 8k/1k: 1P1D TP4 STP + wide-EP (DEP4 prefill / DEP16 decode) from 1P1D up to 8P1D, recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp8/"
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2137
 
+- config-keys:
+    - minimaxm3-fp8-mi355x-vllm
+  description:
+    - "Bump the MiniMax-M3 MXFP8 MI355X vLLM image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287)"
+    - "Enable AITER page-16 sparse PA (VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1, with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4 where num_kv_heads == 1 per rank): AITER derives page-16 K/V views from the page-128 KV cache and routes decode/prefill through AITER Gluon paged attention over MiniMax-M3's top-k 128-token sparse blocks. Kernel-level speedup of the same computation, no FLOP reduction"
+    - "Concurrency-gate the fast path to the long-context high-concurrency tail (isl>=8192 && conc>=64). Sparse PA is a long-context/high-batch optimization and adds overhead at short context (1k1k) or low concurrency, so both VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT and --linear-backend emulation are enabled only in that regime; everything else (all 1k1k, and 8k1k conc<64) falls back to the prior #2003 path (non-shuffled Triton attention + native linear). Net effect vs #2003: 8k1k tput/gpu conc128 +3.3%, conc256 +2.5%, conc512 +4.7% (peak), neutral elsewhere"
+    - "Switch quick all-reduce quantization to INT4 for all concurrencies (#2003 used INT6); accuracy is guarded by the 8k1k evals. In the high-conc fast path also apply the AITER tuning knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256"
+    - "Deliberately do NOT pass the #47269 --hf-overrides use_index_cache/index_topk_freq cross-layer indexer-skip override: it reduces model-architecture FLOPs, which is disallowed by docs/PR_REVIEW_CHECKLIST.md"
+    - "Serving flags are otherwise unchanged (--block-size 128, --language-model-only, --moe-backend aiter, --kv-cache-dtype fp8, --attention-backend TRITON_ATTN, minimax_m3 parsers); TP4 conc 1-512 sweep at 1k1k and 8k1k unchanged"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2187
+
 - config-keys:
     - kimik2.5-fp4-mi355x-atom-disagg
   description:
