Skip to content

Commit af285ca

Browse files
xinli-swAnkur-singh
xinli-sw
and
Ankur-singh
authored
[NV] minimax-m3-b200/b300-mtp: update fp8 MTP to vLLM nightly / [NV] minimax-m3-b200/b300-mtp：将 FP8 MTP 更新至 vLLM nightly (#2337)
* [NV] minimax-m3-b200/b300-mtp: vLLM nightly image, GQA draft head, FLASH_ATTN, stream-interval 32 Single-node MTP (minimaxm3-fp8-b200-vllm-mtp, minimaxm3-fp8-b300-vllm-mtp): - image -> vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - EAGLE3 draft head -> Inferact/MiniMax-M3-EAGLE3-GQA - speculative attention_backend TRITON_ATTN -> FLASH_ATTN; num_speculative_tokens kept at 3 - consolidate --attention-config with indexer_kv_dtype fp8; --stream-interval 32 - drop the sparse_attention_msa contiguity patch (shipped in the nightly image) * [NV] minimax-m3-b200/b300-mtp: add perf-changelog pr-link * [NV] minimax-m3-b200/b300-mtp: drop tp4-ep4-DEP (dp-attn) arm from the sweep --------- Co-authored-by: Ankur-singh <ankusingh@nvidia.com>
1 parent 85606a7 commit af285ca

4 files changed

Lines changed: 24 additions & 86 deletions

File tree

‎benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b200_mtp.sh‎

Lines changed: 7 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,15 @@
33
# MiniMax-M3 MXFP8 B200 single-node vLLM recipe with EAGLE3 speculative
44
# decoding — the repo's spec-decoding=mtp variant of minimaxm3_fp8_b200.sh
55
# (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3). Adds the
6-
# Inferact/MiniMax-M3-EAGLE3 draft head via --speculative-config with 3
6+
# Inferact/MiniMax-M3-EAGLE3-GQA draft head via --speculative-config with 3
77
# speculative tokens. Everything else keeps the non-MTP serve shape:
88
# --block-size 128 is mandatory (MSA sparse_block_size is 128; the default 16
99
# misaligns sparse indexing), and --language-model-only skips the vision
1010
# encoder for the text-only benchmark. dp-attn=true maps to DP×EP (DEP);
1111
# ep>1 maps to TP+EP (TEP).
1212
#
13-
# The target uses the FlashInfer TRT-LLM attention path. The EAGLE3 drafter is
14-
# pinned separately to TRITON_ATTN.
13+
# The target uses the FlashInfer TRT-LLM attention path. The EAGLE3-GQA drafter
14+
# is pinned separately to FLASH_ATTN.
1515

1616
source "$(dirname "$0")/../../benchmark_lib.sh"
1717

@@ -27,40 +27,7 @@ check_env_vars \
2727
RANDOM_RANGE_RATIO \
2828
RESULT_FILENAME
2929

30-
# The 0618 image keeps MiniMax M3 top-k indices in a persistent
31-
# [head_kv, max_tokens, topK] buffer for CUDA graphs. Slicing that buffer to
32-
# the actual prefill length is non-contiguous when TP leaves multiple local KV
33-
# heads, and the MSA CSR builder rejects it. Materialize the slice until the
34-
# image includes this fix.
35-
python3 - <<'PYEOF' || { echo "MiniMax M3 MSA contiguity patch failed" >&2; exit 1; }
36-
import importlib.util
37-
import pathlib
38-
39-
spec = importlib.util.find_spec("vllm")
40-
if spec is None or not spec.submodule_search_locations:
41-
raise RuntimeError("Could not locate the installed vllm package")
42-
43-
target = (
44-
pathlib.Path(next(iter(spec.submodule_search_locations)))
45-
/ "models"
46-
/ "minimax_m3"
47-
/ "nvidia"
48-
/ "sparse_attention_msa.py"
49-
)
50-
src = target.read_text()
51-
old = " prefill_topk = topk[:, nd:num_tokens, :]\n"
52-
new = " prefill_topk = topk[:, nd:num_tokens, :].contiguous()\n"
53-
54-
if new in src:
55-
print(f"[minimax-m3-msa-patch] already applied: {target}")
56-
elif src.count(old) == 1:
57-
target.write_text(src.replace(old, new, 1))
58-
print(f"[minimax-m3-msa-patch] patched: {target}")
59-
else:
60-
raise RuntimeError(f"Expected exactly one patch anchor in {target}")
61-
PYEOF
62-
63-
DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3"
30+
DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3-GQA"
6431

6532
if [[ -n "$SLURM_JOB_ID" ]]; then
6633
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
@@ -116,14 +83,13 @@ $PARALLEL_ARGS \
11683
--gpu-memory-utilization 0.90 \
11784
--max-model-len $MAX_MODEL_LEN \
11885
--block-size 128 \
119-
--attention-config '{"backend": "FLASHINFER", "use_trtllm_attention": true}' \
120-
--attention-config.indexer_kv_dtype "fp8" \
86+
--attention-config '{"backend": "FLASHINFER", "use_trtllm_attention": true, "indexer_kv_dtype": "fp8"}' \
12187
--kv-cache-dtype fp8 \
12288
--language-model-only \
12389
--max-cudagraph-capture-size 2048 \
12490
--max-num-batched-tokens "$((ISL * 2 ))" \
125-
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"TRITON_ATTN\"}" \
126-
--stream-interval 20 --no-enable-prefix-caching \
91+
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"FLASH_ATTN\"}" \
92+
--stream-interval 32 --no-enable-prefix-caching \
12793
--trust-remote-code > $SERVER_LOG 2>&1 &
12894

12995
SERVER_PID=$!

‎benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300_mtp.sh‎

Lines changed: 7 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33
# MiniMax-M3 MXFP8 B300 single-node vLLM recipe with EAGLE3 speculative
44
# decoding — the repo's spec-decoding=mtp variant of minimaxm3_fp8_b300.sh
55
# (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3). Adds the
6-
# Inferact/MiniMax-M3-EAGLE3 draft head via --speculative-config with 3
6+
# Inferact/MiniMax-M3-EAGLE3-GQA draft head via --speculative-config with 3
77
# speculative tokens. Everything else keeps the non-MTP serve shape:
88
# --block-size 128 is mandatory (MSA sparse/index cache); the benchmark is
99
# text-only, so --language-model-only frees the vision encoder's VRAM.
1010
#
11-
# The target uses the FlashInfer TRT-LLM attention path. The EAGLE3 drafter is
12-
# pinned separately to TRITON_ATTN.
11+
# The target uses the FlashInfer TRT-LLM attention path. The EAGLE3-GQA drafter
12+
# is pinned separately to FLASH_ATTN.
1313

1414
source "$(dirname "$0")/../../benchmark_lib.sh"
1515

@@ -25,40 +25,7 @@ check_env_vars \
2525
RANDOM_RANGE_RATIO \
2626
RESULT_FILENAME
2727

28-
# The 0618 image keeps MiniMax M3 top-k indices in a persistent
29-
# [head_kv, max_tokens, topK] buffer for CUDA graphs. Slicing that buffer to
30-
# the actual prefill length is non-contiguous when TP leaves multiple local KV
31-
# heads, and the MSA CSR builder rejects it. Materialize the slice until the
32-
# image includes this fix.
33-
python3 - <<'PYEOF' || { echo "MiniMax M3 MSA contiguity patch failed" >&2; exit 1; }
34-
import importlib.util
35-
import pathlib
36-
37-
spec = importlib.util.find_spec("vllm")
38-
if spec is None or not spec.submodule_search_locations:
39-
raise RuntimeError("Could not locate the installed vllm package")
40-
41-
target = (
42-
pathlib.Path(next(iter(spec.submodule_search_locations)))
43-
/ "models"
44-
/ "minimax_m3"
45-
/ "nvidia"
46-
/ "sparse_attention_msa.py"
47-
)
48-
src = target.read_text()
49-
old = " prefill_topk = topk[:, nd:num_tokens, :]\n"
50-
new = " prefill_topk = topk[:, nd:num_tokens, :].contiguous()\n"
51-
52-
if new in src:
53-
print(f"[minimax-m3-msa-patch] already applied: {target}")
54-
elif src.count(old) == 1:
55-
target.write_text(src.replace(old, new, 1))
56-
print(f"[minimax-m3-msa-patch] patched: {target}")
57-
else:
58-
raise RuntimeError(f"Expected exactly one patch anchor in {target}")
59-
PYEOF
60-
61-
DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3"
28+
DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3-GQA"
6229

6330
# `hf download` creates the target dir if missing and is itself idempotent.
6431
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE.
@@ -118,14 +85,13 @@ $PARALLEL_ARGS \
11885
--gpu-memory-utilization 0.90 \
11986
--max-model-len $MAX_MODEL_LEN \
12087
--block-size 128 \
121-
--attention-config '{"backend": "FLASHINFER", "use_trtllm_attention": true}' \
122-
--attention-config.indexer_kv_dtype "fp8" \
88+
--attention-config '{"backend": "FLASHINFER", "use_trtllm_attention": true, "indexer_kv_dtype": "fp8"}' \
12389
--kv-cache-dtype fp8 \
12490
--language-model-only \
12591
--max-cudagraph-capture-size 2048 \
12692
--max-num-batched-tokens "$((ISL * 2 ))" \
127-
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"TRITON_ATTN\"}" \
128-
--stream-interval 20 --no-enable-prefix-caching \
93+
--speculative-config "{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL_PATH\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"FLASH_ATTN\"}" \
94+
--stream-interval 32 --no-enable-prefix-caching \
12995
--trust-remote-code > $SERVER_LOG 2>&1 &
13096

13197
SERVER_PID=$!

‎configs/nvidia-master.yaml‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7729,7 +7729,7 @@ minimaxm3-fp4-b200-vllm:
77297729
# precedent: spec decode pays off at low/mid concurrency while acceptance
77307730
# dilutes in big batches, and the draft weights + draft KV shave headroom.
77317731
minimaxm3-fp8-b200-vllm-mtp:
7732-
image: vllm/vllm-openai:minimax-m3-0618-x86_64-cu130
7732+
image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9
77337733
model: MiniMaxAI/MiniMax-M3-MXFP8
77347734
model-prefix: minimaxm3
77357735
runner: b200-dgxc
@@ -7745,7 +7745,6 @@ minimaxm3-fp8-b200-vllm-mtp:
77457745
- { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp }
77467746
- { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp }
77477747
- { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp }
7748-
- { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128, spec-decoding: mtp }
77497748
- { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp }
77507749

77517750
# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of MiniMax-M3 NVFP4
@@ -7785,7 +7784,7 @@ minimaxm3-fp4-b200-vllm-mtp:
77857784
# big batches, and the draft weights + draft KV shave headroom — tp2-ep2 is
77867785
# dropped entirely since its KV headroom was already thin without a draft.
77877786
minimaxm3-fp8-b300-vllm-mtp:
7788-
image: vllm/vllm-openai:minimax-m3-0618-x86_64-cu130
7787+
image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9
77897788
model: MiniMaxAI/MiniMax-M3-MXFP8
77907789
model-prefix: minimaxm3
77917790
runner: b300
@@ -7801,7 +7800,6 @@ minimaxm3-fp8-b300-vllm-mtp:
78017800
- { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp }
78027801
- { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp }
78037802
- { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp }
7804-
- { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128, spec-decoding: mtp }
78057803
- { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp }
78067804

78077805
# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of

‎perf-changelog.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5230,3 +5230,11 @@
52305230
- "Bump image to vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 and dynamo to 1.3.0.dev20260710; drop the minimax-m3-gb300-vllm-fixes.sh setup patch (shipped in the nightly image)"
52315231
- "Collapse the 2xDEP8 decode into a single DEP16 worker (decode_workers 2->1, gpus_per_decode 8->16, data-parallel-size 8->16); recipe renamed 3p2d-dep2-dep8 -> 3p1d-dep2-dep16 (conc 512). The remaining decodes are 1xDEP8 or TEP and are unchanged."
52325232
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2340
5233+
5234+
- config-keys:
5235+
- minimaxm3-fp8-b200-vllm-mtp
5236+
- minimaxm3-fp8-b300-vllm-mtp
5237+
description:
5238+
- "Bump image to vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9; set --stream-interval 32; consolidate --attention-config with indexer_kv_dtype fp8; drop the sparse_attention_msa contiguity patch (shipped in the nightly image)"
5239+
- "EAGLE3 draft head Inferact/MiniMax-M3-EAGLE3 -> Inferact/MiniMax-M3-EAGLE3-GQA; speculative attention_backend TRITON_ATTN -> FLASH_ATTN; num_speculative_tokens kept at 3"
5240+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2337

0 commit comments

Comments
 (0)