Skip to content

Commit fbade41

Browse files
ajith-sirra-amdseungrokjclaudecquil11github-actions[bot]
authored
[AMD][AgentX] MINIMAX-M3 FP4 MI355X agentX vLLM (#2118)
* [AMD] MINIMAX-M3 FP4 vLLM Agentic Support Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Added PR Link Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Search Space Modification. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Search Space Modification. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Search Space Modification. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Search Space Modification & Fixing MODEL_PATH Arg in Serve Command. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Search Space Modification & Fixing MODEL_PATH Arg in Serve Command. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Search Space Modification. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * chore: remove LMCache backend from MiniMax-M3 FP4 MI355X agentic benchmark Drop lmcache case branch, helper functions, and config entry; keep native-only offload path. Add OFFLOAD_ARGS fallback, enable prefix caching and hybrid KV cache manager, bump max-num-seqs to 2x CONC. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: revert max-num-seqs to CONC and drop hybrid KV cache flag Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Search Space Modification. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Removing redundant prefix cache flag. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * [AMD] MINIMAX-M3 FP4 vLLM Agentic Support. Modifying Search Space. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> * fix(agentic): preserve matrix CPU DRAM value * fix(agentic): use explicit MI355X server port * fix(agentic): remove recipe env fallbacks * fix(configs): restore indentation of dsv4 decode additional-settings The extra leading space broke YAML parsing of amd-master.yaml. Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(changelog): omit null optional metadata * fix: support EVAL_ONLY mode in minimaxm3_fp4_mi355x.sh Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Updating Docker Image Signed-off-by: Sirra <asirra@amd.com> * fix: restore origin/main's dsv4-fp4-mi355x-sglang-disagg-agentic-hicache config Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Updating Docker Image Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Updating Script to remove undefined MODEL_PATH Flags. Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Updating Script to remove undefined MODEL_PATH Flags. Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Updating Script to remove undefined MODEL_PATH Flags. Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Updating Script. Adding Native vLLM Offload only. Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Correcting Offload Backend name in the benchmark script. Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Resolving conflict in Perf change log. Signed-off-by: Sirra <asirra@amd.com> * [AgentX] MiniMax-M3 FP4 MI355X vLLM agentic: increase eval step limit and timeout Raise SWEBENCH_AGENT_STEP_LIMIT from 75 to 100 and SWEBENCH_AGENT_TIMEOUT from 4h to 6h. The prior run scored 36.8% (below the 50% threshold) because 71 instances exhausted their 75-turn budget without submitting a patch and 88 instances never started before the 4h deadline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix Signed-off-by: seungrokj <seungrok.jung@amd.com> * swebench: expose SWEBENCH_AGENT_DEPLOYMENT_TIMEOUT for Modal sandbox lifetime The Modal sandbox deployment_timeout was left at the swerex default of 3600s while only runtime_timeout was configurable, so long agent runs outlived the 1h sandbox and every later command failed with "Cannot connect to host ...modal.host", yielding empty patches even when the fix was correct. Make the sandbox hard lifetime tunable (default 3600). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Removing unwanted CONC from list. Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] [WIP] Filtering CONC Changes Signed-off-by: Sirra <asirra@amd.com> * [AMD] [AGENTX] [MINIMAX-M3] Fix tool-calling loops: keep <mm:think> across turns Drop --reasoning-parser minimax_m3 so the interleaved <mm:think> block stays inline in message.content and is round-tripped by the mini-swe-agent client; stripping it caused the model to lose its plan and degenerate into command loops until the step limit (empty patches). Add SWEBENCH_AGENT_FILTER plumbing to run an instance-ID subset, and restrict the recipe to two previously-looping instances for validation. Restrict agentic conc-list to [16]; bump aiperf. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] [AGENTX] [MINIMAX-M3] Revert debug SWEBENCH_AGENT_FILTER plumbing Remove the temporary instance-ID filter used to validate the <mm:think> round-trip fix on two instances: drop SWEBENCH_AGENT_FILTER/ SWEBENCH_EXPECTED_INSTANCES exports from the recipe and the --filter plumbing from benchmark_lib.sh, restoring the full sweep. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] [AGENTX] [MINIMAX-M3] Force lm-eval framework for minimaxm3 agentic recipe - minimaxm3_fp4_mi355x.sh: export EVAL_FRAMEWORK=lm-eval so run_eval never falls back to the agentic swebench default. - benchmark_lib.sh: drop the Modal deployment_timeout override. - amd-master.yaml: restore agentic conc-list to [1, 4, 8, 16]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com> Signed-off-by: Sirra <asirra@amd.com> Signed-off-by: seungrokj <seungrok.jung@amd.com> Co-authored-by: seungrokj <seungrok.jung@amd.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com> Co-authored-by: Sirra <asirra@amd.com>
1 parent d121d11 commit fbade41

3 files changed

Lines changed: 169 additions & 0 deletions

File tree

‎benchmarks/single_node/agentic/minimaxm3_fp4_mi355x.sh‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
set -x
4+
5+
# Agentic trace replay benchmark for Minimax-M3 FP4 on MI355X using vLLM.
6+
#
7+
# Required env vars:
8+
# MODEL, MODEL_PATH, TP, CONC, KV_OFFLOADING, KV_OFFLOAD_BACKEND,
9+
# TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, EP_SIZE, DP_ATTENTION
10+
11+
source "$(dirname "$0")/../../benchmark_lib.sh"
12+
13+
# Force the eval framework to lm-eval for this recipe. run_eval derives its
14+
# default as swebench for agentic scenarios (scenario_default=swebench when
15+
# IS_AGENTIC/SCENARIO_TYPE=agentic-coding), but EVAL_FRAMEWORK takes precedence
16+
# over that default (benchmark_lib.sh: framework=${EVAL_FRAMEWORK:-...}), so
17+
# setting it here makes the effective framework always lm-eval, never swebench.
18+
export EVAL_FRAMEWORK="lm-eval"
19+
20+
check_env_vars MODEL TP CONC KV_OFFLOADING KV_OFFLOAD_BACKEND TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION
21+
22+
echo "MODEL=$MODEL TP=$TP CONC=$CONC KV_OFFLOADING=$KV_OFFLOADING TOTAL_CPU_DRAM_GB=$TOTAL_CPU_DRAM_GB RESULT_DIR=$RESULT_DIR DURATION=$DURATION EP_SIZE=$EP_SIZE DP_ATTENTION=$DP_ATTENTION"
23+
24+
PORT=8888
25+
26+
if [[ -n "${SLURM_JOB_ID+x}" ]]; then
27+
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
28+
fi
29+
30+
# ROCR/HIP visibility for vLLM 0.14+
31+
if [[ -n "${ROCR_VISIBLE_DEVICES+x}" ]]; then
32+
export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES"
33+
fi
34+
35+
if [[ -n "${MODEL_PATH:-}" ]]; then
36+
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
37+
hf download "$MODEL" --local-dir "$MODEL_PATH"
38+
fi
39+
else
40+
hf download "$MODEL"
41+
export MODEL_PATH="$MODEL"
42+
fi
43+
44+
rocm-smi || true
45+
amd-smi || true
46+
47+
resolve_trace_source
48+
install_agentic_deps
49+
50+
# ---- Server config ----------------------------------------------------------
51+
SERVER_LOG="$RESULT_DIR/server.log"
52+
LMCACHE_LOG="$RESULT_DIR/lmcache_server.log"
53+
mkdir -p "$RESULT_DIR"
54+
55+
OFFLOAD_ARGS=(--no-enable-prefix-caching)
56+
57+
case "$KV_OFFLOAD_BACKEND" in
58+
vllm-simple)
59+
unset VLLM_USE_SIMPLE_KV_OFFLOAD
60+
# Use vLLM's regular native KV-offload path (OffloadingConnector),
61+
# NOT the SimpleCPUOffloadConnector. The "native" backend resolves to
62+
# OffloadingConnector by default; setting VLLM_USE_SIMPLE_KV_OFFLOAD=1
63+
# would switch it to SimpleCPUOffloadConnector. We intentionally leave
64+
# that env var UNSET here so the regular OffloadingConnector path is
65+
# used. The shortcut --kv_offloading_backend native + --kv_offloading_size
66+
# form constructs the KVTransferConfig at engine startup
67+
# (vllm/config/vllm.py:662).
68+
69+
# Remove --disable-hybrid-kv-cache-manager and enable hybrid kv cache manager (default)
70+
# This gives extra cache hit than disabling hybrid kv cache manager
71+
OFFLOAD_ARGS=(
72+
--kv_offloading_backend native
73+
--kv_offloading_size "$TOTAL_CPU_DRAM_GB"
74+
)
75+
;;
76+
esac
77+
78+
# ---- LLM server config ----------------------------------------------------------
79+
PARALLEL_ARGS=(--tensor-parallel-size "$TP")
80+
if [ "${DP_ATTENTION}" = "true" ]; then
81+
PARALLEL_ARGS=(
82+
--tensor-parallel-size 1
83+
--data-parallel-size "$TP"
84+
--enable-expert-parallel
85+
)
86+
elif [ "$EP_SIZE" -gt 1 ]; then
87+
PARALLEL_ARGS+=(--enable-expert-parallel)
88+
fi
89+
90+
echo "Starting vllm server..."
91+
export PYTHONNOUSERSITE=1
92+
93+
export VLLM_ENGINE_READY_TIMEOUT_S=3600
94+
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
95+
export VLLM_ROCM_USE_AITER=1
96+
export VLLM_ROCM_USE_AITER_MOE=1
97+
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
98+
# INT4 quantized all-reduce for the (~1.5 MB) decode all-reduces, which are the
99+
# single biggest decode kernel at high concurrency. The MIN_SIZE_KB override is
100+
# required: vLLM's default INT4 quick-reduce size gate for (bf16, TP4) is 16 MB,
101+
# so it never fires for decode-sized tensors without it.
102+
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
103+
export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0
104+
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256
105+
106+
VLLM_CMD=(
107+
vllm serve "$MODEL_PATH"
108+
--served-model-name "$MODEL"
109+
--host 0.0.0.0
110+
--port "$PORT"
111+
"${PARALLEL_ARGS[@]}"
112+
--trust-remote-code
113+
--block-size 128
114+
--gpu-memory-utilization 0.85
115+
--language-model-only
116+
--attention-backend TRITON_ATTN
117+
--moe-backend aiter
118+
--kv-cache-dtype fp8
119+
--tool-call-parser minimax_m3
120+
--enable-auto-tool-choice
121+
# NOTE: --reasoning-parser minimax_m3 is intentionally OMITTED.
122+
# MiniMax-M3 is an interleaved-thinking model: its <mm:think>...</mm:think>
123+
# block MUST be round-tripped back into the conversation history every turn
124+
# or multi-turn quality collapses (the model loses its plan and degenerates
125+
# into repeating the same command until the step limit -> empty patch).
126+
# The reasoning parser moves <mm:think> out of message.content into the
127+
# response-only reasoning_content field, which the mini-swe-agent/litellm
128+
# OpenAI client does NOT resend. Leaving the parser off keeps the think block
129+
# inline in message.content, so the client preserves it across turns. The
130+
# tool-call parser above still extracts tool calls from the full output.
131+
--max-num-seqs "$CONC"
132+
"${OFFLOAD_ARGS[@]}"
133+
)
134+
printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt"
135+
printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt"
136+
"${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 &
137+
SERVER_PID=$!
138+
echo "Server PID: $SERVER_PID"
139+
140+
wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"
141+
142+
# ---- Run benchmark ----------------------------------------------------------
143+
if [ "${EVAL_ONLY}" = "true" ]; then
144+
run_eval --port "$PORT"
145+
else
146+
build_replay_cmd "$RESULT_DIR"
147+
run_agentic_replay_and_write_outputs "$RESULT_DIR"
148+
fi

‎configs/amd-master.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2233,6 +2233,20 @@ minimaxm3-fp8-mi325x-vllm-agentic:
22332233
- { tp: 8, ep: 8, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 32] }
22342234
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [24, 32, 36, 40, 44, 48, 52, 56, 60, 64, 72, 80, 96], router: { name: vllm-router, version: "0.1.14" } }
22352235

2236+
minimaxm3-fp4-mi355x-vllm-agentic:
2237+
image: vllm/vllm-openai-rocm:nightly-dcfebf93f4eccf30f71872283331eee757915daf
2238+
model: amd/MiniMax-M3-MXFP4
2239+
model-prefix: minimaxm3
2240+
runner: cluster:mi355x-amds
2241+
precision: fp4
2242+
framework: vllm
2243+
multinode: false
2244+
scenarios:
2245+
agentic-coding:
2246+
- dram-utilization: 0.80
2247+
search-space:
2248+
- { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [1, 4, 8, 16] }
2249+
22362250
dsv4-fp4-mi355x-sglang-disagg-agentic-hicache:
22372251
image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
22382252
model: deepseek-ai/DeepSeek-V4-Pro

‎perf-changelog.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5061,6 +5061,13 @@
50615061
- "Exclude known-bad nodes mia1-p01-g09,g14 from the disagg node pool"
50625062
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2301
50635063

5064+
- config-keys:
5065+
- minimaxm3-fp4-mi355x-vllm-agentic
5066+
description:
5067+
- "Add Minimax-M3 FP4 vLLM Single Node Agentic Support"
5068+
- "Image: vllm/vllm-openai-rocm:nightly-dcfebf93f4eccf30f71872283331eee757915daf"
5069+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2118
5070+
50645071
- config-keys:
50655072
- minimaxm3-fp4-b300-dynamo-vllm-mtp
50665073
- minimaxm3-fp4-b300-dynamo-vllm-mtp-legacy-dep4

0 commit comments

Comments
 (0)