Skip to content

Commit 1376407

Browse files
seungrokjclaudechunfangamd
authored
[AMD] MiniMax-M3 FP4/FP8 MI355X ATOMESH (disagg): (0701) refactor config (#2000)
* [AMD] refactor server_atom.sh and models_atom.yaml for model-specific ATOM config; add minimaxm3-fp4-mi355x-atom-disagg Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] add perf-changelog entry for minimaxm3-fp4-mi355x-atom-disagg and server_atom.sh refactor (PR #1940) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] add env dump in server_atom.sh and minimaxm3-fp4-mi355x-atom-disagg launch script Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] fix server_atom.sh YAML loading: safe source, EP string compare, SPEC_DECODING guard - Replace fragile eval "$(python3 -c "...")" with heredoc + source tempfile to avoid nested quote escaping issues that caused MODEL_ENVS to be empty at runtime - Fix PREFILL/DECODE_ENABLE_EP comparison from numeric -gt 1 to string = "true" to match the "true"/"false" values set by launch scripts - Fix SPEC_DECODING guard from hardcoded "mtp" to any non-none/non-empty value so EAGLE3 and future methods also activate SPEC_ARGS from models_atom.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] cap minimaxm3-fp8-mi355x-atom-disagg conc to 256; fix missing newline in models_atom.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] update amd-master.yaml: image bumps, search space tweaks for MiniMax-M3 ATOM recipes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] restore minimaxm3-fp4/fp8-mi355x-atom recipes; bump all ATOM images to 20260623 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] clean up minimaxm3-fp4-mi355x-atom search space; revert fp8-disagg image to 20260622 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [AMD] add amd-master.yaml config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] remove amd-master.yaml config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] refactor ATOM disagg config: split per-role flags, move model defaults to YAML - Split MODEL_TP_DP_FLAGS and MODEL_EP_DP_FLAGS into prefill/decode variants - Move BLOCK_SIZE, MEM_FRAC_STATIC, MAX_MODEL_LEN, MAX_NUM_SEQS, MAX_NUM_BATCHED_TOKENS from launch scripts into models_atom.yaml - Add hf_overrides and online_quant_config (with DPA variant) to YAML - Remove SPEC_DECODING gate; use MODEL_MTP_FLAGS + DECODE_MTP_SIZE > 0 - Add minimaxm3-fp4/fp8-mi355x-atom-disagg-mtp recipes to amd-master.yaml Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] fix YAML server-tuning defaults never taking effect Shell defaults (BLOCK_SIZE=16, MEM_FRAC_STATIC=0.85) were set before YAML loading, so the YAML values (128, 0.8) were never substituted. Use three-tier fallback: env var > YAML > shell default. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] add perf-changelog entry for MiniMax-M3 ATOM disagg refactor (PR #2000) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] remove hf_overrides from models_atom.yaml and server_atom.sh Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] fix server-tuning: YAML values must override job.slurm Docker defaults job.slurm injects BLOCK_SIZE=16, MEM_FRAC_STATIC=0.85, MAX_NUM_SEQS=256 as Docker env vars with hardcoded defaults. The previous env-first fallback (env > YAML > default) meant YAML values were always shadowed. Flip all five server-tuning vars to YAML > env > default so models_atom.yaml entries (e.g. block_size=128 for MiniMax-M3-MXFP4) actually take effect. Also add set -x before YAML parsing for CI debuggability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] fix MiniMax-M3 ATOM disagg: TBO prefill flag, cudagraph step-8, add spec-decoding keys - models_atom.yaml: change MiniMax-M3-MXFP4 prefill_tp_dp_flags from "--enable-tbo" to "--enable-tbo prefill" to match Slurm reference - server_atom.sh: switch decode cudagraph-capture-sizes from step-4 (65 entries) to step-8 (33 entries) to match Slurm reference - amd-master.yaml: add missing spec-decoding: "mtp" to all MiniMax M3 fp8/fp4 atom-disagg-mtp search-space entries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD] raise ulimit, drop MTP recipes, fix changelog for MiniMax-M3 ATOM disagg - Raise file descriptor limit (ulimit -n 65536) in server_atom.sh to prevent lm-eval "Too many open files" crashes with high num_concurrent - Remove minimaxm3-fp4/fp8-mi355x-atom-disagg-mtp recipes from master config - Update perf-changelog to reflect removed MTP entries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Chun Fang <chun.fang@amd.com>
1 parent 01af32e commit 1376407

7 files changed

Lines changed: 225 additions & 172 deletions

File tree

‎benchmarks/multi_node/amd_utils/env_atom.sh‎

Lines changed: 8 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -32,36 +32,24 @@ else
3232
fi
3333
export IBDEVICES
3434

35-
export SAFETENSORS_FAST_GPU=1
36-
export VLLM_LOG_LEVEL=WARNING
37-
export ATOM_LOG_LEVEL=WARNING
38-
export AITER_LOG_LEVEL=WARNING
39-
export LOG_LEVEL=WARNING
40-
export LOGLEVEL=WARNING
41-
4235
# =============================================================================
4336
# ATOM/mooncake-specific environment
4437
# =============================================================================
4538

4639
# mooncake RDMA KV transfer library path
4740
export LD_LIBRARY_PATH=/opt/venv/lib/python3.10/site-packages/mooncake:/opt/rocm/lib:${LD_LIBRARY_PATH:-}
4841

49-
50-
# ATOM_HOST_IP is set per-node in server_atom.sh (= host_ip, used as handshake IP)
42+
# faster model loading (safetensors only)
43+
export SAFETENSORS_FAST_GPU=1
5144

5245
# aiter logging (WARNING to reduce noise; use DEBUG for troubleshooting)
46+
export VLLM_LOG_LEVEL=WARNING
47+
export ATOM_LOG_LEVEL=WARNING
5348
export AITER_LOG_LEVEL=WARNING
54-
55-
if [[ "$MODEL_NAME" == "DeepSeek-V4-Pro" ]]; then
56-
# ATOM MoE gather/scatter interleave optimization
57-
export ATOM_MOE_GU_ITLV=1
58-
# Disable bf16->fp8 MoE bound (only for DeepSeek-V4-Pro)
59-
export AITER_BF16_FP8_MOE_BOUND=0
60-
fi
61-
62-
# Clear stale ATOM cache on startup (server_atom.sh handles this via rm -rf)
63-
# No env var needed; documented here for reference.
49+
export LOG_LEVEL=WARNING
50+
export LOGLEVEL=WARNING
6451

6552
set +x
6653

67-
echo "[INFO] ATOM env: IBDEVICES=$IBDEVICES LD_LIBRARY_PATH includes mooncake"
54+
# ATOM_HOST_IP is set per-node in server_atom.sh (= host_ip, used as handshake IP)
55+
echo "[INFO] ATOM env: IBDEVICES=$IBDEVICES LD_LIBRARY_PATH includes mooncake"

‎benchmarks/multi_node/amd_utils/models_atom.yaml‎

Lines changed: 61 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Model-specific SGLang server configurations for disaggregated inference.
1+
# Model-specific ATOM server configurations for disaggregated inference.
22
#
33
# Each top-level key is a MODEL_NAME value (must match the directory name under MODEL_DIR).
44
#
@@ -7,50 +7,69 @@
77
#
88
# Schema:
99
# <model-name>:
10-
# base_flags: str # Common flags for both prefill and decode
11-
# mtp_flags: str # Appended to decode when DECODE_MTP_SIZE > 0
12-
# dp_flags: str # Appended when DP is enabled (prefill or decode)
13-
# prefill:
14-
# mem_fraction_static: float
15-
# disable_radix_cache: bool
16-
# dp: # Config when data-parallel attention is enabled
17-
# max_running_requests: int
18-
# chunked_prefill_size: str # Can be integer or bash arithmetic expression
19-
# cuda_graph_bs: str # Space-separated values
20-
# no_dp: # Config when data-parallel attention is disabled
21-
# max_running_requests: int
22-
# chunked_prefill_size: int
23-
# cuda_graph_bs_range: str # "start-end" expanded via seq
24-
# decode:
25-
# mem_fraction_static: float
26-
# prefill_round_robin_balance: bool
27-
# dp:
28-
# max_running_requests: int
29-
# chunked_prefill_size: str
30-
# cuda_graph_bs_range: str
31-
# ep_only: # Config when EP is enabled but DP is disabled
32-
# max_running_requests: int
33-
# chunked_prefill_size: int
34-
# cuda_graph_bs_range: str
35-
# no_dp:
36-
# max_running_requests: int
37-
# chunked_prefill_size: int
38-
# cuda_graph_bs_range: str
10+
# env: str # Space-separated KEY=VALUE pairs exported unconditionally
11+
# tp_dp_flags: str # Shared TP+DPA flags (fallback when prefill/decode-specific keys are absent)
12+
# prefill_tp_dp_flags: str # TP+DPA flags for prefill only (overrides tp_dp_flags)
13+
# decode_tp_dp_flags: str # TP+DPA flags for decode only (overrides tp_dp_flags)
14+
# tp_dp_env: str # Space-separated KEY=VALUE pairs exported only in TP+DPA mode
15+
# ep_dp_flags: str # Shared EP+DPA flags (fallback when prefill/decode-specific keys are absent)
16+
# prefill_ep_dp_flags: str # EP+DPA flags for prefill only (overrides ep_dp_flags)
17+
# decode_ep_dp_flags: str # EP+DPA flags for decode only (overrides ep_dp_flags)
18+
# ep_dp_env: str # Space-separated KEY=VALUE pairs exported only in EP+DPA mode
19+
# mtp_flags: str # Flags passed to SPEC_ARGS before $DECODE_MTP_SIZE (e.g. "--method mtp --num-speculative-tokens")
20+
# kv_cache_flags: str # Full --kv_cache_dtype flag string (e.g. "--kv_cache_dtype fp8", or "" for none)
21+
# online_quant_config: str # JSON string passed to --online_quant_config (used when DPA is disabled)
22+
# online_quant_dpa_config: str # JSON string passed to --online_quant_config when DPA is enabled (falls back to online_quant_config)
23+
# block_size: str # --block-size value (overrides server_atom.sh default of 16)
24+
# mem_frac_static: str # --gpu-memory-utilization value (overrides default of 0.85)
25+
# max_model_len: str # --max-model-len value (overrides default of unset)
26+
# max_num_seqs: str # --max-num-seqs value (overrides default of 256)
27+
# max_num_batched_tokens: str # --max-num-batched-tokens value (overrides default of unset)
3928

4029
DeepSeek-V4-Pro:
41-
# ATOM engine (atom-disagg): server_atom.sh uses MEM_FRACTION/KV_CACHE_DTYPE/BLOCK_SIZE/MAX_NUM_SEQS
42-
# directly from env vars (defaulting to 0.85/fp8/16/256). base_flags/dp_flags are not used by
43-
# server_atom.sh; they are kept here for documentation and potential future use.
44-
base_flags: ""
45-
mtp_flags: ""
46-
dp_flags: ""
30+
env: "ATOM_MOE_GU_ITLV=1 AITER_BF16_FP8_MOE_BOUND=0"
31+
kv_cache_flags: "--kv_cache_dtype fp8"
32+
tp_dp_flags: "--enable-dp-attention --enable-tbo"
33+
prefill_tp_dp_flags: "--enable-dp-attention --enable-tbo"
34+
decode_tp_dp_flags: "--enable-dp-attention --enable-tbo"
35+
tp_dp_env: "GPU_MAX_HW_QUEUES=5 ATOM_CPU_AFFINITY=1"
36+
ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
37+
prefill_ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
38+
decode_ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
39+
mtp_flags: "--method mtp --num-speculative-tokens"
4740

4841
MiniMax-M3-MXFP4:
49-
base_flags: ""
50-
mtp_flags: ""
51-
dp_flags: ""
42+
env: "AITER_QUICK_REDUCE_QUANTIZATION=INT4 ATOM_FORCE_ATTN_TRITON=1"
43+
kv_cache_flags: "--kv_cache_dtype fp8"
44+
tp_dp_flags: "--enable-dp-attention"
45+
prefill_tp_dp_flags: "--enable-dp-attention --enable-tbo prefill"
46+
decode_tp_dp_flags: "--enable-dp-attention"
47+
ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
48+
prefill_ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
49+
decode_ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
50+
mtp_flags: "--method eagle3 --draft-model Inferact/MiniMax-M3-EAGLE3 --num-speculative-tokens"
51+
online_quant_config: '{"global_quant_config":"ptpc_fp8","exclude_layer":["lm_head","model.embed_tokens","vision_tower","multi_modal_projector","patch_merge_mlp","*block_sparse_moe"]}'
52+
online_quant_dpa_config: '{"global_quant_config":"ptpc_fp8","exclude_layer":["lm_head","model.embed_tokens","vision_tower","multi_modal_projector","patch_merge_mlp","*block_sparse_moe"]}'
53+
block_size: "128"
54+
mem_frac_static: "0.8"
55+
max_model_len: "32768"
56+
max_num_seqs: "256"
57+
max_num_batched_tokens: "32768"
5258

5359
MiniMax-M3-MXFP8:
54-
base_flags: ""
55-
mtp_flags: ""
56-
dp_flags: ""
60+
env: "AITER_QUICK_REDUCE_QUANTIZATION=INT4 ATOM_FORCE_ATTN_TRITON=1"
61+
kv_cache_flags: "--kv_cache_dtype fp8"
62+
tp_dp_flags: "--enable-dp-attention"
63+
prefill_tp_dp_flags: "--enable-dp-attention --enable-tbo prefill"
64+
decode_tp_dp_flags: "--enable-dp-attention"
65+
ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
66+
prefill_ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
67+
decode_ep_dp_flags: "--enable-expert-parallel --enable-dp-attention"
68+
mtp_flags: "--method eagle3 --draft-model Inferact/MiniMax-M3-EAGLE3 --num-speculative-tokens"
69+
online_quant_config: '{"global_quant_config":"ptpc_fp8","exclude_layer":["lm_head","model.embed_tokens","vision_tower","multi_modal_projector","patch_merge_mlp","*block_sparse_moe"]}'
70+
online_quant_dpa_config: '{"global_quant_config":"ptpc_fp8","exclude_layer":["lm_head","model.embed_tokens","vision_tower","multi_modal_projector","patch_merge_mlp","*.gate.*","*.block_sparse_moe.experts*"]}'
71+
block_size: "128"
72+
mem_frac_static: "0.8"
73+
max_model_len: "32768"
74+
max_num_seqs: "256"
75+
max_num_batched_tokens: "32768"

0 commit comments

Comments
 (0)