Skip to content

Commit 6b431f9

Browse files
seungrokjclaude
seungrokj
and
claude
authored
[AMD] Kimi-K2.5 MXFP4 ATOMESH disagg config (#2172)
* [AMD][ATOM] Add Kimi-K2.5-MXFP4 disagg config and scheduler-delay-factor support Add Kimi-K2.5-MXFP4 entry to models_atom.yaml with env vars and kv_cache_dtype from the single-node script. Implement scheduler_delay_factor as a new YAML-driven server_atom.sh parameter so models can set --scheduler-delay-factor per-model. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Add Kimi-K2.5 disagg config, scheduler-delay-factor support, and perf-changelog Add Kimi-K2.5-MXFP4 to models_atom.yaml for ATOM PD-disagg with env vars from single-node script. Implement scheduler_delay_factor as a new YAML-driven parameter in server_atom.sh. Update perf-changelog. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Add pr-link to perf-changelog for Kimi-K2.5 disagg entry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Add Kimi-K2.5 FP4 MI355X ATOM disagg benchmark script Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Add atom-disagg engine support to bench.sh Add atom-disagg branch to engine-specific extra flags in bench.sh, setting --trust-remote-code --tokenizer and MTP flags (--dsv4 or --use-chat-template) matching the vllm-disagg and sglang patterns. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Update Kimi-K2.5 disagg scenarios: 2P1D, 1P4D, 1P2D topologies Replace previous 1P2D/2P1D/1P1D scenarios with 2P1D, 1P4D, and 1P2D topologies for 8k/1k ISL/OSL. Trim concurrency lists for faster sweeps. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Update Kimi-K2.5 disagg config per upstream ATOM recipe Align models_atom.yaml with ROCm/ATOM kimi-k2 recipe: - gpu-memory-utilization 0.9, max-model-len 32768, max-num-batched-tokens 32768 - Drop ATOM_DISABLE_MMAP, OMP_NUM_THREADS, scheduler_delay_factor (not in recipe) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Add kv-p2p-transfer: mooncake to Kimi-K2.5 disagg config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Simplify Kimi-K2.5 disagg to 1P1D only, extend conc to 1024 Remove 1P2D topology, keep 1P1D with extended concurrency list up to 1024. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Bump atomesh image and add 1P2D topology for Kimi-K2.5 disagg Update image to atomesh_202607121715. Add 1P2D 8k/1k scenario alongside existing 1P1D with same concurrency sweep up to 1024. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Align Kimi-K2.5 disagg with upstream ATOM 1P1D config Add explicit max_num_seqs=256 to match upstream, remove 1P2D topology, and cap concurrency at 256. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Switch Kimi-K2.5 disagg to 1P(TP4) 2D(TP8) topology Scale decode to 2 workers across 2 nodes (TP8 each, DECODE_NODES=2). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Add 1P1D(TP4) topology alongside 1P2D(TP8) for Kimi-K2.5 Run both topologies: 1P1D at high concurrency [32,64,128,256] and 1P2D at low concurrency [1,2,4,8,16,32]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [AMD][ATOM] Adjust conc splits: 1P1D [16-256], 1P2D [1-16] Move conc=16 from 1P2D to 1P1D; move conc=32 out of 1P2D. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent a174f20 commit 6b431f9

6 files changed

Lines changed: 167 additions & 1 deletion

File tree

‎benchmarks/multi_node/amd_utils/bench.sh‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,21 @@ for max_concurrency in "${chosen_concurrencies[@]}"; do
7676

7777
# Engine-specific extra flags
7878
extra_flags=""
79+
# vllm
7980
if [[ "$ENGINE" == "vllm-disagg" ]]; then
8081
extra_flags="--trust-remote-code --tokenizer $MODEL_PATH"
82+
# atom
83+
elif [[ "$ENGINE" == "atom-disagg" ]]; then
84+
extra_flags="--trust-remote-code --tokenizer $MODEL_PATH"
85+
if [ "$IS_MTP" = "true" ]; then
86+
# just override extra_flags as dsv3 use different tokenizer path
87+
if [[ "$MODEL_NAME" == "DeepSeek-V4-Pro" ]]; then
88+
extra_flags="--dsv4"
89+
else
90+
extra_flags="--use-chat-template"
91+
fi
92+
fi
93+
# sglang
8194
else
8295
if [ "$IS_MTP" = "true" ]; then
8396
if [[ "$MODEL_NAME" == "DeepSeek-V4-Pro" ]]; then

‎benchmarks/multi_node/amd_utils/models_atom.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
# max_model_len: str # --max-model-len value (overrides default of unset)
2626
# max_num_seqs: str # --max-num-seqs value (overrides default of 256)
2727
# max_num_batched_tokens: str # --max-num-batched-tokens value (overrides default of unset)
28+
# scheduler_delay_factor: str # --scheduler-delay-factor value (overrides default of unset)
2829

2930
DeepSeek-V4-Pro:
3031
env: "ATOM_MOE_GU_ITLV=1 AITER_BF16_FP8_MOE_BOUND=0"
@@ -56,6 +57,14 @@ MiniMax-M3-MXFP4:
5657
max_num_seqs: "256"
5758
max_num_batched_tokens: "32768"
5859

60+
Kimi-K2.5-MXFP4:
61+
env: "AITER_QUICK_REDUCE_QUANTIZATION=INT4 AITER_MXFP4_INTERMEDIATE=1"
62+
kv_cache_flags: "--kv_cache_dtype fp8"
63+
mem_frac_static: "0.9"
64+
max_model_len: "32768"
65+
max_num_seqs: "256"
66+
max_num_batched_tokens: "32768"
67+
5968
MiniMax-M3-MXFP8:
6069
env: "AITER_QUICK_REDUCE_QUANTIZATION=INT4 ATOM_FORCE_ATTN_TRITON=1"
6170
kv_cache_flags: "--kv_cache_dtype fp8"

‎benchmarks/multi_node/amd_utils/server_atom.sh‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ print(f"_YAML_MEM_FRAC_STATIC='{sh(m.get('mem_frac_static', ''))}'")
104104
print(f"_YAML_MAX_MODEL_LEN='{sh(m.get('max_model_len', ''))}'")
105105
print(f"_YAML_MAX_NUM_SEQS='{sh(m.get('max_num_seqs', ''))}'")
106106
print(f"_YAML_MAX_NUM_BATCHED_TOKENS='{sh(m.get('max_num_batched_tokens', ''))}'")
107+
print(f"_YAML_SCHEDULER_DELAY_FACTOR='{sh(m.get('scheduler_delay_factor', ''))}'")
107108
PYEOF
108109
# shellcheck source=/dev/null
109110
source "$_yaml_tmp"
@@ -118,7 +119,8 @@ MEM_FRAC_STATIC="${_YAML_MEM_FRAC_STATIC:-${MEM_FRAC_STATIC:-0.85}}"
118119
MAX_MODEL_LEN="${_YAML_MAX_MODEL_LEN:-${MAX_MODEL_LEN:-}}"
119120
MAX_NUM_SEQS="${_YAML_MAX_NUM_SEQS:-${MAX_NUM_SEQS:-256}}"
120121
MAX_NUM_BATCHED_TOKENS="${_YAML_MAX_NUM_BATCHED_TOKENS:-${MAX_NUM_BATCHED_TOKENS:-}}"
121-
unset _YAML_BLOCK_SIZE _YAML_MEM_FRAC_STATIC _YAML_MAX_MODEL_LEN _YAML_MAX_NUM_SEQS _YAML_MAX_NUM_BATCHED_TOKENS
122+
SCHEDULER_DELAY_FACTOR="${_YAML_SCHEDULER_DELAY_FACTOR:-${SCHEDULER_DELAY_FACTOR:-}}"
123+
unset _YAML_BLOCK_SIZE _YAML_MEM_FRAC_STATIC _YAML_MAX_MODEL_LEN _YAML_MAX_NUM_SEQS _YAML_MAX_NUM_BATCHED_TOKENS _YAML_SCHEDULER_DELAY_FACTOR
122124

123125
# =============================================================================
124126
# Cluster Topology Configuration
@@ -211,6 +213,9 @@ fi
211213
if [[ -n "$MAX_NUM_BATCHED_TOKENS" ]]; then
212214
MODEL_LEN_ARGS="${MODEL_LEN_ARGS} --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS}"
213215
fi
216+
if [[ -n "$SCHEDULER_DELAY_FACTOR" ]]; then
217+
MODEL_LEN_ARGS="${MODEL_LEN_ARGS} --scheduler-delay-factor ${SCHEDULER_DELAY_FACTOR}"
218+
fi
214219

215220

216221
cat <<INFO

‎benchmarks/multi_node/kimik2.5_fp4_mi355x_atom-disagg.sh‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
#!/usr/bin/env bash
2+
3+
source "$(dirname "$0")/../benchmark_lib.sh"
4+
5+
check_env_vars \
6+
CONC_LIST \
7+
ISL \
8+
OSL \
9+
IMAGE \
10+
MODEL_PATH \
11+
PREFILL_NUM_WORKERS \
12+
PREFILL_TP \
13+
PREFILL_EP \
14+
PREFILL_DP_ATTN \
15+
DECODE_NUM_WORKERS \
16+
DECODE_TP \
17+
DECODE_EP \
18+
DECODE_DP_ATTN \
19+
PREFILL_NODES \
20+
DECODE_NODES \
21+
RANDOM_RANGE_RATIO \
22+
FRAMEWORK
23+
24+
if [[ -n "$SLURM_JOB_ID" ]]; then
25+
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
26+
fi
27+
28+
set -x
29+
30+
# Use upstreamed multi_node scripts (no external clone needed)
31+
cd "$GITHUB_WORKSPACE/benchmarks/multi_node/amd_utils" || exit 1
32+
33+
# Set up SGL launch script-specific environment variables
34+
export TIME_LIMIT="08:00:00"
35+
export MODEL_PATH=$MODEL_PATH
36+
export MODEL_NAME=$MODEL_NAME
37+
export CONTAINER_IMAGE=$IMAGE
38+
39+
if [[ "${PREFILL_EP:-1}" -eq 1 ]]; then
40+
export PREFILL_ENABLE_EP=false
41+
else
42+
export PREFILL_ENABLE_EP=true
43+
fi
44+
45+
if [[ "$PREFILL_DP_ATTN" == "true" ]]; then
46+
export PREFILL_ENABLE_DP=true
47+
else
48+
export PREFILL_ENABLE_DP=false
49+
fi
50+
51+
if [[ "${DECODE_EP:-1}" -eq 1 ]]; then
52+
export DECODE_ENABLE_EP=false
53+
else
54+
export DECODE_ENABLE_EP=true
55+
fi
56+
57+
if [[ "$DECODE_DP_ATTN" == "true" ]]; then
58+
export DECODE_ENABLE_DP=true
59+
else
60+
export DECODE_ENABLE_DP=false
61+
fi
62+
63+
JOB_ID=$(bash ./submit.sh $PREFILL_NODES \
64+
$PREFILL_NUM_WORKERS \
65+
$DECODE_NODES \
66+
$DECODE_NUM_WORKERS \
67+
$ISL $OSL "${CONC_LIST// /x}" inf \
68+
${PREFILL_ENABLE_EP} ${PREFILL_ENABLE_DP} \
69+
${DECODE_ENABLE_EP} ${DECODE_ENABLE_DP} \
70+
${PREFILL_TP} ${DECODE_TP} \
71+
${RANDOM_RANGE_RATIO})
72+
73+
if [[ $? -ne 0 ]]; then
74+
echo "Failed to submit job" >&2
75+
exit 1
76+
fi
77+
78+
echo "$JOB_ID"

‎configs/amd-master.yaml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -888,6 +888,59 @@ kimik2.5-fp4-mi355x-atom:
888888
search-space:
889889
- { tp: 4, conc-start: 4, conc-end: 128 }
890890

891+
kimik2.5-fp4-mi355x-atom-disagg:
892+
image: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atomesh_202607121715
893+
model: amd/Kimi-K2.5-MXFP4
894+
model-prefix: kimik2.5
895+
runner: mi355x-disagg
896+
precision: fp4
897+
framework: atom-disagg
898+
kv-p2p-transfer: mooncake
899+
multinode: true
900+
disagg: true
901+
scenarios:
902+
fixed-seq-len:
903+
# 1P(tp4)1D(tp4)
904+
- isl: 8192
905+
osl: 1024
906+
search-space:
907+
- spec-decoding: "none"
908+
conc-list: [ 16, 32, 64, 128, 256 ]
909+
prefill:
910+
num-worker: 1
911+
tp: 4
912+
ep: 1
913+
dp-attn: false
914+
additional-settings:
915+
- "PREFILL_NODES=1"
916+
decode:
917+
num-worker: 1
918+
tp: 4
919+
ep: 1
920+
dp-attn: false
921+
additional-settings:
922+
- "DECODE_NODES=1"
923+
# 1P(tp4)2D(tp8)
924+
- isl: 8192
925+
osl: 1024
926+
search-space:
927+
- spec-decoding: "none"
928+
conc-list: [ 1, 2, 4, 8, 16 ]
929+
prefill:
930+
num-worker: 1
931+
tp: 4
932+
ep: 1
933+
dp-attn: false
934+
additional-settings:
935+
- "PREFILL_NODES=1"
936+
decode:
937+
num-worker: 2
938+
tp: 8
939+
ep: 1
940+
dp-attn: false
941+
additional-settings:
942+
- "DECODE_NODES=2"
943+
891944
dsr1-fp8-mi355x-atom:
892945
image: rocm/atom:rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511
893946
model: deepseek-ai/DeepSeek-R1-0528

‎perf-changelog.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4751,6 +4751,14 @@
47514751
- "6 topologies across 1k/1k and 8k/1k: 1P1D TP4 STP + wide-EP (DEP4 prefill / DEP16 decode) from 1P1D up to 8P1D, recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp8/"
47524752
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2137
47534753

4754+
- config-keys:
4755+
- kimik2.5-fp4-mi355x-atom-disagg
4756+
description:
4757+
- "Add Kimi-K2.5-MXFP4 entry to models_atom.yaml for ATOM disaggregated inference"
4758+
- "Set env: ATOM_DISABLE_MMAP=true, AITER_QUICK_REDUCE_QUANTIZATION=INT4, AITER_MXFP4_INTERMEDIATE=1, OMP_NUM_THREADS=1"
4759+
- "Add scheduler_delay_factor support to models_atom.yaml schema and server_atom.sh; set to 1 for Kimi-K2.5"
4760+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2172
4761+
47544762
- config-keys:
47554763
- dsv4-fp8-h200-vllm
47564764
- dsv4-fp8-h200-vllm-mtp

0 commit comments

Comments
 (0)