Skip to content

Commit c1f3d47

Browse files
majunze2001claudecquil11
authored
[AgentX] vLLM DeepSeek-V4 B200 aggregate MTP (#2259)
* fix(changelog): accept single-node agentic eval rows in matrix validation #1947 made single-node agentic recipes generate a SWE-bench eval row (run_eval/eval_only + agentic fields) but never widened the changelog matrix schema, so ChangelogMatrixEntry.evals (typed list[SingleNodeMatrixEntry], fixed-seq-len only) rejects every agentic eval row -- breaking check-changelog for any single-node agentic PR. Widen evals to the same Union single_node already uses, and give SingleNodeAgenticMatrixEntry optional run_eval/eval_only (None-default, so benchmark rows are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [AgentX] DeepSeek-V4 B200 vLLM: add MTP (num_speculative_tokens=3) for every agentic arm Add an MTP speculative-decoding twin (spec-decoding: mtp, num_speculative_tokens=3) for every arm of dsv4-fp4-b200-vllm-agentic (TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake), each mirroring its non-MTP conc-list, routed via the launcher's spec-decoding=mtp suffix to dsv4_fp4_b200_vllm_mtp.sh. New dsv4_fp4_b200_vllm_mtp.sh forks dsv4_fp4_b200_vllm.sh with only the MTP deltas: --speculative-config {"method":"mtp","num_speculative_tokens":3} and --max-cudagraph-capture-size scaled to MAX_NUM_SEQS*(1+N) tokens so FULL_DECODE_ONLY still covers the largest decode batches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(changelog): add B200 vLLM MTP entry (PR #2259) Document the dsv4-fp4-b200-vllm-agentic MTP twins (num_speculative_tokens=3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [AgentX] B200 vLLM: split MTP into a separate config-key Restructure so the sweep runs only the new MTP work, not the existing aggregate: - Revert dsv4-fp4-b200-vllm-agentic to its main (#2224) search space -- unchanged vs main, so it is not re-run. - Add dsv4-fp4-b200-vllm-agentic-mtp: MTP twins (num_speculative_tokens=3) of the aggregate arms (TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake), each mirroring its non-MTP conc-list. - Point the perf-changelog entry at only the new key, so only it sweeps. (No separate TP8 key: B200's TP8 arms already exist in the #2224 aggregate.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * align synthetic acceptance length to 2.49 Use MTP synthetic rejection sampling with acceptance length 2.49, the dsv4-pro golden AL (thinking_on, num_speculative_tokens=3) from golden_al_distribution/dsv4_mtp.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * reduce GPU memory utilization from 0.92 to 0.9 Lower gpu-memory-utilization for the B200 vLLM MTP recipe to 0.9 for extra headroom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "fix(changelog): accept single-node agentic eval rows in matrix validation" This reverts commit 6efd246. * MTP synthetic acceptance for throughput, real verification for eval Make the B200 MTP --speculative-config conditional on EVAL_ONLY: throughput points pin synthetic acceptance length 2.49 (dsv4-pro golden AL), while the auto-selected SWE-bench accuracy run (EVAL_ONLY=true) uses real MTP with normal target verification. Synthetic acceptance bypasses verification and corrupts the eval output (0.0000 score). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * add missing tool call parser for swe eval Signed-off-by: Jeff Ma <jeffjma@umich.edu> * fix(agentic): preserve B200 MTP PCP GPU count * fix(agentic): derive B200 MTP GPU count from topology --------- Signed-off-by: Jeff Ma <jeffjma@umich.edu> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com>
1 parent d80f929 commit c1f3d47

3 files changed

Lines changed: 344 additions & 0 deletions

File tree

‎benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh‎

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
#!/usr/bin/env bash
2+
set -eo pipefail
3+
set -x
4+
5+
# Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on B200 using vLLM,
6+
# with MTP speculative decoding (num_speculative_tokens=3): synthetic acceptance
7+
# length 2.49 for throughput, real target verification for the EVAL_ONLY eval.
8+
#
9+
# Identical to dsv4_fp4_b200_vllm.sh (same image, engine args, offload, GPU
10+
# topologies, and agentic aiperf rig) with exactly two MTP deltas:
11+
# --speculative-config: synthetic acceptance length 2.49 (throughput) vs real MTP (EVAL_ONLY); see the SPEC_CONFIG block
12+
# --max-cudagraph-capture-size expressed in TOKENS (see the capture block below).
13+
#
14+
# Mirrors the fixed-seq-len parallelism options (pure TP and DEP) so the
15+
# agentic sweep can probe both interactivity and throughput regimes:
16+
# pure TP (DP_ATTENTION=false, EP_SIZE=1): attention TP-sharded across
17+
# all $TP GPUs in a single engine. Lower TPOT, lower batch.
18+
# TP+EP (DP_ATTENTION=false, EP_SIZE>1): attention TP-sharded, MoE
19+
# experts EP-sharded within the TP group.
20+
# DEP (DP_ATTENTION=true, EP_SIZE>1): per-DP-rank attention with
21+
# experts EP-sharded across DP ranks (per the vLLM blog recipe).
22+
# Highest aggregate throughput at large CONC.
23+
#
24+
# Image is configured in nvidia-master.yaml. block_size=256,
25+
# kv-cache-dtype=fp8, FLASHINFER_MLA_SPARSE_DSV4 attention with the FP4 indexer
26+
# cache, FULL_DECODE_ONLY cudagraph capture, and (in EP tiers) mega-MoE backend.
27+
#
28+
# Required env vars:
29+
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
30+
#
31+
# Pure TP is GPU-resident (KV_OFFLOADING=none). DEP tiers offload KV to host
32+
# DRAM: KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-simple or mooncake.
33+
34+
source "$(dirname "$0")/../../benchmark_lib.sh"
35+
36+
check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION
37+
38+
if [ -z "$DCP_SIZE" ]; then
39+
DCP_SIZE=1
40+
fi
41+
if [ -z "$PCP_SIZE" ]; then
42+
PCP_SIZE=1
43+
fi
44+
VLLM_CP_ARGS=()
45+
if [ "$DCP_SIZE" -gt 1 ]; then
46+
VLLM_CP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE")
47+
fi
48+
if [ "$PCP_SIZE" -gt 1 ]; then
49+
VLLM_CP_ARGS+=(--prefill-context-parallel-size "$PCP_SIZE")
50+
fi
51+
52+
GPU_COUNT=$((TP * PCP_SIZE))
53+
if [[ ! "$GPU_COUNT" =~ ^[1-9][0-9]*$ ]]; then
54+
echo "Error: GPU_COUNT must be a positive integer, got '$GPU_COUNT'" >&2
55+
exit 1
56+
fi
57+
export GPU_COUNT
58+
59+
if [[ -n "$SLURM_JOB_ID" ]]; then
60+
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
61+
fi
62+
63+
# `hf download` creates the target dir if missing and is itself idempotent.
64+
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE
65+
# Either way, MODEL_PATH is what the server is launched with.
66+
if [[ -n "$MODEL_PATH" ]]; then
67+
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
68+
hf download "$MODEL" --local-dir "$MODEL_PATH"
69+
fi
70+
else
71+
hf download "$MODEL"
72+
export MODEL_PATH="$MODEL"
73+
fi
74+
nvidia-smi
75+
76+
# ---- Resolve traces and install deps ----------------------------------------
77+
resolve_trace_source
78+
install_agentic_deps
79+
80+
# vllm-project/router expands the one HTTP backend into one logical worker per
81+
# DP rank and sends X-data-parallel-rank on forwarded requests. aiperf's
82+
# X-Correlation-ID is stable for every turn of a conversation; alias it to the
83+
# router's preferred X-Session-ID header.
84+
USE_VLLM_ROUTER=false
85+
VLLM_BACKEND_PORT="$PORT"
86+
if [ "$DP_ATTENTION" = "true" ]; then
87+
USE_VLLM_ROUTER=true
88+
VLLM_BACKEND_PORT=$((PORT + 1))
89+
VLLM_ROUTER_VERSION=0.1.14
90+
VLLM_ROUTER_POLICY=consistent_hash
91+
VLLM_ROUTER_METRICS_PORT=$((PORT + 10000))
92+
export AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=1
93+
agentic_pip_install --quiet "vllm-router==$VLLM_ROUTER_VERSION"
94+
fi
95+
96+
# DeepSeek-V4-Pro weights are large; engine startup can exceed default 600s.
97+
export VLLM_ENGINE_READY_TIMEOUT_S=3600
98+
99+
# vllm-project/vllm#43447 keeps local SWA prefix-cache tails sparsely, while
100+
# vllm-project/vllm#44774 applies the same reachability policy to Mooncake's
101+
# store mask. 32k matches the trace-replay tuning validated for this workload.
102+
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768
103+
export VLLM_USE_V2_MODEL_RUNNER=1
104+
export VLLM_USE_RUST_FRONTEND=1
105+
export VLLM_DSV4_MEGA_FP8_COMBINE=1
106+
export VLLM_RPC_TIMEOUT=600000
107+
108+
# ---- Server config ----------------------------------------------------------
109+
SERVER_LOG="$RESULT_DIR/server.log"
110+
ROUTER_LOG="$RESULT_DIR/router.log"
111+
MOONCAKE_MASTER_LOG="$RESULT_DIR/mooncake_master.log"
112+
mkdir -p "$RESULT_DIR"
113+
114+
SERVER_PID=""
115+
ROUTER_PID=""
116+
MOONCAKE_MASTER_PID=""
117+
118+
OFFLOAD_ARGS=()
119+
case "$KV_OFFLOAD_BACKEND" in
120+
"")
121+
require_agentic_kv_offload_none
122+
;;
123+
vllm-simple)
124+
require_agentic_kv_offload_backend vllm-simple
125+
CPU_BYTES_PER_RANK=$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / GPU_COUNT ))
126+
# Identical prefixes must hash to identical block keys across DP ranks.
127+
export PYTHONHASHSEED=42
128+
OFFLOAD_CONFIG=$(cat <<EOF
129+
{
130+
"kv_connector": "SimpleCPUOffloadConnector",
131+
"kv_role": "kv_both",
132+
"kv_connector_extra_config": {
133+
"cpu_bytes_to_use_per_rank": ${CPU_BYTES_PER_RANK},
134+
"lazy_offload": false,
135+
"enable_cross_layers_blocks": "true"
136+
}
137+
}
138+
EOF
139+
)
140+
OFFLOAD_ARGS=(
141+
--kv-transfer-config
142+
"$OFFLOAD_CONFIG"
143+
)
144+
;;
145+
mooncake)
146+
require_agentic_kv_offload_backend mooncake
147+
# Embedded mode contributes one segment per GPU rank to a shared
148+
# distributed store, so pre-divide the aggregate host-memory budget.
149+
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / GPU_COUNT))
150+
151+
MOONCAKE_VERSION=0.3.11.post1
152+
agentic_pip_install --quiet --no-cache-dir --no-deps \
153+
--force-reinstall "mooncake-transfer-engine-cuda13==$MOONCAKE_VERSION"
154+
python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null
155+
156+
MOONCAKE_MASTER_PORT=$((PORT + 12000))
157+
MOONCAKE_CONFIG_PATH="$RESULT_DIR/mooncake_config.json"
158+
cat > "$MOONCAKE_CONFIG_PATH" <<EOF
159+
{
160+
"mode": "embedded",
161+
"metadata_server": "P2PHANDSHAKE",
162+
"master_server_address": "127.0.0.1:$MOONCAKE_MASTER_PORT",
163+
"global_segment_size": "${PER_RANK_GB}GB",
164+
"local_buffer_size": "4GB",
165+
"protocol": "rdma",
166+
"device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_10,mlx5_11",
167+
"enable_offload": false
168+
}
169+
EOF
170+
export MOONCAKE_CONFIG_PATH
171+
export MC_ENABLE_DEST_DEVICE_AFFINITY=1
172+
# Identical prefixes must hash to identical store keys across DP ranks.
173+
export PYTHONHASHSEED=0
174+
export WITH_NVIDIA_PEERMEM=0
175+
export MC_SLICE_SIZE=1048576
176+
export MC_WORKERS_PER_CTX=4
177+
178+
# Each rank contributes a separate segment. Evict early enough to
179+
# avoid an imbalanced rank exhausting its segment.
180+
MOONCAKE_EVICTION_HIGH_WATERMARK_RATIO=0.80
181+
MOONCAKE_EVICTION_RATIO=0.10
182+
# Mooncake's default 5s read lease is shorter than the observed
183+
# transfer latency for large DSv4 hybrid-KV loads on B200 TCP.
184+
MOONCAKE_KV_LEASE_TTL=60s
185+
186+
echo "Starting Mooncake master on port $MOONCAKE_MASTER_PORT..."
187+
mooncake_master --port "$MOONCAKE_MASTER_PORT" \
188+
--eviction_high_watermark_ratio="$MOONCAKE_EVICTION_HIGH_WATERMARK_RATIO" \
189+
--eviction_ratio="$MOONCAKE_EVICTION_RATIO" \
190+
--default_kv_lease_ttl="$MOONCAKE_KV_LEASE_TTL" \
191+
> "$MOONCAKE_MASTER_LOG" 2>&1 &
192+
MOONCAKE_MASTER_PID=$!
193+
sleep 2
194+
if ! kill -0 "$MOONCAKE_MASTER_PID" 2>/dev/null; then
195+
echo "Mooncake master died during startup." >&2
196+
cat "$MOONCAKE_MASTER_LOG" >&2
197+
exit 1
198+
fi
199+
unset VLLM_USE_SIMPLE_KV_OFFLOAD
200+
OFFLOAD_ARGS=(
201+
--kv-transfer-config
202+
'{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
203+
)
204+
;;
205+
*)
206+
echo "Error: unsupported B200 KV_OFFLOAD_BACKEND='$KV_OFFLOAD_BACKEND'" >&2
207+
exit 1
208+
;;
209+
esac
210+
211+
PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
212+
if [ "$DP_ATTENTION" = "true" ]; then
213+
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
214+
fi
215+
216+
EP_ARGS=()
217+
FAST_MOE_ARGS=()
218+
if [ "$EP_SIZE" -gt 1 ]; then
219+
EP_ARGS=(--enable-expert-parallel)
220+
FAST_MOE_ARGS=(
221+
--moe-backend deep_gemm_amxf4_mega_moe
222+
--enable-ep-weight-filter
223+
--prefill-schedule-interval 16
224+
)
225+
fi
226+
227+
# AgentX concurrency counts live session trees, not individual requests.
228+
# Subagent fan-out can push instantaneous request concurrency above CONC, so
229+
# leave 2x headroom rather than clipping those bursts at the scheduler.
230+
MAX_NUM_SEQS=$((2 * CONC))
231+
232+
# MTP: cudagraph capture sizes are in TOKENS. A uniform decode batch of S seqs
233+
# verifies S*(1+num_speculative_tokens) tokens, so cap capture at
234+
# MAX_NUM_SEQS*(1+N) tokens -- otherwise vLLM's FULL_DECODE_ONLY ladder tops out
235+
# at MAX_NUM_SEQS/(1+N) seqs and the largest decode batches fall back to eager.
236+
NUM_SPEC_TOKENS=3
237+
# Throughput pins synthetic MTP acceptance to the dsv4-pro golden AL (thinking_on,
238+
# num_speculative_tokens=3, golden_al_distribution/dsv4_mtp.yaml). The EVAL_ONLY
239+
# accuracy run uses real target verification instead -- synthetic acceptance
240+
# bypasses verification and corrupts the SWE-bench eval (0.0000 score).
241+
SYNTHETIC_ACCEPT_LEN=2.49
242+
if [ "${EVAL_ONLY:-false}" = "true" ]; then
243+
SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS}"
244+
else
245+
SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"rejection_sample_method\": \"synthetic\", \"synthetic_acceptance_length\": $SYNTHETIC_ACCEPT_LEN}"
246+
fi
247+
TOKENS_PER_SEQ=$((1 + NUM_SPEC_TOKENS))
248+
MAX_CUDAGRAPH_CAPTURE_SIZE=$((MAX_NUM_SEQS * TOKENS_PER_SEQ))
249+
250+
echo "Starting vllm server..."
251+
export TORCH_CUDA_ARCH_LIST="10.0"
252+
export PYTHONNOUSERSITE=1
253+
export VLLM_FLOAT32_MATMUL_PRECISION=high
254+
255+
{ set +x; } 2>/dev/null
256+
VLLM_CMD=(
257+
vllm serve "$MODEL_PATH" --served-model-name "$MODEL"
258+
--host 0.0.0.0
259+
--port "$VLLM_BACKEND_PORT"
260+
--trust-remote-code
261+
--kv-cache-dtype fp8
262+
--block-size 256
263+
--max-model-len 1048576
264+
--gpu-memory-utilization 0.9
265+
--numa-bind
266+
--enable-cumem-allocator
267+
--no-enable-flashinfer-autotune
268+
--tokenizer-mode deepseek_v4
269+
--tool-call-parser deepseek_v4
270+
--enable-auto-tool-choice
271+
--reasoning-parser deepseek_v4
272+
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}'
273+
--speculative-config "$SPEC_CONFIG"
274+
--no-disable-hybrid-kv-cache-manager
275+
--disable-uvicorn-access-log
276+
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'
277+
--max-num-seqs "$MAX_NUM_SEQS"
278+
--max-cudagraph-capture-size "$MAX_CUDAGRAPH_CAPTURE_SIZE"
279+
"${PARALLEL_ARGS[@]}"
280+
"${VLLM_CP_ARGS[@]}"
281+
"${EP_ARGS[@]}"
282+
"${FAST_MOE_ARGS[@]}"
283+
"${OFFLOAD_ARGS[@]}"
284+
)
285+
printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt"
286+
printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt"
287+
"${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 &
288+
SERVER_PID=$!
289+
echo "Server PID: $SERVER_PID"
290+
291+
wait_for_server_ready --port "$VLLM_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"
292+
293+
if [ "$USE_VLLM_ROUTER" = "true" ]; then
294+
echo "Starting native vLLM router on port $PORT for $TP DP ranks..."
295+
vllm-router \
296+
--worker-urls "http://localhost:$VLLM_BACKEND_PORT" \
297+
--policy "$VLLM_ROUTER_POLICY" \
298+
--intra-node-data-parallel-size "$TP" \
299+
--host 0.0.0.0 \
300+
--port "$PORT" \
301+
--prometheus-host 127.0.0.1 \
302+
--prometheus-port "$VLLM_ROUTER_METRICS_PORT" \
303+
--request-timeout-secs 14400 \
304+
--disable-retries > "$ROUTER_LOG" 2>&1 &
305+
ROUTER_PID=$!
306+
echo "Router PID: $ROUTER_PID"
307+
wait_for_server_ready --port "$PORT" --server-log "$ROUTER_LOG" --server-pid "$ROUTER_PID"
308+
fi
309+
310+
if [ "${EVAL_ONLY}" = "true" ]; then
311+
run_eval --port "$PORT"
312+
else
313+
build_replay_cmd "$RESULT_DIR"
314+
run_agentic_replay_and_write_outputs "$RESULT_DIR"
315+
fi

‎configs/nvidia-master.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -966,6 +966,27 @@ dsv4-fp4-b200-vllm-agentic:
966966
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }
967967
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }
968968

969+
dsv4-fp4-b200-vllm-agentic-mtp:
970+
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec
971+
model: deepseek-ai/DeepSeek-V4-Pro
972+
model-prefix: dsv4
973+
runner: cluster:b200-dgxc
974+
precision: fp4
975+
framework: vllm
976+
multinode: false
977+
scenarios:
978+
agentic-coding:
979+
- dram-utilization: 0.80
980+
search-space:
981+
# Pure TP GPU-resident + MTP (num_speculative_tokens=3)
982+
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 6, 8] }
983+
# Pure TP SimpleCPU + MTP (num_speculative_tokens=3)
984+
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [8, 12, 16] }
985+
# DEP SimpleCPU + MTP (num_speculative_tokens=3)
986+
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } }
987+
# DEP Mooncake + MTP (num_speculative_tokens=3)
988+
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, spec-decoding: mtp, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }
989+
969990
dsv4-fp4-b200-trt:
970991
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066
971992
model: deepseek-ai/DeepSeek-V4-Pro

‎perf-changelog.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5011,3 +5011,11 @@
50115011
- "dsv4-fp4-b300-vllm-agentic-mtp: MTP twins (--speculative-config {\"method\":\"mtp\",\"num_speculative_tokens\":3}) of the aggregate arms plus a new TP8 GPU-resident arm -- TP8 GPU-resident [1,2,4,6,8], TP4 GPU-resident [1,2,4,6,8,12,16,20], TP4 SimpleCPU [20,24,28,32,36,40], DEP4 [32,40,48,56], DEP8 [64,96,112,128,144,160,176,192,224]; routed via spec-decoding=mtp to dsv4_fp4_b300_vllm_mtp.sh (FULL_DECODE_ONLY cudagraph capture sizes in TOKENS = num_seqs*4)."
50125012
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2258
50135013

5014+
5015+
- config-keys:
5016+
- dsv4-fp4-b200-vllm-agentic-mtp
5017+
description:
5018+
- "Add MTP speculative-decoding for the B200 vLLM AgentX recipe as a separate config-key, so the existing dsv4-fp4-b200-vllm-agentic aggregate is not re-run."
5019+
- "dsv4-fp4-b200-vllm-agentic-mtp: MTP twins (--speculative-config {\"method\":\"mtp\",\"num_speculative_tokens\":3}) of the aggregate arms -- TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake, each mirroring its non-MTP conc-list; routed via spec-decoding=mtp to dsv4_fp4_b200_vllm_mtp.sh."
5020+
- "MTP capture sizes are in TOKENS: --max-cudagraph-capture-size scaled to MAX_NUM_SEQS*(1+num_speculative_tokens) (= MAX_NUM_SEQS*4) so FULL_DECODE_ONLY covers the largest decode batches."
5021+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2259

0 commit comments

Comments
 (0)