Skip to content

Commit d80f929

Browse files
majunze2001cquil11ivaniumclaude
authored
[AgentX] vLLM DeepSeek-V4 B300 aggregate MTP (#2258)
* feat(agentic): split DeepSeek-V4 B300 vLLM recipe Carry the B300-only launcher, config search space, and changelog scope from PR #2202. 中文：拆分 DeepSeek-V4 B300 vLLM AgentX 配方，仅保留 PR #2202 中的 B300 启动器、配置搜索空间和变更日志范围。 * docs(changelog): link B300 replacement PR Point the split B300 changelog entry to PR #2225. 中文：将拆分后的 B300 变更日志条目链接到 PR #2225。 * chore: update conc * update dep8 args Signed-off-by: Jeff Ma <jeffjma@umich.edu> * add more tp configs Signed-off-by: Jeff Ma <jeffjma@umich.edu> * dep8 reduce gpu mem util to 0.92 Signed-off-by: Jeff Ma <jeffjma@umich.edu> * chore: update conc list * docs(changelog): point B300 entry to PR #2241 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LjHXfE1FRN9c5XQqgeTdez * fix(changelog): keep perf-changelog additions-only Restore upstream lines whitespace-for-whitespace and append the B300 entry so the changelog diff contains no deletions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LjHXfE1FRN9c5XQqgeTdez * add mtp * [AgentX] DeepSeek-V4 B300 vLLM: MTP (num_speculative_tokens=3) across the recipe Finalize the B300 AgentX vLLM search space (dsv4-fp4-b300-vllm-agentic): - Add a TP8 GPU-resident arm at conc [1,2,4,6,8]. - Trim TP4 GPU-resident to conc [1,2,4,6,8,12,16,20]; extend TP4 SimpleCPU lazy-offload down to conc [20,24,28,32,36,40]. - Add an MTP speculative-decoding twin (num_speculative_tokens=3) for every topology (TP8/TP4 GPU-resident, TP4 SimpleCPU, DEP4, DEP8), each mirroring its non-MTP conc-list, routed via the launcher's spec-decoding=mtp suffix to dsv4_fp4_b300_vllm_mtp.sh. MTP script: NUM_SPEC_TOKENS=3 -> TOKENS_PER_SEQ=4, so FULL_DECODE_ONLY cudagraph capture sizes (num_seqs*TOKENS_PER_SEQ) scale to num_seqs*4. Sync the tail with dsv4_fp4_b300_vllm.sh to restore the EVAL_ONLY branch (added upstream by #1947) so MTP configs also run the SWE-bench Lite accuracy eval. Add the perf-changelog entry for dsv4-fp4-b300-vllm-agentic (PR #2258). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(changelog): accept single-node agentic eval rows in matrix validation #1947 made single-node agentic recipes generate a SWE-bench eval row (run_eval/eval_only + agentic fields) but never widened the changelog matrix schema, so ChangelogMatrixEntry.evals (typed list[SingleNodeMatrixEntry], fixed-seq-len only) rejects every agentic eval row -- breaking check-changelog for any single-node agentic PR. Widen evals to the same Union single_node already uses, and give SingleNodeAgenticMatrixEntry optional run_eval/eval_only (None-default, so benchmark rows are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [AgentX] B300 vLLM: split MTP + TP8 into separate config-keys Restructure so the sweep runs only the new work, not the existing aggregate: - Revert dsv4-fp4-b300-vllm-agentic to its main (#2241) search space (TP4 [1..32], SimpleCPU [28..40], DEP4, DEP8) -- unchanged vs main, so it is not re-run. - Add dsv4-fp4-b300-vllm-agentic-mtp: MTP twins of the aggregate arms (num_speculative_tokens=3), reshaped TP4 [1..20] / SimpleCPU [20..40]. - Add dsv4-fp4-b300-vllm-agentic-tp8: new TP8 GPU-resident topology at conc [1,2,4,6,8], both non-MTP and MTP. - Point the perf-changelog entry at only the two new keys, so only they sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * align synthetic acceptance length Inject rejection_sample_method=synthetic + synthetic_acceptance_length=2.49 into the dsv4-fp4-b300-vllm MTP speculative-config, matching the dsv4-pro golden AL (thinking_on, num_speculative_tokens=3) and the sglang gb300 MTP recipe (SGLANG_SIMULATE_ACC_LEN=2.49). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * reduce GPU memory utilization to 0.95 (except DEP8) TP4, DEP4, and TP8 use gpu-memory-utilization 0.95 (down from 0.96); DEP8 stays at 0.92 for its larger prefill token budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * trim B300 MTP search space: fold TP8 into -mtp, drop -tp8 key, trim DEP4 Remove the dsv4-fp4-b300-vllm-agentic-tp8 config-key entirely; its TP8 GPU-resident MTP arm moves into dsv4-fp4-b300-vllm-agentic-mtp (the non-MTP TP8 baseline is dropped). Also drop conc 64 and 72 from the DEP4 arm (now [32,40,48,56]). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "fix(changelog): accept single-node agentic eval rows in matrix validation" This reverts commit 5887e1a. * MTP synthetic acceptance for throughput, real verification for eval Make the MTP --speculative-config conditional on EVAL_ONLY: throughput points pin synthetic acceptance length 2.49 (dsv4-pro golden AL), while the auto-selected SWE-bench accuracy run (EVAL_ONLY=true) uses real MTP with normal target verification. Synthetic acceptance bypasses verification and corrupts the eval output (0.0000 score). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(agentic): wire B300 MTP context parallelism * fix(agentic): derive B300 MTP GPU count from topology * Update perf-changelog.yaml * fix(changelog): restore PR 2258 entry indentation --------- Signed-off-by: Jeff Ma <jeffjma@umich.edu> Co-authored-by: Cam Quilici <cjquilici@gmail.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 01e9e73 commit d80f929

3 files changed

Lines changed: 381 additions & 0 deletions

File tree

‎benchmarks/single_node/agentic/dsv4_fp4_b300_vllm_mtp.sh‎

Lines changed: 350 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,350 @@
1+
#!/usr/bin/env bash
2+
set -eo pipefail
3+
set -x
4+
5+
# Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on B300 using vLLM,
6+
# with MTP speculative decoding (num_speculative_tokens=3): synthetic acceptance
7+
# length 2.49 for throughput, real target verification for the EVAL_ONLY eval.
8+
#
9+
# Identical to dsv4_fp4_b300_vllm.sh (same image, engine args, offload, GPU
10+
# topologies, and agentic aiperf rig) with exactly two MTP deltas:
11+
# --speculative-config: synthetic acceptance length 2.49 (throughput) vs real MTP (EVAL_ONLY); see the SPEC_CONFIG block
12+
# cudagraph capture sizes expressed in TOKENS (see the capture block below).
13+
#
14+
# Image is configured in nvidia-master.yaml. The recipe uses FP8 KV cache,
15+
# sparse DeepSeek-V4 FlashInfer attention with an FP4 indexer cache, mega-MoE,
16+
# and FULL_DECODE_ONLY CUDA graphs with every decode batch captured explicitly.
17+
#
18+
# Required env vars:
19+
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
20+
#
21+
# TP4, TP8, and DEP8 (TP8 + DP-attention) are GPU-resident (KV_OFFLOADING=none).
22+
# DEP4 uses KV_OFFLOADING=dram with KV_OFFLOAD_BACKEND=vllm-simple or mooncake.
23+
24+
source "$(dirname "$0")/../../benchmark_lib.sh"
25+
26+
check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION
27+
28+
DCP_SIZE="${DCP_SIZE:-1}"
29+
PCP_SIZE="${PCP_SIZE:-1}"
30+
VLLM_CP_ARGS=()
31+
if [ "$DCP_SIZE" -gt 1 ]; then
32+
VLLM_CP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE")
33+
fi
34+
if [ "$PCP_SIZE" -gt 1 ]; then
35+
VLLM_CP_ARGS+=(--prefill-context-parallel-size "$PCP_SIZE")
36+
fi
37+
38+
GPU_COUNT=$((TP * PCP_SIZE))
39+
if [[ ! "$GPU_COUNT" =~ ^[1-9][0-9]*$ ]]; then
40+
echo "Error: GPU_COUNT must be a positive integer, got '$GPU_COUNT'" >&2
41+
exit 1
42+
fi
43+
export GPU_COUNT
44+
45+
# Under DP-attention the DP world size equals TP, and the DEP recipe sizes
46+
# per-rank batch as MAX_NUM_SEQS = 2*CONC/TP, which must be an integer.
47+
if [ "$DP_ATTENTION" = "true" ] && [ $((2 * CONC % TP)) -ne 0 ]; then
48+
echo "Error: DEP requires 2*CONC divisible by TP, got CONC='$CONC' and TP='$TP'" >&2
49+
exit 1
50+
fi
51+
52+
# DEP8 (TP8 + DP-attention) is a GPU-resident, high-concurrency arm that is
53+
# tuned separately from the smaller DEP4 arm (larger prefill token budget,
54+
# long-prefill chunking, and a lower GPU-memory-utilization headroom).
55+
IS_DEP8=false
56+
if [ "$DP_ATTENTION" = "true" ] && [ "$TP" -eq 8 ]; then
57+
IS_DEP8=true
58+
fi
59+
60+
if [[ -n "$SLURM_JOB_ID" ]]; then
61+
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
62+
fi
63+
64+
# `hf download` creates the target dir if missing and is itself idempotent.
65+
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE.
66+
# Either way, MODEL_PATH is what the server is launched with.
67+
if [[ -n "$MODEL_PATH" ]]; then
68+
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
69+
hf download "$MODEL" --local-dir "$MODEL_PATH"
70+
fi
71+
else
72+
hf download "$MODEL"
73+
export MODEL_PATH="$MODEL"
74+
fi
75+
nvidia-smi
76+
77+
# ---- Resolve traces and install deps ----------------------------------------
78+
resolve_trace_source
79+
install_agentic_deps
80+
81+
# vllm-project/router expands the one HTTP backend into one logical worker per
82+
# DP rank. Bind every turn of a conversation to the same rank by mapping
83+
# AIPerf's stable correlation ID to the router's X-Session-ID header.
84+
USE_VLLM_ROUTER=false
85+
VLLM_BACKEND_PORT="$PORT"
86+
if [ "$DP_ATTENTION" = "true" ]; then
87+
USE_VLLM_ROUTER=true
88+
VLLM_BACKEND_PORT=$((PORT + 1))
89+
VLLM_ROUTER_VERSION=0.1.14
90+
VLLM_ROUTER_POLICY=consistent_hash
91+
VLLM_ROUTER_METRICS_PORT=$((PORT + 10000))
92+
export AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=1
93+
agentic_pip_install --quiet "vllm-router==$VLLM_ROUTER_VERSION"
94+
fi
95+
96+
# Match the environment used by v4pro-b300.yaml.
97+
export VLLM_USE_V2_MODEL_RUNNER=1
98+
export VLLM_ENGINE_READY_TIMEOUT_S=3600
99+
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768
100+
export VLLM_DSV4_MEGA_FP8_COMBINE=1
101+
export NCCL_NVLS_ENABLE=1
102+
export VLLM_USE_RUST_FRONTEND=1
103+
104+
# ---- Server config ----------------------------------------------------------
105+
SERVER_LOG="$RESULT_DIR/server.log"
106+
ROUTER_LOG="$RESULT_DIR/router.log"
107+
MOONCAKE_MASTER_LOG="$RESULT_DIR/mooncake_master.log"
108+
mkdir -p "$RESULT_DIR"
109+
110+
SERVER_PID=""
111+
ROUTER_PID=""
112+
MOONCAKE_MASTER_PID=""
113+
114+
# The generated TOTAL_CPU_DRAM_GB budget is proportional to allocated GPUs.
115+
# On cluster:b300-nv, dram-utilization=0.80 and DEP4 resolve to roughly the
116+
# source recipe's 280 GiB per DP rank. TP4 remains GPU-resident.
117+
OFFLOAD_ARGS=()
118+
case "$KV_OFFLOAD_BACKEND" in
119+
"")
120+
require_agentic_kv_offload_none
121+
;;
122+
vllm-simple)
123+
require_agentic_kv_offload_backend vllm-simple
124+
CPU_BYTES_PER_RANK=$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / GPU_COUNT ))
125+
# Identical prefixes must hash to identical block keys across DP ranks.
126+
export PYTHONHASHSEED=42
127+
# The plain-TP (non-DP-attention) offload ladder uses lazy offload;
128+
# DEP keeps eager offload for cross-rank block-hash stability.
129+
SIMPLE_LAZY_OFFLOAD=false
130+
if [ "$DP_ATTENTION" != "true" ]; then
131+
SIMPLE_LAZY_OFFLOAD=true
132+
fi
133+
OFFLOAD_CONFIG=$(cat <<EOF
134+
{
135+
"kv_connector": "SimpleCPUOffloadConnector",
136+
"kv_role": "kv_both",
137+
"kv_connector_extra_config": {
138+
"cpu_bytes_to_use": ${CPU_BYTES_PER_RANK},
139+
"enable_cross_layers_blocks": "true",
140+
"lazy_offload": ${SIMPLE_LAZY_OFFLOAD}
141+
}
142+
}
143+
EOF
144+
)
145+
OFFLOAD_ARGS=(
146+
--kv-transfer-config
147+
"$OFFLOAD_CONFIG"
148+
)
149+
;;
150+
mooncake)
151+
require_agentic_kv_offload_backend mooncake
152+
# Embedded mode contributes one global segment per DP rank to the
153+
# shared store, so divide the aggregate host budget across ranks.
154+
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / GPU_COUNT))
155+
MOONCAKE_VERSION=0.3.11.post1
156+
agentic_pip_install --quiet --no-cache-dir --no-deps \
157+
--force-reinstall "mooncake-transfer-engine-cuda13==$MOONCAKE_VERSION"
158+
python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null
159+
160+
MOONCAKE_MASTER_PORT=$((PORT + 12000))
161+
MOONCAKE_CONFIG_PATH="$RESULT_DIR/mooncake_config.json"
162+
cat > "$MOONCAKE_CONFIG_PATH" <<EOF
163+
{
164+
"mode": "embedded",
165+
"metadata_server": "P2PHANDSHAKE",
166+
"master_server_address": "127.0.0.1:$MOONCAKE_MASTER_PORT",
167+
"global_segment_size": "${PER_RANK_GB}GB",
168+
"local_buffer_size": "4GB",
169+
"protocol": "rdma",
170+
"device_name": "",
171+
"enable_offload": false
172+
}
173+
EOF
174+
export MOONCAKE_CONFIG_PATH
175+
export MC_ENABLE_DEST_DEVICE_AFFINITY=1
176+
# Identical prefixes must hash to identical store keys across DP ranks.
177+
export PYTHONHASHSEED=0
178+
export WITH_NVIDIA_PEERMEM=0
179+
export MC_SLICE_SIZE=1048576
180+
export MC_WORKERS_PER_CTX=4
181+
182+
# The store is shared, but each rank contributes a separate segment.
183+
# Start eviction before an imbalanced rank exhausts its segment, and
184+
# reclaim enough space for several concurrent multi-GB batch puts.
185+
MOONCAKE_EVICTION_HIGH_WATERMARK_RATIO=0.80
186+
MOONCAKE_EVICTION_RATIO=0.10
187+
188+
echo "Starting Mooncake master on port $MOONCAKE_MASTER_PORT..."
189+
mooncake_master --port "$MOONCAKE_MASTER_PORT" \
190+
--eviction_high_watermark_ratio="$MOONCAKE_EVICTION_HIGH_WATERMARK_RATIO" \
191+
--eviction_ratio="$MOONCAKE_EVICTION_RATIO" \
192+
> "$MOONCAKE_MASTER_LOG" 2>&1 &
193+
MOONCAKE_MASTER_PID=$!
194+
sleep 2
195+
if ! kill -0 "$MOONCAKE_MASTER_PID" 2>/dev/null; then
196+
echo "Mooncake master died during startup." >&2
197+
cat "$MOONCAKE_MASTER_LOG" >&2
198+
exit 1
199+
fi
200+
201+
unset VLLM_USE_SIMPLE_KV_OFFLOAD
202+
OFFLOAD_CONFIG='{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
203+
OFFLOAD_ARGS=(--kv-transfer-config "$OFFLOAD_CONFIG")
204+
;;
205+
*)
206+
echo "Error: unsupported B300 KV_OFFLOAD_BACKEND='$KV_OFFLOAD_BACKEND'" >&2
207+
exit 1
208+
;;
209+
esac
210+
211+
PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
212+
if [ "$DP_ATTENTION" = "true" ]; then
213+
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
214+
fi
215+
216+
TP_ARGS=()
217+
if [ "$DP_ATTENTION" = "true" ]; then
218+
export PYTORCH_ALLOC_CONF=expandable_segments:True
219+
else
220+
export VLLM_ALLREDUCE_USE_FLASHINFER=1
221+
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=auto
222+
TP_ARGS+=(--disable-custom-all-reduce)
223+
fi
224+
225+
MODE_ARGS=()
226+
if [ "$EP_SIZE" -gt 1 ]; then
227+
MODE_ARGS+=(
228+
--enable-expert-parallel
229+
--enable-ep-weight-filter
230+
--moe-backend deep_gemm_amxf4_mega_moe
231+
)
232+
fi
233+
if [ "$DP_ATTENTION" = "true" ]; then
234+
MODE_ARGS+=(--prefill-schedule-interval 8)
235+
if [ "$IS_DEP8" = "true" ]; then
236+
# GPU-resident DEP8 gets a larger prefill token budget and chunks long
237+
# prefills so decode latency stays bounded at high concurrency.
238+
MODE_ARGS+=(
239+
--max-num-batched-tokens 16384
240+
--long-prefill-token-threshold 4096
241+
)
242+
else
243+
MODE_ARGS+=(--max-num-batched-tokens 8192)
244+
fi
245+
fi
246+
247+
if [ "$DP_ATTENTION" = "true" ]; then
248+
# The DEP source recipe enforces 2*CONC = DP_WORLD_SIZE*MAX_NUM_SEQS.
249+
MAX_NUM_SEQS=$((2 * CONC / TP))
250+
else
251+
# Preserve the previous TP4 scheduler headroom for agentic fan-out.
252+
MAX_NUM_SEQS=$((2 * CONC))
253+
fi
254+
# MTP: cudagraph capture sizes are in TOKENS. With num_speculative_tokens=N,
255+
# every uniform decode batch of S seqs verifies S*(1+N) tokens, so capture the
256+
# explicit multiples (1+N), 2*(1+N), ..., MAX_NUM_SEQS*(1+N) -- one graph per
257+
# decode batch of 1..MAX_NUM_SEQS seqs. vLLM rounds configured sizes up to
258+
# multiples of (1+N) and dedups (adjust_cudagraph_sizes_for_spec_decode), so a
259+
# plain 1..MAX_NUM_SEQS list would collapse to coverage of only
260+
# MAX_NUM_SEQS/(1+N) seqs and drop the largest decode batches to eager.
261+
NUM_SPEC_TOKENS=3
262+
TOKENS_PER_SEQ=$((1 + NUM_SPEC_TOKENS))
263+
# Throughput pins synthetic MTP acceptance to the dsv4-pro golden AL (thinking_on,
264+
# num_speculative_tokens=3, golden_al_distribution/dsv4_mtp.yaml). The EVAL_ONLY
265+
# accuracy run uses real target verification instead -- synthetic acceptance
266+
# bypasses verification and corrupts the SWE-bench eval (0.0000 score).
267+
if [ "${EVAL_ONLY:-false}" = "true" ]; then
268+
SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS}"
269+
else
270+
SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"rejection_sample_method\": \"synthetic\", \"synthetic_acceptance_length\": 2.49}"
271+
fi
272+
CUDA_GRAPH_CAPTURE_SIZES=""
273+
for ((num_seqs = 1; num_seqs <= MAX_NUM_SEQS; num_seqs++)); do
274+
if [ -n "$CUDA_GRAPH_CAPTURE_SIZES" ]; then
275+
CUDA_GRAPH_CAPTURE_SIZES+=","
276+
fi
277+
CUDA_GRAPH_CAPTURE_SIZES+="$((num_seqs * TOKENS_PER_SEQ))"
278+
done
279+
COMPILATION_CONFIG="{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[${CUDA_GRAPH_CAPTURE_SIZES}],\"mode\":0}"
280+
281+
echo "Starting vllm server..."
282+
export TORCH_CUDA_ARCH_LIST="10.0"
283+
export PYTHONNOUSERSITE=1
284+
export VLLM_FLOAT32_MATMUL_PRECISION=high
285+
286+
# DEP8 leaves more headroom for its larger prefill token budget; all other
287+
# topologies (TP4/DEP4/TP8) use 0.95.
288+
GPU_MEM_UTIL=0.95
289+
if [ "$IS_DEP8" = "true" ]; then
290+
GPU_MEM_UTIL=0.92
291+
fi
292+
293+
{ set +x; } 2>/dev/null
294+
VLLM_CMD=(
295+
vllm serve "$MODEL_PATH" --served-model-name "$MODEL"
296+
--host 0.0.0.0
297+
--port "$VLLM_BACKEND_PORT"
298+
--gpu-memory-utilization "$GPU_MEM_UTIL"
299+
--trust-remote-code
300+
--no-enable-flashinfer-autotune
301+
--no-disable-hybrid-kv-cache-manager
302+
--max-num-seqs "$MAX_NUM_SEQS"
303+
--kv-cache-dtype fp8
304+
--block-size 256
305+
--max-model-len 1048576
306+
--attention-config '{"use_fp4_indexer_cache":true,"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}'
307+
--speculative-config "$SPEC_CONFIG"
308+
--disable-uvicorn-access-log
309+
--tokenizer-mode deepseek_v4
310+
--tool-call-parser deepseek_v4
311+
--enable-auto-tool-choice
312+
--reasoning-parser deepseek_v4
313+
--compilation-config "$COMPILATION_CONFIG"
314+
"${PARALLEL_ARGS[@]}"
315+
"${VLLM_CP_ARGS[@]}"
316+
"${TP_ARGS[@]}"
317+
"${MODE_ARGS[@]}"
318+
"${OFFLOAD_ARGS[@]}"
319+
)
320+
printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt"
321+
printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt"
322+
"${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 &
323+
SERVER_PID=$!
324+
echo "Server PID: $SERVER_PID"
325+
326+
wait_for_server_ready --port "$VLLM_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"
327+
328+
if [ "$USE_VLLM_ROUTER" = "true" ]; then
329+
echo "Starting native vLLM router on port $PORT for $TP DP ranks..."
330+
vllm-router \
331+
--worker-urls "http://localhost:$VLLM_BACKEND_PORT" \
332+
--policy "$VLLM_ROUTER_POLICY" \
333+
--intra-node-data-parallel-size "$TP" \
334+
--host 0.0.0.0 \
335+
--port "$PORT" \
336+
--prometheus-host 127.0.0.1 \
337+
--prometheus-port "$VLLM_ROUTER_METRICS_PORT" \
338+
--request-timeout-secs 14400 \
339+
--disable-retries > "$ROUTER_LOG" 2>&1 &
340+
ROUTER_PID=$!
341+
echo "Router PID: $ROUTER_PID"
342+
wait_for_server_ready --port "$PORT" --server-log "$ROUTER_LOG" --server-pid "$ROUTER_PID"
343+
fi
344+
345+
if [ "${EVAL_ONLY}" = "true" ]; then
346+
run_eval --port "$PORT"
347+
else
348+
build_replay_cmd "$RESULT_DIR"
349+
run_agentic_replay_and_write_outputs "$RESULT_DIR"
350+
fi

‎configs/nvidia-master.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1767,6 +1767,29 @@ dsv4-fp4-b300-vllm-agentic:
17671767
# DEP8 LMCache (vs GPU-resident DEP8)
17681768
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [72, 104, 120, 136, 152, 168, 184, 200], router: { name: vllm-router, version: "0.1.14" } }
17691769

1770+
dsv4-fp4-b300-vllm-agentic-mtp:
1771+
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec
1772+
model: deepseek-ai/DeepSeek-V4-Pro
1773+
model-prefix: dsv4
1774+
runner: cluster:b300-nv
1775+
precision: fp4
1776+
framework: vllm
1777+
multinode: false
1778+
scenarios:
1779+
agentic-coding:
1780+
- dram-utilization: 0.80
1781+
search-space:
1782+
# TP8 GPU-resident + MTP (num_speculative_tokens=3)
1783+
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 6, 8] }
1784+
# TP4 GPU-resident + MTP (num_speculative_tokens=3)
1785+
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 6, 8, 12, 16, 20] }
1786+
# TP4 SimpleCPU + MTP (num_speculative_tokens=3)
1787+
- { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [20, 24, 28, 32, 36, 40] }
1788+
# DEP4 SimpleCPU + MTP (num_speculative_tokens=3)
1789+
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [32, 40, 48, 56], router: { name: vllm-router, version: "0.1.14" } }
1790+
# DEP8 SimpleCPU + MTP (num_speculative_tokens=3)
1791+
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 96, 112, 128, 144, 160, 176, 192, 224], router: { name: vllm-router, version: "0.1.14" } }
1792+
17701793
dsv4-fp4-b300-trt:
17711794
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066
17721795
model: deepseek-ai/DeepSeek-V4-Pro

‎perf-changelog.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5003,3 +5003,11 @@
50035003
- "LMCache MP server (lmcache_driven transfer mode) + LMCacheMPConnector per PR #2153; L1 pool derated to 75% of TOTAL_CPU_DRAM_GB; --enable-cumem-allocator dropped on the lmcache arm only (cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache server)"
50045004
- "LMCache TP8 points mirror the vllm-simple ladder (conc [8, 12, 16]); LMCache DEP8 points mirror the Mooncake ladder (conc [12, 20, 28, 36, 44, 52, 60, 68, 76]) for a like-for-like backend comparison; the PR #2231 bring-up sweep validated the recipe across DEP8 conc 8-80 (peak ~25.7k total tok/s/GPU at conc 72, 96-98% cache hit)"
50055005
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2231
5006+
5007+
- config-keys:
5008+
- dsv4-fp4-b300-vllm-agentic-mtp
5009+
description:
5010+
- "Add MTP speculative-decoding for the B300 vLLM AgentX recipe as a separate config-key, so the existing dsv4-fp4-b300-vllm-agentic aggregate is not re-run."
5011+
- "dsv4-fp4-b300-vllm-agentic-mtp: MTP twins (--speculative-config {\"method\":\"mtp\",\"num_speculative_tokens\":3}) of the aggregate arms plus a new TP8 GPU-resident arm -- TP8 GPU-resident [1,2,4,6,8], TP4 GPU-resident [1,2,4,6,8,12,16,20], TP4 SimpleCPU [20,24,28,32,36,40], DEP4 [32,40,48,56], DEP8 [64,96,112,128,144,160,176,192,224]; routed via spec-decoding=mtp to dsv4_fp4_b300_vllm_mtp.sh (FULL_DECODE_ONLY cudagraph capture sizes in TOKENS = num_seqs*4)."
5012+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2258
5013+

0 commit comments

Comments
 (0)