Skip to content

Use official TRT-LLM image (1.3.0rc15.post1) for DSv4 B300 TRT (non-MTP + MTP) #1636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Oseltamivir wants to merge 18 commits into
base: main
Choose a base branch
from
+26 −2
Open

Use official TRT-LLM image (1.3.0rc15.post1) for DSv4 B300 TRT (non-MTP + MTP) #1636

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4bcefb7
Update DSv4 TRT image for B200/B300 (non-MTP) to feat-deepseek_v4-2dd…
Oseltamivir Jun 1, 2026
6b7558c
Backfill PR number in changelog pr-link
Oseltamivir Jun 1, 2026
bd3c94c
Merge branch 'main' into update-dsv4-trt-image-2dd03e6
Oseltamivir Jun 1, 2026
f441f9f
Try official TRT-LLM release image 1.3.0rc15.post1 for DSv4 B200/B300…
Oseltamivir Jun 1, 2026
4bc5592
Revert to custom feat/deepseek_v4-2dd03e6 image for DSv4 B200/B300 (n…
Oseltamivir Jun 2, 2026
1b0afeb
Merge branch 'main' into update-dsv4-trt-image-2dd03e6
Oseltamivir Jun 2, 2026
14a1bb3
Point DSv4 B200/B300 TRT (non-MTP) at the SWA-scratch-fix image
Oseltamivir Jun 2, 2026
242ab88
Merge remote-tracking branch 'origin/main' into update-dsv4-trt-image…
Oseltamivir Jun 2, 2026
e23a541
Revert DSv4 B200/B300 TRT (non-MTP) to 2dd03e6 + disable SWA scratch …
Oseltamivir Jun 2, 2026
6118a76
Merge remote-tracking branch 'origin/main' into update-dsv4-trt-image…
Oseltamivir Jun 3, 2026
5adfeb3
Scope DSv4 TRT non-MTP change to B300 only
Oseltamivir Jun 3, 2026
ad529fb
Use official TRT-LLM image (1.3.0rc15.post1) for DSv4 B300 TRT (non-M…
Oseltamivir Jun 3, 2026
c2381b7
Re-add TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 on rc15.post1 to test D…
Oseltamivir Jun 5, 2026
b09619e
Merge branch 'main' into update-dsv4-trt-image-2dd03e6
Oseltamivir Jun 5, 2026
65bddbb
Re-enable SWA scratch reuse (confirmed not the cause of rc15.post1 DP…
Oseltamivir Jun 5, 2026
285b79a
Merge branch 'main' into update-dsv4-trt-image-2dd03e6
Oseltamivir Jun 5, 2026
30c340b
Switch B300 DSv4 TRT image to nvcr.io/nvidia/ai-dynamo/tensorrtllm-ru…
github-actions[bot] Jun 8, 2026
485aacb
Merge branch 'main' into update-dsv4-trt-image-2dd03e6
Oseltamivir Jun 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3049,7 +3049,7 @@ dsv4-fp4-b300-vllm-agentic:
- { tp: 8, ep: 8, dp-attn: true, offloading: cpu, conc-list: [128, 256, 512] }

dsv4-fp4-b300-trt:
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-9aa3715
image: nvcr.io#nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: b300
Expand All @@ -3072,7 +3072,7 @@ dsv4-fp4-b300-trt:
- { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 1024 }

dsv4-fp4-b300-trt-mtp:
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-9aa3715
image: nvcr.io#nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: b300
Expand Down
5 changes: 5 additions & 0 deletions benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@ sanitize_slurm_mpi_env_for_trtllm
export NCCL_NVLS_ENABLE="${NCCL_NVLS_ENABLE:-0}"
echo "NCCL_NVLS_ENABLE: $NCCL_NVLS_ENABLE"

# Disable DSv4 SWA scratch reuse to test whether the rc15.post1 DPA crash
# is the same SWA-scratch bug or a separate FMHA kernel issue.
export TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-1}"

@cursor cursor Bot Jun 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SWA default contradicts disable comment

Medium Severity

The new launcher comments say DSv4 SWA scratch reuse is disabled to isolate the rc15.post1 DPA crash, but TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE defaults to 1 when unset, which keeps scratch reuse enabled. Sweep runs without an override therefore do not match the stated experiment.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 65bddbb. Configure here.

echo "TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE: $TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE"

nvidia-smi

SERVER_LOG="$PWD/server.log"
Expand Down
5 changes: 5 additions & 0 deletions benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ sanitize_slurm_mpi_env_for_trtllm
export NCCL_NVLS_ENABLE="${NCCL_NVLS_ENABLE:-0}"
echo "NCCL_NVLS_ENABLE: $NCCL_NVLS_ENABLE"

# Disable DSv4 SWA scratch reuse to test whether the rc15.post1 DPA crash
# is the same SWA-scratch bug or a separate FMHA kernel issue.
export TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-1}"
echo "TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE: $TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE"

nvidia-smi

SERVER_LOG="$PWD/server.log"
Expand Down
14 changes: 14 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3445,6 +3445,13 @@
- "Add 1k1k/8k1k minimax recipe set under benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m2.5/"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1641

- config-keys:
- dsv4-fp4-b300-trt
- dsv4-fp4-b300-trt-mtp
description:
- "Switch B300 DSv4 TRT (non-MTP + MTP) to official rc15.post1 image with TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 to test whether DPA crash is same SWA-scratch bug"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1636

- config-keys:
- dsv4-fp4-b200-vllm
description:
Expand Down Expand Up @@ -3502,6 +3509,13 @@
- "Update GPT-OSS model for MI355X vLLM from amd/gpt-oss-120b-w-mxfp4-a-fp8 to openai/gpt-oss-120b"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1670

- config-keys:
- dsv4-fp4-b300-trt
- dsv4-fp4-b300-trt-mtp
description:
- "Switch B300 DSv4 TRT (non-MTP + MTP) image from rc15.post1 to nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1636

- config-keys:
- kimik2.5-fp4-b300-vllm
description:
Expand Down
Loading