Revert DSv4 B200/B300 TRT (non-MTP) to 2dd03e6 + disable SWA scratch reuse

The c914d6d image's kv_cache_manager_v2 patch was wrong: freeing SWA scratch
slots on the attention-DP revert->resize(shrink) path hits finish_event=None
(a deferred request never forwarded), crashing every dpa=true job and hanging
the engine. Root cause is a V2-scheduler / SWA-scratch-reuse conflict: the V2
scheduler grows a context request's KV cache (incl. SWA scratch) before delay
batching can defer it, so revert_allocate_context -> resize(shrink) must release
scratch slots that have no finish_event.

Revert both non-MTP images to feat-deepseek_v4-2dd03e6 and set
TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 in the launchers so no scratch slots are
allocated and the revert shrinks cleanly. MTP configs untouched (9aa3715).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: Oseltamivir

.github/configs/nvidia-master.yaml

@@ -1801,7 +1801,7 @@ dsv4-fp4-b200-vllm-agentic:
       - { tp: 8, ep: 8, dp-attn: true, offloading: cpu,  conc-list: [64, 128, 256] }
 
 dsv4-fp4-b200-trt:
-  image: ghcr.io#semianalysisai/trtllm-deepseek-v4:fix-dsv4-swa-scratch-revert-shrink-c914d6d
+  image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-2dd03e6
   model: deepseek-ai/DeepSeek-V4-Pro
   model-prefix: dsv4
   runner: b200-dsv4
@@ -3049,7 +3049,7 @@ dsv4-fp4-b300-vllm-agentic:
       - { tp: 8, ep: 8, dp-attn: true, offloading: cpu,  conc-list: [128, 256, 512] }
 
 dsv4-fp4-b300-trt:
-  image: ghcr.io#semianalysisai/trtllm-deepseek-v4:fix-dsv4-swa-scratch-revert-shrink-c914d6d
+  image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-2dd03e6
   model: deepseek-ai/DeepSeek-V4-Pro
   model-prefix: dsv4
   runner: b300

benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_trt.sh

@@ -47,6 +47,13 @@ sanitize_slurm_mpi_env_for_trtllm
 export NCCL_NVLS_ENABLE="${NCCL_NVLS_ENABLE:-0}"
 echo "NCCL_NVLS_ENABLE: $NCCL_NVLS_ENABLE"
 
+# Disable DSv4 SWA scratch reuse: with attention-DP the V2 scheduler grows ctx KV
+# (incl. SWA scratch) before delay batching defers a request, and the resulting
+# revert_allocate_context -> resize(shrink) can't release the scratch of a
+# never-forwarded request (no finish_event), crashing every dpa=true job.
+export TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-0}"
+echo "TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE: $TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE"
+
 if [[ "$MODEL" != /* ]]; then
     hf download "$MODEL"
 fi

benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt.sh

@@ -59,6 +59,13 @@ sanitize_slurm_mpi_env_for_trtllm
 export NCCL_NVLS_ENABLE="${NCCL_NVLS_ENABLE:-0}"
 echo "NCCL_NVLS_ENABLE: $NCCL_NVLS_ENABLE"
 
+# Disable DSv4 SWA scratch reuse: with attention-DP the V2 scheduler grows ctx KV
+# (incl. SWA scratch) before delay batching defers a request, and the resulting
+# revert_allocate_context -> resize(shrink) can't release the scratch of a
+# never-forwarded request (no finish_event), crashing every dpa=true job.
+export TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-0}"
+echo "TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE: $TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE"
+
 nvidia-smi
 
 SERVER_LOG="$PWD/server.log"

perf-changelog.yaml

@@ -3382,7 +3382,7 @@
     - dsv4-fp4-b200-trt
     - dsv4-fp4-b300-trt
   description:
-    - "Update the TensorRT-LLM DeepSeek-V4-Pro image to ghcr.io/semianalysisai/trtllm-deepseek-v4:fix-dsv4-swa-scratch-revert-shrink-c914d6d (TRT-LLM feat/deepseek_v4 @ 084cf2ba + kv_cache_manager_v2 fix: free SWA scratch slots on shrink instead of asserting, which crashed the engine on attention-DP context/generation reverts at high concurrency, e.g. b300 8k1k conc>=512)"
+    - "Revert the non-MTP TensorRT-LLM DeepSeek-V4-Pro image to ghcr.io/semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-2dd03e6 and disable DSv4 SWA scratch reuse via TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 in the launcher. Root cause of the prior attention-DP hangs/crashes: the V2 scheduler grows a context request's KV cache (incl. SWA scratch slots) before delay batching can defer it, so revert_allocate_context -> resize(shrink) must release scratch slots of a never-forwarded request, which has no finish_event -> crash on every dpa=true job. Disabling scratch reuse stops those slots from being allocated so the revert shrinks cleanly."
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1636
 
 - config-keys: