[LMCache] Add LMCache arm on tuned DSV4 FP4 B300 vLLM AgentX recipe / 在调优后的 DSV4 FP4 B300 vLLM AgentX 配方上新增 LMCache 分支 (#2232)

ApostaC · claude · cquil11 · web-flow · commit 9ecc56d25533 · 2026-07-21T10:19:59.000-05:00
* feat(agentic): add LMCache arm to tuned DSV4 FP4 B300 vLLM AgentX recipe Combine the tuned B300 recipe from PR #2225 (nightly image, sparse DSV4 FlashInfer attention, FULL_DECODE_ONLY CUDA graphs, AMXF4 mega-MoE) with the LMCache 0.5.1 KV-offload backend from PR #2153, mirroring the B200 arm in PR #2231. The lmcache arm keeps the stock PyTorch caching allocator instead of PYTORCH_ALLOC_CONF=expandable_segments:True (expandable-segment cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache MP server, the same failure mode as --enable-cumem-allocator on B200) and runs otherwise identical serving flags, so backends are directly comparable. Adds a standalone dsv4-fp4-b300-vllm-agentic-lmcache config section mirroring the vllm-simple DEP4/DEP8 concurrency ladders and a perf-changelog entry triggering it. 中文：将 PR #2225 的 B300 调优配方（nightly 镜像、稀疏 DSV4 FlashInfer 注意力、FULL_DECODE_ONLY CUDA graph、AMXF4 mega-MoE）与 PR #2153 的 LMCache 0.5.1 KV 卸载后端合并，对应 B200 的 PR #2231。lmcache 分支仅以 默认 PyTorch 缓存分配器替代 PYTORCH_ALLOC_CONF=expandable_segments:True （expandable segments 的 cuMem/VMM 分配无法通过 CUDA IPC 导出给 LMCache MP server，与 B200 上 --enable-cumem-allocator 的失败模式相同），其余 serving 参数与其他分支保持一致，便于卸载后端间直接对比。新增独立的 dsv4-fp4-b300-vllm-agentic-lmcache 配置（测试点与 vllm-simple 的 DEP4/DEP8 阶梯对齐）及触发它的 perf-changelog 条目。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fill in perf-changelog pr-link with PR #2232 中文：将 perf-changelog 条目的 pr-link 填写为 PR #2232。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(agentic): tier lmcache GPU mem util and add pure-TP lmcache arms Bring-up sweep on PR #2232 (run 29463061871) showed the lmcache arm has <1 GiB GPU headroom at the recipe's gpu-memory-utilization 0.96 (LMCache MP server GPU worker + DeepGEMM/TileLang JIT driver allocations + no expandable segments): every DEP8 point OOMed deterministically growing the torch pool, and 2/9 DEP4 points failed at the margin (JIT module load driver OOM at startup; cuBLAS workspace failure mid-run). Derate DEP8 to the B200-proven 0.92 and DEP4/pure-TP to 0.94. Also add pure-TP LMCache arms (TP8 and TP4, conc [8, 12, 16]) mirroring the upper end of the parent's GPU-resident TP ladders and the B200 lmcache TP8 ladder, for a direct GPU-cache vs LMCache-offload comparison. 中文：PR #2232 的调试扫描（run 29463061871）显示 lmcache 分支在配方的 gpu-memory-utilization 0.96 下 GPU 余量不足 1 GiB（LMCache MP server 的 GPU worker + DeepGEMM/TileLang JIT 驱动层分配 + 无 expandable segments）：所有 DEP8 测试点在扩展 torch 内存池时必然 OOM，DEP4 有 2/9 测试点在边缘失败（启动时 JIT 模块加载驱动 OOM；运行中 cuBLAS workspace 失败）。将 DEP8 降额至 B200 验证过的 0.92，DEP4 与纯 TP 降额至 0.94。 另新增纯 TP 的 LMCache 分支（TP8 与 TP4，并发 [8, 12, 16]），与父配置 GPU 驻留 TP 阶梯的高段及 B200 lmcache TP8 阶梯对齐，便于 GPU 缓存与 LMCache 卸载的直接对比。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(matrix): accept agentic SWE-bench eval rows in the changelog matrix schema PR #1947 made process_changelog's eval pass select single-node agentic (SWE-bench) eval rows via mark_eval_entries --evals-only, but ChangelogMatrixEntry.evals still only accepted SingleNodeMatrixEntry, so any changelog entry touching an agentic-coding config now fails validation (missing isl/osl, extra kv-offloading/scenario-type/duration). Accept Union[SingleNodeMatrixEntry, SingleNodeAgenticMatrixEntry] in evals and add run-eval/eval-only (default false) to the agentic entry model, with a regression test. 中文：PR #1947 使 process_changelog 的评估生成步骤通过 mark_eval_entries --evals-only 选择单节点 agentic（SWE-bench）评估行，但 ChangelogMatrixEntry.evals 仍只接受 SingleNodeMatrixEntry，导致任何涉及 agentic-coding 配置的变更日志条目都无法通过校验（缺少 isl/osl，多出 kv-offloading/scenario-type/duration 字段）。将 evals 改为接受 Union[SingleNodeMatrixEntry, SingleNodeAgenticMatrixEntry]，并为 agentic 条目模型添加 run-eval/eval-only（默认 false）字段，附回归测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "fix(matrix): accept agentic SWE-bench eval rows in the changelog matrix schema" This reverts commit bdc33bab6a72a0847c26be0dbe4e10bd8fe4a2eb to keep this PR scoped to the B300 LMCache recipe. The PR #1947 schema regression (agentic changelog entries fail matrix validation) needs a standalone fix PR; until that merges, check-changelog on this PR is expected to fail. 中文：回退 bdc33ba，使本 PR 仅保留 B300 LMCache 配方相关改动。PR #1947 造成的 schema 回归（agentic 变更日志条目无法通过矩阵校验）需通过独立 PR 修复；在该修复合并前，本 PR 的 check-changelog 预期会失败。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(agentic): fold B300 lmcache arms into the official agentic config Per review: run the LMCache sweep together with the official dsv4-fp4-b300-vllm-agentic config instead of a separate section. The standalone dsv4-fp4-b300-vllm-agentic-lmcache config is removed; the parent now carries one lmcache arm per official arm at a +4 conc offset (TP4 [32, 36, 40, 44]; DEP4 [36, 44, 52, 60, 68, 76]; DEP8 [68, 100, 116, 132, 148, 164, 180, 196, 228]), all inside the ranges validated in the PR #2232 bring-up sweeps. The changelog entry now triggers only the parent config. 中文：按评审意见，LMCache 与官方 dsv4-fp4-b300-vllm-agentic 配置在同一 扫描中运行，不再单列配置段。移除独立的 dsv4-fp4-b300-vllm-agentic-lmcache 配置；父配置为每个官方分支各增加一个 +4 并发偏移的 lmcache 分支（TP4 [32, 36, 40, 44]；DEP4 [36, 44, 52, 60, 68, 76]；DEP8 [68, 100, 116, 132, 148, 164, 180, 196, 228]），全部处于 PR #2232 调试扫描验证过的范围内。变更日志条目现仅触发 父配置。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(agentic): DEP8 lmcache ladder at +16 conc offset capped at 208 Per review: offset the DEP8 lmcache arm by +16 conc from the official GPU-resident DEP8 ladder and cap the maximum at 208, giving [80, 112, 128, 144, 160, 176, 192, 208]. TP4/DEP4 keep the +4 offset. 中文：按评审意见，DEP8 lmcache 分支相对官方 GPU 驻留 DEP8 阶梯偏移 +16 并发，最大并发限制为 208，即 [80, 112, 128, 144, 160, 176, 192, 208]。 TP4/DEP4 保持 +4 偏移。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(agentic): DEP8 lmcache ladder at +8 conc offset capped at 208 Correct the DEP8 lmcache offset from +16 to +8 (still capped at 208), giving [72, 104, 120, 136, 152, 168, 184, 200]. 中文：将 DEP8 lmcache 分支的并发偏移从 +16 更正为 +8（最大并发仍限制为 208），即 [72, 104, 120, 136, 152, 168, 184, 200]。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update perf-changelog.yaml --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cameron Quilici <cjquilici@gmail.com>
diff --git a/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh b/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh
@@ -3,7 +3,8 @@ set -eo pipefail
 set -x
 
 # Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on B300 using vLLM.
-# v4pro-b300.yaml TP4, DEP4, and DEP8 recipe. SimpleCPUOffload / MooncakeStore
+# v4pro-b300.yaml TP4, DEP4, and DEP8 recipe. SimpleCPUOffload /
+# MooncakeStore / LMCache
 #
 # Image is configured in nvidia-master.yaml. The recipe uses FP8 KV cache,
 # sparse DeepSeek-V4 FlashInfer attention with an FP4 indexer cache, mega-MoE,
@@ -12,8 +13,9 @@ set -x
 # Required env vars:
 #   MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
 #
-# TP4, TP8, and DEP8 (TP8 + DP-attention) are GPU-resident (KV_OFFLOADING=none).
-# DEP4 uses KV_OFFLOADING=dram with KV_OFFLOAD_BACKEND=vllm-simple or mooncake.
+# GPU-resident arms (TP4 and DEP8) use KV_OFFLOADING=none. DRAM offload
+# arms (TP and DEP) use KV_OFFLOADING=dram with
+# KV_OFFLOAD_BACKEND=vllm-simple, mooncake, or lmcache.
 
 source "$(dirname "$0")/../../benchmark_lib.sh"
 
@@ -33,8 +35,8 @@ if [ "$DP_ATTENTION" = "true" ] && [ $((2 * CONC % TP)) -ne 0 ]; then
     exit 1
 fi
 
-# DEP8 (TP8 + DP-attention) is a GPU-resident, high-concurrency arm that is
-# tuned separately from the smaller DEP4 arm (larger prefill token budget,
+# DEP8 (TP8 + DP-attention) is a high-concurrency arm that is tuned
+# separately from the smaller DEP4 arm (larger prefill token budget,
 # long-prefill chunking, and a lower GPU-memory-utilization headroom).
 IS_DEP8=false
 if [ "$DP_ATTENTION" = "true" ] && [ "$TP" -eq 8 ]; then
@@ -89,11 +91,13 @@ export VLLM_USE_RUST_FRONTEND=1
 SERVER_LOG="$RESULT_DIR/server.log"
 ROUTER_LOG="$RESULT_DIR/router.log"
 MOONCAKE_MASTER_LOG="$RESULT_DIR/mooncake_master.log"
+LMCACHE_SERVER_LOG="$RESULT_DIR/lmcache_server.log"
 mkdir -p "$RESULT_DIR"
 
 SERVER_PID=""
 ROUTER_PID=""
 MOONCAKE_MASTER_PID=""
+LMCACHE_SERVER_PID=""
 
 # The generated TOTAL_CPU_DRAM_GB budget is proportional to allocated GPUs.
 # On cluster:b300-nv, dram-utilization=0.80 and DEP4 resolve to roughly the
@@ -186,6 +190,89 @@ EOF
         OFFLOAD_CONFIG='{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
         OFFLOAD_ARGS=(--kv-transfer-config "$OFFLOAD_CONFIG")
         ;;
+    lmcache)
+        require_agentic_kv_offload_backend lmcache
+        # The LMCache MP server owns the host-DRAM KV pool as one shared
+        # tier; vLLM ranks attach via LMCacheMPConnector, so the aggregate
+        # host budget is passed through undivided (unlike Mooncake's
+        # per-rank segments). Follows the LMCache DeepSeek-V4 recipe
+        # (docs.lmcache.ai/recipes/deepseek_v4_flash.html); LMCache handles
+        # DSV4's Sparse-MLA hybrid KV geometries automatically.
+        LMCACHE_VERSION=0.5.1
+        agentic_pip_install --quiet --no-cache-dir "lmcache==$LMCACHE_VERSION"
+        python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null
+
+        LMCACHE_HOST=127.0.0.1
+        LMCACHE_PORT=$((PORT + 12000))
+        LMCACHE_HTTP_PORT=$((PORT + 13000))
+        # LMCacheMPConnector concatenates lmcache.mp.host and port into the
+        # ZMQ endpoint. Bind the server to a raw host, but pass the connector
+        # a ZMQ-style host string.
+        LMCACHE_CONNECT_HOST="tcp://$LMCACHE_HOST"
+        # Pool target derated to 75% of the aggregate budget: pinned host
+        # memory is unswappable and also consumes GPU-side mapping
+        # resources, so leave headroom for vLLM host buffers and the OS.
+        # Full-budget targets OOM-killed the node (host OOM-killer or
+        # cudaErrorMemoryAllocation) as the cache filled past ~2 TB during
+        # PR #2153 bring-up.
+        LMCACHE_L1_SIZE_GB=$((TOTAL_CPU_DRAM_GB * 3 / 4))
+        # The pool grows lazily from the initial allocation, so the full
+        # --l1-size-gb target is not pinned at startup.
+        LMCACHE_L1_INIT_SIZE_GB=20
+        LMCACHE_MQ_TIMEOUT=300
+        # Identical prefixes must hash to identical cache keys across DP ranks.
+        export PYTHONHASHSEED=0
+        # Per-engine scheduler stats every 5s, to diagnose per-DP-rank KV
+        # cache imbalance under the session-sticky router.
+        export VLLM_LOG_STATS_INTERVAL=5
+
+        echo "Starting LMCache MP server on port $LMCACHE_PORT..."
+        # One GPU-side transfer worker avoids concurrent-GPU-transfer stalls
+        # under heavy async-load pressure; CPU-side workers stay at 8.
+        lmcache server \
+            --host "$LMCACHE_HOST" \
+            --port "$LMCACHE_PORT" \
+            --http-host "$LMCACHE_HOST" \
+            --http-port "$LMCACHE_HTTP_PORT" \
+            --l1-size-gb "$LMCACHE_L1_SIZE_GB" \
+            --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" \
+            --max-gpu-workers 1 \
+            --max-cpu-workers 8 \
+            --chunk-size 1024 \
+            --l1-align-bytes 16384 \
+            --eviction-trigger-watermark 0.85 \
+            --eviction-ratio 0.10 \
+            --eviction-policy LRU \
+            --supported-transfer-mode lmcache_driven \
+            --no-separate-object-groups \
+            > "$LMCACHE_SERVER_LOG" 2>&1 &
+        LMCACHE_SERVER_PID=$!
+        LMCACHE_READY=0
+        for _ in $(seq 1 60); do
+            if ! kill -0 "$LMCACHE_SERVER_PID" 2>/dev/null; then
+                echo "LMCache server died during startup." >&2
+                cat "$LMCACHE_SERVER_LOG" >&2
+                exit 1
+            fi
+            if curl --output /dev/null --silent --fail \
+                "http://127.0.0.1:$LMCACHE_HTTP_PORT/healthcheck"; then
+                LMCACHE_READY=1
+                break
+            fi
+            sleep 2
+        done
+        if [ "$LMCACHE_READY" -ne 1 ]; then
+            echo "LMCache server did not become healthy in time." >&2
+            cat "$LMCACHE_SERVER_LOG" >&2
+            exit 1
+        fi
+
+        unset VLLM_USE_SIMPLE_KV_OFFLOAD
+        OFFLOAD_ARGS=(
+            --kv-transfer-config
+            "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"$LMCACHE_CONNECT_HOST\",\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}"
+        )
+        ;;
     *)
         echo "Error: unsupported B300 KV_OFFLOAD_BACKEND='$KV_OFFLOAD_BACKEND'" >&2
         exit 1
@@ -199,7 +286,15 @@ fi
 
 TP_ARGS=()
 if [ "$DP_ATTENTION" = "true" ]; then
-    export PYTORCH_ALLOC_CONF=expandable_segments:True
+    # LMCacheMPConnector exports the KV cache to the LMCache server through
+    # legacy CUDA IPC handles, and expandable-segment (cuMem/VMM) allocations
+    # cannot be exported that way (register_kv_caches fails with
+    # cudaErrorInvalidValue, same failure mode as --enable-cumem-allocator on
+    # the B200 lmcache arm in PR #2231), so the lmcache arm keeps the stock
+    # caching allocator.
+    if [ "$KV_OFFLOAD_BACKEND" != "lmcache" ]; then
+        export PYTORCH_ALLOC_CONF=expandable_segments:True
+    fi
 else
     export VLLM_ALLREDUCE_USE_FLASHINFER=1
     export VLLM_FLASHINFER_ALLREDUCE_BACKEND=auto
@@ -217,8 +312,8 @@ fi
 if [ "$DP_ATTENTION" = "true" ]; then
     MODE_ARGS+=(--prefill-schedule-interval 8)
     if [ "$IS_DEP8" = "true" ]; then
-        # GPU-resident DEP8 gets a larger prefill token budget and chunks long
-        # prefills so decode latency stays bounded at high concurrency.
+        # DEP8 gets a larger prefill token budget and chunks long prefills
+        # so decode latency stays bounded at high concurrency.
         MODE_ARGS+=(
             --max-num-batched-tokens 16384
             --long-prefill-token-threshold 4096
@@ -254,6 +349,16 @@ GPU_MEM_UTIL=0.96
 if [ "$IS_DEP8" = "true" ]; then
     GPU_MEM_UTIL=0.92
 fi
+# The lmcache arm needs extra headroom on the other topologies too: the
+# LMCache MP server keeps a GPU worker (~0.8 GiB CUDA context + staging
+# buffers) on every GPU and the arm cannot run expandable segments (the KV
+# cache must stay legacy-CUDA-IPC-exportable), so at 0.96 the PR #2232
+# bring-up sweep hit DeepGEMM workspace/JIT-loader OOMs. 0.94 was validated
+# for lmcache TP and DEP4 in the same sweep; lmcache DEP8 keeps the 0.92
+# above (also validated).
+if [ "$KV_OFFLOAD_BACKEND" = "lmcache" ] && [ "$IS_DEP8" != "true" ]; then
+    GPU_MEM_UTIL=0.94
+fi
 
 { set +x; } 2>/dev/null
 VLLM_CMD=(
diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml
@@ -1753,6 +1753,17 @@ dsv4-fp4-b300-vllm-agentic:
       - { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [32, 40, 48, 56, 64, 72], router: { name: vllm-router, version: "0.1.14" } }
       # DEP8 SimpleCPU
       - { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [64, 96, 112, 128, 144, 160, 176, 192, 224], router: { name: vllm-router, version: "0.1.14" } }
+      # LMCache arms: offset each official ladder above so the backends
+      # sample distinct points on the same recipe — TP4/DEP4 at +4 conc,
+      # DEP8 at +8 conc capped at 208. Bring-up validated in the PR #2232
+      # sweeps (runs 29463061871 and 29535333851): TP/DEP4 at
+      # gpu-memory-utilization 0.94, DEP8 at 0.92.
+      # TP4 LMCache (vs TP4 SimpleCPU)
+      - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [32, 36, 40, 44] }
+      # DEP4 LMCache (vs DEP4 SimpleCPU)
+      - { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }
+      # DEP8 LMCache (vs GPU-resident DEP8)
+      - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [72, 104, 120, 136, 152, 168, 184, 200], router: { name: vllm-router, version: "0.1.14" } }
 
 dsv4-fp4-b300-trt:
   image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066
diff --git a/perf-changelog.yaml b/perf-changelog.yaml
@@ -4982,3 +4982,14 @@
     - "DEP arm runs conc-list [48] only (was [48, 64, 96, 128, 192, 256, 512]): conc 48 is the frontier peak (163 tok/s/gpu out, ttft 2.8s); conc >=64 overflows the HBM+host radix cache (GPU hit 0.93->0.57 at 48->64) and thrashes on re-prefill, so it is strictly dominated"
     - "Low-latency TP8 arm (conc [1, 2, 4, 8, 16, 32], fp8 KV + cutedsl bf16 GEMM) unchanged; DEP mirrors the proven dsv4-fp4-b300-sglang tp8/ep8/dp-attn DEP arm"
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2281
+
+- config-keys:
+    - dsv4-fp4-b300-vllm-agentic
+  scenario-type:
+    - agentic-coding
+  description:
+    - "Add LMCache 0.5.1 DRAM KV-offload arms to the official B300 AgentX config from PR #2241 (sparse DSV4 FlashInfer attention, AMXF4 mega-MoE, FULL_DECODE_ONLY CUDA graphs, image vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec), so LMCache is benchmarked in the same sweep as the SimpleCPU and GPU-resident arms"
+    - "LMCache MP server (lmcache_driven transfer mode) + LMCacheMPConnector per PR #2153/#2231; L1 pool derated to 75% of TOTAL_CPU_DRAM_GB; PYTORCH_ALLOC_CONF=expandable_segments:True dropped on the lmcache arms only (VMM allocations cannot be CUDA-IPC-exported to the LMCache server, same failure mode as cuMem on B200); gpu-memory-utilization 0.92 on lmcache DEP8 (matches the official DEP8 derate) and 0.94 on lmcache TP4/DEP4 (at 0.96 the PR #2232 bring-up sweeps hit torch-pool, DeepGEMM-JIT, and cuBLAS-workspace OOMs; 23 lmcache points validated green in runs 29463061871/29535333851 after the derate)"
+    - "LMCache ladders offset the official arms: TP4 +4 conc [32, 36, 40, 44] vs SimpleCPU [28, 32, 36, 40]; DEP4 +4 conc [36, 44, 52, 60, 68, 76] vs SimpleCPU [32, 40, 48, 56, 64, 72]; DEP8 +8 conc capped at 208 [72, 104, 120, 136, 152, 168, 184, 200] vs GPU-resident [64, 96, 112, 128, 144, 160, 176, 192, 224]"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2232
+  
