[LMCache] Add LMCache arm to official DSV4 FP4 B200 vLLM AgentX sweep / 将 LMCache 分支并入官方 DSV4 FP4 B200 vLLM AgentX 扫描 (#2231)

* feat(agentic): add LMCache arm to official DSV4 FP4 B200 vLLM AgentX sweep

Rebased onto main after PR #2224 merged the tuned B200 recipe. The
LMCache points now live directly in the official
dsv4-fp4-b200-vllm-agentic search space alongside the vllm-simple and
Mooncake arms (no standalone config section). LMCache 0.5.1 MP server
(lmcache_driven transfer mode) + LMCacheMPConnector per PR #2153; the
lmcache arm drops --enable-cumem-allocator because cuMem/VMM
allocations cannot be CUDA-IPC-exported to the LMCache server. Ladder
validated in the PR #2231 bring-up sweep: peak ~25.7k total tok/s/GPU
at DEP8 conc 72 with 96-98% cache hit.

中文：在 PR #2224 的调优配方合并后 rebase 到 main。LMCache 测试点直接并入
官方 dsv4-fp4-b200-vllm-agentic 搜索空间，与 vllm-simple、Mooncake 分支
并列（不再使用独立配置段）。LMCache 0.5.1 MP server（lmcache_driven
传输模式）+ LMCacheMPConnector（沿用 PR #2153）；lmcache 分支去掉
--enable-cumem-allocator（cuMem/VMM 分配无法通过 CUDA IPC 导出给
LMCache server）。测试点阶梯已在 PR #2231 调试扫描中验证：DEP8 并发 72
达到峰值约 25.7k total tok/s/GPU，缓存命中率 96–98%。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(agentic): align LMCache DEP8 conc-list with the Mooncake ladder

Change the LMCache DEP8 points from the vllm-simple ladder to the
Mooncake ladder ([12, 20, 28, 36, 44, 52, 60, 68, 76]) so the two
external KV stores are benchmarked at identical concurrency points.
TP8 stays on [8, 12, 16] (Mooncake has no TP8 arm).

中文：将 LMCache DEP8 测试点从 vllm-simple 阶梯改为 Mooncake 阶梯
（[12, 20, 28, 36, 44, 52, 60, 68, 76]），使两个外部 KV 存储在完全相同的
并发点上进行基准测试。TP8 保持 [8, 12, 16]（Mooncake 无 TP8 分支）。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cam Quilici <cjquilici@gmail.com>

Author: 3 people

benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh

@@ -21,7 +21,8 @@ set -x
 #   MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
 #
 # Pure TP is GPU-resident (KV_OFFLOADING=none). DEP tiers offload KV to host
-# DRAM: KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-simple or mooncake.
+# DRAM: KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-simple, mooncake,
+# or lmcache.
 
 source "$(dirname "$0")/../../benchmark_lib.sh"
 
@@ -101,13 +102,18 @@ export VLLM_RPC_TIMEOUT=600000
 SERVER_LOG="$RESULT_DIR/server.log"
 ROUTER_LOG="$RESULT_DIR/router.log"
 MOONCAKE_MASTER_LOG="$RESULT_DIR/mooncake_master.log"
+LMCACHE_SERVER_LOG="$RESULT_DIR/lmcache_server.log"
 mkdir -p "$RESULT_DIR"
 
 SERVER_PID=""
 ROUTER_PID=""
 MOONCAKE_MASTER_PID=""
+LMCACHE_SERVER_PID=""
 
 OFFLOAD_ARGS=()
+# vLLM's cuMem/VMM allocator is the recipe default; the lmcache arm clears
+# this (see the lmcache case below).
+CUMEM_ARGS=(--enable-cumem-allocator)
 case "$KV_OFFLOAD_BACKEND" in
     "")
         require_agentic_kv_offload_none
@@ -194,6 +200,94 @@ EOF
             '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
         )
         ;;
+    lmcache)
+        require_agentic_kv_offload_backend lmcache
+        # The LMCache MP server owns the host-DRAM KV pool as one shared
+        # tier; vLLM ranks attach via LMCacheMPConnector, so the aggregate
+        # host budget is passed through undivided (unlike Mooncake's
+        # per-rank segments). Follows the LMCache DeepSeek-V4 recipe
+        # (docs.lmcache.ai/recipes/deepseek_v4_flash.html); LMCache handles
+        # DSV4's Sparse-MLA hybrid KV geometries automatically.
+        LMCACHE_VERSION=0.5.1
+        agentic_pip_install --quiet --no-cache-dir "lmcache==$LMCACHE_VERSION"
+        python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null
+
+        LMCACHE_HOST=127.0.0.1
+        LMCACHE_PORT=$((PORT + 12000))
+        LMCACHE_HTTP_PORT=$((PORT + 13000))
+        # LMCacheMPConnector concatenates lmcache.mp.host and port into the
+        # ZMQ endpoint. Bind the server to a raw host, but pass the connector
+        # a ZMQ-style host string.
+        LMCACHE_CONNECT_HOST="tcp://$LMCACHE_HOST"
+        # Pool target derated to 75% of the aggregate budget: pinned host
+        # memory is unswappable and also consumes GPU-side mapping
+        # resources, so leave headroom for vLLM host buffers and the OS.
+        # Full-budget targets OOM-killed the node (host OOM-killer or
+        # cudaErrorMemoryAllocation) as the cache filled past ~2 TB during
+        # PR #2153 bring-up.
+        LMCACHE_L1_SIZE_GB=$((TOTAL_CPU_DRAM_GB * 3 / 4))
+        # The pool grows lazily from the initial allocation, so the full
+        # --l1-size-gb target is not pinned at startup.
+        LMCACHE_L1_INIT_SIZE_GB=20
+        LMCACHE_MQ_TIMEOUT=300
+        # Identical prefixes must hash to identical cache keys across DP ranks.
+        export PYTHONHASHSEED=0
+        # LMCacheMPConnector exports the KV cache to the LMCache server
+        # through legacy CUDA IPC handles, and cuMem/VMM allocations cannot
+        # be exported that way (register_kv_caches fails with
+        # cudaErrorInvalidValue), so drop the allocator for this arm only.
+        CUMEM_ARGS=()
+        # Per-engine scheduler stats every 5s, to diagnose per-DP-rank KV
+        # cache imbalance under the session-sticky router.
+        export VLLM_LOG_STATS_INTERVAL=5
+
+        echo "Starting LMCache MP server on port $LMCACHE_PORT..."
+        # One GPU-side transfer worker avoids concurrent-GPU-transfer stalls
+        # under heavy async-load pressure; CPU-side workers stay at 8.
+        lmcache server \
+            --host "$LMCACHE_HOST" \
+            --port "$LMCACHE_PORT" \
+            --http-host "$LMCACHE_HOST" \
+            --http-port "$LMCACHE_HTTP_PORT" \
+            --l1-size-gb "$LMCACHE_L1_SIZE_GB" \
+            --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" \
+            --max-gpu-workers 1 \
+            --max-cpu-workers 8 \
+            --chunk-size 1024 \
+            --l1-align-bytes 16384 \
+            --eviction-trigger-watermark 0.85 \
+            --eviction-ratio 0.10 \
+            --eviction-policy LRU \
+            --supported-transfer-mode lmcache_driven \
+            --no-separate-object-groups \
+            > "$LMCACHE_SERVER_LOG" 2>&1 &
+        LMCACHE_SERVER_PID=$!
+        LMCACHE_READY=0
+        for _ in $(seq 1 60); do
+            if ! kill -0 "$LMCACHE_SERVER_PID" 2>/dev/null; then
+                echo "LMCache server died during startup." >&2
+                cat "$LMCACHE_SERVER_LOG" >&2
+                exit 1
+            fi
+            if curl --output /dev/null --silent --fail \
+                "http://127.0.0.1:$LMCACHE_HTTP_PORT/healthcheck"; then
+                LMCACHE_READY=1
+                break
+            fi
+            sleep 2
+        done
+        if [ "$LMCACHE_READY" -ne 1 ]; then
+            echo "LMCache server did not become healthy in time." >&2
+            cat "$LMCACHE_SERVER_LOG" >&2
+            exit 1
+        fi
+
+        unset VLLM_USE_SIMPLE_KV_OFFLOAD
+        OFFLOAD_ARGS=(
+            --kv-transfer-config
+            "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"$LMCACHE_CONNECT_HOST\",\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}"
+        )
+        ;;
     *)
         echo "Error: unsupported B200 KV_OFFLOAD_BACKEND='$KV_OFFLOAD_BACKEND'" >&2
         exit 1
@@ -237,7 +331,7 @@ VLLM_CMD=(
     --max-model-len 1048576
     --gpu-memory-utilization 0.92
     --numa-bind
-    --enable-cumem-allocator
+    "${CUMEM_ARGS[@]}"
     --no-enable-flashinfer-autotune
     --tokenizer-mode deepseek_v4
     --reasoning-parser deepseek_v4

configs/nvidia-master.yaml

@@ -960,9 +960,11 @@ dsv4-fp4-b200-vllm-agentic:
       # Pure TP at low concurrency.
       - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8] }
       - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 12, 16] }
+      - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [8, 12, 16] }
       # DEP
       - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } }
       - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }
+      - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }
 
 dsv4-fp4-b200-trt:
   image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066

perf-changelog.yaml

@@ -4993,3 +4993,13 @@
     - "LMCache ladders offset the official arms: TP4 +4 conc [32, 36, 40, 44] vs SimpleCPU [28, 32, 36, 40]; DEP4 +4 conc [36, 44, 52, 60, 68, 76] vs SimpleCPU [32, 40, 48, 56, 64, 72]; DEP8 +8 conc capped at 208 [72, 104, 120, 136, 152, 168, 184, 200] vs GPU-resident [64, 96, 112, 128, 144, 160, 176, 192, 224]"
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2232
 
+
+- config-keys:
+    - dsv4-fp4-b200-vllm-agentic
+  scenario-type:
+    - agentic-coding
+  description:
+    - "Add LMCache 0.5.1 DRAM KV-offload arm to the official B200 AgentX sweep alongside the vllm-simple and Mooncake arms, on the same tuned recipe and image (vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec) from PR #2224"
+    - "LMCache MP server (lmcache_driven transfer mode) + LMCacheMPConnector per PR #2153; L1 pool derated to 75% of TOTAL_CPU_DRAM_GB; --enable-cumem-allocator dropped on the lmcache arm only (cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache server)"
+    - "LMCache TP8 points mirror the vllm-simple ladder (conc [8, 12, 16]); LMCache DEP8 points mirror the Mooncake ladder (conc [12, 20, 28, 36, 44, 52, 60, 68, 76]) for a like-for-like backend comparison; the PR #2231 bring-up sweep validated the recipe across DEP8 conc 8-80 (peak ~25.7k total tok/s/GPU at conc 72, 96-98% cache hit)"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2231