GLM-5.2 NVFP4 B300 SGLang high-throughput arm to DEP8, cap conc at 64 (#2281)

Author: Oseltamivir

benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh

@@ -8,7 +8,8 @@ set -x
 # (https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2), STP only:
 # the cookbook's EAGLE MTP variants are intentionally not wired up yet.
 #   DP_ATTENTION=false -> low-latency arm (TP8, fp8 KV, cutedsl bf16 GEMM)
-#   DP_ATTENTION=true  -> high-throughput arm (TP8 + DP8 attention-DP)
+#   DP_ATTENTION=true  -> high-throughput DEP arm (TP8 + DP8 attention-DP +
+#                         EP_SIZE expert-parallel MoE via --ep-size)
 #
 # Required env vars:
 #   MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION,
@@ -105,11 +106,6 @@ if [ "$DP_ATTENTION" = "true" ]; then
     # while KV usage sat at ~0.01). Use the cookbook's own dp8 lever from
     # the B200 cells (32768 = ~4096/rank).
     CHUNKED_PREFILL_SIZE=32768
-    # At conc 512 the saturation working set outlives the default 1800s
-    # warmup drain grace: the drain converges healthily (~0.45 req/s, zero
-    # errors) but needs ~2500s end to end. 3600 is a maximum wait, not a
-    # fixed sleep — lower-conc DPA points still finish as fast as they drain.
-    export AGENTIC_WARMUP_GRACE_PERIOD=3600
     PARALLEL_ARGS+=(
         --dp "$TP"
         --enable-dp-attention

configs/nvidia-master.yaml

@@ -7858,9 +7858,10 @@ qwen3.5-fp8-gb300-dynamo-sglang:
 # single-node recipes (https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2),
 # STP only (the cookbook's EAGLE MTP variants are deferred). The TP8 arm is the
 # cookbook low-latency recipe (fp8 KV, cutedsl bf16 GEMM) and covers the
-# interactivity end; the TP8/DP8 attention-DP arm is the cookbook
-# high-throughput recipe behind sglang-router consistent hashing for session
-# affinity. Conc lists are disjoint between arms so exp-names stay unique.
+# interactivity end; the TP8/DP8/EP8 DEP arm (attention-DP + expert-parallel
+# MoE) is the cookbook high-throughput recipe behind sglang-router consistent
+# hashing for session affinity. Conc lists are disjoint between arms so
+# exp-names stay unique.
 glm5.2-fp4-b300-sglang-agentic:
   image: lmsysorg/sglang:v0.5.15.post1-cu130
   model: nvidia/GLM-5.2-NVFP4
@@ -7874,4 +7875,8 @@ glm5.2-fp4-b300-sglang-agentic:
     - dram-utilization: 0.80
       search-space:
       - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 2, 4, 8, 16, 32] }
-      - { tp: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [48, 64, 96, 128, 192, 256, 512], router: { name: sglang-router, version: "0.3.2" } }
+      # DEP (attention-DP + EP8 expert-parallel MoE) throughput arm at conc 48 — the
+      # frontier peak (163 tok/s/gpu out, ttft 2.8s). conc >=64 overflows the HBM+host
+      # radix cache (GPU hit 0.93->0.57 at 48->64) and thrashes on re-prefill, so it is
+      # strictly dominated by conc 48 on both throughput and interactivity.
+      - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [48], router: { name: sglang-router, version: "0.3.2" } }

perf-changelog.yaml

@@ -4974,3 +4974,11 @@
     - "Run 29651235293 showed the 1M-context corpus working set outgrowing the HBM KV pool past conc 8 (TP8) / conc 64 (DP8): gpu_kv_cache_usage pinned at 1.0 and the radix hit rate collapsed from a ~0.97 theoretical ceiling to 0.04-0.06, so every post-knee turn re-prefilled its full history and throughput fell together with interactivity"
     - "HiCache spills evicted prefixes to host DRAM and restores them at C2C bandwidth instead of recomputing; sizing follows the qwen3.5-fp8-b300-sglang-agentic-hicache recipe (GLM-5.2 is plain GQA: one host pool per rank, GB-based --hicache-size)"
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2280
+
+- config-keys:
+    - glm5.2-fp4-b300-sglang-agentic
+  description:
+    - "Convert the high-throughput arm to DEP: add ep: 8 to the TP8/DP8 attention-DP arm so the MoE layers run expert-parallel across all 8 ranks (--ep-size 8) instead of TP-sharded (the arm previously left ep unset -> EP_SIZE=1). Attention stays data-parallel behind sglang-router consistent hashing; the recipe already passes --ep-size EP_SIZE unconditionally, so only the master-config search space changes"
+    - "DEP arm runs conc-list [48] only (was [48, 64, 96, 128, 192, 256, 512]): conc 48 is the frontier peak (163 tok/s/gpu out, ttft 2.8s); conc >=64 overflows the HBM+host radix cache (GPU hit 0.93->0.57 at 48->64) and thrashes on re-prefill, so it is strictly dominated"
+    - "Low-latency TP8 arm (conc [1, 2, 4, 8, 16, 32], fp8 KV + cutedsl bf16 GEMM) unchanged; DEP mirrors the proven dsv4-fp4-b300-sglang tp8/ep8/dp-attn DEP arm"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2281