Skip to content

Commit e47ca81

Browse files
committed
feat: trim glm5.2 b300 DEP arm to conc 48 only / 将 glm5.2 b300 DEP 分支收敛为仅 conc 48
The sweep (run 29717273540) showed DEP conc 48 is the frontier peak (163 tok/s/gpu out, ttft 2.8s, GPU radix hit 0.93) while conc 64 overflows the HBM+host radix cache (hit collapses to 0.57), thrashing on re-prefill down to 41 tok/s/gpu at 56s ttft. conc 64 is strictly dominated by conc 48 on both throughput and interactivity and cannot be recovered without more host-pool capacity than the 3TB node can give (ratio > 0.75 OOMs, per #2279). Drop it: DEP arm conc-list is now [48]. The agentic eval also moves to conc 48 (off the conc-64 point whose eval hung). 中文:扫描(run 29717273540)显示 DEP conc 48 是前沿峰值(输出 163 tok/s/gpu, ttft 2.8s,GPU radix 命中 0.93),而 conc 64 会使 HBM+host radix 缓存溢出(命中 率跌至 0.57),陷入 re-prefill 抖动,降至 41 tok/s/gpu、ttft 56s。conc 64 在吞吐 与交互性两个维度上都被 conc 48 严格支配,且在 3TB 节点上无法通过增大 host pool 恢复(ratio > 0.75 会 OOM,见 #2279)。故移除:DEP 分支 conc-list 现为 [48]。 agentic 评估点也随之移到 conc 48(避开评估卡住的 conc 64 点)。
1 parent 0e363a6 commit e47ca81

2 files changed

Lines changed: 6 additions & 3 deletions

File tree

configs/nvidia-master.yaml

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7875,5 +7875,8 @@ glm5.2-fp4-b300-sglang-agentic:
78757875
- dram-utilization: 0.80
78767876
search-space:
78777877
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 2, 4, 8, 16, 32] }
7878-
# DEP (attention-DP + EP8 expert-parallel MoE) throughput arm, capped at conc 64.
7879-
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [48, 64], router: { name: sglang-router, version: "0.3.2" } }
7878+
# DEP (attention-DP + EP8 expert-parallel MoE) throughput arm at conc 48 — the
7879+
# frontier peak (163 tok/s/gpu out, ttft 2.8s). conc >=64 overflows the HBM+host
7880+
# radix cache (GPU hit 0.93->0.57 at 48->64) and thrashes on re-prefill, so it is
7881+
# strictly dominated by conc 48 on both throughput and interactivity.
7882+
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [48], router: { name: sglang-router, version: "0.3.2" } }

perf-changelog.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4971,6 +4971,6 @@
49714971
- glm5.2-fp4-b300-sglang-agentic
49724972
description:
49734973
- "Convert the high-throughput arm to DEP: add ep: 8 to the TP8/DP8 attention-DP arm so the MoE layers run expert-parallel across all 8 ranks (--ep-size 8) instead of TP-sharded (the arm previously left ep unset -> EP_SIZE=1). Attention stays data-parallel behind sglang-router consistent hashing; the recipe already passes --ep-size EP_SIZE unconditionally, so only the master-config search space changes"
4974-
- "Cap the DEP arm concurrency at 64: conc-list [48, 64] (was [48, 64, 96, 128, 192, 256, 512])"
4974+
- "DEP arm runs conc-list [48] only (was [48, 64, 96, 128, 192, 256, 512]): conc 48 is the frontier peak (163 tok/s/gpu out, ttft 2.8s); conc >=64 overflows the HBM+host radix cache (GPU hit 0.93->0.57 at 48->64) and thrashes on re-prefill, so it is strictly dominated"
49754975
- "Low-latency TP8 arm (conc [1, 2, 4, 8, 16, 32], fp8 KV + cutedsl bf16 GEMM) unchanged; DEP mirrors the proven dsv4-fp4-b300-sglang tp8/ep8/dp-attn DEP arm"
49764976
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2281

0 commit comments

Comments
 (0)