You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix(agentx): keep Dynamo discovery alive during cold start
Apply ETCD_LEASE_TTL=7200 globally in both GB200 AgentX recipes so the frontend and all workers retain discovery leases through the two-hour health window.
中文:在两个 GB200 AgentX 配方中全局设置 ETCD_LEASE_TTL=7200,确保前端及所有 worker 在最长两小时的健康检查窗口内维持 Dynamo 服务发现租约。
Copy file name to clipboardExpand all lines: benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb200-3p1d-dep8-dep16-agentic.yaml
+5-11Lines changed: 5 additions & 11 deletions
Original file line number
Diff line number
Diff line change
@@ -14,6 +14,11 @@ dynamo:
14
14
15
15
setup_script: vllm-container-deps.sh
16
16
17
+
environment:
18
+
# Apply the lease to the frontend as well as every prefill/decode worker so
19
+
# discovery survives the long CPU-saturated cold start.
20
+
ETCD_LEASE_TTL: "7200"
21
+
17
22
slurm:
18
23
time_limit: "8:00:00"
19
24
@@ -65,14 +70,6 @@ backend:
65
70
mode: "embedded"
66
71
enable_offload: false
67
72
prefill_environment:
68
-
# dynamo's primary etcd lease defaults to a 10s TTL. During the ~380s
69
-
# weight load the process is CPU-saturated and the Rust keep-alive task
70
-
# cannot renew in time, so etcd revokes the lease and worker registration
71
-
# later fails with "requested lease not found" (workers never go ready ->
72
-
# health check times out at 2h). 600s covers the weight load + cudagraph
73
-
# stalls; longer TTL only delays dead-worker cleanup, which the health
74
-
# check / fail-fast monitor handle independently.
75
-
ETCD_LEASE_TTL: "600"
76
73
DG_JIT_CACHE_DIR: "/tmp/dg-cache-dsv4-agentx"
77
74
TORCH_SYMMMEM: "NVSHMEM"
78
75
VLLM_USE_NCCL_SYMM_MEM: "1"
@@ -101,9 +98,6 @@ backend:
101
98
# MC_STORE_CLIENT_METRIC_INTERVAL: "5"
102
99
# MC_TE_METRIC: "0"
103
100
decode_environment:
104
-
# See prefill_environment: raise dynamo's primary etcd lease TTL from the
105
-
# 10s default so the lease survives the CPU-saturated cold start.
0 commit comments