Skip to content

Commit 99a28ab

Browse files
committed
fix(agentx): keep Dynamo discovery alive during cold start
Apply ETCD_LEASE_TTL=7200 globally in both GB200 AgentX recipes so the frontend and all workers retain discovery leases through the two-hour health window. 中文:在两个 GB200 AgentX 配方中全局设置 ETCD_LEASE_TTL=7200,确保前端及所有 worker 在最长两小时的健康检查窗口内维持 Dynamo 服务发现租约。
1 parent 8248e47 commit 99a28ab

2 files changed

Lines changed: 10 additions & 12 deletions

File tree

benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/agg-gb200-tp8-agentic.yaml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@ dynamo:
1414

1515
setup_script: vllm-container-deps.sh
1616

17+
environment:
18+
# The frontend shares Grace CPU capacity with the long TP8 cold start. Keep
19+
# Dynamo discovery alive until the outer two-hour health check expires.
20+
ETCD_LEASE_TTL: "7200"
21+
1722
slurm:
1823
time_limit: "8:00:00"
1924

@@ -59,7 +64,6 @@ backend:
5964
enable_offload: false
6065
aggregated_environment:
6166
DYN_REQUEST_PLANE: "tcp"
62-
ETCD_LEASE_TTL: "600"
6367
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
6468
VLLM_RPC_TIMEOUT: "600000"
6569
TILELANG_CLEANUP_TEMP_FILES: "1"

benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb200-3p1d-dep8-dep16-agentic.yaml

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@ dynamo:
1414

1515
setup_script: vllm-container-deps.sh
1616

17+
environment:
18+
# Apply the lease to the frontend as well as every prefill/decode worker so
19+
# discovery survives the long CPU-saturated cold start.
20+
ETCD_LEASE_TTL: "7200"
21+
1722
slurm:
1823
time_limit: "8:00:00"
1924

@@ -65,14 +70,6 @@ backend:
6570
mode: "embedded"
6671
enable_offload: false
6772
prefill_environment:
68-
# dynamo's primary etcd lease defaults to a 10s TTL. During the ~380s
69-
# weight load the process is CPU-saturated and the Rust keep-alive task
70-
# cannot renew in time, so etcd revokes the lease and worker registration
71-
# later fails with "requested lease not found" (workers never go ready ->
72-
# health check times out at 2h). 600s covers the weight load + cudagraph
73-
# stalls; longer TTL only delays dead-worker cleanup, which the health
74-
# check / fail-fast monitor handle independently.
75-
ETCD_LEASE_TTL: "600"
7673
DG_JIT_CACHE_DIR: "/tmp/dg-cache-dsv4-agentx"
7774
TORCH_SYMMMEM: "NVSHMEM"
7875
VLLM_USE_NCCL_SYMM_MEM: "1"
@@ -101,9 +98,6 @@ backend:
10198
# MC_STORE_CLIENT_METRIC_INTERVAL: "5"
10299
# MC_TE_METRIC: "0"
103100
decode_environment:
104-
# See prefill_environment: raise dynamo's primary etcd lease TTL from the
105-
# 10s default so the lease survives the CPU-saturated cold start.
106-
ETCD_LEASE_TTL: "600"
107101
DG_JIT_CACHE_DIR: "/tmp/dg-cache-dsv4-agentx"
108102
TORCH_SYMMMEM: "NVSHMEM"
109103
VLLM_USE_NCCL_SYMM_MEM: "1"

0 commit comments

Comments
 (0)