docs(gateway): improve cluster deployment guide

- Fix subnet scheme to use SUBNET_INDEX 0-based allocation (10.8.0.0/18,
  10.8.64.0/18, ...) matching the deploy script behavior
- Add SUBNET_INDEX mapping table for quick reference
- Add cluster sync and peer discovery section explaining bootnode vs
  auto-discovery mechanisms
- Add image version note about sync enable logic differences
- Fix gateway-1 config to use empty bootnode (first node doesn't need one)
- Expand verify section with Status API and health check criteria
- Add Cloudflare DNS:Edit permission note for API token setup

Author: kvinwang

gateway/docs/cluster-deployment.md

@@ -75,36 +75,65 @@ address = "unix:/run/dstack/admin.sock"  # Use Unix Domain Socket
 
 | Node | node_id | Gateway IP | Client IP range | bootnode |
 |------|---------|------------|-----------------|----------|
-| gateway-1 | 1 | 10.8.128.1/16 | 10.8.128.0/18 | gateway-2 |
-| gateway-2 | 2 | 10.8.0.1/16 | 10.8.0.0/18 | gateway-1 |
+| gateway-1 | 1 | 10.8.0.1/16 | 10.8.0.0/18 | (none or gateway-2) |
+| gateway-2 | 2 | 10.8.64.1/16 | 10.8.64.0/18 | gateway-1 |
 
 Notes:
-- Each node's node_id must be unique
-- Each node's Client IP range should not overlap (used for allocating IPs to different CVMs)
-- bootnode is configured as another node's RPC URL, used for cluster discovery at startup
+- Each node's `node_id` must be unique and greater than 0
+- Each node's Client IP range must not overlap (used for allocating IPs to different CVMs)
+- `bootnode` is optional — it speeds up initial peer discovery but is not required. Without a bootnode, a node will auto-discover peers when they connect via sync RPC
+- If a bootnode is set, its hostname must be resolvable before cluster bootstrap
 
 ### 2.2 CIDR Description
 
 Client IP range (/18):
 - /18 means the first 18 bits are the network prefix
-- For example, 10.8.128.0/18 covers the address range 10.8.128.0 ~ 10.8.191.255
+- For example, 10.8.0.0/18 covers the address range 10.8.0.0 ~ 10.8.63.255
 - Each Gateway's /18 range does not overlap, so each Gateway can allocate IPs locally without syncing with other Gateways
+- With 4 possible /18 ranges in a /16 network, a cluster supports up to 4 gateway nodes
 
 Gateway IP (/16):
 - Gateway IP uses /16 netmask to allow network routing to cover the larger 10.8.0.0/16 address space
 - This way, when another Gateway allocates an address in a /18 subnet, traffic can still be correctly routed
 
+Subnet mapping:
+
+| SUBNET_INDEX | Gateway IP | Client IP range | Address range | Usable IPs |
+|-------------|------------|-----------------|---------------|------------|
+| 0 | 10.8.0.1/16 | 10.8.0.0/18 | 10.8.0.0 ~ 10.8.63.255 | 16,382 |
+| 1 | 10.8.64.1/16 | 10.8.64.0/18 | 10.8.64.0 ~ 10.8.127.255 | 16,382 |
+| 2 | 10.8.128.1/16 | 10.8.128.0/18 | 10.8.128.0 ~ 10.8.191.255 | 16,382 |
+| 3 | 10.8.192.1/16 | 10.8.192.0/18 | 10.8.192.0 ~ 10.8.255.255 | 16,382 |
+
 ### 2.3 WireGuard Configuration Fields
 
 Key fields in the `[core.wg]` section:
 
-- `ip`: Gateway's own WireGuard address in CIDR format (e.g., 10.8.128.1/16)
-- `client_ip_range`: Address pool range for allocating to CVMs (e.g., 10.8.128.0/18)
-- `reserved_net`: Reserved address range that will not be allocated to CVMs (e.g., 10.8.128.1/32, reserving the gateway's own address)
+- `ip`: Gateway's own WireGuard address in CIDR format (e.g., 10.8.0.1/16)
+- `client_ip_range`: Address pool range for allocating to CVMs (e.g., 10.8.0.0/18)
+- `reserved_net`: Reserved address range that will not be allocated to CVMs (e.g., 10.8.0.1/32, reserving the gateway's own address)
 
 Recommendation: Design client_ip_range and reserved_net to ensure clear address pool planning for each Gateway, avoiding address conflicts.
 
-### 2.4 Configuration File Examples
+### 2.4 Cluster Sync and Peer Discovery
+
+> **Image version note**: The sync enable logic varies by gateway image version. In `dstacktee/dstack-gateway:0.5.8`, sync is enabled when `NODE_ID > 0` (regardless of `BOOTNODE_URL`). In some custom-built images, sync may only be enabled when `BOOTNODE_URL` is non-empty. Check your image's `entrypoint.sh` to confirm the behavior. When in doubt, set `NODE_ID > 0` and provide a `BOOTNODE_URL` on at least one node.
+
+Gateway nodes discover each other through two mechanisms:
+
+1. **Bootnode discovery** (active): A node with `bootnode` configured will fetch the peer list from the bootnode at startup, then periodically retry until peers are found.
+
+2. **Auto-discovery** (passive): When a remote node sends a sync request, the local node automatically adds it as a peer. This means the first node in a cluster does not need a bootnode — it will be discovered when the second node connects to it.
+
+This allows a simple deployment order:
+1. Start gateway-1 with `bootnode = ""` (no bootnode)
+2. Start gateway-2 with `bootnode = "https://rpc.gateway-1:9012"`
+3. Gateway-2 fetches peers from gateway-1 and starts syncing
+4. Gateway-1 auto-discovers gateway-2 from the incoming sync request
+
+> Note: `bootnode` is only used for initial discovery. Once peers are discovered, they are persisted in the KV store and survive restarts.
+
+### 2.5 Configuration File Examples
 
 gateway-1.toml:
 
@@ -141,17 +170,17 @@ enabled = true
 interval = "30s"
 timeout = "60s"
 my_url = "https://rpc.gateway-1.demo.dstack.org:9012"
-bootnode = "https://rpc.gateway-2.demo.dstack.org:9012"
+bootnode = ""
 node_id = 1
 data_dir = "/var/lib/gateway/data"
 
 [core.wg]
 private_key = "<node1-private-key>"
 public_key = "<node1-public-key>"
 listen_port = 9013
-ip = "10.8.128.1/16"
-reserved_net = ["10.8.128.1/32"]
-client_ip_range = "10.8.128.0/18"
+ip = "10.8.0.1/16"
+reserved_net = ["10.8.0.1/32"]
+client_ip_range = "10.8.0.0/18"
 config_path = "/var/lib/gateway/wg.conf"
 interface = "wg-gw1"
 endpoint = "<host ip>:9013"
@@ -205,9 +234,9 @@ data_dir = "/var/lib/gateway/data"
 private_key = "<node2-private-key>"
 public_key = "<node2-public-key>"
 listen_port = 9013
-ip = "10.8.0.1/16"
-reserved_net = ["10.8.0.1/32"]
-client_ip_range = "10.8.0.0/18"
+ip = "10.8.64.1/16"
+reserved_net = ["10.8.64.1/32"]
+client_ip_range = "10.8.64.0/18"
 config_path = "/var/lib/gateway/wg.conf"
 interface = "wg-gw2"
 endpoint = "<host ip>:9013"
@@ -218,13 +247,22 @@ listen_port = 9014
 external_port = 443
 ```
 
-### 2.5 Verify Cluster Sync
+### 2.6 Verify Cluster Sync
 
 ```bash
-# Check sync status on any node
+# Check sync status on any node (replace port with your admin port)
 curl -s http://localhost:9016/prpc/Admin.WaveKvStatus | jq .
+
+# List known cluster nodes
+curl -s http://localhost:9016/prpc/Admin.Status | jq '.nodes'
 ```
 
+A healthy cluster sync shows:
+- `enabled: true` on all nodes
+- Each node appears in every other node's `.nodes` array
+- `last_seen` timestamps are recent (within the sync interval)
+- `peer_ack` values are close to `local_ack` (no large lag)
+
 ## 3. Adding Reverse Proxy Domains
 
 Gateway supports automatic TLS certificate management via the ACME protocol.
@@ -245,6 +283,8 @@ curl -X POST "http://localhost:9016/prpc/Admin.SetCertbotConfig" \
 
 Gateway uses DNS-01 validation, which requires configuring DNS provider API credentials.
 
+The Cloudflare API token needs the **DNS:Edit** permission on the target zone. Create one at [Cloudflare API Tokens](https://dash.cloudflare.com/profile/api-tokens) with the "Edit zone DNS" template.
+
 Cloudflare example:
 
 ```bash