Skip to content

Commit 0c6b262

Browse files
kvinwangkvinwang
committed
gw: Add cluster-deployment.md
1 parent dc394c9 commit 0c6b262

1 file changed

Lines changed: 333 additions & 0 deletions

File tree

‎gateway/docs/cluster-deployment.md‎

Lines changed: 333 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,333 @@
1+
# dstack-gateway Cluster Deployment Guide
2+
3+
This document describes how to deploy a dstack-gateway cluster, including single-node and multi-node configurations.
4+
5+
## Table of Contents
6+
7+
1. [Overview](#1-overview)
8+
2. [Cluster Deployment (2-Node Example)](#2-cluster-deployment-2-node-example)
9+
3. [Adding Reverse Proxy Domains](#3-adding-reverse-proxy-domains)
10+
4. [Operations and Monitoring](#4-operations-and-monitoring)
11+
12+
## 1. Overview
13+
14+
dstack-gateway is a distributed reverse proxy gateway for dstack services. Key features include:
15+
16+
- TLS termination and SNI routing: Automatically selects certificates and routes traffic based on SNI
17+
- Automatic certificate management: Automatically requests and renews certificates via ACME protocol (Let's Encrypt)
18+
- Multi-node cluster: Multiple gateway nodes automatically sync state for high availability
19+
- WireGuard tunnels: Provides secure network access for CVM instances
20+
21+
### Architecture Diagram
22+
23+
```mermaid
24+
flowchart TB
25+
subgraph Internet
26+
LB[Load Balancer]
27+
end
28+
29+
subgraph Gateway Cluster
30+
G1[Gateway 1<br/>node_id=1]
31+
G2[Gateway 2<br/>node_id=2]
32+
G1 <-->|Sync| G2
33+
end
34+
35+
subgraph CVM Pool
36+
CVM1[CVM 1<br/>App A]
37+
CVM2[CVM 2<br/>App B]
38+
end
39+
40+
LB --> G1
41+
LB --> G2
42+
43+
G1 -.->|WireGuard| CVM1
44+
G1 -.->|WireGuard| CVM2
45+
G2 -.->|WireGuard| CVM1
46+
G2 -.->|WireGuard| CVM2
47+
```
48+
49+
When a CVM starts, it registers with one of the Gateways. The Gateway cluster automatically syncs the CVM's information (including WireGuard public key), enabling all Gateway nodes to establish WireGuard tunnel connections to that CVM.
50+
51+
### Port Description
52+
53+
| Default Port | Protocol | Purpose | Security Recommendation |
54+
|--------------|----------|---------|-------------------------|
55+
| 9012 | HTTPS | RPC port for inter-node sync communication | Internal network only |
56+
| 9013 | UDP | WireGuard tunnel port | Internal network only |
57+
| 9014 | HTTPS | Proxy port for external TLS proxy service | Can be exposed to public |
58+
| 9015 | HTTP | Debug port for health checks and debugging | Must be disabled in production |
59+
| 9016 | HTTP | Admin port for management API | Do not expose to public, recommend using Unix Domain Socket |
60+
61+
Production security configuration example:
62+
63+
```toml
64+
[core.debug]
65+
insecure_enable_debug_rpc = false # Disable Debug port
66+
67+
[core.admin]
68+
enabled = true
69+
address = "unix:/run/dstack/admin.sock" # Use Unix Domain Socket
70+
```
71+
72+
## 2. Cluster Deployment (2-Node Example)
73+
74+
### 2.1 Node Planning
75+
76+
| Node | node_id | Gateway IP | Client IP range | bootnode |
77+
|------|---------|------------|-----------------|----------|
78+
| gateway-1 | 1 | 10.8.128.1/16 | 10.8.128.0/18 | gateway-2 |
79+
| gateway-2 | 2 | 10.8.0.1/16 | 10.8.0.0/18 | gateway-1 |
80+
81+
Notes:
82+
- Each node's node_id must be unique
83+
- Each node's Client IP range should not overlap (used for allocating IPs to different CVMs)
84+
- bootnode is configured as another node's RPC URL, used for cluster discovery at startup
85+
86+
### 2.2 CIDR Description
87+
88+
Client IP range (/18):
89+
- /18 means the first 18 bits are the network prefix
90+
- For example, 10.8.128.0/18 covers the address range 10.8.128.0 ~ 10.8.191.255
91+
- Each Gateway's /18 range does not overlap, so each Gateway can allocate IPs locally without syncing with other Gateways
92+
93+
Gateway IP (/16):
94+
- Gateway IP uses /16 netmask to allow network routing to cover the larger 10.8.0.0/16 address space
95+
- This way, when another Gateway allocates an address in a /18 subnet, traffic can still be correctly routed
96+
97+
### 2.3 WireGuard Configuration Fields
98+
99+
Key fields in the `[core.wg]` section:
100+
101+
- `ip`: Gateway's own WireGuard address in CIDR format (e.g., 10.8.128.1/16)
102+
- `client_ip_range`: Address pool range for allocating to CVMs (e.g., 10.8.128.0/18)
103+
- `reserved_net`: Reserved address range that will not be allocated to CVMs (e.g., 10.8.128.1/32, reserving the gateway's own address)
104+
105+
Recommendation: Design client_ip_range and reserved_net to ensure clear address pool planning for each Gateway, avoiding address conflicts.
106+
107+
### 2.4 Configuration File Examples
108+
109+
gateway-1.toml:
110+
111+
```toml
112+
log_level = "info"
113+
address = "0.0.0.0"
114+
port = 9012
115+
116+
[tls]
117+
key = "/var/lib/gateway/certs/gateway-rpc.key"
118+
certs = "/var/lib/gateway/certs/gateway-rpc.cert"
119+
120+
[tls.mutual]
121+
ca_certs = "/var/lib/gateway/certs/gateway-ca.cert"
122+
mandatory = false
123+
124+
[core]
125+
kms_url = "https://kms.demo.dstack.org"
126+
rpc_domain = "rpc.gateway-1.demo.dstack.org"
127+
128+
[core.admin]
129+
enabled = true
130+
port = 9016
131+
address = "0.0.0.0"
132+
133+
[core.debug]
134+
insecure_enable_debug_rpc = true
135+
insecure_skip_attestation = false
136+
port = 9015
137+
address = "0.0.0.0"
138+
139+
[core.sync]
140+
enabled = true
141+
interval = "30s"
142+
timeout = "60s"
143+
my_url = "https://rpc.gateway-1.demo.dstack.org:9012"
144+
bootnode = "https://rpc.gateway-2.demo.dstack.org:9012"
145+
node_id = 1
146+
data_dir = "/var/lib/gateway/data"
147+
148+
[core.wg]
149+
private_key = "<node1-private-key>"
150+
public_key = "<node1-public-key>"
151+
listen_port = 9013
152+
ip = "10.8.128.1/16"
153+
reserved_net = ["10.8.128.1/32"]
154+
client_ip_range = "10.8.128.0/18"
155+
config_path = "/var/lib/gateway/wg.conf"
156+
interface = "wg-gw1"
157+
endpoint = "<host ip>:9013"
158+
159+
[core.proxy]
160+
listen_addr = "0.0.0.0"
161+
listen_port = 9014
162+
external_port = 443
163+
```
164+
165+
gateway-2.toml:
166+
167+
```toml
168+
log_level = "info"
169+
address = "0.0.0.0"
170+
port = 9012
171+
172+
[tls]
173+
key = "/var/lib/gateway/certs/gateway-rpc.key"
174+
certs = "/var/lib/gateway/certs/gateway-rpc.cert"
175+
176+
[tls.mutual]
177+
ca_certs = "/var/lib/gateway/certs/gateway-ca.cert"
178+
mandatory = false
179+
180+
[core]
181+
kms_url = "https://kms.demo.dstack.org"
182+
rpc_domain = "rpc.gateway-2.demo.dstack.org"
183+
184+
[core.admin]
185+
enabled = true
186+
port = 9016
187+
address = "0.0.0.0"
188+
189+
[core.debug]
190+
insecure_enable_debug_rpc = true
191+
insecure_skip_attestation = false
192+
port = 9015
193+
address = "0.0.0.0"
194+
195+
[core.sync]
196+
enabled = true
197+
interval = "30s"
198+
timeout = "60s"
199+
my_url = "https://rpc.gateway-2.demo.dstack.org:9012"
200+
bootnode = "https://rpc.gateway-1.demo.dstack.org:9012"
201+
node_id = 2
202+
data_dir = "/var/lib/gateway/data"
203+
204+
[core.wg]
205+
private_key = "<node2-private-key>"
206+
public_key = "<node2-public-key>"
207+
listen_port = 9013
208+
ip = "10.8.0.1/16"
209+
reserved_net = ["10.8.0.1/32"]
210+
client_ip_range = "10.8.0.0/18"
211+
config_path = "/var/lib/gateway/wg.conf"
212+
interface = "wg-gw2"
213+
endpoint = "<host ip>:9013"
214+
215+
[core.proxy]
216+
listen_addr = "0.0.0.0"
217+
listen_port = 9014
218+
external_port = 443
219+
```
220+
221+
### 2.5 Verify Cluster Sync
222+
223+
```bash
224+
# Check sync status on any node
225+
curl -s http://localhost:9016/prpc/Admin.WaveKvStatus | jq .
226+
```
227+
228+
## 3. Adding Reverse Proxy Domains
229+
230+
Gateway supports automatic TLS certificate management via the ACME protocol.
231+
232+
### 3.1 Configure ACME Service
233+
234+
```bash
235+
# Set ACME URL (Let's Encrypt production)
236+
curl -X POST "http://localhost:9016/prpc/Admin.SetCertbotConfig" \
237+
-H "Content-Type: application/json" \
238+
-d '{"acme_url": "https://acme-v02.api.letsencrypt.org/directory"}'
239+
240+
# For testing, use Let's Encrypt Staging
241+
# "acme_url": "https://acme-staging-v02.api.letsencrypt.org/directory"
242+
```
243+
244+
### 3.2 Configure DNS Credential
245+
246+
Gateway uses DNS-01 validation, which requires configuring DNS provider API credentials.
247+
248+
Cloudflare example:
249+
250+
```bash
251+
curl -X POST "http://localhost:9016/prpc/Admin.CreateDnsCredential" \
252+
-H "Content-Type: application/json" \
253+
-d '{
254+
"name": "cloudflare-prod",
255+
"provider_type": "cloudflare",
256+
"cf_api_token": "your-cloudflare-api-token",
257+
"set_as_default": true
258+
}'
259+
```
260+
261+
### 3.3 Add Domain
262+
263+
Call the Admin.AddZtDomain API to add a domain. Gateway will automatically request a *.domain wildcard certificate.
264+
265+
Parameter description:
266+
267+
| Parameter | Type | Required | Description |
268+
|-----------|------|----------|-------------|
269+
| domain | string | Yes | Base domain (e.g., example.com), certificate will be issued for *.example.com |
270+
| port | uint32 | Yes | External service port for this domain (usually 443) |
271+
| dns_cred_id | string | No | DNS credential ID, leave empty to use default credential |
272+
| node | uint32 | No | Bind to specific node (node_id), leave empty for any node to serve this domain |
273+
| priority | int32 | No | Priority for selecting default base_domain (higher value = higher priority, default is 0) |
274+
275+
Basic usage (using default DNS credential):
276+
277+
```bash
278+
curl -X POST "http://localhost:9016/prpc/Admin.AddZtDomain" \
279+
-H "Content-Type: application/json" \
280+
-d '{"domain": "example.com", "port": 443}'
281+
```
282+
283+
Specifying DNS credential and node binding:
284+
285+
```bash
286+
curl -X POST "http://localhost:9016/prpc/Admin.AddZtDomain" \
287+
-H "Content-Type: application/json" \
288+
-d '{
289+
"domain": "internal.example.com",
290+
"port": 443,
291+
"dns_cred_id": "cloudflare-prod",
292+
"node": 1,
293+
"priority": 10
294+
}'
295+
```
296+
297+
Response example:
298+
299+
```json
300+
{
301+
"config": {
302+
"domain": "example.com",
303+
"port": 443,
304+
"priority": 0
305+
},
306+
"cert_status": {
307+
"has_cert": false,
308+
"not_after": 0,
309+
"issued_by": 0,
310+
"issued_at": 0
311+
}
312+
}
313+
```
314+
315+
Note: After adding a domain, the certificate is not issued immediately. Gateway will request the certificate asynchronously in the background. You can check certificate status via section 3.5, or manually trigger certificate request via section 3.4.
316+
317+
### 3.4 Manually Trigger Certificate Renewal
318+
319+
```bash
320+
curl -X POST "http://localhost:9016/prpc/Admin.RenewZtDomainCert" \
321+
-H "Content-Type: application/json" \
322+
-d '{"domain": "example.com", "force": true}'
323+
```
324+
325+
### 3.5 Check Certificate Status
326+
327+
```bash
328+
curl -s http://localhost:9016/prpc/Admin.Status | jq '.zt_domains'
329+
```
330+
331+
### 3.6 Web UI
332+
333+
All the above command-line operations can also be performed via Web UI by visiting http://localhost:9016 in a browser.

0 commit comments

Comments
 (0)