Skip to content

Commit 0f912d7

Browse files
h4x3rotabclaude
andcommitted
chore(consul-postgres-ha): pin provider exactly + document disk-loss rejoin
Two small follow-ups after verifying the v0.2.0-beta.3 env-update path against the live cluster: 1. Provider pin in cluster.tf changed from `>= 0.2.0-beta.3` to `0.2.0-beta.3` exactly. Terraform's `>=` operator does NOT include later prerelease versions, so `>= 0.2.0-beta.3` only matches stable `>= 0.2.0` — `terraform init` failed with "no available releases match the given constraints". Pin exactly until we hit a stable. 2. FAILOVER.md gains a disk-loss rejoin section: stop patroni, wipe the patroni-pgdata volume, restart, watch Patroni's bootstrap path pull a full pg_basebackup from the leader over mesh-conn's QUIC tunnel. Measured 5.2 MB / 7s end-to-end on the live cluster (handshake-dominated for a small dataset; the real throughput number remains the ~25 MB/s pg_basebackup observed during the soft -kill section). Closes the last "What this demo does NOT cover" item. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 4b7f962 commit 0f912d7

2 files changed

Lines changed: 39 additions & 10 deletions

File tree

consul-postgres-ha/stage4/FAILOVER.md

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,38 @@ signals.
172172
new timeline. No pg_basebackup, no multi-MB re-copy through
173173
mesh-conn.
174174

175+
## Disk-loss rejoin (full pg_basebackup variant)
176+
177+
A replica whose pgdata is wiped goes through Patroni's bootstrap path
178+
and pulls a full pg_basebackup from the leader, all over mesh-conn's
179+
QUIC tunnel. Recipe (run on a non-leader CVM):
180+
181+
```bash
182+
docker stop -t 5 dstack-patroni-1
183+
rm -rf /var/lib/docker/volumes/dstack_patroni-pgdata/_data/*
184+
docker start dstack-patroni-1
185+
```
186+
187+
### Measured timeline (run from 2026-05-03)
188+
189+
```
190+
T_wipe 21:13:41 docker stop + rm -rf pgdata on worker-5
191+
T_restart 21:13:42 docker start
192+
T_basebackup 21:13:47 "trying to bootstrap from leader 'worker-4'"
193+
T_complete 21:13:54 "replica has been created using basebackup" +7s
194+
T_streaming 21:13:58 service registered, streaming WAL +16s total
195+
```
196+
197+
5.2 MB pgdata transferred in ~7 seconds end-to-end. Note the dataset
198+
is small enough that handshake/startup overhead dominates — for a
199+
realistic throughput number, see the soft-kill section's pg_basebackup
200+
trace at ~25 MB/s sustained on the QUIC path.
201+
202+
The path itself is the proof point: Patroni correctly detects empty
203+
pgdata, picks `bootstrap from leader` (not WAL replay), pulls the full
204+
backup over mesh-conn, transitions to streaming on the current
205+
timeline. No operator intervention.
206+
175207
## What this demo does NOT cover
176208

177209
* **CVM reboot or kernel panic**`reboot`/`poweroff` from inside
@@ -182,8 +214,3 @@ signals.
182214
* **Network partition**: split-brain isolation between coordinators
183215
vs workers. Patroni + Consul should handle it, but worth a separate
184216
test before claiming partition-tolerance.
185-
* **Disk loss on rejoin**: if the ex-leader's pgdata is wiped, rejoin
186-
WILL trigger a full pg_basebackup through mesh-conn. The
187-
~25 MB/s throughput and the QUIC transport mean even a 10 GB
188-
rebuild takes ~7 minutes (acceptable), but it's a different code
189-
path than the cheap rejoin shown above.

consul-postgres-ha/stage4/cluster-example/cluster.tf

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,13 @@ terraform {
1515
required_providers {
1616
phala = {
1717
source = "phala-network/phala"
18-
# >= 0.2.0-beta.3 is required for in-place env-block updates to
19-
# actually take effect — earlier versions silently no-op'd them
20-
# (Phala-Network/phala-cloud#246, fixed in
21-
# Phala-Network/terraform-provider-phala#8).
22-
version = ">= 0.2.0-beta.3"
18+
# 0.2.0-beta.3 is the first version where in-place env-block
19+
# updates actually take effect — earlier betas silently no-op'd
20+
# them (Phala-Network/phala-cloud#246, fixed in
21+
# Phala-Network/terraform-provider-phala#8). Pin exactly because
22+
# Terraform's `>=` operator doesn't include later prerelease
23+
# versions; bump this line by hand when a newer beta ships.
24+
version = "0.2.0-beta.3"
2325
}
2426
}
2527
}

0 commit comments

Comments
 (0)