Skip to content

Commit 85cf941

Browse files
h4x3rotabclaude
andcommitted
chore(consul-postgres-ha): drop implementation-process labels from docs + code
Removes the "Stage 1 / Stage 2 / Stage 4 / Phase 0" labels that crept into comments, doc titles, and inline notes during development. Those were process markers for ourselves, not vocabulary anyone reading this example cold should care about. Substantive content (what's a workaround, what's the principled fix, where to find the design) preserved verbatim — only the labels are gone. Also: coordinator/docker-compose.yaml was referencing a deleted `../phase0/icetest` build context for the signaling service. Repointed to the published GHCR image (`consul-postgres-ha-signaling:latest`), matching how the rest of the example pulls its images. Verified: mesh-conn tests pass, bootstrap-secrets/signaling/quic-on-ice all build, terraform validate clean, shell entrypoints parse. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 808836c commit 85cf941

18 files changed

Lines changed: 82 additions & 79 deletions

File tree

consul-postgres-ha/ARCHITECTURE.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -333,9 +333,9 @@ format.
333333
- …all confidential traffic above it is **already encrypted by Envoy
334334
mTLS** (Layer 3), so the wire is safe even if someone could see the
335335
UDP datagrams.
336-
- **Consul gossip** is encrypted via `-encrypt` (Stage-1 workaround:
337-
key generated in Terraform and broadcast via env; Stage-2
338-
attestation will replace this with TEE-rooted material). RPC is
336+
- **Consul gossip** is encrypted via `-encrypt` (workaround: key
337+
generated in Terraform and broadcast via env; attestation-rooted
338+
admission will replace this with TEE-derived material). RPC is
339339
plaintext. Both are confined to inside the overlay, but a full
340340
setup would also configure TLS for RPC. See [ROBUSTNESS.md](ROBUSTNESS.md).
341341

consul-postgres-ha/FAILOVER.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
# Stage 4 — failover demo
1+
# Failover demo
22

33
A reproducible recipe for the soft-kill leader-failover scenario, plus the
44
measured timeline from a real run on the live cluster (2026-05-03).
5-
This demonstrates that stage 4's HA story is end-to-end working: Patroni
5+
This demonstrates that the HA story is end-to-end working: Patroni
66
elects via Consul KV when the leader's lock expires, a replica is
77
promoted, writes resume on the new leader, and the old leader rejoins
88
cheaply (WAL replay + streaming, no full pg_basebackup) once it comes

consul-postgres-ha/PUBLISHING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
# Stage 4 — image publishing & verification
1+
# Image publishing & verification
22

3-
The stage-4 example needs four container images deployed in lockstep:
3+
This example needs four container images deployed in lockstep:
44
`mesh-sidecar`, `patroni`, `webdemo`, `signaling`. CI publishes them to
55
GHCR with Sigstore-backed GitHub Build Provenance; consumers pin by
66
tag (or, better, by digest) and verify provenance with

consul-postgres-ha/README.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ peers' ICE candidates; no data ever passes through it once peers
2727
connect. Per-CVM and per-cluster secrets are split: the TURN HMAC is
2828
derived per-CVM from dstack KMS, while cluster-wide-identical
2929
material (gossip key, Patroni passwords) is generated in Terraform
30-
and broadcast via env until Stage-2 attestation-admission lands.
30+
and broadcast via env until attestation-rooted admission lands.
3131
Connect CA uses Consul's built-in CA provider — root in Raft, no
3232
external derivation needed.
3333

@@ -142,11 +142,11 @@ and the Terraform structure as-is.
142142
admission credential is the next architectural step.
143143
* The cluster's gossip key, Patroni superuser password, and
144144
replication password are **generated in Terraform and broadcast
145-
via env to every CVM**. This is a Stage-1 workaround those
146-
bytes live in `terraform.tfstate` and pass through whoever runs
147-
`apply`. Stage-2 attestation-admission
145+
via env to every CVM** — a workaround, because those bytes live
146+
in `terraform.tfstate` and pass through whoever runs `apply`. The
147+
attestation-rooted admission design
148148
([`design/attestation-admission.md`](design/attestation-admission.md))
149-
replaces this with TEE-rooted material that no human ever sees.
149+
replaces this with TEE-derived material that no human ever sees.
150150

151151
## Filed upstream
152152

consul-postgres-ha/ROBUSTNESS.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ ours, they're the parts that *must* be made robust by hand.
5151
| Coordinator host dies | New peers can't bootstrap. **Existing ICE pairs keep working** (no data flows through this box once handshake is done). New retries from existing peers fail until it's back. | bring it back; peers reconnect on their own. |
5252
| Coordinator ufw / network change | Same as above. | restore ports 3478/udp+tcp, 5349/tcp, 7000/tcp, 49152-49999/udp. |
5353
| TURN shared secret leaks | Anyone can use the box as an open TURN relay (cost / abuse risk). | rotate `TURN_SHARED_SECRET` in coordinator + every CVM env, redeploy. |
54-
| Signalling broker is unauthenticated | Any external actor can publish/poll messages, spoof candidates, intercept ICE handshakes. Currently low-impact only because we're solo. | gate `/publish` + `/poll` on attestation-derived identity (Stage 4 work). |
54+
| Signalling broker is unauthenticated | Any external actor can publish/poll messages, spoof candidates, intercept ICE handshakes. Currently low-impact only because we're solo. | gate `/publish` + `/poll` on attestation-derived identity. |
5555
| dstack provider NAT changes type (e.g. cone → symmetric) | ICE picks TURN relay path. ~150 ms RTT instead of ~6 ms. **Functionality unchanged.** | none needed; coturn covers this fallback. |
5656
| Underlying CVM dies | That peer's services drop out. Consul will mark it `failed` after gossip timeout, Envoy LB removes it within seconds. | redeploy; the rest of the cluster is unaffected. |
5757

@@ -200,7 +200,7 @@ The mitigations are mostly testing discipline:
200200
| Envoy sidecar dies | All in-flight mTLS connections through it drop. App's calls to `127.0.0.1:19000` get connection refused. | container restart. ~5 s downtime per peer. |
201201
| Connect CA root expiry | All sidecar leaf certs go invalid; whole mesh stops. | `consul connect ca set-config` to rotate root, or default 5-year root won't bite us in this experiment. |
202202
| Connect intention misconfigured (e.g. accidental deny) | Some traffic blocked silently. Sidecar denies are reported as `RBAC: access denied` in Envoy logs. | rotate intention; xDS picks it up in seconds (already demoed). |
203-
| **RPC TLS not set** | RPC is plaintext on the overlay. Threat is bounded by the QUIC overlay below it (peer ⇄ peer end-to-end encrypted), so on-the-wire taps don't see it; in-CVM containers that bind `127.0.0.1` to the agent's RPC port would. | Stage-2 work: derive a small CA from attestation-rooted material and configure Consul TLS using it. |
203+
| **RPC TLS not set** | RPC is plaintext on the overlay. Threat is bounded by the QUIC overlay below it (peer ⇄ peer end-to-end encrypted), so on-the-wire taps don't see it; in-CVM containers that bind `127.0.0.1` to the agent's RPC port would. | Derive a small CA from attestation-rooted material and configure Consul TLS using it — design lives in `design/attestation-admission.md`. |
204204
| ttl.sh image expiry | After 24h, a CVM restart can't pull our images. New deploys silently fail to pull. | move to a real registry (GHCR, Phala internal, local registry on the public box). |
205205

206206
### Risk shape
@@ -232,13 +232,13 @@ depth.
232232
- **Real registry** — Sigstore-attested GHCR images via
233233
`.github/workflows/consul-postgres-ha-publish.yml`. See
234234
`PUBLISHING.md`.
235-
- **Gossip key wired in (Stage-1 workaround)**`cluster.tf`
235+
- **Gossip key wired in (workaround)**`cluster.tf`
236236
generates a `random_bytes` and broadcasts it to every CVM via
237237
env; `mesh-sidecar/entrypoint.sh` passes it as
238238
`consul agent -encrypt=…`. Same shape used for the Patroni
239239
superuser + replication passwords. The keys live in
240240
`terraform.tfstate`; eliminating that exposure is part of the
241-
Stage-2 attestation-admission work
241+
attestation-admission work
242242
(`design/attestation-admission.md`).
243243

244244
## Cross-layer concerns
@@ -312,7 +312,7 @@ are the next plateau.
312312

313313
The deeper open question — **anyone with `terraform.tfstate` can
314314
read the cluster's gossip key and Patroni passwords** — is
315-
deliberately deferred to Stage 2 (attestation admission), where
315+
deliberately deferred to the attestation-admission work, where
316316
peers prove TEE residency and shared cluster material is rooted
317317
in attestation rather than handed in by the deployer.
318318

@@ -328,9 +328,9 @@ in attestation rather than handed in by the deployer.
328328
- **Real registry** — Sigstore-attested GHCR images via
329329
`.github/workflows/consul-postgres-ha-publish.yml`.
330330
- **Gossip key + Patroni passwords are now cluster-wide identical
331-
(Stage-1 workaround)** — generated in Terraform and broadcast
332-
to every phala_app via env. Stage-2 attestation will replace
333-
this with TEE-rooted material.
331+
(workaround)** — generated in Terraform and broadcast to every
332+
phala_app via env. Attestation-rooted admission will replace
333+
this with TEE-derived material.
334334

335335
## "Are we playing too many tricks?"
336336

consul-postgres-ha/bootstrap-secrets/main.go

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// bootstrap-secrets — stage 4 init container.
1+
// bootstrap-secrets — init container.
22
//
33
// One-shot. Runs to completion before any other service starts on a CVM.
44
// Responsibilities:
@@ -28,9 +28,8 @@
2828
// `condition: service_completed_successfully` can release the
2929
// next services.
3030
//
31-
// The keystone of the stage-4 design is here: this is the only piece
32-
// that holds plaintext secret material, and it does so entirely
33-
// inside the TEE. The deploy host never sees them.
31+
// This is the only piece that holds plaintext secret material, and
32+
// it does so entirely inside the TEE. The deploy host never sees them.
3433

3534
package main
3635

@@ -71,14 +70,14 @@ func main() {
7170

7271
// 2. Derive per-app secrets via dstack KMS.
7372
//
74-
// Stage-1 WORKAROUND scope note: gossip key, Patroni superuser
75-
// password, and Patroni replication password are NOT derived
76-
// here — they would be different bytes on every CVM (each CVM
77-
// is its own phala_app with its own app_id, and GetKey is rooted
78-
// at app_id), so per-CVM derivation can't produce the
79-
// cluster-wide identical material those consumers need. They
80-
// come from Terraform-generated env instead. See cluster.tf and
81-
// design/attestation-admission.md for the Stage-2 fix.
73+
// WORKAROUND scope note: gossip key, Patroni superuser password,
74+
// and Patroni replication password are NOT derived here — they
75+
// would be different bytes on every CVM (each CVM is its own
76+
// phala_app with its own app_id, and GetKey is rooted at app_id),
77+
// so per-CVM derivation can't produce the cluster-wide identical
78+
// material those consumers need. They come from Terraform-
79+
// generated env instead. See cluster.tf and
80+
// design/attestation-admission.md for the principled fix.
8281
//
8382
// What's still derived here is intentionally narrow:
8483
// turn: handed to mesh-conn for coturn auth (the actual
@@ -216,7 +215,7 @@ func mustEnv(k string) string {
216215
// First match wins. Returns the slot index. Slot ownership is
217216
// permanent for the InstanceID's lifetime; cleanup of stale slots
218217
// (when an instance is permanently retired) is a separate operator
219-
// task — note in stage-4 README.
218+
// task — see the example README.
220219
func claimOrdinal(cfg *Config, instanceID string) (int, error) {
221220
if cfg.ConsulHTTPAddr == "" {
222221
return 0, fmt.Errorf("CONSUL_HTTP_ADDR required for non-coordinator role")

consul-postgres-ha/cluster-example/cluster.tf

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ terraform {
3131

3232
provider "phala" {}
3333

34-
# ---------- Cluster-wide shared secrets (Stage-1 WORKAROUND) ----------
34+
# ---------- Cluster-wide shared secrets (WORKAROUND) ----------
3535
#
3636
# These three secrets MUST be byte-identical across every CVM that
3737
# joins the cluster (gossip auth, Patroni replication, Patroni superuser).
@@ -41,11 +41,12 @@ provider "phala" {}
4141
# phala_app.coordinator/worker) and dstack's GetKey() is rooted at
4242
# app_id, so per-CVM derivation produces DIFFERENT bytes on each peer.
4343
#
44-
# Until Stage-2 attestation-rooted admission lands (see
44+
# Until attestation-rooted admission lands (see
4545
# `consul-postgres-ha/design/attestation-admission.md`), we generate
4646
# them in Terraform and hand the same bytes to every phala_app via
4747
# env. Trade-off accepted: anyone with read access to terraform.tfstate
48-
# (or the apply host's memory) sees plaintext keys. Stage 2 closes this.
48+
# (or the apply host's memory) sees plaintext keys. The attestation
49+
# work closes this.
4950
#
5051
# Connect CA root is NOT in this list — Consul's built-in CA provider
5152
# generates the root in Raft on first quorum, no external derivation
@@ -193,7 +194,7 @@ resource "phala_app" "coordinator" {
193194
TURN_HOST = var.external_coordinator_host
194195
TURN_SHARED_SECRET = var.external_turn_secret
195196
MESH_SIDECAR_IMAGE = var.mesh_sidecar_image
196-
# Stage-1 WORKAROUND — see `random_bytes` block at top of file.
197+
# WORKAROUND — see `random_bytes` block at top of file.
197198
GOSSIP_KEY = random_bytes.gossip_key.base64
198199
MESH_CONN_RELAY_ONLY = var.mesh_conn_relay_only
199200
MESH_CONN_DEBUG_ICE = var.mesh_conn_debug_ice
@@ -237,7 +238,7 @@ resource "phala_app" "worker" {
237238
MESH_SIDECAR_IMAGE = var.mesh_sidecar_image
238239
WEBDEMO_IMAGE = var.webdemo_image
239240
PATRONI_IMAGE = var.patroni_image
240-
# Stage-1 WORKAROUND — see `random_bytes` block at top of file.
241+
# WORKAROUND — see `random_bytes` block at top of file.
241242
GOSSIP_KEY = random_bytes.gossip_key.base64
242243
PATRONI_SUPERUSER_PW = random_bytes.patroni_superuser_pw.hex
243244
PATRONI_REPLICATION_PW = random_bytes.patroni_replication_pw.hex

consul-postgres-ha/cluster-example/rollout.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env bash
2-
# Stage 4 — workload-aware rolling update driver.
2+
# Workload-aware rolling update driver.
33
#
44
# Until phala-cloud#243 lands `phala_app.update_policy`, the platform's
55
# in-place app update touches all replicas in unspecified order. That's

consul-postgres-ha/compose/coordinator.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,8 +55,8 @@ services:
5555
- SIGNALING_URL=${SIGNALING_URL}
5656
- TURN_HOST=${TURN_HOST}
5757
- TURN_SHARED_SECRET=${TURN_SHARED_SECRET}
58-
# Stage-1 WORKAROUND: gossip key generated in Terraform and
59-
# broadcast identically to every CVM. See cluster.tf.
58+
# WORKAROUND: gossip key generated in Terraform and broadcast
59+
# identically to every CVM. See cluster.tf.
6060
- GOSSIP_KEY=${GOSSIP_KEY}
6161
# See worker.yaml for the rationale on MESH_CONN_RELAY_ONLY.
6262
- MESH_CONN_RELAY_ONLY=${MESH_CONN_RELAY_ONLY:-}

consul-postgres-ha/compose/worker.yaml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ services:
4040
- SIGNALING_URL=${SIGNALING_URL}
4141
- TURN_HOST=${TURN_HOST}
4242
- TURN_SHARED_SECRET=${TURN_SHARED_SECRET}
43-
# Stage-1 WORKAROUND: gossip key generated in Terraform and
44-
# broadcast identically to every CVM. See cluster.tf.
43+
# WORKAROUND: gossip key generated in Terraform and broadcast
44+
# identically to every CVM. See cluster.tf.
4545
- GOSSIP_KEY=${GOSSIP_KEY}
4646
# MESH_CONN_RELAY_ONLY=1 forces ICE to gather only Relay
4747
# candidates, routing all peer traffic through the coturn server.
@@ -71,8 +71,8 @@ services:
7171
# CLUSTER_NAME drives Patroni's `scope` — every peer's patroni
7272
# must use the same value to land in the same cluster. Postgres
7373
# binds the canonical 127.0.0.1:5432; Patroni REST on 127.0.0.1:8008.
74-
# Stage-1 WORKAROUND: superuser + replication passwords generated
75-
# in Terraform and broadcast identically to every worker. See
74+
# WORKAROUND: superuser + replication passwords generated in
75+
# Terraform and broadcast identically to every worker. See
7676
# cluster.tf.
7777
environment:
7878
- CLUSTER_NAME=${CLUSTER_NAME}

0 commit comments

Comments
 (0)