Skip to content

Commit c2db690

Browse files
h4x3rotabclaude
andcommitted
docs(consul-postgres-ha): rewrite "Adapting to your own workload" + retire design brief
README's adaptation guide now walks a worked example (add a fictional billing:9090 service) end-to-end. ARCHITECTURE notes the allowlist is platform-generated. design/declarative-services.md is deleted per the hand-off section — the code + updated README are the surviving artifacts; the index entry comes out too. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 79490dc commit c2db690

4 files changed

Lines changed: 86 additions & 283 deletions

File tree

consul-postgres-ha/ARCHITECTURE.md

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -78,9 +78,10 @@ above. Two carved-out loopback `/24`s, allocated cluster-wide:
7878
worker consumes (e.g. webdemo, postgres-master, postgres-replica).
7979
Identifies *what service* an app is calling.
8080

81-
mesh-conn binds a small **static infra-port allowlist** on every
81+
mesh-conn binds a small **platform-supplied allowlist** on every
8282
*other* peer's VIP and forwards each accepted connection to the right
83-
remote peer over the QUIC link:
83+
remote peer over the QUIC link. With the 3-service example
84+
(webdemo + postgres-master/-replica) the allowlist is:
8485

8586
| Port | Used by | Proto |
8687
|-------|-------------------------------------------------------|-------------|
@@ -89,11 +90,16 @@ remote peer over the QUIC link:
8990
| 8300 | Consul server RPC (server-to-server, client-to-server)| TCP |
9091
| 8301 | Consul serf-LAN gossip | UDP + TCP |
9192

92-
The allowlist is static and small by design — mesh-conn knows
93-
**peers, not services**. Apps never dial peer VIPs; only Envoy and
94-
Consul-agent do, and both speak well-known platform ports. Adding a
95-
port to the allowlist is a code change in `mesh-conn/main.go`, not
96-
a runtime catalog watch.
93+
The allowlist is intentionally minimal — mesh-conn knows **peers,
94+
not services**. Apps never dial peer VIPs; only Envoy and Consul-agent
95+
do, and both speak well-known platform ports.
96+
97+
The allowlist is **platform-generated, not Go-const**: per-service
98+
Connect-sidecar ports come from `local.services` in `cluster.tf`, get
99+
collapsed into per-backend `sidecar_port`s, and the platform sidecar
100+
emits `MESH_CONN_ALLOWLIST` env (JSON `[{port, udp}, …]`) at startup.
101+
mesh-conn reads it once and binds accordingly. Adding a service is an
102+
HCL edit in `cluster.tf`; mesh-conn never has to be rebuilt.
97103

98104
```
99105
inside worker-3 CVM (network_mode: host)

consul-postgres-ha/README.md

Lines changed: 73 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -122,20 +122,81 @@ consul-postgres-ha/
122122

123123
## Adapting to your own workload
124124

125-
Three things make this opinionated for Patroni; everything else is
126-
generic platform plumbing.
125+
The mesh is **declarative**: `local.services` in `cluster.tf` is the
126+
single source of truth, and the platform sidecar generates Consul
127+
registrations, Envoy supervise loops, loopback aliases, `/etc/hosts`
128+
entries, and `mesh-conn`'s peer-VIP allowlist from it. Adding a
129+
microservice is one HCL block plus an image; no edits to `mesh-conn`,
130+
the sidecar entrypoint, your app's source, or the CI workflow.
131+
132+
### Add a service: worked example
133+
134+
Say you want a billing service on port 9090 that the existing
135+
`webdemo` calls into. End-to-end:
136+
137+
1. **Declare it in `cluster.tf`** — append one entry to
138+
`local.services`:
139+
140+
```hcl
141+
services_raw = [
142+
{ name = "webdemo", port = 8080, subset = null },
143+
{ name = "postgres-master", port = 5432, subset = "master" },
144+
{ name = "postgres-replica", port = 5432, subset = "replica" },
145+
{ name = "billing", port = 9090, subset = null }, # ← new
146+
]
147+
```
148+
149+
This is the only edit to the platform.
150+
151+
2. **Add the image** to `terraform.tfvars` (`billing_image = "..."`),
152+
wire a `BILLING_IMAGE` variable + env entry on `phala_app.worker`,
153+
and add a `billing` service to `compose/worker.yaml` (modelled on
154+
`webdemo` — it just binds `127.0.0.1:9090` and serves).
155+
156+
3. **`terraform apply`**. The provider's in-place env update pushes
157+
the new `SERVICES_JSON` to every CVM:
158+
159+
- `mesh-conn`'s allowlist extends to `{21000, 21001, 21002,
160+
8300, 8301}` automatically — `MESH_CONN_ALLOWLIST` is computed
161+
from `SERVICES_JSON` in `mesh-sidecar/entrypoint.sh`.
162+
- Workers provision `127.10.0.13/32 dev lo`, append
163+
`127.10.0.13 billing` to `/etc/hosts`, and start a third Envoy
164+
supervise loop with `--base-id 3 -admin-bind 127.0.0.1:19002`.
165+
- Coordinator-0 writes a default-allow intention for `billing`
166+
and any subset/redirect resolvers implied by the declaration.
167+
168+
4. **From any container on any peer**, `curl http://billing:9090/...`
169+
load-balances across all peers' `billing` instances over Connect
170+
mTLS. No application-side service-discovery code.
171+
172+
### What the convention does for you
173+
174+
| Field on each entry | What it controls |
175+
|---|---|
176+
| `name` | Consul service name + `/etc/hosts` alias. Apps dial `${name}:${port}`. |
177+
| `port` | Canonical app port. App binds `127.0.0.1:port`. Two entries sharing a port collapse onto one **backend** (one Envoy supervise loop, one `sidecar_port`, one Connect-mTLS endpoint) — that's how `postgres-master` and `postgres-replica` ride the same Patroni instance. |
178+
| `subset` | Optional Consul service-subset filter (matches `Service.Tags`). Each subset-bearing entry generates a redirect resolver to the parent backend. Patroni's role-watcher (in `patroni/entrypoint.sh`) updates those tags on role flips. |
179+
180+
VIP octets and `sidecar_port` numbers are computed in HCL from
181+
declaration order — first service gets `vip=10 sidecar_port=21000`,
182+
second `vip=11 sidecar_port=21001`, and so on. Plan-time validation
183+
catches duplicate `(name, subset)` tuples.
184+
185+
### Workload-specific pieces remaining
186+
187+
Only two files contain workload-specific logic after this refactor:
127188

128-
| Patroni-specific | Lives in |
189+
| Workload-specific | Lives in |
129190
|---|---|
130-
| The Patroni image itself | `patroni/` |
131-
| Patroni env block + REST/Postgres port choices | `compose/worker.yaml` |
132-
| Postgres service VIPs + Connect upstreams | `cluster-example/cluster.tf` (`local.service_vips`) |
133-
| Postgres Connect sidecar registration + Envoy launch | `mesh-sidecar/entrypoint.sh` |
134-
135-
To run something else (a Redis cluster, a Kafka broker, your own
136-
stateful service): swap those three pieces, leave `mesh-conn`,
137-
`bootstrap-secrets`, `consul`, `sidecar`, the coordinator topology,
138-
and the Terraform structure as-is.
191+
| The Patroni image itself + role-watcher loop | `patroni/` |
192+
| Patroni env block (`CLUSTER_NAME`, replication passwords, etc.) | `compose/worker.yaml` |
193+
194+
`mesh-sidecar/entrypoint.sh` contains zero per-workload code paths —
195+
grep it for `patroni` or `webdemo` and you get nothing. To run
196+
something other than Patroni (Redis, Kafka, your own service): replace
197+
the `patroni` compose service and image with your own, edit
198+
`local.services` to declare its names + ports, and leave the rest of
199+
the platform plumbing untouched.
139200

140201
## Key operational properties
141202

consul-postgres-ha/design/README.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ in their face.
1212

1313
| Doc | What |
1414
|---|---|
15-
| [`declarative-services.md`](declarative-services.md) | Adding a microservice should be one HCL block in `cluster.tf`. Today's `mesh-conn` allowlist is hardcoded Go; sidecar registration lives in app binaries; Envoy supervise loops are copy-pasted. This pass makes the service list declarative and generates the rest. |
1615
| [`attestation-admission.md`](attestation-admission.md) | Use dstack TEE attestation as the mesh-conn admission credential, replacing/augmenting the shared TURN HMAC. Phased plan: per-app-id first, Consul-KV-rooted policy later. |
1716

1817
Each doc includes:

0 commit comments

Comments
 (0)