Skip to content

Commit 57705ad

Browse files
h4x3rotabclaude
andcommitted
fix(consul-postgres-ha): role-watcher + set-e isolation in supervise loops
Two related fixes: 1. **Role watcher**: service-resolver subset filters in Connect EDS apply to the SIDECAR registration's Tags, not to the parent service's tags (Patroni's stage4 instance tags don't propagate to the postgres-sidecar-proxy registration). Added a polling loop in entrypoint.sh that watches Patroni's REST API at `127.0.0.1:8008/` and re-PUTs the postgres-sidecar registration with `Tags=["master"]` or `["replica"]` whenever Patroni's role changes. Initial registration starts with Tags=[]; the watcher fills it in within ~5s of Patroni REST coming up. 2. **Supervise-loop isolation**: each Envoy + the role-watcher run in a `(... &) | prefix` subshell. They inherited the outer entrypoint's `set -euo pipefail`, so a single failed `curl` (Patroni REST not yet up) or an Envoy crash (no Consul leader yet → bootstrap config can't be fetched) propagated as subshell exit → `wait -n` returned → container teardown cascade. Added `set +eo pipefail` at the start of each supervise subshell so the while-true retry loop survives expected transient failures. This is what was missing for the first end-to-end Patroni-via-Connect to fire: without role tags on the sidecar, postgres-master EDS returns empty; without supervise-loop isolation, the first Envoy crash kills the container. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 9bb3295 commit 57705ad

1 file changed

Lines changed: 44 additions & 0 deletions

File tree

consul-postgres-ha/mesh-sidecar/entrypoint.sh

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,7 @@ if [ "$ROLE" = "worker" ]; then
174174
"Name": "postgres-sidecar-proxy",
175175
"Address": "127.50.0.${SELF_VIP}",
176176
"Port": 21001,
177+
"Tags": [],
177178
"Proxy": {
178179
"DestinationServiceName": "${CLUSTER_NAME}",
179180
"LocalServiceAddress": "127.0.0.1",
@@ -223,7 +224,12 @@ EOF
223224
# list, but the bootstrap config is fine). Webdemo registers its own
224225
# SidecarService block from webdemo/main.go; the bootstrap call
225226
# blocks until that registration is in place.
227+
#
228+
# `set +e` because envoy crashes mid-startup count as expected events
229+
# the while loop handles by retrying — outer `set -e` would bubble
230+
# them up as a "child exited" → container teardown.
226231
(
232+
set +eo pipefail
227233
while true; do
228234
if consul connect envoy \
229235
-sidecar-for="webdemo-${PEER_ID}" \
@@ -242,6 +248,7 @@ EOF
242248
# Postgres Envoy supervise loop, same pattern. Bootstrap from the
243249
# standalone connect-proxy registration above.
244250
(
251+
set +eo pipefail
245252
while true; do
246253
if consul connect envoy \
247254
-proxy-id="postgres-sidecar-${PEER_ID}" \
@@ -256,6 +263,43 @@ EOF
256263
done
257264
) 2>&1 | prefix envoy-postgres &
258265
ENVOYS+=("$!")
266+
267+
# Role watcher: mirror Patroni's current role into the postgres-
268+
# sidecar registration's Tags. Service-resolver subset filters
269+
# (Service.Tags contains "master") apply to the SIDECAR registration,
270+
# not to Patroni's parent service. So we poll Patroni's REST API
271+
# (127.0.0.1:8008, shared via network_mode: host) and re-PUT the
272+
# sidecar with Tags=["master"] / ["replica"] on role change. Patroni
273+
# itself drives the role via its DCS leader-lock; this loop just
274+
# mirrors that into a shape Connect EDS understands.
275+
#
276+
# Initial state: Tags=[] (the registration above). Until the watcher
277+
# runs at least once, postgres-master and postgres-replica EDS
278+
# return empty. The first iteration runs ~5s after Patroni REST
279+
# comes up; consumers pre-watch will reconnect when EDS populates.
280+
(
281+
set +eo pipefail
282+
PREV=""
283+
while true; do
284+
sleep 5
285+
ROLE=$(curl -fsS --max-time 2 http://127.0.0.1:8008/ 2>/dev/null \
286+
| jq -r '.role // empty' 2>/dev/null)
287+
case "$ROLE" in
288+
master|primary) TAG="master" ;;
289+
replica|standby_leader|sync_standby) TAG="replica" ;;
290+
*) continue ;;
291+
esac
292+
[ "$TAG" = "$PREV" ] && continue
293+
jq --arg t "$TAG" '.Tags = [$t]' /tmp/postgres-sidecar.json \
294+
> /tmp/postgres-sidecar.tagged.json
295+
if curl -fsS -X PUT --data-binary @/tmp/postgres-sidecar.tagged.json \
296+
http://127.0.0.1:8500/v1/agent/service/register; then
297+
echo "[role-watcher] postgres-sidecar tag: $PREV -> $TAG (patroni role=$ROLE)"
298+
PREV="$TAG"
299+
fi
300+
done
301+
) 2>&1 | prefix role-watcher &
302+
ENVOYS+=("$!")
259303
fi
260304

261305
# ---- 6. coordinator-0: write Connect config entries (idempotent) ----

0 commit comments

Comments
 (0)