fix(consul-postgres-ha): unify Connect registration on inline SidecarService

h4x3rotab · claude · h4x3rotab · commit ddb2fa23d663 · 2026-05-14T10:27:01.000Z
The previous design carved Connect registration into two patterns: A
(platform owns the parent service + sidecar inline) and B (workload
owns the parent, platform registers a standalone `Kind: connect-proxy`
pointing at it). Pattern B is what the Patroni-style HA workload was
supposed to use.

What that shape misses: Consul's discovery chain does NOT return a
proxy as a Connect endpoint for its parent unless the parent
registration carries the inline `Connect.SidecarService` field. With
two-table-entries Pattern B, `proxycfg`'s local-destination watch
satisfies (the proxy points at a known DestinationServiceID) — but
`/v1/health/connect/postgres-master` returns an empty endpoint set,
Envoy's upstream cluster gets zero hosts, and every consumer-side
listener accepts then immediately closes, surfacing as
`pg_basebackup: server closed the connection unexpectedly` on every
replica startup.

Unify both patterns onto the inline shape: one registration per
backend, with the proxy declared as `Connect.SidecarService` on the
parent service. The platform sidecar pre-registers the parent +
inline sidecar idempotently at boot; for Patroni's HA case, Patroni's
own `register_service` is set to `false` so it doesn't deregister
the platform's parent every loop iteration, and a role-watcher in
`patroni/entrypoint.sh` re-PUTs the same parent service_id with
`Tags=["master"]` or `["replica"]` (the full spec, including the
inline SidecarService block, so the proxy registration survives the
tag flip).

Three secondary fixes that fell out:

- Service IDs use hyphens only (`${parent}-${peer-id}`). Patroni's
  natural `${scope}/${name}` convention contains a `/`, which trips
  up `consul connect envoy -sidecar-for`'s URL handling.
- Role-watcher polls `/patroni` instead of `/` on the Patroni REST.
  `/` returns 503 on non-leaders, which made the role-watcher's
  `curl -fsS http://:8008/` fail forever on every replica and the
  `replica` tag never propagated.
- `patroni/Dockerfile` adds `curl` (the role-watcher's HTTP client).

End-to-end verified on a fresh 6-CVM cluster: 1 leader + 2 streaming
replicas inside ~3 min of apply, zero persistent Consul proxycfg
errors, soft-kill leader → 33s failover RTO (matches the documented
baseline), old leader rejoins via WAL replay.

Co-Authored-By: Claude Opus 4.7 (1M context) &lt;noreply@anthropic.com&gt;
diff --git a/consul-postgres-ha/compose/worker.yaml b/consul-postgres-ha/compose/worker.yaml
@@ -78,10 +78,10 @@ services:
     # CLUSTER_NAME drives Patroni's `scope` — every peer's patroni
     # must use the same value to land in the same cluster. Postgres
     # binds the canonical 127.0.0.1:5432; Patroni REST on 127.0.0.1:8008.
-    # The Patroni entrypoint also hosts the role-watcher loop that
-    # mirrors Patroni's current role into the postgres-sidecar
-    # registration's Service.Tags (workload-specific, lives here
-    # instead of in the platform sidecar).
+    # Patroni's native Consul integration registers the parent service
+    # `${scope}/${peer}` with role-aware tags; the platform sidecar
+    # pre-registers a stub at the same service_id so consumer-side
+    # Envoy can come up before Patroni has bootstrapped.
     # WORKAROUND: superuser + replication passwords generated in
     # Terraform and broadcast identically to every worker. See
     # cluster.tf.
@@ -91,11 +91,10 @@ services:
       - PATRONI_REPLICATION_PW=${PATRONI_REPLICATION_PW}
       # SERVICES_JSON: service VIPs for /etc/hosts (patroni needs to
       # resolve postgres-master to its local Envoy upstream so
-      # primary_conninfo routes through the mesh), AND the sidecar
-      # registration shape the role-watcher re-PUTs with tags.
+      # primary_conninfo routes through the mesh).
       - SERVICES_JSON=${SERVICES_JSON}
       # PEERS_JSON: role-watcher needs SELF_VIP to set Address on the
-      # sidecar registration.
+      # inline SidecarService entry.
       - PEERS_JSON=${PEERS_JSON}
     volumes:
       - /tmp/dstack-runtime/instance:/run/instance:ro
diff --git a/consul-postgres-ha/mesh-sidecar/entrypoint.sh b/consul-postgres-ha/mesh-sidecar/entrypoint.sh
@@ -39,11 +39,18 @@
 #   ]
 #
 # Services with the same canonical `port` collapse onto one backend
-# (one sidecar_port, one Envoy). `registers_parent=false` means the
-# workload auto-registers its parent service under its own scope; the
-# platform only registers a standalone connect-proxy pointing at it,
-# and the workload owns any role-tag mutation on the sidecar via the
-# Consul agent HTTP API.
+# (one sidecar_port, one Envoy). For `registers_parent=false`, the
+# workload owns the *authoritative* parent service registration
+# (tags, health check) under its own `scope` — e.g. Patroni's native
+# Consul integration registers `${scope}/${peer}` with role tags and a
+# REST-API health check. To avoid the chicken-and-egg where the
+# workload can't run (no consumer-side Envoy listener yet) before it
+# has registered (Patroni only registers after pg_basebackup, which
+# needs the listener), the platform pre-registers a *stub* parent
+# service at the same service_id so Consul's proxycfg can populate
+# the connect-proxy's state immediately. The workload's later PUT
+# overwrites the stub in place (same service_id ⇒ idempotent), adding
+# its check and role tags.
 #
 # Supervision policy: any one inner process dying takes the whole
 # container down. Compose `restart: on-failure` brings it back in
@@ -240,78 +247,76 @@ if [ "$ROLE" = "worker" ]; then
     SPEC_FILE="/tmp/sidecar-${PARENT}.json"
     ENVOY_BOOT="/tmp/envoy-${PARENT}.json"
 
+    # Both patterns use a single registration with inline SidecarService:
+    # that's the shape Consul recognises as a Connect-aware service +
+    # its proxy. A bare `Kind: connect-proxy` plus a separate parent
+    # without `Connect.SidecarService` is not enough — Consul won't
+    # return the proxy as a Connect endpoint for the parent, so the
+    # discovery chain resolves to zero hosts and consumer Envoys close
+    # incoming connections immediately.
+    #
+    # The two patterns differ only in service_id and what overwrites
+    # the registration over time:
+    #
+    #   pattern A (webdemo): platform-only. Parent ID is
+    #     "${parent}-${peer}", tagged "peer=${peer}". Nothing else
+    #     touches this registration.
+    #
+    #   pattern B (Patroni): platform pre-registers a "stub" with no
+    #     role tags. Patroni's entrypoint role-watcher re-PUTs the
+    #     same service_id with tags ["master"] / ["replica"] tracked
+    #     from Patroni REST. We keep `register_service: false` in
+    #     patroni.yml so Patroni's own integration stays out of the
+    #     service catalog (it only uses Consul as DCS); see
+    #     patroni/entrypoint.sh for the role-watcher.
+    # Service ID format is `${parent}-${peer}` for both patterns so it
+    # never contains a slash. Slashes in service IDs trip up
+    # `consul connect envoy -sidecar-for` URL handling.
+    PARENT_ID="${PARENT}-${PEER_ID}"
     if [ "$REGISTERS_PARENT" = "true" ]; then
-      # Pattern A: platform owns the parent service AND its sidecar.
-      # TCP check on the canonical port — generic across HTTP/non-HTTP
-      # workloads; passes once the app's listener is up. SidecarService
-      # inline means a single PUT registers both records.
-      jq -n \
-        --arg name "$PARENT" \
-        --arg id "${PARENT}-${PEER_ID}" \
-        --arg peer "$PEER_ID" \
-        --argjson port "$LOCAL_PORT" \
-        --arg vip "$SELF_VIP" \
-        --argjson sport "$SIDECAR_PORT" \
-        --argjson upstreams "$UPSTREAMS" '{
-          Name: $name,
-          ID: $id,
-          Address: "127.0.0.1",
-          Port: $port,
-          Tags: ["peer=" + $peer],
-          Check: {
-            TCP: ("127.0.0.1:" + ($port|tostring)),
-            Interval: "10s",
-            Timeout: "2s",
-            DeregisterCriticalServiceAfter: "1m"
-          },
-          Connect: {
-            SidecarService: {
-              Address: ("127.50.0." + $vip),
-              Port: $sport,
-              Proxy: {
-                LocalServiceAddress: "127.0.0.1",
-                LocalServicePort: $port,
-                Upstreams: $upstreams
-              }
-            }
-          }
-        }' > "$SPEC_FILE"
-      ENVOY_ARGS=(-sidecar-for="${PARENT}-${PEER_ID}")
+      TAGS_JSON='["peer='"$PEER_ID"'"]'
     else
-      # Pattern B: workload auto-registers the parent under its
-      # `scope`; platform registers a standalone connect-proxy pointing
-      # at it. Role/state tags on the sidecar (e.g. master/replica) are
-      # managed by the workload itself via the Consul agent HTTP API;
-      # the subset resolvers below filter on those tags.
-      jq -n \
-        --arg id "$SIDECAR_ID" \
-        --arg name "${PARENT}-sidecar-proxy" \
-        --arg parent "$PARENT" \
-        --arg vip "$SELF_VIP" \
-        --argjson port "$LOCAL_PORT" \
-        --argjson sport "$SIDECAR_PORT" \
-        --argjson upstreams "$UPSTREAMS" '{
-          Kind: "connect-proxy",
-          ID: $id,
-          Name: $name,
-          Address: ("127.50.0." + $vip),
-          Port: $sport,
-          Tags: [],
-          Proxy: {
-            DestinationServiceName: $parent,
-            LocalServiceAddress: "127.0.0.1",
-            LocalServicePort: $port,
-            Upstreams: $upstreams
-          }
-        }' > "$SPEC_FILE"
-      ENVOY_ARGS=(-proxy-id="$SIDECAR_ID")
+      TAGS_JSON='[]'
     fi
 
+    jq -n \
+      --arg name "$PARENT" \
+      --arg id "$PARENT_ID" \
+      --argjson port "$LOCAL_PORT" \
+      --arg vip "$SELF_VIP" \
+      --argjson sport "$SIDECAR_PORT" \
+      --argjson tags "$TAGS_JSON" \
+      --argjson upstreams "$UPSTREAMS" '{
+        Name: $name,
+        ID: $id,
+        Address: "127.0.0.1",
+        Port: $port,
+        Tags: $tags,
+        Check: {
+          TCP: ("127.0.0.1:" + ($port|tostring)),
+          Interval: "10s",
+          Timeout: "2s",
+          DeregisterCriticalServiceAfter: "1m"
+        },
+        Connect: {
+          SidecarService: {
+            Address: ("127.50.0." + $vip),
+            Port: $sport,
+            Proxy: {
+              LocalServiceAddress: "127.0.0.1",
+              LocalServicePort: $port,
+              Upstreams: $upstreams
+            }
+          }
+        }
+      }' > "$SPEC_FILE"
+    ENVOY_ARGS=(-sidecar-for="$PARENT_ID")
+
     # Re-register on every boot via the agent HTTP API (PUT-idempotent).
-    log "registering ${PARENT} sidecar (port=${SIDECAR_PORT}, base-id=${BASE_ID})"
+    log "registering ${PARENT} (id=${PARENT_ID}) + sidecar (port=${SIDECAR_PORT}, base-id=${BASE_ID})"
     until curl -fsS -X PUT --data-binary @"$SPEC_FILE" \
              http://127.0.0.1:8500/v1/agent/service/register; do
-      log "${PARENT} sidecar register failed; retrying"
+      log "${PARENT} register failed; retrying"
       sleep 2
     done
 
diff --git a/consul-postgres-ha/patroni/Dockerfile b/consul-postgres-ha/patroni/Dockerfile
@@ -6,7 +6,7 @@
 # /etc/patroni.yml and execs `patroni`.
 FROM postgres:16-alpine
 
-RUN apk add --no-cache python3 py3-pip py3-psycopg2 py3-yaml jq tini su-exec
+RUN apk add --no-cache python3 py3-pip py3-psycopg2 py3-yaml jq tini su-exec curl
 
 # python-consul + python-etcd are optional; we only ship the consul DCS.
 RUN pip install --no-cache-dir --break-system-packages \
diff --git a/consul-postgres-ha/patroni/entrypoint.sh b/consul-postgres-ha/patroni/entrypoint.sh
@@ -65,8 +65,17 @@ restapi:
 
 consul:
   host: ${CONSUL_HTTP}
-  register_service: true
-  service_check_interval: 5s
+  # Patroni's native register_service is set to false because the
+  # platform sidecar owns the parent service registration (Address,
+  # Port, check). The role-watcher loop below polls Patroni's REST API
+  # and re-PUTs the parent registration with tags ["master"] or
+  # ["replica"] so service-resolver subset filters match. We do this
+  # because Patroni deregisters the service every loop iteration while
+  # `role=uninitialized` (the entire pre-bootstrap window on a
+  # replica), which wipes the registration our connect-proxy depends
+  # on for proxycfg state. With register_service:false Patroni stays
+  # out of the agent's service catalog and just uses Consul as DCS.
+  register_service: false
 
 bootstrap:
   dcs:
@@ -121,26 +130,19 @@ tags:
   clonefrom: false
 EOF
 
-# Role watcher: mirror Patroni's current role into the postgres-sidecar
-# registration's Tags. Lives here (not in the platform sidecar) because
-# the role <-> tag mapping is Patroni-specific knowledge — the platform
-# sidecar registers the proxy with no tags, and only Patroni knows when
-# it has flipped master/replica.
+# Role-watcher: poll Patroni's REST API and re-PUT the parent service
+# registration `${CLUSTER}-${PEER_ID}` with role-aware tags. Consul's
+# /v1/agent/service/register is a complete replace, so each PUT must
+# carry the full spec — same shape the platform sidecar pre-registers,
+# including the inline `Connect.SidecarService` that links the proxy
+# to this parent. Without that field Consul doesn't return the proxy
+# as a Connect endpoint for the parent and the discovery chain
+# resolves to zero hosts.
 #
-# Service-resolver subset filters (`Service.Tags contains "master"`)
-# apply to the SIDECAR registration, not Patroni's parent service. We
-# re-PUT the sidecar via the local consul agent's HTTP API; the PUT is
-# idempotent on (Name, ID), so it acts as an in-place tag update.
-PEERS_JSON_FOR_ROLE="${PEERS_JSON:?PEERS_JSON required (for role-watcher to compute self vip)}"
-SELF_VIP=$(echo "$PEERS_JSON_FOR_ROLE" | jq -r --arg id "$PEER_ID" '.[] | select(.id == $id) | .vip')
-[ -n "$SELF_VIP" ] || { echo "FATAL: $PEER_ID not in PEERS_JSON" >&2; exit 1; }
-
-# Same registration shape the platform sidecar emits in pattern B
-# (standalone connect-proxy, parent=CLUSTER auto-registered by Patroni).
-# We rebuild it here rather than relying on a shared file because the
-# platform sidecar lives in a different container with its own /tmp;
-# reconstructing from SERVICES_JSON + PEERS_JSON keeps this loop
-# self-contained.
+# The subset filters in the service-resolver
+# (`Service.Tags contains "master"` / "replica") match on the parent's
+# tags, so flipping `$TAG` here is what makes `postgres-master:5432`
+# follow the current leader live.
 ROLE_BACKEND=$(
   echo "$SERVICES_JSON" \
     | jq -c --arg parent "$CLUSTER" '
@@ -156,18 +158,22 @@ ROLE_BACKEND=$(
       '
 )
 
+PEERS_JSON_FOR_ROLE="${PEERS_JSON:?PEERS_JSON required (role-watcher needs SELF_VIP)}"
+SELF_VIP=$(echo "$PEERS_JSON_FOR_ROLE" | jq -r --arg id "$PEER_ID" '.[] | select(.id == $id) | .vip')
+[ -n "$SELF_VIP" ] || { echo "FATAL: $PEER_ID not in PEERS_JSON" >&2; exit 1; }
+
 if [ "$(echo "$ROLE_BACKEND" | jq -r '.sidecar_port // "null"')" = "null" ]; then
   echo "[role-watcher] no SERVICES_JSON entry with parent=$CLUSTER — skipping watcher"
 else
-  # Background subshell — tini reaps it when this script's process tree
-  # tears down on container stop. We don't trap or `wait`: the original
-  # design had this loop as a sibling background process in the platform
-  # sidecar, and we preserve that lifecycle here.
   (
+    echo "[role-watcher] starting (service_id=${CLUSTER}-${PEER_ID}, rest=127.0.0.1:${REST_PORT})"
     PREV=""
     while true; do
       sleep 5
-      PATRONI_ROLE=$(curl -fsS --max-time 2 http://127.0.0.1:${REST_PORT}/ 2>/dev/null \
+      # `/` returns 503 on non-leaders (libpq's idiomatic way to advertise
+      # role to load balancers); `/patroni` is 200 on every node and
+      # carries the same JSON.
+      PATRONI_ROLE=$(curl -fsS --max-time 2 "http://127.0.0.1:${REST_PORT}/patroni" 2>/dev/null \
                       | jq -r '.role // empty' 2>/dev/null)
       case "$PATRONI_ROLE" in
         master|primary)                       TAG="master"  ;;
@@ -176,28 +182,38 @@ else
       esac
       [ "$TAG" = "$PREV" ] && continue
       SPEC=$(jq -n \
-        --arg id "${CLUSTER}-sidecar-${PEER_ID}" \
-        --arg name "${CLUSTER}-sidecar-proxy" \
-        --arg parent "$CLUSTER" \
-        --arg vip "$SELF_VIP" \
+        --arg name "$CLUSTER" \
+        --arg id "${CLUSTER}-${PEER_ID}" \
         --arg tag "$TAG" \
-        --argjson backend "$ROLE_BACKEND" '{
-          Kind: "connect-proxy",
-          ID: $id,
+        --arg vip "$SELF_VIP" \
+        --argjson backend "$ROLE_BACKEND" \
+        --argjson port "$PG_PORT" '{
           Name: $name,
-          Address: ("127.50.0." + $vip),
-          Port: $backend.sidecar_port,
+          ID: $id,
+          Address: "127.0.0.1",
+          Port: $port,
           Tags: [$tag],
-          Proxy: {
-            DestinationServiceName: $parent,
-            LocalServiceAddress:    "127.0.0.1",
-            LocalServicePort:       $backend.port,
-            Upstreams:              $backend.upstreams
+          Check: {
+            TCP: ("127.0.0.1:" + ($port|tostring)),
+            Interval: "5s",
+            Timeout: "2s",
+            DeregisterCriticalServiceAfter: "1m"
+          },
+          Connect: {
+            SidecarService: {
+              Address: ("127.50.0." + $vip),
+              Port: $backend.sidecar_port,
+              Proxy: {
+                LocalServiceAddress: "127.0.0.1",
+                LocalServicePort:    $backend.port,
+                Upstreams:           $backend.upstreams
+              }
+            }
           }
         }')
       if printf '%s' "$SPEC" | curl -fsS -X PUT --data-binary @- \
-                                http://${CONSUL_HTTP}/v1/agent/service/register; then
-        echo "[role-watcher] ${CLUSTER}-sidecar tag: $PREV -> $TAG (patroni role=$PATRONI_ROLE)"
+                                "http://${CONSUL_HTTP}/v1/agent/service/register"; then
+        echo "[role-watcher] ${CLUSTER}-${PEER_ID} tag: ${PREV:-<none>} -> $TAG (patroni role=$PATRONI_ROLE)"
         PREV="$TAG"
       fi
     done
