docs(consul-postgres-ha): explain why subsets exist (Patroni-style HA)

A user reading the declarative-services config can see `subset` as a
field but not understand *why* it's there. The reason is load-bearing
and worth being explicit about: subsets are how Consul tracks
dynamic state (a Patroni instance's master/replica role) without
re-registration. Workloads with built-in Consul integration rewrite
their own tags on role changes; consumers track those changes via
EDS push, transparent to the app. That's what makes
`postgres-master:5432` always reach whoever is leader right now,
sub-second.

Three places of doc, increasing in depth:

1. cluster.tf: a comment block at `services_raw` explains the
   "why" at first contact with the config.
2. README "Adapting to your own workload": new section walks the
   two patterns explicitly — Consul-blind workload (Pattern A,
   webdemo) vs Consul-native (Pattern B, Patroni) — with a decision
   table for which to pick.
3. ARCHITECTURE.md: new section "How the master tag flows through
   EDS on failover" traces the chain end-to-end with a timing
   breakdown.

Config syntax stays at native Consul vocabulary (`subset`,
`service-resolver`) — the audience for this template is people who
know or are learning Consul service mesh, and the 1-to-1 mapping
between our config and Consul's docs is worth more than a prettier
alias.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: h4x3rotab

consul-postgres-ha/ARCHITECTURE.md

@@ -228,6 +228,53 @@ followers). The subset filter strips down EDS endpoints to the right
 peer's sidecar, and the `postgres-master` / `postgres-replica`
 service-resolvers redirect to the right subset of `postgres`.
 
+### How the master tag flows through EDS on failover
+
+The subset filter isn't just a static routing rule — it's what makes
+failover transparent at the consumer side. The chain is short and
+worth tracing:
+
+```
+T+0    Patroni promotes worker-5 (leadership lock moves in Consul KV).
+T+~1s  worker-5's role-watcher polls Patroni REST, sees role=master,
+       PUTs the sidecar registration with Tags=["master"].
+       (The old leader's role-watcher does the symmetric flip:
+       Tags=["replica"].)
+T+~1s  Local Consul agent on worker-5 anti-entropies the new tags up
+       to a Consul server.
+T+~1s  Consul server pushes an EDS update to every consumer's Envoy:
+       "for cluster postgres-master, the subset master is now
+       {worker-5's sidecar:21001} — the previous member is gone."
+T+~1s  Each consumer Envoy retargets the postgres-master listener at
+       127.10.0.20:5432. Existing in-flight connections to the old
+       endpoint die; the next connection lands on worker-5.
+```
+
+The whole loop is bounded by Patroni's `loop_wait` (10s default but
+we set the role-watcher to 5s) plus EDS push latency (sub-second on a
+healthy Consul). Failover RTO from the app's perspective is "the time
+my retry-on-disconnect loop takes" — no service-discovery code, no
+DNS update, no client-library failover string. Just an Envoy upstream
+endpoint set that changes underneath the listener.
+
+Two non-obvious consequences of doing it this way:
+
+- **No re-registration churn.** Patroni's instance stays continuously
+  registered in Consul's catalog as it gains and loses leadership.
+  Only the tag changes. A deregister/re-register flow would race
+  health checks and risk a "no member" window.
+- **The platform sidecar is unaware of any of this.** It registered
+  the Connect sidecar-proxy once at startup with no tags; the
+  workload (Patroni) owns the dynamic state. Pattern A workloads
+  (webdemo) never touch the role-watcher path — `subset = null`
+  means there's no tag-dependent EDS filter to update.
+
+If you write your own Consul-native workload (Pattern B — Vault,
+Nomad, custom Raft), the same shape applies: register your parent
+once under `CLUSTER_NAME`, write the tag-flipping loop in your
+workload's entrypoint, declare role-aware `service-resolver` subsets
+in `local.services`, and Connect's EDS plumbing handles the rest.
+
 ## mesh-conn × QUIC — how they work together
 
 The bit that's worth being precise about: mesh-conn is built on top

consul-postgres-ha/README.md

@@ -182,6 +182,88 @@ declaration order — first service gets `vip=10 sidecar_port=21000`,
 second `vip=11 sidecar_port=21001`, and so on. Plan-time validation
 catches duplicate `(name, subset)` tuples.
 
+### Two patterns: Consul-blind vs. Consul-native workloads
+
+A workload's relationship to Consul determines how its service-mesh
+entry is shaped. Two patterns cover the realistic cases:
+
+#### Pattern A — Consul-blind workload (`webdemo` is the example)
+
+The app is a plain process that listens on a port. It doesn't know
+Consul exists; it never opens a connection to the local Consul agent;
+it has no leader/follower concept. The **platform sidecar** registers
+the service on the app's behalf, with the canonical port as the
+backend and a standalone Connect-proxy in front of it. There's only
+one parent service, named the same as the entry, no subsets.
+
+Declaration:
+
+```hcl
+{ name = "webdemo", port = 8080, subset = null }
+```
+
+This is the default for "I just wrote a microservice and I want it on
+the mesh." If you have no opinions about leader election, this is the
+shape to use.
+
+#### Pattern B — Consul-native workload (`patroni` is the example)
+
+The app integrates with Consul as part of how it operates — Patroni
+uses Consul's KV store as its leader-election lock and registers
+itself in Consul's service catalog under its `scope` (which we set to
+`CLUSTER_NAME` = `"demo"`). On every failover, Patroni rewrites tags
+(`master` / `replica`) on its own registration. From a single parent
+service we get *multiple* logical names — one per role — each as a
+service-resolver that filters the parent's tags into the subset its
+consumer wants:
+
+```hcl
+{ name = "postgres-master",  port = 5432, subset = "master"  }
+{ name = "postgres-replica", port = 5432, subset = "replica" }
+```
+
+Two entries, same canonical port → one Envoy public listener shared
+across both, two service-resolvers with different subset filters. The
+platform sidecar does **not** register a parent for this case (Patroni
+already did it); it only stamps the Connect sidecar-proxy. The
+role-watcher in `patroni/entrypoint.sh` is the loop that maintains
+tag consistency between Patroni's view of leadership and Consul's
+catalog.
+
+Failover round-trips through this in <1 second:
+
+```
+Patroni promotes worker-5 to leader
+      ↓
+worker-5's role-watcher writes Tags=["master"] on worker-5's sidecar
+      ↓
+Consul EDS push: subset master = [worker-5's sidecar:21001]
+      ↓
+every consumer's Envoy retargets `postgres-master` to worker-5
+      ↓
+next psql connection lands on worker-5
+```
+
+No DNS update, no service deregistration, no client-side retry-loop.
+That's what `subset` buys you, and it's why this field exists in our
+config even though only one of the three example services uses it.
+
+#### When to pick which
+
+| Question | Pattern A | Pattern B |
+|---|---|---|
+| Does your workload have a built-in Consul integration? | No | Yes |
+| Does your workload register itself with Consul? | No — platform does | Yes — workload does |
+| Does your workload have leader election? | No | Probably yes |
+| Will tags on the service change at runtime? | No | Yes (the workload rewrites them) |
+| Number of `local.services` entries per workload? | One | One per role |
+| Need a `subset` field? | No (`subset = null`) | One per role |
+
+If you're not sure, you're probably building Pattern A. Pattern B is
+specifically for "this thing has its own opinions about leader
+election that need to surface to consumers" — Patroni, Vault, Nomad,
+custom raft-based services.
+
 ### Workload-specific pieces remaining
 
 Only two files contain workload-specific logic after this refactor:

consul-postgres-ha/cluster-example/cluster.tf

@@ -164,12 +164,39 @@ locals {
   #   name    Consul service name (also the /etc/hosts alias).
   #   port    Canonical app port. App binds 127.0.0.1:port; the local
   #           Envoy upstream listener binds 127.10.0.<vip>:port.
-  #   subset  Optional. When set, this entry is a subset filter on a
-  #           shared backend (the postgres-master/-replica split is the
-  #           only case today). Entries sharing the same `port` collapse
-  #           into one producer-side sidecar — same Envoy public
-  #           listener, same Connect-mTLS endpoint, just different
-  #           service-resolver names with different subset filters.
+  #   subset  Optional. Set this when a single backend hosts multiple
+  #           role-aware logical services (the postgres-master/replica
+  #           split is the only case today). See below.
+  #
+  # ## Why `subset` exists
+  #
+  # `subset` is native Consul vocabulary — `service-resolver` config
+  # entries filter a parent service's instances by tag, and each
+  # filter is called a subset. We expose the same field here because
+  # the audience for this template is people who use (or are learning)
+  # Consul Connect, and importing Consul's vocabulary lets them map
+  # 1-to-1 between our config and Consul's docs.
+  #
+  # The reason this matters for HA Postgres specifically: Patroni
+  # uses Consul as its distributed lock store (DCS) and registers
+  # itself with the local Consul agent on startup. As leadership
+  # changes, Patroni rewrites the `master` / `replica` tag on its own
+  # service instance — no re-registration, no destroy/recreate. A
+  # `service-resolver` with `Subsets { master = Filter("...master") }`
+  # then routes consumer traffic by tag, transparent to the app.
+  # That's what makes `postgres-master:5432` always reach whoever is
+  # leader right now, with sub-second failover via Consul's EDS push.
+  #
+  # So `subset` is not decoration — it's the field that turns a
+  # static service name (`postgres-master`) into a live, leader-
+  # aware route. Workloads that don't have a leader concept
+  # (webdemo) leave it null and Consul Connect treats the entry as
+  # a plain service-resolver redirect.
+  #
+  # See README.md "Adapting to your own workload" for the
+  # two-pattern walkthrough (Consul-blind vs Consul-native), and
+  # ARCHITECTURE.md for how tags flow through the EDS data plane on
+  # failover.
   services_raw = [
     { name = "webdemo", port = 8080, subset = null },
     { name = "postgres-master", port = 5432, subset = "master" },