docs(consul-postgres-ha): tfvars credential-trap fix + ICE auth-race bug writeup

Two findings from live verification of the declarative-services refactor:

1. README's PHALA_CLOUD_API_KEY snippet silently followed
   `~/.phala-cloud/credentials.json`'s `current_profile`, so the
   deploy billed whichever account was last `phala auth login`-ed
   into. Replace with an explicit `export PHALA_CLOUD_API_KEY=phak_...`
   instruction so the user picks a specific token. Caught when a
   verification run deployed onto an unrelated profile.

2. mesh-conn's "fresh auth supersedes consumed" cancellation
   (commit cc5a12d) is symmetric — when both peers restart in
   close succession, each side keeps cancelling the other's
   in-flight dial as fresher auth arrives, and the 5-second
   back-off doesn't break the symmetry. Observed in live testing:
   multiple worker↔coordinator pairs flapping >7 minutes without
   converging, blocking Consul RPC and replica pg_basebackup.

   Adds an "ICE auth-race convergence bug" section to
   ROBUSTNESS.md walking the symptom, the diagnosis, and three
   possible fix shapes (asymmetric back-off keyed on peer-id
   lexicographic order is the cheapest experiment). Promotes it
   to top of the Layer 1 recommended-fixes list.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: h4x3rotab

consul-postgres-ha/README.md

@@ -54,7 +54,11 @@ inside the app.
 
 Prerequisites:
 
-- A Phala Cloud account with API credentials at `~/.phala-cloud/credentials.json`.
+- A Phala Cloud API key for the account you want this cluster billed to.
+  Get one from the Phala dashboard (Settings → API Keys) or use the
+  one already in your `~/.phala-cloud/credentials.json` profile if
+  you've used `phala` CLI before — but **pick a specific token**,
+  don't let it follow whatever profile is currently selected.
 - A Linux box with a public IP for the external coordinator (coturn + signaling).
 - The four container images (`mesh-sidecar`, `patroni`, `webdemo`,
   `signaling`) either already published to GHCR (via the CI workflow
@@ -66,9 +70,11 @@ cd consul-postgres-ha/cluster-example
 cp terraform.tfvars.example terraform.tfvars
 $EDITOR terraform.tfvars   # set gateway_domain, image refs, external_*
 
-export PHALA_CLOUD_API_KEY=$(python3 -c "
-import json; d=json.load(open('$HOME/.phala-cloud/credentials.json'))
-print(d['profiles'][d['current_profile']]['token'])")
+# Set this explicitly to a specific account's token. Do NOT auto-pull
+# from credentials.json's `current_profile` — that follows whichever
+# account you last `phala auth login`-ed into, and a wrong-account
+# deploy is silent until you spot it in the dashboard.
+export PHALA_CLOUD_API_KEY=phak_...
 
 terraform init
 terraform apply -parallelism=1   # phala-cloud#247 needs serial creates

consul-postgres-ha/ROBUSTNESS.md

@@ -97,13 +97,80 @@ whose direct path goes down can't fail over to TURN until it's back".
 | Resource exhaustion (many TCP streams) | QUIC per-session limits kick in (256 streams default); new TCP streams to that peer fail. UDP and existing TCP unaffected. | bump `AcceptBacklog` / `MaxIncomingStreams` if it ever hits us at scale. |
 | Head-of-line blocking | A big TCP write on one stream briefly delays a UDP datagram or another TCP stream. Imperceptible at Consul scale. | None needed today. If a future workload becomes jitter-sensitive, split into two ICE conns per pair (UDP-only + TCP-only). |
 
+### The ICE auth-race convergence bug — the one real open issue
+
+`dialICE` both publishes its local auth (ufrag+pwd) via the signalling
+broker and waits on the remote auth from the broker. On a peer
+restart or fresh bring-up, both sides republish in close succession.
+mesh-conn's protection against consuming a stale auth is to compare
+the most-recently-seen remote auth against the one it's currently
+dialing against; if a fresher auth arrives mid-dial, the in-flight
+attempt is cancelled and `runPeerLink` retries after 5 s.
+
+What goes wrong is that **both peers do this symmetrically**:
+
+```
+peer-A consumed=X publishes auth Y → dials with peer-B's X
+peer-B sees A's Y, supersedes its X-bound dial, publishes new auth Z
+peer-A sees B's Z, supersedes its Y-bound dial
+peer-B sees A's next auth, supersedes again
+…
+```
+
+The 5 s back-off doesn't break the symmetry, so the loop never
+converges. Observed in live testing on 2026-05-13: multiple worker↔
+coordinator and worker↔worker pairs flapping for 7+ minutes without
+stabilizing, even with `MESH_CONN_RELAY_ONLY=1` set (so it's not a
+NAT-direct-path issue). Per-peer log signature:
+
+```
+[<peer>] ice state: Checking
+[<peer>] fresh auth (ufrag=Y) supersedes consumed (ufrag=X) — aborting attempt
+[<peer>] link failed: ice: connecting canceled by caller — retrying in 5s
+[<peer>] ice state: Closed
+```
+
+The previous commit message acknowledged this: `cc5a12d fix(consul-
+postgres-ha): partial — cancel in-flight ICE dial on fresher peer
+auth`. The cancellation logic is correct; the convergence isn't.
+
+Practical impact: when this happens, the affected peer pair never
+forms a working link, which means the Consul agent on one side
+can't reach the server tier (Raft RPC on `:8300`), Envoy can't fetch
+its xDS config, and any cross-peer Connect call through that pair
+fails. Patroni leader election still works (Consul KV is on the
+working pairs), but replica `pg_basebackup` fails because the
+replica's local Envoy has no endpoint for `postgres-master`.
+
+**Possible fix shapes**:
+
+- **Asymmetric back-off keyed on peer ID lexicographic order.** The
+  side with the *higher* peer-id waits 5 s before redialing; the
+  *lower* side dials immediately. Breaks the symmetry; one side
+  always wins the auth race within one cycle.
+- **Don't supersede if the new auth came from a poll happening
+  during the in-flight dial.** Today's logic supersedes on any newer
+  auth; a tighter rule would only supersede if the auth changed for
+  a reason other than "the remote peer also just restarted." Harder
+  to define; the asymmetric back-off is simpler.
+- **A "settling" window after publishing auth.** Don't act on a
+  remote auth that arrives in the first ~1 s after we publish ours;
+  the remote is probably reacting to our publish, not to a peer
+  restart we should respect.
+
+The first of these is the cheapest experiment. Needs a focused
+mesh-conn debugging session with `MESH_CONN_DEBUG_ICE=1` traces from
+a fresh repro, looking at which side initiates each supersession.
+
 ### Recommended fixes
 
-1. **Set a QUIC read deadline on the UDP-stream pumps**, so if a
+1. **The ICE auth-race convergence bug above** — top priority. Blocks
+   end-to-end live verification of every recent refactor.
+2. **Set a QUIC read deadline on the UDP-stream pumps**, so if a
    stream silently stalls (QUIC keep-alive happens at session
    level, not stream level), the pump returns and `runPeerLink`
    restarts.
-2. **Tune QUIC `MaxStreamWindowSize`** if we ever need higher
+3. **Tune QUIC `MaxStreamWindowSize`** if we ever need higher
    throughput; default is 256 KB which is fine for now.
 
 ### Already shipped