Commit 808836c
fix(consul-postgres-ha): converge ICE in ~8s after peer mesh-conn restart
When one peer's mesh-conn process restarts (container hot-patch, crash
loop, etc.) the surviving peers used to wait ~60s for QUIC's idle
timeout before noticing the link was dead and retrying. Each retry
also had a candidate-loss race where the restarted peer's freshly-
published candidates landed on the survivors' soon-to-be-discarded
agent and never made it onto the next attempt's agent. Net effect:
recovery was slow and often required a manual restart of the survivor.
Three coordinated fixes in the per-peer session, with one ICE agent
constructed per attempt (not persistent):
1. abortAttempt: pollLoop tracks the remote ufrag we Dial'd against
and cancels the entire attempt context the moment peer publishes
a fresher one. Trips ICE's stale-creds dead-end immediately rather
than waiting for the 30s connectivity-check timeout.
2. candidateBuffer: pollLoop appends every received remote candidate
to a session-owned slice in addition to AddRemoteCandidate'ing it
onto the live agent. dialICE drains the slice into each freshly-
built agent before Dial, so candidates that arrived while we were
between attempts (no agent installed yet) aren't lost. Buffer is
cleared on fresh peer auth (old candidates won't match new ufrag).
3. authCh always wins-on-latest: pollLoop drain-then-pushes peer auth
independent of the abort decision, so the next dialICE iteration
reads the freshest value when it wakes — no risk of dropping the
message that would have unblocked it.
Considered persisting the ice.Agent across attempts via Agent.Restart()
but pion doesn't reset agent.startedCh: a second Dial returns
ErrMultipleStart. Restart() is designed for the WebRTC pattern where
the ice.Conn outlives the credential cycle and the agent re-pairs
silently. Our pattern (re-Dial + fresh QUIC per attempt) needs a new
agent each time. Documented inline.
Verified live on a 3+3 stage4 cluster: hot-patching worker-1's
sidecar container, peers detected the abort 0s after the kill and
linked back up 8s after T_kill. Patroni cluster stayed healthy
through the cycle (no failover); consul members all alive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent a3a63b1 commit 808836c
1 file changed
Lines changed: 205 additions & 114 deletions
0 commit comments