Skip to content

Commit 808836c

Browse files
h4x3rotabclaude
andcommitted
fix(consul-postgres-ha): converge ICE in ~8s after peer mesh-conn restart
When one peer's mesh-conn process restarts (container hot-patch, crash loop, etc.) the surviving peers used to wait ~60s for QUIC's idle timeout before noticing the link was dead and retrying. Each retry also had a candidate-loss race where the restarted peer's freshly- published candidates landed on the survivors' soon-to-be-discarded agent and never made it onto the next attempt's agent. Net effect: recovery was slow and often required a manual restart of the survivor. Three coordinated fixes in the per-peer session, with one ICE agent constructed per attempt (not persistent): 1. abortAttempt: pollLoop tracks the remote ufrag we Dial'd against and cancels the entire attempt context the moment peer publishes a fresher one. Trips ICE's stale-creds dead-end immediately rather than waiting for the 30s connectivity-check timeout. 2. candidateBuffer: pollLoop appends every received remote candidate to a session-owned slice in addition to AddRemoteCandidate'ing it onto the live agent. dialICE drains the slice into each freshly- built agent before Dial, so candidates that arrived while we were between attempts (no agent installed yet) aren't lost. Buffer is cleared on fresh peer auth (old candidates won't match new ufrag). 3. authCh always wins-on-latest: pollLoop drain-then-pushes peer auth independent of the abort decision, so the next dialICE iteration reads the freshest value when it wakes — no risk of dropping the message that would have unblocked it. Considered persisting the ice.Agent across attempts via Agent.Restart() but pion doesn't reset agent.startedCh: a second Dial returns ErrMultipleStart. Restart() is designed for the WebRTC pattern where the ice.Conn outlives the credential cycle and the agent re-pairs silently. Our pattern (re-Dial + fresh QUIC per attempt) needs a new agent each time. Documented inline. Verified live on a 3+3 stage4 cluster: hot-patching worker-1's sidecar container, peers detected the abort 0s after the kill and linked back up 8s after T_kill. Patroni cluster stayed healthy through the cycle (no failover); consul members all alive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent a3a63b1 commit 808836c

1 file changed

Lines changed: 205 additions & 114 deletions

File tree

  • consul-postgres-ha/mesh-conn

0 commit comments

Comments
 (0)