Recover cluster leader from Postgres failover#280
Merged
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prune in-memory nodes that have left the store, but never add store-only nodes (preserving the split-brain guard). Refactor verify() onto the shared clusterIntegrityErr() and demote its logging to debug. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Evaluate cluster health on the sweep tick; after StepDownTimeout of continuous failure, return from lead() so the supervisor re-elects. Log transient mismatches at debug and sustained failures at error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
evaluateHealth now reconciles the in-memory map against the store on every sweep tick, so the sweep step must seed the store with the surviving node. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CDRayn
reviewed
Jun 2, 2026
CDRayn
reviewed
Jun 2, 2026
Per review feedback: surfacing leader membership changes at INFO captures them in default logs during customer escalations without requiring a log level change and restart. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the onset and recovery of an unhealthy episode once at INFO so they appear in default logs for customer escalations, while keeping per-tick detail at debug to avoid noise on routine self-healing mismatches. Sustained failures still escalate to error (when step-down is enabled). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CDRayn
reviewed
Jun 2, 2026
| "errors" | ||
| "time" | ||
|
|
||
| "gopkg.in/check.v1" |
Contributor
There was a problem hiding this comment.
This doesn't need to be changed or fixed here, but it looks like rstudio/platform-lib still uses gopkg.in/check.v1 in its tests, while PPM has mostly migrated away from that package. I'll add an issue to track migrating off gopkg.in/check.v1 and we can tackle when we have capacity for it.
CDRayn
approved these changes
Jun 2, 2026
CDRayn
left a comment
CDRayn
left a comment
Contributor
There was a problem hiding this comment.
Looks good to me. Approved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the Postgres database behind a multi-node PPM cluster fails over (e.g. an AWS RDS maintenance window),
PgxLeadercan wedge: it keeps emitting pings and considers itself leader, but stops running scheduled work. Two collaborating bugs cause this:Lead()never returns on persistent failure — its only exit was the stop channel, so the supervisor's leader→follower recovery path was never taken.verify()'s strict length check then fails permanently, silently skipping every scheduled task.This is the underlying cause of repeated production incidents (
node list length differs ...) downstream in Posit Package Manager (rstudio/package-manager#17889).Changes
clusterIntegrityErr()— a shared cluster-integrity check that reconciles the in-memory node map asymmetrically: prune nodes that have left the store (safe), but never add store-only nodes. Reachable nodes are re-added byhandlePingResponsewhen they answer a ping, which preserves the split-brain guard.verify()becomes a thin wrapper over it.evaluateHealth()+ time-based step-down — evaluated on the existing sweep tick. AfterStepDownTimeoutof continuous integrity failure the leader returns fromlead(), soLead()returns nil and the consumer's supervisor re-elects.StepDownTimeout == 0disables step-down (legacy behavior).New config field:
PgxLeaderConfig.StepDownTimeout.Tests
New Postgres-free
RecoverySuitecovers the asymmetric reconcile (prune vs. error-without-adding), the step-down threshold boundary, reset-on-recovery, and the disabled (0) case. The existingTestLeaderLeadInternalintegrity block was updated to the new semantics. Full Postgres-backed suite runs in CI.