fix(operator): TCP keepalives + per-query timeout on operator-side libpq#55
Merged
Conversation
The operator's tokio-postgres clients run over raw TCP sockets opened with no keepalives, so a NAT-evicted or silently-dead connection (e.g. between reconciles, between queries on the same client, or while a long-running DROP SCHEMA CASCADE is in flight) wedges the reconcile indefinitely. Other replicas keep reconciling, but the stuck one never makes progress. Mirror the libpq URI keepalives we already use in the migration Job (60s idle / 10s interval / 3 retries — detects dead sockets within ~90s) by setting them on the raw TcpStream via socket2 before handing the stream to tokio-postgres::connect_raw. As a backstop for the case where keepalives still ACK but the query itself stalls, wrap each query/execute call in a 5-minute tokio::time::timeout. The timeout is generous on purpose — intent is runaway detection, not a query SLA — but bounded enough that the reconcile fails and the controller retries instead of sitting forever. Port-forwarded connections (used out-of-cluster, e.g. CI / dev) go through the kube API server WebSocket and keep their own keepalive handling there; no socket-level change is meaningful for that path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖
Summary
The operator's own `tokio-postgres` clients open raw `TcpStream`s with no keepalives, so a NAT-evicted or silently-dead socket — between reconciles, between queries on the same client, or while a long-running `DROP SCHEMA ... CASCADE` is in flight — wedges that reconcile indefinitely. The Controller keeps processing other replicas, but the stuck one never makes progress (`observedGeneration` falls behind, status fields freeze) until the operator pod is restarted.
Fix
Two complementary safeguards in `postgres.rs`:
The port-forward path (used out-of-cluster, e.g. CI / dev) goes through the kube API server WebSocket — it has its own keepalive handling, so no socket-level change is meaningful there.
Shape