Skip to content

fix(operator): TCP keepalives + per-query timeout on operator-side libpq#55

Merged
passcod merged 1 commit into
mainfrom
operator-pg-keepalives
Jun 5, 2026
Merged

fix(operator): TCP keepalives + per-query timeout on operator-side libpq#55
passcod merged 1 commit into
mainfrom
operator-pg-keepalives

Conversation

@passcod

@passcod passcod commented Jun 5, 2026

Copy link
Copy Markdown
Member

🤖

Summary

The operator's own `tokio-postgres` clients open raw `TcpStream`s with no keepalives, so a NAT-evicted or silently-dead socket — between reconciles, between queries on the same client, or while a long-running `DROP SCHEMA ... CASCADE` is in flight — wedges that reconcile indefinitely. The Controller keeps processing other replicas, but the stuck one never makes progress (`observedGeneration` falls behind, status fields freeze) until the operator pod is restarted.

Fix

Two complementary safeguards in `postgres.rs`:

  1. TCP keepalives on the direct in-cluster path (`connect_via_tcp`). Mirrors the libpq URI keepalives we already use in the schema-migration Job: 60s idle, 10s interval, 3 retries — detects dead sockets within ~90s. Implemented via `socket2::SockRef::from(&stream).set_tcp_keepalive(...)` before handing the stream to `Config::connect_raw`.
  2. Per-query timeout (`QUERY_TIMEOUT = 5 min`) wrapping every `query / query_one / query_opt / execute` call, as a backstop for the case where the connection ACKs keepalive probes but the query itself stalls. Generous because `DROP SCHEMA CASCADE` on a populated dbt schema can legitimately take minutes; intent is runaway detection, not a query SLA.

The port-forward path (used out-of-cluster, e.g. CI / dev) goes through the kube API server WebSocket — it has its own keepalive handling, so no socket-level change is meaningful there.

Shape

  • New helpers `query_timeout` / `query_one_timeout` / `query_opt_timeout` / `execute_timeout` that wrap the corresponding `tokio_postgres::Client` methods in `tokio::time::timeout`.
  • All four operator-side query sites in `postgres.rs` migrated to the timeout helpers.
  • New `socket2` direct dependency (already transitive via `hyper-util`).

The operator's tokio-postgres clients run over raw TCP sockets opened
with no keepalives, so a NAT-evicted or silently-dead connection (e.g.
between reconciles, between queries on the same client, or while a
long-running DROP SCHEMA CASCADE is in flight) wedges the reconcile
indefinitely. Other replicas keep reconciling, but the stuck one
never makes progress.

Mirror the libpq URI keepalives we already use in the migration Job
(60s idle / 10s interval / 3 retries — detects dead sockets within
~90s) by setting them on the raw TcpStream via socket2 before handing
the stream to tokio-postgres::connect_raw. As a backstop for the
case where keepalives still ACK but the query itself stalls, wrap
each query/execute call in a 5-minute tokio::time::timeout. The
timeout is generous on purpose — intent is runaway detection, not
a query SLA — but bounded enough that the reconcile fails and the
controller retries instead of sitting forever.

Port-forwarded connections (used out-of-cluster, e.g. CI / dev) go
through the kube API server WebSocket and keep their own keepalive
handling there; no socket-level change is meaningful for that path.
@passcod passcod merged commit 00bf8d5 into main Jun 5, 2026
18 checks passed
@passcod passcod deleted the operator-pg-keepalives branch June 5, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant