Skip to content

fix(operator): drop wall-clock query timeout, keep TCP keepalives#56

Merged
passcod merged 1 commit into
mainfrom
operator-drop-query-timeout
Jun 5, 2026
Merged

fix(operator): drop wall-clock query timeout, keep TCP keepalives#56
passcod merged 1 commit into
mainfrom
operator-drop-query-timeout

Conversation

@passcod

@passcod passcod commented Jun 5, 2026

Copy link
Copy Markdown
Member

🤖

Summary

Follow-up to #55. The per-query 5-minute `tokio::time::timeout` introduced alongside TCP keepalives turned out to be too tight for a legitimately slow statement: `DROP SCHEMA dbt CASCADE` on a populated dbt schema runs for tens of minutes. With the timeout in place, every retry hits the same wall and the reconcile loop never converges — worse than the original wedge it was meant to mitigate.

Why keepalives alone are enough

The original failure mode was a NAT-evicted (or otherwise silently-dead) socket: the operator's tokio task waits for a server reply that will never come. TCP keepalives fire on the kernel's own timer regardless of in-flight application traffic, so:

  • A legitimately long `DROP SCHEMA … CASCADE` keeps working — the socket is healthy, keepalive probes get ACKed.
  • A dead socket fails the keepalive probe and the kernel reports the socket dead within ~90s; the query returns an error and the reconcile retries.

So the wall-clock timeout is unnecessary and actively harmful.

Shape

  • Remove `QUERY_TIMEOUT`, `query_timeout`, `query_one_timeout`, `query_opt_timeout`, `execute_timeout` from `postgres.rs`.
  • Revert the four call sites to direct `tokio_postgres::Client` methods.
  • Keep `set_tcp_keepalive` and the keepalive constants — these are the actual fix.
  • Add a `Lessons learned` section to `AGENTS.md` recording the keepalives-vs-wall-clock-timeout reasoning so the next iteration doesn't re-add the timeout.

The per-query 5-minute timeout introduced alongside TCP keepalives in
0.3.19 turned out to be too tight for a legitimately slow operation:
DROP SCHEMA dbt CASCADE on a populated dbt schema can run for tens of
minutes. With the timeout in place, every retry hits the same wall
and the reconcile loop never converges, which is worse than the
original wedge it was meant to mitigate.

TCP keepalives alone are sufficient to detect the failure mode we
saw (NAT-evicted / silently-dead sockets): the kernel sends probes
on its own timer regardless of whether a query is in flight, so a
dead socket errors out within ~90s while a long-running statement
keeps working. Remove the QUERY_TIMEOUT plumbing and revert the
four call sites back to direct tokio_postgres methods.

Also record the lesson in AGENTS.md so the next iteration doesn't
re-add the timeout.
@passcod passcod merged commit 3abc7fb into main Jun 5, 2026
18 checks passed
@passcod passcod deleted the operator-drop-query-timeout branch June 5, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant