Skip to content

fix(operator): terminate other client backends before drop_schemas_on#57

Merged
passcod merged 1 commit into
mainfrom
operator-terminate-other-backends
Jun 5, 2026
Merged

fix(operator): terminate other client backends before drop_schemas_on#57
passcod merged 1 commit into
mainfrom
operator-terminate-other-backends

Conversation

@passcod

@passcod passcod commented Jun 5, 2026

Copy link
Copy Markdown
Member

🤖

Summary

Before each `DROP SCHEMA … CASCADE` during schema-migration prep, ask postgres to terminate every other client backend on the target database. The operator owns the restore database fully until handover, so any other session is necessarily a stray or transient client whose lock-holding would queue the DDL indefinitely.

Real failure mode this prevents: an external session got stuck mid-`CREATE TABLE` in the target restore while it was still in the operator's `Switching` phase, holding namespace locks that every subsequent DROP SCHEMA attempt queued behind for 20+ hours. Each operator-restart left another DROP queued, eventually 20+ orphaned sessions. The `DROP`'s patient wait (now correct since we removed the wall-clock timeout) made things worse here because there was no time bound either.

Shape

  • New `terminate_other_backends(pg)` helper in `postgres.rs` running:
    ```sql
    SELECT count(*) FILTER (WHERE pg_terminate_backend(pid))
    FROM pg_stat_activity
    WHERE datname = current_database()
    AND backend_type = 'client backend'
    AND pid <> pg_backend_pid()
    ```
  • Called at the top of `drop_schemas_on` so every drop run clears the deck first. Returns the count for logging.

Caveats

  • Best-effort: backends stuck in uninterruptible kernel waits won't exit on SIGTERM (the case that triggered this eventually needed a pod restart). But those are rare; the 99% case is a normal live client and `pg_terminate_backend` handles it.
  • A follow-up structural fix is planned: gating the Service selector on an operator-set `ready-for-traffic` label so external clients can't reach a restore at all while it's still in `Switching`. That eliminates the precondition for this whole scenario.

Before each DROP SCHEMA ... CASCADE run during schema-migration prep,
ask postgres to terminate every other client backend connected to
the target database. The operator owns the restore database fully
until handover, so any other session is necessarily a stray client
whose lock-holding would queue our DDL indefinitely.

Real scenario that triggered this: a long-running external psql
session got stuck mid-CREATE TABLE in the target restore while it
was still in the operator's Switching phase, holding namespace
locks that all subsequent DROP SCHEMA attempts queued behind for
20+ hours. With this change, the operator clears the deck before
attempting the DROP.

Best-effort by design: backends stuck in uninterruptible kernel
waits won't exit on SIGTERM (the case above eventually needed a
pod restart). But those are rare; the 99% case is a normal live
client and pg_terminate_backend takes care of it.
@passcod passcod enabled auto-merge June 5, 2026 05:45
@passcod passcod merged commit ecba368 into main Jun 5, 2026
18 checks passed
@passcod passcod deleted the operator-terminate-other-backends branch June 5, 2026 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant