fix(operator): terminate other client backends before drop_schemas_on#57
Merged
Conversation
Before each DROP SCHEMA ... CASCADE run during schema-migration prep, ask postgres to terminate every other client backend connected to the target database. The operator owns the restore database fully until handover, so any other session is necessarily a stray client whose lock-holding would queue our DDL indefinitely. Real scenario that triggered this: a long-running external psql session got stuck mid-CREATE TABLE in the target restore while it was still in the operator's Switching phase, holding namespace locks that all subsequent DROP SCHEMA attempts queued behind for 20+ hours. With this change, the operator clears the deck before attempting the DROP. Best-effort by design: backends stuck in uninterruptible kernel waits won't exit on SIGTERM (the case above eventually needed a pod restart). But those are rare; the 99% case is a normal live client and pg_terminate_backend takes care of it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖
Summary
Before each `DROP SCHEMA … CASCADE` during schema-migration prep, ask postgres to terminate every other client backend on the target database. The operator owns the restore database fully until handover, so any other session is necessarily a stray or transient client whose lock-holding would queue the DDL indefinitely.
Real failure mode this prevents: an external session got stuck mid-`CREATE TABLE` in the target restore while it was still in the operator's `Switching` phase, holding namespace locks that every subsequent DROP SCHEMA attempt queued behind for 20+ hours. Each operator-restart left another DROP queued, eventually 20+ orphaned sessions. The `DROP`'s patient wait (now correct since we removed the wall-clock timeout) made things worse here because there was no time bound either.
Shape
```sql
SELECT count(*) FILTER (WHERE pg_terminate_backend(pid))
FROM pg_stat_activity
WHERE datname = current_database()
AND backend_type = 'client backend'
AND pid <> pg_backend_pid()
```
Caveats