feat(operator): bound schema migration at 20% of cron interval#61
Merged
Conversation
The schema migration step inside switchover used to wait indefinitely for its Job to complete. A genuinely-stuck postgres backend on the target restore (e.g. a CREATE TABLE spinning at 100% CPU forever) therefore wedged the entire replica: switchover blocked, replica stuck in Switching, the old Active restore serves increasingly stale data, and every subsequent scheduled restore queues behind the wedge. Cap the migration at 20% of the time between consecutive cron firings (72 min for a 6-hourly schedule, 5h for daily). On timeout the operator drops the persistent_schemas on the new restore via DROP SCHEMA … CASCADE, records a SchemaMigrationTimedOut Warning event, sets schemaMigrationPhase = timeout-skipped, and proceeds with the switchover. The replica comes up serving snapshot contents without the persistent schemas; the next cycle re-attempts migration if the schemas have regenerated on the source. A usable replica beats carrying the schema through indefinitely. Document the budget + timeout behaviour in the README's new "Persistent schemas" section.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖
Summary
Cap the schema-migration step at 20% of the time between consecutive cron firings. If the migration Job hasn't finished within that budget, the operator drops the persistent schemas on the new restore and proceeds with the switchover anyway. The replica comes up serving snapshot contents without the carried-over schemas; the next restore cycle re-attempts the migration if the schemas have regenerated upstream.
Rationale: a usable replica beats carrying the schema through indefinitely. Today a genuinely-stuck postgres backend on the new restore (e.g. a CREATE TABLE spinning at 100% CPU) wedges the whole replica — switchover blocks, the old Active serves increasingly stale data, every subsequent restore queues behind the wedge.
Budget
A healthy migration takes seconds; the budget is a backstop, not a tight SLA.
Timeout behaviour
When the budget is exceeded, the operator:
Shape
Tests
Three new unit tests covering the budget calculation: