Skip to content

feat(operator): bound schema migration at 20% of cron interval#61

Merged
passcod merged 1 commit into
mainfrom
operator-schema-migration-timeout
Jun 5, 2026
Merged

feat(operator): bound schema migration at 20% of cron interval#61
passcod merged 1 commit into
mainfrom
operator-schema-migration-timeout

Conversation

@passcod

@passcod passcod commented Jun 5, 2026

Copy link
Copy Markdown
Member

🤖

Summary

Cap the schema-migration step at 20% of the time between consecutive cron firings. If the migration Job hasn't finished within that budget, the operator drops the persistent schemas on the new restore and proceeds with the switchover anyway. The replica comes up serving snapshot contents without the carried-over schemas; the next restore cycle re-attempts the migration if the schemas have regenerated upstream.

Rationale: a usable replica beats carrying the schema through indefinitely. Today a genuinely-stuck postgres backend on the new restore (e.g. a CREATE TABLE spinning at 100% CPU) wedges the whole replica — switchover blocks, the old Active serves increasingly stale data, every subsequent restore queues behind the wedge.

Budget

  • 6-hourly cron (`0 */6 * * *`) → 72 min
  • Daily cron → ~5 h
  • Hourly cron → 12 min
  • Falls back to 1 h if the cron expression can't be parsed.

A healthy migration takes seconds; the budget is a backstop, not a tight SLA.

Timeout behaviour

When the budget is exceeded, the operator:

  1. Cancels the migration Job (background propagation).
  2. Connects to the new restore and runs `DROP SCHEMA IF EXISTS CASCADE` for each `persistentSchemas` entry.
  3. Records a `SchemaMigrationTimedOut` Warning event on the replica.
  4. Sets `status.schemaMigrationPhase = "timeout-skipped"` so it's distinguishable from `complete` / `partial` / `failed`.
  5. Proceeds with the switchover.

Shape

  • New `PostgresPhysicalReplica::schema_migration_timeout()` in `replica/scheduling.rs` — parses the cron, finds two consecutive firings, returns 20% of the interval.
  • New `migration_exceeded_budget(replica, job)` and `timeout_schema_migration(...)` helpers in `replica.rs`. The Active branch of the existing schema-migration reconcile checks the budget and calls the timeout helper before returning "still running".
  • README: new `Persistent schemas` section documenting the budget and timeout behaviour; updated status field table to include `timeout-skipped`.

Tests

Three new unit tests covering the budget calculation:

  • 6-hourly cron → 4320s
  • daily cron → 17280s
  • invalid cron → 3600s fallback

The schema migration step inside switchover used to wait indefinitely
for its Job to complete. A genuinely-stuck postgres backend on the
target restore (e.g. a CREATE TABLE spinning at 100% CPU forever)
therefore wedged the entire replica: switchover blocked, replica stuck
in Switching, the old Active restore serves increasingly stale data,
and every subsequent scheduled restore queues behind the wedge.

Cap the migration at 20% of the time between consecutive cron firings
(72 min for a 6-hourly schedule, 5h for daily). On timeout the
operator drops the persistent_schemas on the new restore via
DROP SCHEMA … CASCADE, records a SchemaMigrationTimedOut Warning
event, sets schemaMigrationPhase = timeout-skipped, and proceeds with
the switchover. The replica comes up serving snapshot contents
without the persistent schemas; the next cycle re-attempts migration
if the schemas have regenerated on the source. A usable replica
beats carrying the schema through indefinitely.

Document the budget + timeout behaviour in the README's new
"Persistent schemas" section.
@passcod passcod enabled auto-merge June 5, 2026 12:56
@passcod passcod merged commit 82e468a into main Jun 5, 2026
18 checks passed
@passcod passcod deleted the operator-schema-migration-timeout branch June 5, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant