Skip to content

feat(operator): gate Service traffic on operator-set ready label#58

Merged
passcod merged 1 commit into
mainfrom
operator-ready-for-traffic-gate
Jun 5, 2026
Merged

feat(operator): gate Service traffic on operator-set ready label#58
passcod merged 1 commit into
mainfrom
operator-ready-for-traffic-gate

Conversation

@passcod

@passcod passcod commented Jun 5, 2026

Copy link
Copy Markdown
Member

🤖

Summary

Adds a second label `pgro.bes.au/ready-for-traffic=true` to the per-replica postgres Service's selector, and has the operator only patch that label onto a restore's pod once it's safe for external traffic. Until the operator sets the label, the new restore is unreachable via the Service even though its postgres pod is healthy — so DDL prep (DROP SCHEMA, schema-migration Job, etc.) runs without external clients racing to grab locks.

Why

Today the Service routes to any pod matching `pgro.bes.au/restore=`, which means a new restore becomes reachable the moment its pod goes Ready — including during the operator's `Switching` phase, before schema migration completes. External workloads (dbt sync, application connectors, etc.) that hit the Service during that window can start work on the very schemas the operator is about to drop and rewrite. Locks pile up, migration wedges, and only a pod restart clears the mess.

This change closes that hole structurally instead of relying on session-cleanup (which is the companion #57 belt-and-braces fix).

Shape

  • `replica/resources.rs`:
    • New constant `READY_FOR_TRAFFIC_LABEL = "pgro.bes.au/ready-for-traffic"`.
    • New method `PostgresPhysicalRestore::mark_pod_ready_for_traffic` that finds the restore's running pod and patches the label via SSA. Idempotent; warns and returns Ok if the pod isn't running yet (next reconcile will retry).
    • `update_service_selector` now patches the Service selector to require both `pgro.bes.au/restore` and the new label.
  • `replica.rs` (Switching → Active transition): call `mark_pod_ready_for_traffic` immediately before `update_service_selector`. The Service flips to the new restore only after its pod carries the label.
  • `restore.rs` (`reconcile_active`): re-apply the label every Active-reconcile pass. Covers two cases: a postgres pod restart that drops the label, and the upgrade path where existing pre-upgrade Active pods don't have the label yet.

Test coverage

`tests/switchover.rs` already exercises the Service-selector rewrite during switchover. Added assertions that:

  • the Service selector carries `ready-for-traffic=true` both before and after a switchover, and
  • the second restore's pod carries `pgro.bes.au/ready-for-traffic=true` after switchover.

No new test file, so no integration matrix entry needed.

Caveats

  • The label is applied to the pod directly, not the deployment template, to avoid forcing a rollout on every set. Trade-off: a postgres pod restart (OOM, eviction, node loss — rare) drops the label, and the Service stops routing until `reconcile_active` runs (~5 min worst case). For an analytics replica this is acceptable; if it becomes a problem we can shorten the requeue interval or add a Pod watch source.
  • Stacks naturally with fix(operator): terminate other client backends before drop_schemas_on #57 (terminate other backends) but doesn't depend on it — independent PRs.

@passcod passcod enabled auto-merge June 5, 2026 05:52
The per-replica postgres Service used to route to any pod matching
just `pgro.bes.au/restore=<active-name>`, which meant a new restore
became reachable to external clients the moment its pod went Ready
— including during the operator's Switching phase, before schema
migration completes. External workloads that connect at that point
can grab locks on the very schemas the operator is about to drop
and rewrite, wedging the migration.

Add a second label `pgro.bes.au/ready-for-traffic=true` to the
Service selector, and have the operator only patch that label
onto a restore's pod once it's safe for external traffic — after
schema migration completes, immediately before the Service
selector is pointed at the new restore. Until then, the new
restore is unreachable via the Service even though its pod is
healthy, so DDL prep runs without external interference.

Also re-apply the label every Active reconcile pass so it's
restored if a pod restart drops it, and so existing pre-upgrade
Active pods get the label on the first reconcile after deploy.
@passcod passcod force-pushed the operator-ready-for-traffic-gate branch from e234c51 to 71bfb91 Compare June 5, 2026 10:50
@passcod passcod merged commit 3cbd137 into main Jun 5, 2026
13 checks passed
@passcod passcod deleted the operator-ready-for-traffic-gate branch June 5, 2026 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant