feat(canopy): materialise PostgresPhysicalReplica CRs from worklist#82
Merged
Conversation
…pshot_id status
Step 1 of docs/plans/canopy-cr-materialisation.md. Purely additive:
existing kopiaSecretRef replicas keep working.
- Spec: kopiaSecretRef becomes Option; new canopySource { group, type }
alternative. Reconciler validates exactly one is set — surfaces
SecretRefAndCanopySource / NoCredentialSource / CanopySourced via the
KopiaSecretValid condition.
- Status: new canopyDesiredSnapshotId field (populated by the canopy
worklist syncer in a follow-up commit; consumed by the reconciler in
step 4 to drive Restore CR creation without a snapshot-list Job).
- Job builders (build_snapshot_list_job, build_restore_job) still
unwrap kopia_secret_ref via .expect() — the credential-source
validation guarantees it's set before these are called. Step 3 wires
the canopy proxy-sidecar branch through a KopiaSource enum.
- crds.yaml regenerated; README spec+status tables updated.
Introduce a KopiaSource enum that captures whether kopia auth goes through a static Secret (legacy) or through a canopy-proxy sidecar. build_restore_job now branches on that: the canopy path prepends a shell prelude that waits for the sidecar's proxy-port file, exports KOPIA_ENDPOINT/KOPIA_DISABLE_TLS, and adds the canopy-proxy container + shared emptyDir Volume + proxy-sidecar Pod label so the operator broker NetworkPolicy admits it. The env-from-Secret block is unchanged in shape; canopy's materialised Secret carries dummy AWS creds because the proxy re-signs upstream. Callers pass an Option<&CanopyProxyArgs> — required whenever replica.spec.canopy_source is set (enforced by expect()).
When `replica.spec.canopy_source` is set, the reconciler no longer creates or processes a snapshot-list Job. Instead it reads `status.canopyDesiredSnapshotId` (written by the canopy worklist syncer in a later step) and calls `create_restore_for_snapshot` directly when the desired snapshot differs from the currently active one. Canopy replicas rely on intent-driven `storage_size_override` for PVC sizing since the snapshot size isn't known ahead of the restore; a condition `SnapshotAvailable=False (CanopyPending)` surfaces the wait state when the syncer hasn't set the desired snapshot yet.
Introduce `src/controllers/canopy/intent.rs` with the three supported
intents — verify, analytics-dev, analytics-dbt — and their fixed
configuration (resources, read_only, minimum_ttl, persistent_schemas,
service_annotations, switchover_grace_period, storage_size_override).
IntentConfig::to_replica_spec materialises a PostgresPhysicalReplicaSpec
from a WorklistEntry + intent config; {name} in service annotations is
substituted with the entry's name at materialisation time.
The operator's SUPPORTED_INTENTS constant now re-exports the intent
module's SUPPORTED slice, so adding an intent only touches one place.
Adjust the reconciler so a change to status.canopyDesiredSnapshotId
becomes a restore trigger on the canopy path (in addition to schedule,
never-restored, active-restore-deleted). Extract PostgresPhysicalReplica::
within_minimum_ttl so the canopy trigger honours minimum_ttl even when
the schedule cron isn't the driver — analytics-dbt's 2h TTL still gates
canopy-provided newer snapshots.
Retire the parallel canopy Job/PVC/Service builders and the reporter alongside them. In their place the syncer now: - lists Namespaces labelled `pgro.bes.au/managed-by=pgro-canopy`; - for each worklist entry, ensures a labelled Namespace, a `canopy-replica` PostgresPhysicalReplica CR (spec built via IntentConfig::to_replica_spec), and a namespace-local canopy-creds Secret carrying the entry's bucket/region/prefix + repo password + dummy AWS keys; - patches `status.canopyDesiredSnapshotId` on the CR whenever canopy offers a snapshot newer than what the CR has seen; - tears down Namespaces (cascades everything inside) that no longer have a matching worklist entry. Restore Job creation, Deployment, verification, and switchover now go through the same CR machinery legacy replicas use. RestoreVerification POSTs to canopy will land as a notification target in a follow-up commit — the /api/v1/canopy-stats callback route the proxy sidecar posts to is unchanged.
Add `controllers::canopy::verification` — one `report` fn that builds a `bestool_canopy::RestoreVerification` from a PostgresPhysicalReplica + PostgresPhysicalRestore and POSTs it via `ctx.canopy.restore_verification`. - No-op when replica.spec.canopy_source is None or the canopy client isn't configured. - Reads group/server/type/intent/declaration-id from CR labels (set by the canopy syncer at Provision time). - Pulls S3 traffic stats out of `ctx.canopy_stats` (populated by the proxy sidecar's shutdown callback). - Success path fires from the switchover-complete branch in replica.rs; failure path fires from `fail_restore` in restore.rs.
The parallel canopy path used a fixed pgdata PVC size; the CR path uses per-intent `storage_size_override` instead (see intent module), so `CANOPY_PGDATA_PVC_SIZE` is no longer read anywhere.
The step-4 shortcut (skip snapshot-list, use canopyDesiredSnapshotId directly with size=0) meant every canopy replica ran with the intent's static storage_size_override regardless of the actual snapshot size, and kopia_content_cache_mb floored out. Instead, run the snapshot-list Job on both paths (canopy adds the proxy-sidecar + shared emptyDir volume, same shape as build_restore_job). On the result side: - legacy: apply snapshot_filter + latest_snapshot as before; - canopy: pick the snapshot whose id matches canopyDesiredSnapshotId, producing a real total_size_bytes for PVC + cache sizing. Surface a canopy-specific condition (SnapshotAvailable=False, CanopyDesiredSnapshotMissing) when the desired snapshot isn't in the kopia listing yet — usually a fresh backup canopy is offering ahead of its indexer catching up.
fail_restore now takes a short reason string. Each call site supplies what actually went wrong (kopia Job failed after backoff, version detection Job failed, deployment not Ready within timeout, etc.), and the canopy verification target passes it to canopy in the RestoreVerification's `error` field. Operators see the reason on canopy's UI without having to trawl the operator's k8s events.
The file is regenerated from k8s-openapi schemas whose descriptions use 'ANDed' / 'ORed' style jargon that typos flags as misspellings. Exclude the whole file rather than allowlisting each upstream term.
Ships across three PRs (#82 + #83 + #84). Two conscious deviations from the plan: - Step 4: snapshot-list runs on both paths (not skipped on canopy) so canopy replicas also get real PVC + kopia-cache sizing from the snapshot metadata. Result-picking is by canopyDesiredSnapshotId on the canopy path. - Step 8: canopy RestoreVerification uses a direct hook (controllers::canopy::verification::report) rather than a NotificationConfig variant. No retry pipeline. Rationale: canopy reporting is coupled to spec.canopy_source, not user-configurable, and doesn't semantically belong alongside webhook / graphQL. Deferred follow-ups (out of scope per plan): - stub-canopy server + wire canopy_integration test into CI - ops handoff snippet: canopy declarations -> pgro CRs mapping
passcod
force-pushed
the
feat/canopy-cr-materialisation
branch
from
c026bba to
696fc6c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Replaces the parallel canopy Job/PVC/Service/Deployment path with a
worklist-syncer that materialises pgro's own
PostgresPhysicalReplicaCRs. The existing CR machinery (restore Job, snapshot-list, blue/green
switchover, persistent_schemas migration, notification hooks) now runs
end-to-end for canopy-backed replicas — no more strictly-worse parallel
reality.
What the syncer does now
pgro.bes.au/managed-by=pgro-canopy.canopy-replicaPostgresPhysicalReplica CR (spec built viaIntentConfig::to_replica_spec), and a namespace-local canopy-credsSecret carrying the entry's bucket/region/prefix + repo password +
dummy AWS keys.
status.canopyDesiredSnapshotIdwhenever canopy offers asnapshot newer than what the CR has seen.
CR-side changes
spec.canopySource({group, type}), mutually exclusive withkopiaSecretRef. Reconciler surfaces mutex + validity conditions.status.canopyDesiredSnapshotId.KopiaSourceenum threads throughbuild_restore_jobandbuild_snapshot_list_job: canopy path prepends a shell prelude thatwaits for the sidecar's
/var/run/pgro/proxy-portand exportsKOPIA_ENDPOINT/KOPIA_DISABLE_TLS, adds acanopy-proxysidecarcontainer + shared emptyDir volume +
pgro.bes.au/proxy-sidecar=truePod label.
data (PVC + kopia cache) instead of the intent's static override.
Result-side picks by
canopyDesiredSnapshotIdon the canopy path,snapshot_filter+ latest on the legacy path.canopyDesiredSnapshotIdchanging is a first-class restoretrigger alongside schedule / never-restored / active-deleted;
minimum_ttlstill gates.Intents
src/controllers/canopy/intent.rs—IntentConfig+config_for+SUPPORTED=["verify", "analytics-dev", "analytics-dbt"].verify: read-only, minimal resources.analytics-dev: read-only, heavier resources.analytics-dbt: read-only, heavier resources,minimum_ttl: 2h,persistent_schemas: [dbt],switchover_grace: 2m,service_annotations: { tailscale.com/expose: true, tailscale.com/hostname: infra-replica-{name} }.{name}issubstituted from the worklist entry at CR-materialise time.
IntentConfig::to_replica_specproduces the full spec the syncerpatches onto the CR.
Signal 3
New
controllers::canopy::verification::reportbuilds abestool_canopy::RestoreVerificationfrom replica CR labels +restore CR +
ctx.canopy_stats(S3 traffic tallies from the proxysidecar's shutdown callback) and POSTs via
ctx.canopy.restore_verification. Fires from the switchover-completebranch (Success) and from
fail_restore(Failure, with a shortreason string threaded through each call site).
Retired
src/controllers/canopy/builders.rsandsrc/controllers/canopy/reporter.rsare gone — subsumed by the CRpath.
CANOPY_PGDATA_PVC_SIZEenv +canopy_pgdata_pvc_sizeContextfield also removed (intent-driven
storage_size_overridehandles PVCsizing now).