Skip to content

Commit 03d6cee

Browse files
backups(restore): ship + verify restore-drill for pg/mongo
Pro-tier marketing promises automatic backups. The nightly CronJobs (postgres-customers / mongodb / redis-provision in instant-data) have been running since 2026-05-11, but the restore path has never been exercised — RPO + RTO were unverified. This change: - scripts/restore-drill.sh: idempotent restore drill. Asserts kube-context = do-nyc3-instant-prod, pulls the latest backup artifact for pg + mongo, spins up sidecar pods (pgvector/pgvector:pg16 matching prod, mongo:7.0) in a throwaway namespace, restores INTO that sidecar (never over prod), runs smoke queries, prints RPO+RTO, tears down. --service= and --keep-ns flags supported. - BACKUP-RESTORE-RUNBOOK.md: full operational runbook. Pre-conditions, full-cluster restore for pg, single-DB restore, rollback via PV snapshot, smoke queries, restore-in-anger for mongo. Drill log table for future operators to append to. - newrelic/alerts/backup-stale-36h.json: log-based NR alert. WARN at 36h with no "backup OK" line, CRITICAL at 60h. FACET per backup service. Reuses existing log-shipping; no new code surface. - k8s/prometheus-rules.yaml (additive): new `instant-backups` rule group. BackupCronJobStale36h / 60h on kube_cronjob_status_last_ successful_time + BackupCronJobFailedLastRun on kube_job_status_ failed. Activates when kube-state-metrics is installed. Drill executed 2026-05-20T08:52 UTC. Both services PASS: postgres-customers: RPO=20864s (5h47m), RTO=93s, smoke 121 db_* databases + 528 pool_items rows. mongodb: RPO=21038s (5h50m), RTO=42s, smoke 1 non-system DB (matches prod). Findings during drill (captured in runbook): 1. amazon/aws-cli image is distroless → kubectl cp fails (no tar). Switched helper pods to alpine:3.20 + apk add aws-cli. 2. postgres:16-alpine sidecar lacks pgvector → 246 benign ERROR lines on restore (smoke still passed). Switched sidecar image to pgvector/pgvector:pg16, matching prod deployment. 3. Backups are single-region, single-vendor (DO Spaces nyc3 only). Documented as P1 backlog item in runbook §11. Per CLAUDE.md rule 24 (new in this PR): re-run the drill at least monthly and on every change to infra/k8s/backups/*.yaml. Append the result to BACKUP-RESTORE-RUNBOOK.md §12. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 8a2001f commit 03d6cee

4 files changed

Lines changed: 1091 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)