Commit 03d6cee
backups(restore): ship + verify restore-drill for pg/mongo
Pro-tier marketing promises automatic backups. The nightly CronJobs
(postgres-customers / mongodb / redis-provision in instant-data) have
been running since 2026-05-11, but the restore path has never been
exercised — RPO + RTO were unverified.
This change:
- scripts/restore-drill.sh: idempotent restore drill. Asserts
kube-context = do-nyc3-instant-prod, pulls the latest backup
artifact for pg + mongo, spins up sidecar pods (pgvector/pgvector:pg16
matching prod, mongo:7.0) in a throwaway namespace, restores INTO
that sidecar (never over prod), runs smoke queries, prints RPO+RTO,
tears down. --service= and --keep-ns flags supported.
- BACKUP-RESTORE-RUNBOOK.md: full operational runbook. Pre-conditions,
full-cluster restore for pg, single-DB restore, rollback via PV
snapshot, smoke queries, restore-in-anger for mongo. Drill log
table for future operators to append to.
- newrelic/alerts/backup-stale-36h.json: log-based NR alert. WARN at
36h with no "backup OK" line, CRITICAL at 60h. FACET per backup
service. Reuses existing log-shipping; no new code surface.
- k8s/prometheus-rules.yaml (additive): new `instant-backups` rule
group. BackupCronJobStale36h / 60h on kube_cronjob_status_last_
successful_time + BackupCronJobFailedLastRun on kube_job_status_
failed. Activates when kube-state-metrics is installed.
Drill executed 2026-05-20T08:52 UTC. Both services PASS:
postgres-customers: RPO=20864s (5h47m), RTO=93s, smoke 121 db_*
databases + 528 pool_items rows.
mongodb: RPO=21038s (5h50m), RTO=42s, smoke 1 non-system
DB (matches prod).
Findings during drill (captured in runbook):
1. amazon/aws-cli image is distroless → kubectl cp fails (no tar).
Switched helper pods to alpine:3.20 + apk add aws-cli.
2. postgres:16-alpine sidecar lacks pgvector → 246 benign ERROR
lines on restore (smoke still passed). Switched sidecar image
to pgvector/pgvector:pg16, matching prod deployment.
3. Backups are single-region, single-vendor (DO Spaces nyc3 only).
Documented as P1 backlog item in runbook §11.
Per CLAUDE.md rule 24 (new in this PR): re-run the drill at least
monthly and on every change to infra/k8s/backups/*.yaml. Append the
result to BACKUP-RESTORE-RUNBOOK.md §12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 8a2001f commit 03d6cee
4 files changed
Lines changed: 1091 additions & 0 deletions
File tree
- k8s
- newrelic/alerts
- scripts
0 commit comments