Skip to content

Commit 8de5283

Browse files
rdimitrovdomdomegg
rdimitrov
and
domdomegg
authored
Introduce a cron job that syncs the prod base into staging (#633)
<!-- Provide a brief summary of your changes --> ## Motivation and Context <!-- Why is this change needed? What problem does it solve? --> The following PR adds a workflow that should: * Runs daily but also can be manually triggered * Drops the staging database and copies the prod database into it This should improve our release confidence by reducing the likelihood of regressions or unexpected breakages. Copy from Claude: ```bash What This Does: 1. Connects to prod → Extracts backup credentials 2. Switches to staging → Configures backup access 3. Creates restore PVC → 50GB temporary storage 4. Triggers k8up restore → Downloads latest prod backup 5. Waits for restore → With proper job discovery 6. Finds PostgreSQL PVC → Dynamically 7. Scales down DB → Safely 8. Copies data → With validation and backup 9. Scales up DB → With pod creation waiting 10. Verifies DB → Tests tables and queries 11. Cleans up → Removes temporary resources ``` ## How Has This Been Tested? <!-- Have you tested this in a real application? Which scenarios were tested? --> No ## Breaking Changes <!-- Will users need to update their code or configurations? --> No (I assume no one relies on staging keeping its state) ## Types of changes <!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply. --> - [ ] I have read the [MCP Documentation](https://modelcontextprotocol.io) - [ ] My code follows the repository's style guidelines - [ ] New and existing tests pass locally - [ ] I have added appropriate error handling - [ ] I have added or updated documentation as needed ## Additional context <!-- Add any other context, implementation notes, or design decisions --> --------- Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com> Co-authored-by: Adam Jones <adamj@anthropic.com>
1 parent 7d1a6f0 commit 8de5283

1 file changed

Lines changed: 388 additions & 0 deletions

File tree

‎.github/workflows/sync-db.yml‎

Lines changed: 388 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,388 @@
1+
name: Sync Production DB to Staging (from backups)
2+
3+
on:
4+
schedule:
5+
# Run daily at 2 AM UTC (during low-traffic hours)
6+
- cron: '0 2 * * *'
7+
workflow_dispatch: # Allow manual triggering
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
sync-database:
14+
name: Sync Prod DB to Staging from k8up Backups
15+
runs-on: ubuntu-latest
16+
environment: staging
17+
concurrency:
18+
group: sync-staging-database
19+
cancel-in-progress: false
20+
steps:
21+
- name: Authenticate to Google Cloud (Production)
22+
uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093
23+
with:
24+
credentials_json: ${{ secrets.GCP_PROD_SERVICE_ACCOUNT_KEY }}
25+
26+
- name: Setup Google Cloud SDK
27+
uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db
28+
with:
29+
project_id: mcp-registry-prod
30+
install_components: gke-gcloud-auth-plugin
31+
32+
- name: Get backup credentials from prod cluster
33+
id: backup-creds
34+
run: |
35+
gcloud container clusters get-credentials mcp-registry-prod \
36+
--zone=us-central1-b \
37+
--project=mcp-registry-prod
38+
39+
# Store in outputs (GitHub Actions encrypts these automatically)
40+
kubectl get secret k8up-backup-credentials -n default -o json | jq -r '
41+
"access_key=" + (.data.AWS_ACCESS_KEY_ID | @base64d),
42+
"secret_key=" + (.data.AWS_SECRET_ACCESS_KEY | @base64d)
43+
' >> $GITHUB_OUTPUT
44+
45+
- name: Remove all production access (SAFETY MEASURE)
46+
run: |
47+
# Remove production cluster from kubeconfig
48+
kubectl config delete-context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod 2>/dev/null || true
49+
50+
# Revoke gcloud credentials
51+
gcloud auth revoke --all 2>/dev/null || true
52+
53+
# Clear gcloud configuration
54+
gcloud config unset project 2>/dev/null || true
55+
gcloud config unset account 2>/dev/null || true
56+
57+
# Verify no contexts remain
58+
CONTEXT_COUNT=$(kubectl config get-contexts -o name 2>/dev/null | wc -l)
59+
if [ "$CONTEXT_COUNT" -gt 0 ]; then
60+
echo "❌ ERROR: $CONTEXT_COUNT context(s) still exist after cleanup!"
61+
kubectl config get-contexts
62+
exit 1
63+
fi
64+
65+
- name: Switch to staging cluster
66+
uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093
67+
with:
68+
credentials_json: ${{ secrets.GCP_STAGING_SERVICE_ACCOUNT_KEY }}
69+
70+
- name: Configure staging cluster access
71+
run: |
72+
gcloud config set project mcp-registry-staging
73+
gcloud container clusters get-credentials mcp-registry-staging \
74+
--zone=us-central1-b \
75+
--project=mcp-registry-staging
76+
77+
- name: Create secret for prod backup bucket access
78+
run: |
79+
# Create/update secret in staging with access to prod backups
80+
kubectl create secret generic prod-to-staging-sync-credentials \
81+
--from-literal=AWS_ACCESS_KEY_ID="${{ steps.backup-creds.outputs.access_key }}" \
82+
--from-literal=AWS_SECRET_ACCESS_KEY="${{ steps.backup-creds.outputs.secret_key }}" \
83+
--dry-run=client -o yaml | kubectl apply -f -
84+
85+
- name: Create restore PVC
86+
run: |
87+
kubectl apply -f - <<EOF
88+
apiVersion: v1
89+
kind: PersistentVolumeClaim
90+
metadata:
91+
name: restore-data-pvc
92+
namespace: default
93+
spec:
94+
accessModes:
95+
- ReadWriteOnce
96+
resources:
97+
requests:
98+
storage: 50Gi
99+
EOF
100+
101+
- name: Trigger k8up restore from prod backups
102+
id: restore
103+
run: |
104+
RESTORE_NAME="restore-from-prod-$(date +%Y%m%d-%H%M%S)"
105+
echo "restore_name=$RESTORE_NAME" >> $GITHUB_OUTPUT
106+
107+
# Create a k8up Restore resource to restore from prod backups
108+
kubectl apply -f - <<EOF
109+
apiVersion: k8up.io/v1
110+
kind: Restore
111+
metadata:
112+
name: $RESTORE_NAME
113+
namespace: default
114+
spec:
115+
snapshot: latest
116+
restoreMethod:
117+
folder:
118+
claimName: restore-data-pvc
119+
backend:
120+
repoPasswordSecretRef:
121+
name: k8up-repo-password
122+
key: password
123+
s3:
124+
bucket: mcp-registry-prod-backups
125+
endpoint: https://storage.googleapis.com
126+
accessKeyIDSecretRef:
127+
name: prod-to-staging-sync-credentials
128+
key: AWS_ACCESS_KEY_ID
129+
secretAccessKeySecretRef:
130+
name: prod-to-staging-sync-credentials
131+
key: AWS_SECRET_ACCESS_KEY
132+
EOF
133+
134+
- name: Wait for k8up restore to complete
135+
run: |
136+
RESTORE_NAME="${{ steps.restore.outputs.restore_name }}"
137+
138+
echo "Waiting for restore job to start..."
139+
sleep 15
140+
141+
# Find the job created by k8up for this restore
142+
for i in {1..30}; do
143+
JOB_NAME=$(kubectl get jobs -n default -l k8up.io/owned-by=restore -o jsonpath='{.items[?(@.metadata.ownerReferences[0].name=="'$RESTORE_NAME'")].metadata.name}' 2>/dev/null)
144+
if [ -n "$JOB_NAME" ]; then
145+
echo "Found restore job: $JOB_NAME"
146+
break
147+
fi
148+
echo "Waiting for job to be created... ($i/30)"
149+
sleep 2
150+
done
151+
152+
if [ -z "$JOB_NAME" ]; then
153+
echo "ERROR: Restore job not found"
154+
kubectl get restore $RESTORE_NAME -n default -o yaml
155+
exit 1
156+
fi
157+
158+
# Wait for the restore job to complete (max 15 minutes)
159+
kubectl wait --for=condition=complete \
160+
job/$JOB_NAME \
161+
--timeout=900s -n default || {
162+
echo "Restore job failed or timed out"
163+
kubectl describe job/$JOB_NAME -n default
164+
kubectl logs job/$JOB_NAME -n default --tail=100
165+
exit 1
166+
}
167+
168+
- name: Find staging PostgreSQL PVC
169+
id: pgdata-pvc
170+
run: |
171+
# Find the PVC used by the PostgreSQL cluster
172+
PVC_NAME=$(kubectl get pvc -n default -l cnpg.io/cluster=registry-pg -o jsonpath='{.items[0].metadata.name}')
173+
174+
if [ -z "$PVC_NAME" ]; then
175+
echo "ERROR: Could not find PostgreSQL PVC"
176+
kubectl get pvc -n default -l cnpg.io/cluster=registry-pg
177+
exit 1
178+
fi
179+
180+
echo "pvc_name=$PVC_NAME" >> $GITHUB_OUTPUT
181+
182+
- name: Scale down staging PostgreSQL
183+
run: |
184+
echo "Scaling down PostgreSQL cluster..."
185+
kubectl patch cluster registry-pg -n default \
186+
--type merge \
187+
--patch '{"spec":{"instances":0}}'
188+
189+
# Wait for pods to terminate
190+
echo "Waiting for pods to terminate..."
191+
kubectl wait --for=delete pod -l cnpg.io/cluster=registry-pg -n default --timeout=300s || true
192+
193+
- name: Verify we are in staging cluster (SAFETY CHECK)
194+
run: |
195+
# Get current cluster context
196+
CURRENT_CONTEXT=$(kubectl config current-context)
197+
CURRENT_PROJECT=$(gcloud config get-value project)
198+
199+
echo "Current kubectl context: $CURRENT_CONTEXT"
200+
echo "Current GCP project: $CURRENT_PROJECT"
201+
202+
# Verify we're in staging
203+
if ! echo "$CURRENT_CONTEXT" | grep -qi "staging"; then
204+
echo "❌ SAFETY CHECK FAILED: Not in staging cluster"
205+
echo "Context: $CURRENT_CONTEXT"
206+
echo "Expected: staging cluster"
207+
exit 1
208+
fi
209+
210+
if [ "$CURRENT_PROJECT" != "mcp-registry-staging" ]; then
211+
echo "❌ SAFETY CHECK FAILED: Not in staging project"
212+
echo "Project: $CURRENT_PROJECT"
213+
echo "Expected: mcp-registry-staging"
214+
exit 1
215+
fi
216+
217+
- name: Replace staging database with restored backup
218+
id: copy-job
219+
run: |
220+
JOB_NAME="copy-pgdata-$(date +%Y%m%d-%H%M%S)"
221+
echo "job_name=$JOB_NAME" >> $GITHUB_OUTPUT
222+
223+
# Create a job to copy the restored backup data to the staging PVC
224+
kubectl apply -f - <<EOF
225+
apiVersion: batch/v1
226+
kind: Job
227+
metadata:
228+
name: $JOB_NAME
229+
namespace: default
230+
spec:
231+
ttlSecondsAfterFinished: 600
232+
template:
233+
spec:
234+
restartPolicy: Never
235+
containers:
236+
- name: copy-data
237+
image: busybox:latest
238+
command:
239+
- /bin/sh
240+
- -c
241+
- |
242+
set -e
243+
echo "Finding PostgreSQL data in backup..."
244+
echo "Restore structure:"
245+
find /restore -maxdepth 3 -type d 2>/dev/null | head -20
246+
247+
# Try different possible paths for pgdata
248+
PGDATA_SOURCE=""
249+
for path in \$(find /restore -type d -name "pgdata" 2>/dev/null); do
250+
if [ -f "\$path/PG_VERSION" ]; then
251+
PGDATA_SOURCE="\$path"
252+
break
253+
fi
254+
done
255+
256+
if [ -z "\$PGDATA_SOURCE" ]; then
257+
echo "ERROR: Could not find valid pgdata directory with PG_VERSION"
258+
echo "Searched paths:"
259+
find /restore -type d -name "pgdata" 2>/dev/null
260+
exit 1
261+
fi
262+
263+
echo "Found pgdata at: \$PGDATA_SOURCE"
264+
echo "Contents:"
265+
ls -lah \$PGDATA_SOURCE/ | head -10
266+
267+
echo "Backing up existing staging data..."
268+
mkdir -p /pgdata-backup
269+
if [ "\$(ls -A /pgdata)" ]; then
270+
cp -a /pgdata/. /pgdata-backup/ || echo "Warning: Could not backup existing data"
271+
fi
272+
273+
echo "Clearing existing data..."
274+
rm -rf /pgdata/*
275+
276+
echo "Copying backup data to staging PVC..."
277+
cp -a \$PGDATA_SOURCE/. /pgdata/
278+
279+
echo "Setting correct permissions..."
280+
chmod 700 /pgdata
281+
282+
ls -lah /pgdata/ | head -20
283+
echo "PostgreSQL version: \$(cat /pgdata/PG_VERSION)"
284+
volumeMounts:
285+
- name: restore-data
286+
mountPath: /restore
287+
- name: staging-pgdata
288+
mountPath: /pgdata
289+
volumes:
290+
- name: restore-data
291+
persistentVolumeClaim:
292+
claimName: restore-data-pvc
293+
- name: staging-pgdata
294+
persistentVolumeClaim:
295+
claimName: ${{ steps.pgdata-pvc.outputs.pvc_name }}
296+
EOF
297+
298+
- name: Wait for data copy to complete
299+
run: |
300+
JOB_NAME="${{ steps.copy-job.outputs.job_name }}"
301+
302+
# Wait for copy to complete
303+
kubectl wait --for=condition=complete job/$JOB_NAME --timeout=600s -n default || {
304+
echo "Data copy job failed"
305+
kubectl describe job/$JOB_NAME -n default
306+
kubectl logs job/$JOB_NAME -n default --tail=100
307+
exit 1
308+
}
309+
310+
- name: Scale up staging PostgreSQL
311+
run: |
312+
echo "Scaling up PostgreSQL cluster..."
313+
kubectl patch cluster registry-pg -n default \
314+
--type merge \
315+
--patch '{"spec":{"instances":1}}'
316+
317+
# Wait for PostgreSQL pod to be created
318+
echo "Waiting for PostgreSQL pod to be created..."
319+
for i in {1..60}; do
320+
POD_COUNT=$(kubectl get pods -l cnpg.io/cluster=registry-pg -n default --no-headers 2>/dev/null | wc -l)
321+
if [ "$POD_COUNT" -gt 0 ]; then
322+
echo "Pod created"
323+
break
324+
fi
325+
echo "Waiting... ($i/60)"
326+
sleep 2
327+
done
328+
329+
# Wait for PostgreSQL to be ready
330+
echo "Waiting for PostgreSQL to be ready..."
331+
kubectl wait --for=condition=ready pod -l cnpg.io/cluster=registry-pg -n default --timeout=300s
332+
333+
- name: Verify staging DB is functional
334+
run: |
335+
# Create a verification pod
336+
kubectl run pg-verify-$(date +%s) \
337+
--image=postgres:15 \
338+
--rm -i --restart=Never \
339+
--env="PGPASSWORD=$(kubectl get secret registry-pg-superuser -n default -o jsonpath='{.data.password}' | base64 -d)" \
340+
-- bash -c '
341+
echo "Waiting for database to accept connections..."
342+
for i in {1..30}; do
343+
if pg_isready -h registry-pg-rw -U postgres 2>/dev/null; then
344+
break
345+
fi
346+
echo "Waiting... ($i/30)"
347+
sleep 2
348+
done
349+
350+
echo "Querying database..."
351+
TABLE_COUNT=$(psql -h registry-pg-rw -U postgres -d app -tAc "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = '\''public'\'';" 2>&1)
352+
353+
if [ $? -ne 0 ]; then
354+
echo "ERROR: Could not query database"
355+
echo "$TABLE_COUNT"
356+
exit 1
357+
fi
358+
359+
if [ "$TABLE_COUNT" -lt 1 ]; then
360+
echo "ERROR: Staging DB has no tables!"
361+
exit 1
362+
fi
363+
364+
echo "Staging DB has $TABLE_COUNT tables"
365+
echo "Top 10 tables by row count:"
366+
psql -h registry-pg-rw -U postgres -d app \
367+
-c "SELECT schemaname, tablename, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 10;" || true
368+
'
369+
370+
- name: Cleanup
371+
if: always()
372+
run: |
373+
# Clean up jobs first
374+
if [ -n "${{ steps.copy-job.outputs.job_name }}" ]; then
375+
kubectl delete job ${{ steps.copy-job.outputs.job_name }} -n default || true
376+
fi
377+
378+
# Remove restore PVC (will wait for jobs to finish)
379+
kubectl delete pvc restore-data-pvc -n default || true
380+
381+
# Remove prod backup credentials (for security)
382+
kubectl delete secret prod-to-staging-sync-credentials -n default || true
383+
384+
# Clean up old restore resources (keep last 3)
385+
kubectl get restore -n default --sort-by=.metadata.creationTimestamp -o name | head -n -3 | xargs -r kubectl delete || true
386+
387+
# Clean up old copy jobs (keep last 3)
388+
kubectl get jobs -n default --sort-by=.metadata.creationTimestamp -o name | grep 'copy-pgdata-' | head -n -3 | xargs -r kubectl delete -n default || true

0 commit comments

Comments
 (0)