chore: Nightly prod to staging sync (#203)

* Add nightly prod->staging sync CronJob with Auth0 user import

Backup phase: pg_dump prod, azcopy sync prod LFS, Auth0 users-export
all into adr-snapshots storage account.

Apply phase: pg_restore into adr-s Postgres, azcopy sync to adr-s-datalake,
Auth0 users-imports into dev tenant (upsert=true), CKAN solr reindex.

Replaces the AWS-era adx_toolbox sync_development.sh + auth0_backup_users
cron. The same artefacts now serve both staging refresh and DR backups,
governed by Azure Blob lifecycle policy on adr-snapshots.

See deploy/sync/secrets.yaml.template for the secret shape that needs to
exist in adr-s before the CronJob runs.

* Add docs/nightly-sync.md

Architecture, one-time Azure + Auth0 + k8s setup, ops runbook,
failure modes, cost estimate. Cross-references the CKAN
saml2auth process_user() flow and the retired adx_toolbox scripts
that this work supersedes.

* Make pg_restore actually work; drop the users-imports step

pg_restore was silently failing because the refactor moved Postgres
creds to libpq env vars but pg_restore (unlike pg_dump) requires
-d/--dbname on the command line. Now passed explicitly. Restore is
also now loud-fail rather than warn-and-continue — silent partial
restores were how this pipeline could silently rot.

Prod and staging share one Auth0 tenant (canonical
dev-udfgla0l.eu.auth0.com), so the nightly users-imports into a
separate dev tenant was a no-op. Removed the function, the
_sanitise_for_dev helper, the AUTH0_DEV_* config fields, and the
SKIP_AUTH0_IMPORT flag. The Auth0 export still runs to produce a
nightly backup blob.

Other fixes:
- _blob_url() helper builds proper container/prefix URLs so the
  destination LFS path resolves under snapshots/lfs/ rather than
  pointing at a non-existent lfs container.
- pg_env() exports PGHOST/PGUSER/PGPASSWORD/PGDATABASE so the
  password never appears on a subprocess command line (and thus
  not in CalledProcessError tracebacks).
- run() rebuilds CalledProcessError with redacted args before
  re-raising for the same reason.
- ckan search-index rebuild now passes --force and treats the
  rebuild result as advisory — individual datasets with
  serialization quirks shouldn't kill the whole sync.
- ckan -c path is /tmp/production.ini (where base.ini + secrets.ini
  are merged at container startup), not /etc/ckan/.
- SKIP_PG_BACKUP env var lets us iterate on apply-phase work
  without re-dumping prod for ~8 min each run.
- Derive the datastore admin URL from the ckan admin URL by path
  swap; drop the now-redundant PROD/STAGING_DATASTORE_PG_URL env
  vars. The limited 'datastore' role can't SELECT the UUID-named
  resource tables anyway.

* sync: scale ckan+datapusher to 0 around restore; clear Solr on reindex

- Apply phase now scales ckan + datapusher to 0 before DROP DATABASE,
  scales back to 1 after. Datapusher connects as the 'datastore' role,
  which the sync (running as ckan_admin) can't terminate from SQL.
- Drop+recreate filters pg_terminate_backend by usename=current_user
  so an Azure system superuser session can't abort the whole DROP.
- pg_restore now logs the ~7 expected errors (5 benign duplicate-index
  entries from ckanext-harvest + 2 real prod data dups) and continues
  instead of aborting, matching the AWS-era behaviour.
- search-index rebuild now uses --clear; without it stale Solr docs
  point at deleted rows and 500 the home page via ckanext-restricted.
- Bump activeDeadlineSeconds to 4h; datastore restore alone is ~90 min.
- RBAC: grant patch on deployments + deployments/scale.
- Doc updates: architecture (scale-down step), failure modes,
  one-time setup (RBAC note).

* sync: run ckan db upgrade + extension initdb after pg_restore

The AWS-era sync was data-only — it relied on dev and prod schemas
being identical. When staging code adds a new extension or migration
ahead of prod, the post-sync staging DB is missing those tables and
CKAN logs CRITI 'requires database setup' messages until someone
remembers to run initdb by hand.

Run ckan db upgrade plus the per-extension initdb commands
(unaids/versions/validation) as part of the apply phase. All
idempotent; failures are logged but don't abort the sync.

* sync: auto-discover plugin initdb commands

Replace the hardcoded list (unaids/versions/validation) with
discovery from ckan.plugins in the live config. For each enabled
plugin, try ckan <plugin> initdb then init-db; swallow exit 2
('no such Click subcommand') so plugins without an initdb stay
silent. Adding a new extension that needs initdb now works without
editing sync.py — just enable it in ckan.plugins.

Cost: ~5-7 min on a 138-min job (20 plugins x 2 attempts x ~10s
CKAN CLI startup).

* update time and docs

* sync: fix namespace consistency in _scale; capture multi-line ckan.plugins

- _scale() now takes cfg and uses cfg.ckan_namespace instead of reading
  NAMESPACE env var. If CKAN_NAMESPACE is overridden the scale/wait
  calls now follow it instead of hard-coding adr-s.
- _enabled_plugins() previously read only the first line of
  ckan.plugins =; deploy/base.ini continues the value on an indented
  second line, so 6 plugins (incl. text_view, auth) were being skipped
  and their initdb steps never ran after a restore. Awk now captures
  the first match plus any indented continuation lines.

Both raised by Copilot review on PR #203.

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Author: cooper667

deploy/sync/Dockerfile.sync

@@ -0,0 +1,27 @@
+# Image for the nightly prod -> staging sync CronJob.
+# Target: adracr.azurecr.io/adr-sync
+FROM ubuntu:24.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update -qq && apt-get install -y -qq \
+        ca-certificates curl gnupg lsb-release tar python3 python3-pip python3-requests \
+    && install -d /usr/share/postgresql-common/pgdg \
+    && curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
+        -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc \
+    && echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \
+        > /etc/apt/sources.list.d/pgdg.list \
+    && apt-get update -qq \
+    && apt-get install -y -qq postgresql-client-16 \
+    && curl -fsSL https://aka.ms/downloadazcopy-v10-linux -o /tmp/azcopy.tgz \
+    && tar -xzf /tmp/azcopy.tgz -C /tmp \
+    && install /tmp/azcopy_linux_*/azcopy /usr/local/bin/azcopy \
+    && curl -fsSL -o /usr/local/bin/kubectl \
+        "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" \
+    && chmod +x /usr/local/bin/kubectl \
+    && rm -rf /var/lib/apt/lists/* /tmp/azcopy*
+
+COPY sync.py /usr/local/bin/sync.py
+RUN chmod +x /usr/local/bin/sync.py
+
+ENTRYPOINT ["python3", "/usr/local/bin/sync.py"]

deploy/sync/cronjob.yaml

@@ -0,0 +1,77 @@
+# Nightly prod -> staging sync CronJob.
+# Apply with: kubectl apply -f deploy/sync/cronjob.yaml
+# Manual one-off run: kubectl create job --from=cronjob/adr-sync adr-sync-manual-$(date +%s) -n adr-s
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: adr-sync
+  namespace: adr-s
+spec:
+  schedule: "0 1 * * *"  # 01:00 UTC daily
+  timeZone: "Etc/UTC"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 7
+  jobTemplate:
+    spec:
+      backoffLimit: 0  # don't retry — re-run manually after fixing
+      activeDeadlineSeconds: 14400  # 4 hours; datastore restore alone is ~90 min
+      template:
+        spec:
+          restartPolicy: Never
+          serviceAccountName: adr-sync  # needs `kubectl exec` on the ckan deployment
+          containers:
+            - name: sync
+              image: adracr.azurecr.io/adr-sync:latest
+              imagePullPolicy: Always
+              resources:
+                requests:
+                  cpu: "500m"
+                  memory: "1Gi"
+                limits:
+                  cpu: "2"
+                  memory: "4Gi"
+              envFrom:
+                - secretRef:
+                    name: adr-sync-secrets
+              env:
+                - name: CKAN_NAMESPACE
+                  value: "adr-s"
+                - name: CKAN_DEPLOYMENT
+                  value: "deploy/ckan"
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: adr-sync
+  namespace: adr-s
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: adr-sync
+  namespace: adr-s
+rules:
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["get", "patch"]
+  - apiGroups: ["apps"]
+    resources: ["deployments/scale"]
+    verbs: ["get", "patch", "update"]
+  - apiGroups: [""]
+    resources: ["pods", "pods/exec"]
+    verbs: ["get", "list", "create", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: adr-sync
+  namespace: adr-s
+subjects:
+  - kind: ServiceAccount
+    name: adr-sync
+    namespace: adr-s
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: adr-sync

deploy/sync/secrets.yaml.template

@@ -0,0 +1,50 @@
+# Template for the k8s Secret consumed by the adr-sync CronJob.
+#
+# Do NOT commit a populated copy. Either:
+#   1. Use sealed-secrets / external-secrets and commit the encrypted form.
+#   2. Apply manually with `kubectl apply -f` from a local copy that lives
+#      only on the operator's machine.
+#
+# All values are strings. The SAS tokens should be issued with the minimum
+# necessary permissions:
+#   - PROD_LFS_SAS:        read+list on adr-p-datalake
+#   - SNAPSHOTS_SAS:       read+list+write+delete on adr-snapshots (full)
+#   - STAGING_LFS_SAS:     read+list+write+delete on adr-s-datalake
+#
+# Auth0 M2M scopes: read:users create:users (for users-exports job).
+# Use the canonical tenant domain (not the custom domain) for AUTH0_PROD_DOMAIN —
+# the Management API token endpoint validates against the canonical hostname.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: adr-sync-secrets
+  namespace: adr-s
+type: Opaque
+stringData:
+  # Postgres connection URLs — use the ckan_admin (or equivalent) role with
+  # access to BOTH the ckan and datastore databases. The sync derives the
+  # datastore URL by swapping the path component. Do NOT use the limited
+  # `datastore` role here; it can't SELECT the UUID-named resource tables.
+  PROD_CKAN_PG_URL:    "postgresql://ckan_admin:PASS@adr-p-eun-db001.postgres.database.azure.com/ckan?sslmode=require"
+  STAGING_CKAN_PG_URL: "postgresql://ckan_admin:PASS@adr-s-eun-db001.postgres.database.azure.com/ckan?sslmode=require"
+
+  # Storage
+  SNAPSHOTS_ACCOUNT:        "adrsnapshotsta"
+  SNAPSHOTS_CONTAINER:      "snapshots"
+  SNAPSHOTS_SAS:            "sv=...&sig=..."   # full rwl(d) on the container
+  PROD_LFS_ACCOUNT:         "adrpeunsta"
+  PROD_LFS_CONTAINER:       "adr-p-datalake"
+  PROD_LFS_SAS:             "sv=...&sig=..."   # read+list only
+  STAGING_LFS_ACCOUNT:      "adrseunsta"
+  STAGING_LFS_CONTAINER:    "adr-s-datalake"
+  STAGING_LFS_SAS:          "sv=...&sig=..."   # rwl(d)
+
+  # Auth0 — single tenant, M2M creds for the users-exports backup job.
+  # Env var names are AUTH0_PROD_* for backward-compat with earlier secret revisions;
+  # there is no AUTH0_DEV_* anymore because prod and staging share one tenant.
+  AUTH0_PROD_DOMAIN:        "dev-udfgla0l.eu.auth0.com"   # canonical, not the vanity custom domain
+  AUTH0_PROD_CLIENT_ID:     "..."
+  AUTH0_PROD_CLIENT_SECRET: "..."
+
+  # Slack (optional — omit to disable notifications)
+  SLACK_WEBHOOK_URL:        "https://hooks.slack.com/services/..."