ci: split db-backup into preflight/dump/release jobs and switch to zstd -19 --long=31 (#561)

* ci: split db-backup into preflight/dump/release jobs and switch to zstd -19 --long=31

Two changes, bundled:

1. Split the single mega-job into three phases so failures are legible and
   no job holds both powers:
   - preflight (ubuntu-latest): computes the dump filename and release tag
     ONCE as job outputs (per-job date stamping could straddle midnight UTC
     and split a run across two dates); fails fast if the tag already exists.
   - dump (blacksmith-32vcpu): holds only the DB credential, no repo write.
     Keeps pg_dump | zstd | split as one streaming step -- a job boundary
     between dump and compress would force uploading and re-downloading the
     full uncompressed database (tens of GB) as an artifact. Adds a
     post-compress `zstd -t` integrity pass, a SHA256SUMS, and a forensics
     header (nproc/free/df).
   - release (ubuntu-latest): holds only contents:write, no DB credential.
     Verifies checksums, creates the release with SHA256SUMS attached.

2. Fix the failing weekly run (#29166233752) and pick the compressor from a
   benchmark on a real 3 GiB dump sample (ratios conservative -- the sample
   loses cross-region matches the contiguous stream has):
     zstd -12 --long=27  24.4x @ 377 MiB/s   (too weak)
     zstd -19 --long=31  36.7x @  27 MiB/s   <- chosen
     xz -9e dict=192MiB  35.8x @   5 MiB/s   (old prod: worse ratio, 5x slower)
     zstd -22 --ultra    38.9x @   8 MiB/s   (+6% only, ~130 min -> times out)
   The old `xz -9e --dict=192MiB` ran near single-threaded (~5 MiB/s: 9e
   can't fill its 576 MiB blocks fast enough to parallelize on a pipe), so
   the dump ran for hours and the job was cancelled/timed out. zstd -19
   --long=31 beats it on ratio, runs ~5x faster (~40 min on ~60 GB, well
   under the 120 min timeout), and uses a bounded window (no runaway memory).
   Restore docs updated to `zstd -d --long=31`.

A failed release is now re-runnable without repeating the dump.

* ci: right-size db dump runner to 8 vCPU

The 32-vCPU/128-GB runner was sized for xz -9e's memory/thread appetite.
zstd -19 --long=31 uses a bounded ~2 GiB window (a few GiB total across
threads) and the pipeline is likely source-bound (pg_dump over the
network), so the monster runner is unnecessary. Drop to blacksmith-8vcpu;
the forensics header (nproc/free/df) and zstd -v rate on the first real
run confirm sizing and timeout headroom before downsizing further.

* ci: fix db dump OOM -- zstd --long=31 -> --long=27 (bounded memory)

The smoke test (#29176304089) OOM-killed zstd on the 8 vCPU/30 GiB runner
when the large eval_samples table started streaming: broken pipe, exit 143,
~8.5 min in (not the timeout). Root cause: zstd's multithreaded memory
scales with windowSize x thread count, and --long=31's 2 GiB window across
8 -T0 workers exceeded 30 GiB at steady state. The ratio benchmark used a
3 GiB sample that never reached that high-water, so it measured ratio
correctly but missed the memory cost.

--long=27 (128 MiB window) keeps memory to a few GiB regardless of thread
count and still compresses ~32.4x (vs 36.7x for --long=31, 24.4x for the
old -12), while decompressing with plain `zstd -d` (<=27 is within zstd's
default decompressor limit). Reclaiming --long=31's ratio later means
capping threads or a high-memory runner; documented inline.

* ci: [temp] instrument db dump with memory sampler to find the mid-dump VM kill

Two smoke runs died mid-dump on a big-blob table (eval_samples,
benchmark_server_logs) with no OOM line and the VM terminated; --long=27
ruled out zstd as the memory hog. Add a 3s sampler logging total memory +
top-RSS processes so the next run's log shows the memory curve and the
owning process (likely pg_dump/libpq buffering a giant row). Temporary --
revert once the cause is confirmed.

* ci: [temp] back to 32-vCPU runner (blacksmithd needs the headroom) + expand mem sampler

The instrumented run proved the memory hog is blacksmithd (the Blacksmith
runner daemon) at ~28 GiB, not the pipeline (pg_dump 75 MiB, zstd 1.5 GiB).
The 30 GiB 8-vCPU tier OOM-killed on the daemon; the historical 32-vCPU/
128 GiB runner had headroom, so this only broke after the downsize. Revert
to 32 vCPU. Blacksmith has no workflow toggle to disable the cache (support
ticket only), and the ~28 GiB daemon looks like a bug worth reporting.

zstd kept at --long=27 to isolate the runner fix (restore --long=31 for the
better ratio once green). Sampler expanded: mem+swap, tmpfs usage, top-RSS
processes with argv, plus a one-time blacksmithd dump.

* ci: [temp] run db dump on github-hosted ubuntu-latest (blacksmithd fills any VM)

Instrumentation is conclusive: blacksmithd (the Blacksmith runner daemon)
grows to consume the whole VM regardless of size -- ~28 GiB on a 30 GiB
runner, ~110 GiB on a 115 GiB runner -- then OOMs the box mid-dump. The
pipeline is tiny throughout (pg_dump 75 MiB, zstd 1.5 GiB). A bigger
Blacksmith VM can't fix it and the cache has no workflow-level disable, so
move the dump job to a github-hosted ubuntu-latest (no blacksmithd; the
<2 GiB job fits 16 GiB). timeout 120 -> 180 for the 4-core compress on a
growing DB. Instrumentation kept until this passes green.

* ci: remove temp memory instrumentation from db dump

Diagnosis complete (blacksmithd was the OOM cause; fixed by moving to
ubuntu-latest). Drop the forensics step and the per-3s memory sampler;
keep a trimmed comment recording why the job runs on a GitHub-hosted
runner and stays at zstd --long=27.

* ci: strip explanatory comments from db-backup workflow

Remove the prose comments accumulated during debugging. Keep the inline
permission-scope notes (contents: read/write), the two pinned-action
version tags, and a shellcheck disable directive (which also clears
actionlint's last SC2086 warning). Rationale lives in git history.

* ci: restore cron schedule comment

Author: adibarra

.github/workflows/db-backup.yml

@@ -12,21 +12,51 @@ concurrency:
   cancel-in-progress: false
 
 jobs:
-  backup:
-    name: Back up database
-    timeout-minutes: 120
-    runs-on: blacksmith-32vcpu-ubuntu-2404
+  preflight:
+    name: Preflight
+    timeout-minutes: 5
+    runs-on: ubuntu-latest
     permissions:
-      contents: write # Create the database dump release
+      contents: read # Tag-collision check
+    outputs:
+      dump-file: ${{ steps.names.outputs.dump-file }}
+      tag: ${{ steps.names.outputs.tag }}
+      date: ${{ steps.names.outputs.date }}
     steps:
-      - name: Dump and split database
+      - name: Compute dump name and release tag
+        id: names
+        run: |
+          set -euo pipefail
+          DATE="$(date -u +%Y-%m-%d)"
+          {
+            echo "date=${DATE}"
+            echo "dump-file=inferencex-${DATE}.dump.zst"
+            echo "tag=db-dump/${DATE}"
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Fail fast if the release tag already exists
         env:
-          # Use primary; replica recovery can cancel long pg_dump snapshots.
-          DATABASE_URL: ${{ secrets.DATABASE_WRITE_URL }}
+          GH_TOKEN: ${{ github.token }}
+          TAG: ${{ steps.names.outputs.tag }}
         run: |
           set -euo pipefail
-          DUMP_FILE="inferencex-$(date -u +%Y-%m-%d).dump.xz"
+          if gh release view "$TAG" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
+            echo "::error::Release $TAG already exists; delete it first to re-dump for the same date."
+            exit 1
+          fi
 
+  dump:
+    name: Dump and compress
+    needs: preflight
+    timeout-minutes: 180
+    runs-on: ubuntu-latest
+    steps:
+      - name: Dump, compress, and split database
+        env:
+          DATABASE_URL: ${{ secrets.DATABASE_WRITE_URL }}
+          DUMP_FILE: ${{ needs.preflight.outputs.dump-file }}
+        run: |
+          set -euo pipefail
           docker run --rm --env DATABASE_URL \
             postgres:18-alpine@sha256:9a8afca54e7861fd90fab5fdf4c42477a6b1cb7d293595148e674e0a3181de15 \
             sh -ceu 'exec pg_dump \
@@ -37,29 +67,77 @@ jobs:
               --no-privileges \
               --exclude-table-data=public.user_feedback \
               --verbose' \
-            | xz -v -T0 --memlimit-compress=99% --lzma2=preset=9e,dict=192MiB \
+            | zstd -v -T0 -19 --long=27 \
             | split -b 1900m -d -a 2 - "${DUMP_FILE}.part"
 
-          echo "DUMP_FILE=${DUMP_FILE}" >> "$GITHUB_ENV"
-          echo "DUMP_GLOB=${DUMP_FILE}.part*" >> "$GITHUB_ENV"
-          echo "TAG=db-dump/$(date -u +%Y-%m-%d)" >> "$GITHUB_ENV"
+      - name: Verify compressed stream
+        env:
+          DUMP_FILE: ${{ needs.preflight.outputs.dump-file }}
+        run: |
+          set -euo pipefail
+          cat "${DUMP_FILE}".part* | zstd -t --long=27
+
+      - name: Checksum parts
+        env:
+          DUMP_FILE: ${{ needs.preflight.outputs.dump-file }}
+        run: |
+          set -euo pipefail
+          sha256sum "${DUMP_FILE}".part* > SHA256SUMS
+          cat SHA256SUMS
+
+      - name: Upload dump parts
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: db-dump-parts
+          path: |
+            inferencex-*.dump.zst.part*
+            SHA256SUMS
+          if-no-files-found: error
+          retention-days: 1
+          compression-level: 0
+
+  release:
+    name: Create release
+    needs: [preflight, dump]
+    timeout-minutes: 30
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write # Create the database dump release
+    steps:
+      - name: Download dump parts
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          name: db-dump-parts
+
+      - name: Verify checksums
+        run: |
+          set -euo pipefail
+          sha256sum -c SHA256SUMS
 
       - name: Create release
         env:
           GH_TOKEN: ${{ github.token }}
+          TAG: ${{ needs.preflight.outputs.tag }}
+          DATE: ${{ needs.preflight.outputs.date }}
+          DUMP_GLOB: ${{ needs.preflight.outputs.dump-file }}.part*
         run: |
+          set -euo pipefail
           cat > release-notes.md <<'EOF'
           Weekly InferenceX database dump.
 
+          Verify (optional):
+          `sha256sum -c SHA256SUMS`
+
           Reassemble and decompress:
-          `cat *.dump.xz.part* | xz -d > inferencex.dump`
+          `cat *.dump.zst.part* | zstd -d --long=27 > inferencex.dump`
 
           Restore to PostgreSQL or Neon:
           `pg_restore --exit-on-error --clean --if-exists --no-owner --no-privileges --dbname="YOUR_TARGET_DATABASE_URL" inferencex.dump`
           EOF
 
-          # $DUMP_GLOB is intentionally unquoted so the shell expands it to every part file.
-          gh release create "$TAG" $DUMP_GLOB \
-            --title "DB Dump $(date -u +%Y-%m-%d)" \
+          # shellcheck disable=SC2086
+          gh release create "$TAG" $DUMP_GLOB SHA256SUMS \
+            --repo "$GITHUB_REPOSITORY" \
+            --title "DB Dump $DATE" \
             --notes-file release-notes.md \
             --latest=false