fix(db-backup): keep full dump via xz + <2GiB split parts (#419)

Author: adibarra

.claude/skills/inferencex-data/SKILL.md

@@ -5,11 +5,12 @@ description: Download and analyze InferenceX ML inference benchmark data — GPU
 
 # Setup
 
-Download the latest database dump from GitHub releases:
+Download the latest database dump from GitHub releases. It is xz-compressed and split into
+one or more `.tar.xz.part*` files; reassemble them by piping `cat` through `xz` (requires `xz`):
 
 ```bash
-gh release download --repo SemiAnalysisAI/InferenceX-app --pattern 'inferencex-dump-*.zip' --dir .
-unzip inferencex-dump-*.zip
+gh release download --repo SemiAnalysisAI/InferenceX-app --pattern 'inferencex-dump-*.tar.xz.part*' --dir .
+cat inferencex-dump-*.tar.xz.part* | xz -d -T0 | tar -x
 ```
 
 # Data

.github/workflows/db-backup.yml

@@ -8,7 +8,7 @@ on:
 jobs:
   backup:
     timeout-minutes: 30
-    runs-on: ubuntu-latest
+    runs-on: blacksmith-32vcpu-ubuntu-2404
     permissions:
       contents: write
     steps:
@@ -27,17 +27,23 @@ jobs:
         env:
           DATABASE_READONLY_URL: ${{ secrets.DATABASE_READONLY_URL }}
         run: |
+          set -euo pipefail
           DUMP_DIR="inferencex-dump-$(date -u +%Y-%m-%d)"
           pnpm admin:db:dump "$DUMP_DIR"
-          cd packages/db && zip -r "${GITHUB_WORKSPACE}/${DUMP_DIR}.zip" "$DUMP_DIR" && cd -
-          echo "DUMP_ARCHIVE=${DUMP_DIR}.zip" >> "$GITHUB_ENV"
+          # Keep every table intact: tar the dump, compress with xz/lzma2, and split into
+          # <2 GiB parts to stay under GitHub's per-release-asset cap (even as data grows).
+          ( cd packages/db
+            tar -cf - "$DUMP_DIR" \
+              | xz -T0 --lzma2=preset=9e,dict=192MiB \
+              | split -b 1900m -d -a 2 - "${GITHUB_WORKSPACE}/${DUMP_DIR}.tar.xz.part" )
+          echo "DUMP_GLOB=${DUMP_DIR}.tar.xz.part*" >> "$GITHUB_ENV"
           echo "TAG=db-dump/$(date -u +%Y-%m-%d)" >> "$GITHUB_ENV"
-
       - name: Create release
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
-          gh release create "$TAG" "$DUMP_ARCHIVE" \
+          # $DUMP_GLOB is intentionally unquoted so the shell expands it to every part file.
+          gh release create "$TAG" $DUMP_GLOB \
             --title "DB Dump $(date -u +%Y-%m-%d)" \
-            --notes "Weekly database dump." \
+            --notes "Weekly full database dump (xz-compressed, split into <2 GiB parts). Reassemble: cat *.tar.xz.part* | xz -d | tar -x" \
             --latest=false

README.md

@@ -55,17 +55,17 @@ You can run the dashboard against either a live database or a static JSON dump.
 
 #### Option A: JSON Dump (no database required, local dev only)
 
-Download the latest DB dump from [GitHub Releases](https://github.com/SemiAnalysisAI/InferenceX-app/releases), unzip it, and point `DUMP_DIR` at the directory. This only works with `pnpm dev` — production builds require a live database.
+Download the latest DB dump from [GitHub Releases](https://github.com/SemiAnalysisAI/InferenceX-app/releases), unpack it, and point `DUMP_DIR` at the directory. The dump is xz-compressed and split into one or more `.tar.xz.part*` files; reassemble them by piping `cat` through `xz`. This only works with `pnpm dev`; production builds require a live database.
 
 ```bash
 cp .env.example .env
 
-# Download and unzip the latest dump
-gh release download db-dump/2026-03-30 -p '*.zip'
-unzip inferencex-dump-2026-03-30.zip -d inferencex-dump
+# Download and unpack the latest dump (requires xz; `brew install xz` on macOS)
+gh release download db-dump/2026-03-30 -p 'inferencex-dump-*.tar.xz.part*'
+cat inferencex-dump-2026-03-30.tar.xz.part* | xz -d -T0 | tar -x
 
 # Add to .env
-echo 'DUMP_DIR=./inferencex-dump/inferencex-dump-2026-03-30' >> .env
+echo 'DUMP_DIR=./inferencex-dump-2026-03-30' >> .env
 ```
 
 Make sure `DATABASE_READONLY_URL` is not set (or is commented out) in your `.env`.