feat: add scripts to copy production data to preview environment (#574)

# Add scripts for copying production data to preview environment

This PR adds functionality to copy pgflow data from production to preview environments, which helps with testing and debugging using real-world data.

Key additions:
- Added two new scripts:
  - `download-production-dump.sh`: Downloads pgflow schema data from production
  - `restore-dump-to-preview.sh`: Restores downloaded data to preview environment
- Updated documentation in `DEPLOYMENT.md` with instructions for using these scripts
- Added `dumps` directory to `.gitignore` to prevent accidental commits of database dumps

The scripts handle copying flow definitions, run history, and worker registrations while preserving preview environment secrets and schema structure. This allows developers to test with production data without affecting sensitive information in the preview environment.

Author: jumski

apps/demo/.gitignore

@@ -22,3 +22,4 @@ Thumbs.db
 vite.config.js.timestamp-*
 vite.config.ts.timestamp-*
 supabase/functions/_vendor/
+dumps

apps/demo/DEPLOYMENT.md

@@ -209,6 +209,38 @@ pnpm supabase migration up --file 20251230202447_20251225163110_pgflow_add_flow_
 pnpm wrangler deploy
 ```
 
+## Copying Production Data to Preview
+
+To copy pgflow data (flows, runs, step states, etc.) from production to preview:
+
+```bash
+cd apps/demo
+
+# 1. Download pgflow data from production
+./scripts/download-production-dump.sh
+# Creates: dumps/production-data-<timestamp>.sql
+
+# 2. Restore to preview (will ask for confirmation)
+./scripts/restore-dump-to-preview.sh dumps/production-data-<timestamp>.sql
+```
+
+**What gets copied:**
+
+- `pgflow.flows`, `pgflow.steps`, `pgflow.deps` - flow definitions
+- `pgflow.runs`, `pgflow.step_states`, `pgflow.step_tasks` - run history
+- `pgflow.workers`, `pgflow.worker_functions` - worker registrations
+
+**What stays unchanged:**
+
+- Preview vault secrets (API keys, etc.)
+- Schema/migrations - only data is copied
+
+**Requirements:**
+
+- `.env.production` with `SUPABASE_DB_URL` for production
+- `.env.preview` with `SUPABASE_DB_URL` for preview
+- `pg_dump` and `psql` CLI tools installed
+
 ## Troubleshooting
 
 **Edge Functions failing:**

apps/demo/scripts/download-production-dump.sh

@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+#
+# Download data dump from demo-production
+# Run this from apps/demo directory
+#
+
+set -euo pipefail
+
+cd "$(dirname "$0")/.."
+
+# Colors for output
+GREEN='\033[0;32m'
+RED='\033[0;31m'
+NC='\033[0m'
+
+log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
+log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
+
+# Check for required tools
+if ! command -v pg_dump &> /dev/null; then
+  log_error "pg_dump not found. Please install PostgreSQL client tools."
+  exit 1
+fi
+
+# Verify env file exists
+if [ ! -f .env.production ]; then
+  log_error ".env.production not found"
+  exit 1
+fi
+
+# Create output directory
+mkdir -p dumps
+DUMP_FILE="dumps/production-data-$(date +%Y%m%d-%H%M%S).sql"
+
+log_info "Sourcing .env.production..."
+set -a; source .env.production; set +a
+
+if [ -z "${SUPABASE_DB_URL:-}" ]; then
+  log_error "SUPABASE_DB_URL not set in .env.production"
+  exit 1
+fi
+
+log_info "Dumping pgflow schema data from production to $DUMP_FILE"
+
+pg_dump "$SUPABASE_DB_URL" \
+  --data-only \
+  --schema=pgflow \
+  --no-owner \
+  --no-privileges \
+  > "$DUMP_FILE"
+
+log_info "Dump completed successfully!"
+log_info "File: $DUMP_FILE"
+ls -lh "$DUMP_FILE"

apps/demo/scripts/restore-dump-to-preview.sh

@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+#
+# Restore data dump to demo-preview
+# Run this from apps/demo directory after downloading dump
+#
+
+set -euo pipefail
+
+cd "$(dirname "$0")/.."
+
+# Colors for output
+GREEN='\033[0;32m'
+RED='\033[0;31m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+
+log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
+log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
+log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
+
+# Check for required tools
+if ! command -v psql &> /dev/null; then
+  log_error "psql not found. Please install PostgreSQL client tools."
+  exit 1
+fi
+
+# Check for dump file
+if [ $# -eq 0 ]; then
+  log_error "Usage: $0 <dump-file.sql>"
+  log_info "Available dumps in dumps/:"
+  ls -lh dumps/*.sql 2>/dev/null || log_info "  (none)"
+  exit 1
+fi
+
+DUMP_FILE="$1"
+
+if [ ! -f "$DUMP_FILE" ]; then
+  log_error "Dump file not found: $DUMP_FILE"
+  exit 1
+fi
+
+# Verify env file exists
+if [ ! -f .env.preview ]; then
+  log_error ".env.preview not found"
+  exit 1
+fi
+
+log_info "Sourcing .env.preview..."
+set -a; source .env.preview; set +a
+
+if [ -z "${SUPABASE_DB_URL:-}" ]; then
+  log_error "SUPABASE_DB_URL not set in .env.preview"
+  exit 1
+fi
+
+log_warn "This will REPLACE ALL pgflow schema data in preview"
+log_warn "Preview DB: $SUPABASE_DB_URL"
+read -p "Continue? (yes/no): " -r
+if [[ ! $REPLY =~ ^yes$ ]]; then
+  log_info "Aborted"
+  exit 0
+fi
+
+log_info "Step 1: Truncating pgflow schema data..."
+
+psql "$SUPABASE_DB_URL" <<'EOSQL'
+SET session_replication_role = 'replica';
+
+-- Truncate all pgflow tables (order matters due to FKs, CASCADE handles it)
+TRUNCATE TABLE pgflow.step_tasks CASCADE;
+TRUNCATE TABLE pgflow.step_states CASCADE;
+TRUNCATE TABLE pgflow.runs CASCADE;
+TRUNCATE TABLE pgflow.workers CASCADE;
+TRUNCATE TABLE pgflow.worker_functions CASCADE;
+TRUNCATE TABLE pgflow.deps CASCADE;
+TRUNCATE TABLE pgflow.steps CASCADE;
+TRUNCATE TABLE pgflow.flows CASCADE;
+
+SET session_replication_role = 'origin';
+EOSQL
+
+log_info "Step 2: Loading dump..."
+
+psql "$SUPABASE_DB_URL" < "$DUMP_FILE"
+
+log_info "Restore completed successfully!"
+log_info "Preview database now has production pgflow data"