Hotfix/sqlite rescue (#48)

* Add SQLite rescue for Databricks Apps persistence

Databricks Apps containers are ephemeral - SQLite databases are lost on
container restarts during platform updates. This adds a rescue module that:

- Restores DB from Unity Catalog Volume on startup (before migrations)
- Backs up DB to Volume on shutdown (SIGTERM/SIGINT handlers)
- Periodic backup after N write operations (configurable, default 50)
- Exposes rescue status in /health/detailed endpoint

Configuration via environment variables:
- SQLITE_VOLUME_BACKUP_PATH: Volume path for backup
- SQLITE_BACKUP_AFTER_OPS: Write ops before auto-backup

Also documents Databricks Apps authentication (service principals,
automatic credential injection, resource permissions) in the spec.

Co-Authored-By: Claude <noreply@anthropic.com>

* Update deploy recipe to use Databricks CLI directly

The old deploy.sh script was removed. Updated the `just deploy` recipe
to use the Databricks CLI directly for syncing and deploying:

- Syncs files to workspace (excluding .git, node_modules, etc.)
- Deploys app using `databricks apps deploy`
- Uses DATABRICKS_CONFIG_PROFILE and DATABRICKS_APP_NAME from .env.local
- Updated setup message to reference `just deploy`

Co-Authored-By: Claude <noreply@anthropic.com>

* Run npm install in ui-build if dependencies are stale

The ui-build recipe now checks if npm install needs to run by comparing
package.json modification time against node_modules. This ensures builds
don't fail due to missing dependencies while avoiding unnecessary installs.

Co-Authored-By: Claude <noreply@anthropic.com>

* Auto-create Databricks app if it doesn't exist

The deploy recipe now checks if the app exists before deploying.
If not, it creates the app first using `databricks apps create`.

Co-Authored-By: Claude <noreply@anthropic.com>

* Add root package.json for Databricks Apps Node.js build

Databricks Apps looks for package.json at the root to detect Node.js
projects. Added a root package.json that delegates to client/:

- Root package.json with build script that runs client build
- Removed npx from client scripts (not available in Databricks env)
- Simplified deploy recipe to let Databricks handle the build

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix SQLite rescue for Unity Catalog volume paths

The previous implementation failed because:
1. It tried to create parent directories under /Volumes which is not allowed
2. It didn't validate the UC volume path format
3. The volume must already exist - we can't create it

Fixed by:
- Added _validate_volume_path() to check path format early
- Added _get_volume_root() to extract and verify volume accessibility
- Don't call mkdir() for UC volume destinations
- Validate configuration at startup and fail fast with clear error
- Added path_valid, path_error, volume_accessible to status endpoint

Expected path format: /Volumes/<catalog>/<schema>/<volume>/workshop.db

Co-Authored-By: Claude <noreply@anthropic.com>

* Refactor SQLite rescue to use Databricks SDK Files API

Databricks Apps do NOT support FUSE mounts for UC volumes. This refactors
the sqlite_rescue module to use the Databricks SDK Files API instead of
direct filesystem operations.

Changes:
- Use WorkspaceClient.files.download/upload instead of shutil.copy
- Add _get_workspace_client() for cached SDK client
- Add _file_exists_on_volume() using files.get_status()
- Support SQLITE_VOLUME_PATH env var (base path, appends /workshop.db)
- Improve restore logic: try download directly, handle NotFound gracefully
- Add detailed logging to debug file operations
- Update app.yaml with valueFrom configuration example
- Update BUILD_AND_DEPLOY_SPEC.md documentation

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: forrestmurray-db

app.yaml

@@ -1,9 +1,35 @@
-command: 
+command:
   - "gunicorn"
   - "server.app:app"
   - "-w"
   - "2"
   - "--worker-class"
   - "uvicorn.workers.UvicornWorker"
   - "--timeout"
-  - "1800"
+  - "1800"
+
+# SQLite Rescue: Backup/restore DB to Unity Catalog Volume for persistence
+# across container restarts.
+#
+# IMPORTANT: Databricks Apps do NOT support FUSE mounts for UC volumes.
+# This app uses the Databricks SDK Files API for volume access.
+#
+# Setup steps:
+# 1. Add the Unity Catalog volume as an App resource in the Apps UI
+# 2. Grant the app "Can read and write" permission on the volume
+# 3. Give it a resource key (e.g., "db_backup_volume")
+# 4. Uncomment and configure the env section below
+#
+# Note: valueFrom provides the volume base path (e.g., /Volumes/catalog/schema/volume).
+# The app automatically appends "/workshop.db" to construct the full backup path.
+#
+# env:
+#   - name: SQLITE_VOLUME_PATH
+#     valueFrom: db_backup_volume  # Resource key from Apps UI
+#   - name: SQLITE_BACKUP_AFTER_OPS
+#     value: "50"  # Backup after every 50 write operations (default: 50, 0 to disable)
+#
+# Alternative: Specify the full path directly (works but less portable):
+# env:
+#   - name: SQLITE_VOLUME_BACKUP_PATH
+#     value: "/Volumes/your_catalog/your_schema/your_volume/workshop.db"

client/package.json

@@ -4,12 +4,12 @@
   "version": "0.1.0",
   "type": "module",
   "scripts": {
-    "start": "npx vite",
-    "dev": "npx vite",
-    "build": "npx vite build",
-    "preview": "npx vite preview",
-    "test": "npx playwright test",
-    "test:e2e": "npx playwright test",
+    "start": "vite",
+    "dev": "vite",
+    "build": "vite build",
+    "preview": "vite preview",
+    "test": "playwright test",
+    "test:e2e": "playwright test",
     "test:unit": "vitest run",
     "test:unit:watch": "vitest",
     "test:unit:coverage": "vitest run --coverage",

justfile

@@ -30,7 +30,7 @@ setup: setup-uv setup-prereqs setup-python setup-client configure test-connectio
   @echo ""
   @echo "🎯 Virtual environment created at: .venv/"
   @echo ""
-  @echo "Next step: run './deploy.sh' when ready to deploy"
+  @echo "Next step: run 'just deploy' when ready to deploy"
 
 # Install uv
 [group('setup')]
@@ -245,6 +245,15 @@ ui-dev:
 
 [group('dev')]
 ui-build:
+  #!/usr/bin/env bash
+  set -euo pipefail
+
+  # Run npm install if node_modules is missing or package.json is newer
+  if [ ! -d "{{client-dir}}/node_modules" ] || [ "{{client-dir}}/package.json" -nt "{{client-dir}}/node_modules" ]; then
+    echo "📦 Installing frontend dependencies..."
+    npm -C {{client-dir}} install
+  fi
+
   npm -C {{client-dir}} run build
 
 [group('dev')]
@@ -342,9 +351,37 @@ api port="8000":
 
 [group('app')]
 deploy:
-  just ui-build
-  just db-bootstrap
-  SKIP_UI_BUILD=1 ./deploy.sh
+  #!/usr/bin/env bash
+  set -euo pipefail
+
+  PROFILE="${DATABRICKS_CONFIG_PROFILE:-DEFAULT}"
+  APP="${DATABRICKS_APP_NAME:?DATABRICKS_APP_NAME is not set (run \`just configure\`)}"
+
+  echo "📦 Syncing files to workspace..."
+  DATABRICKS_USERNAME=$(databricks --profile "$PROFILE" current-user me | jq -r .userName)
+  WORKSPACE_PATH="/Workspace/Users/$DATABRICKS_USERNAME/$APP"
+
+  databricks --profile "$PROFILE" sync . "$WORKSPACE_PATH" \
+    --exclude ".git" \
+    --exclude "node_modules" \
+    --exclude "__pycache__" \
+    --exclude "*.db" \
+    --exclude ".venv" \
+    --exclude ".e2e-*"
+
+  # Create app if it doesn't exist
+  if ! databricks --profile "$PROFILE" apps get "$APP" &>/dev/null; then
+    echo "📱 Creating app: $APP"
+    databricks --profile "$PROFILE" apps create "$APP"
+  fi
+
+  echo "🚀 Deploying app: $APP"
+  echo "   Databricks will run: npm install → pip install → npm run build → app.yaml command"
+  databricks --profile "$PROFILE" apps deploy "$APP" --source-code-path "$WORKSPACE_PATH"
+
+  echo ""
+  echo "✅ Deployment initiated for $APP"
+  echo "   Run 'just app-info' to check deployment status"
 
 [group('dev')]
 dev api_port="8000" ui_port="5173":

package.json

@@ -0,0 +1,8 @@
+{
+  "name": "human-eval-workshop",
+  "private": true,
+  "description": "Root package.json to enable Databricks Apps Node.js build flow",
+  "scripts": {
+    "build": "npm -C client install && npm -C client run build"
+  }
+}

server/app.py

@@ -14,21 +14,50 @@
 from server.config import ServerConfig
 from server.db_bootstrap import maybe_bootstrap_db_on_startup
 from server.routers import router
+from server.sqlite_rescue import (
+    backup_to_volume,
+    get_rescue_status,
+    install_shutdown_handlers,
+    restore_from_volume,
+)
 
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     """Manage application lifespan with proper startup and shutdown."""
     print("🚀 Application startup - lifespan function called!")
 
+    # SQLite Rescue: Restore from Unity Catalog Volume if configured
+    # This MUST happen before database bootstrap/migrations
+    rescue_status = get_rescue_status()
+    if rescue_status["configured"]:
+        print(f"📦 SQLite rescue configured: {rescue_status['volume_backup_path']}")
+        if restore_from_volume():
+            print("✅ Database restored from Unity Catalog Volume")
+        else:
+            print("ℹ️  No backup to restore (starting fresh or backup not found)")
+
+        # Install signal handlers for graceful shutdown backup
+        install_shutdown_handlers()
+    else:
+        print("⚠️  SQLITE_VOLUME_BACKUP_PATH not configured - database will NOT persist across container restarts")
+
     # NOTE: This is a *fallback* safety net for deployments that don't run `just db-bootstrap`.
     # It is designed to be safe under multi-process servers (e.g., gunicorn with multiple
     # Uvicorn workers) via an inter-process lock.
     maybe_bootstrap_db_on_startup()
 
     print("✅ Application startup complete!")
     yield
+
+    # SQLite Rescue: Backup to Unity Catalog Volume on shutdown
     print("🔄 Application shutting down...")
+    if rescue_status["configured"]:
+        print("💾 Backing up database to Unity Catalog Volume...")
+        if backup_to_volume(force=True):
+            print("✅ Database backed up successfully")
+        else:
+            print("⚠️  Database backup failed or skipped")
 
 
 # Request timing middleware
@@ -113,7 +142,16 @@ async def detailed_health():
             "invalid": getattr(pool, "invalid", lambda: 0)(),  # Handle missing invalid method
         }
 
-        return {"status": "healthy", "database": "connected", "connection_pool": pool_info, "timestamp": time.time()}
+        # Get SQLite rescue status
+        rescue_status = get_rescue_status()
+
+        return {
+            "status": "healthy",
+            "database": "connected",
+            "connection_pool": pool_info,
+            "sqlite_rescue": rescue_status,
+            "timestamp": time.time(),
+        }
     except Exception as e:
         return {"status": "unhealthy", "database": "disconnected", "error": str(e), "timestamp": time.time()}
 

server/database.py

@@ -70,6 +70,18 @@ def set_sqlite_pragma(dbapi_connection, connection_record):
         cursor.execute("PRAGMA synchronous=NORMAL")
         cursor.close()
 
+
+# SQLite Rescue: Track write operations for backup triggering
+# This listener fires after successful commits and notifies the rescue module
+@event.listens_for(engine, "commit")
+def on_commit(conn):
+    """Record write operations for SQLite rescue backup triggering."""
+    try:
+        from server.sqlite_rescue import record_write_operation
+        record_write_operation()
+    except ImportError:
+        pass  # sqlite_rescue not available
+
 # Create session factory with better session management
 SessionLocal = sessionmaker(
     autocommit=False,