benchmarks-website/server: snapshot in Vortex format, not CSV

Switches /api/admin/snapshot from `EXPORT DATABASE … (FORMAT csv)` to
per-table `COPY (SELECT * FROM <table>) TO … (FORMAT vortex)`. Dogfoods
the project's own format and compresses an order of magnitude better
than gzipped CSV on this shape (BIGINT[] runtime arrays + short strings).

- schema.rs grows a `pub const TABLES: &[&str]` so the snapshot loop
  has a stable list to iterate.
- admin::snapshot writes `schema.sql` from `SCHEMA_DDL` verbatim, then
  `INSTALL vortex FROM community; LOAD vortex;` (idempotent — autoload
  is enabled in the bundled libduckdb-sys), then one COPY per table.
- ops/README.md restore section rewritten: untar → `.read schema.sql`
  → `INSERT … SELECT * FROM read_vortex(<file>)` per table.
- Two snapshot tests are marked `#[ignore]` because they need outbound
  network to fetch the vortex extension. Run them by hand before merge:
      cargo test -p vortex-bench-server --test admin -- --ignored

Signed-off-by: Claude <noreply@anthropic.com>

Author: claude

benchmarks-website/ops/README.md

@@ -23,14 +23,14 @@ out-of-tree state — every script and unit lives in
   range touch website-relevant paths it builds, atomically swaps the
   binary, and restarts the server. Otherwise it fast-forwards the
   working tree and exits.
-- A second timer fires hourly, asks the server to `EXPORT DATABASE` to
-  a CSV snapshot, `tar czf`s it into a single archive, and uploads it
-  to `s3://vortex-benchmark-results-database/v3-backups/<UTC ts>.tar.gz`.
-  CSV is the default DuckDB EXPORT format (only format the `bundled`
-  libduckdb-sys feature ships with), and gzip reclaims 5–7× on this
-  shape — most data lands in the BIGINT[] runtime columns which
-  bloat 2–3× as text. Flipping to parquet or a Vortex layout later is
-  a one-line change in [`server/src/admin.rs`](../server/src/admin.rs).
+- A second timer fires hourly, asks the server to write a per-table
+  Vortex snapshot (`schema.sql` + one `<table>.vortex` per table),
+  `tar czf`s it, and uploads to
+  `s3://vortex-benchmark-results-database/v3-backups/<UTC ts>.tar.gz`.
+  The vortex DuckDB extension is auto-installed from the community
+  repo on first call. Vortex compresses the BIGINT[] runtime arrays
+  and string columns roughly an order of magnitude better than
+  gzipped CSV — and dogfoods the project's own format.
 - For ad-hoc reads, `inspect.sh` calls a bearer-gated `/api/admin/sql`
   endpoint instead of stopping the server.
 - For DB-replacing operations (re-running the v2→v3 migration),
@@ -52,7 +52,7 @@ out-of-tree state — every script and unit lives in
 │    bin/                                                              │
 │      vortex-bench-server        ← symlink → versioned binary         │
 │      vortex-bench-server.<ts>   ← versioned, last $KEEP_BINARIES (3) │
-│    snapshots/<ts>/              ← transient EXPORT DATABASE landing  │
+│    snapshots/<ts>/              ← transient vortex-snapshot landing  │
 │    last-deployed-sha            ← stamp file for the deploy timer    │
 │    .deploy.lock                 ← flock guard                        │
 │    ops -> /home/ec2-user/vortex/benchmarks-website/ops               │
@@ -78,8 +78,7 @@ out-of-tree state — every script and unit lives in
                 │     <UTC ts>.tar.gz                   │
                 │       <UTC ts>/                       │
                 │         schema.sql                    │
-                │         load.sql                      │
-                │         <table>.csv                   │
+                │         <table>.vortex                │
                 └───────────────────────────────────────┘
 ```
 
@@ -335,20 +334,35 @@ aws s3 ls s3://vortex-benchmark-results-database/v3-backups/ | tail -20
 ```
 
 Each `<ts>.tar.gz` archive contains a single directory `<ts>/` with
-the artifacts of DuckDB's `EXPORT DATABASE`: a `schema.sql`, a
-`load.sql`, and one CSV file per table. Restore on a fresh box:
+a `schema.sql` (verbatim DDL the server applies on boot) and one
+`<table>.vortex` per table. Restore on a fresh box:
 
 ```bash
 sudo systemctl stop vortex-bench-server
 cd /tmp
 aws s3 cp s3://vortex-benchmark-results-database/v3-backups/<ts>.tar.gz .
 tar xzf <ts>.tar.gz                     # extracts ./<ts>/
-duckdb /var/lib/vortex-bench/bench.duckdb -c "IMPORT DATABASE '/tmp/<ts>'"
+ts=<ts>                                 # e.g. 20260508T010000Z
+sudo -u ec2-user rm -f /var/lib/vortex-bench/bench.duckdb \
+                       /var/lib/vortex-bench/bench.duckdb.wal
+duckdb /var/lib/vortex-bench/bench.duckdb <<EOF
+INSTALL vortex FROM community;
+LOAD vortex;
+.read /tmp/${ts}/schema.sql
+INSERT INTO commits             SELECT * FROM read_vortex('/tmp/${ts}/commits.vortex');
+INSERT INTO query_measurements  SELECT * FROM read_vortex('/tmp/${ts}/query_measurements.vortex');
+INSERT INTO compression_times   SELECT * FROM read_vortex('/tmp/${ts}/compression_times.vortex');
+INSERT INTO compression_sizes   SELECT * FROM read_vortex('/tmp/${ts}/compression_sizes.vortex');
+INSERT INTO random_access_times SELECT * FROM read_vortex('/tmp/${ts}/random_access_times.vortex');
+INSERT INTO vector_search_runs  SELECT * FROM read_vortex('/tmp/${ts}/vector_search_runs.vortex');
+EOF
 sudo systemctl start vortex-bench-server
 ```
 
-(`duckdb` CLI version doesn't strictly have to match — DuckDB's
-import format is forward/backward compatible across recent versions.)
+The `duckdb` CLI version needs to be recent enough that the vortex
+community extension is published for it. If `INSTALL vortex FROM
+community` fails, upgrade the CLI to match (or exceed) the version
+the server was built against (`duckdb` crate `1.10502` ≈ DuckDB 1.5.x).
 
 If you want to take an out-of-band snapshot (e.g. before a risky
 operation), just call the same endpoint the timer does:
@@ -426,7 +440,7 @@ also constitute the public admin contract for any future tooling.
 | Method + path                                                    | Bearer        | Notes                                                                                          |
 |------------------------------------------------------------------|---------------|------------------------------------------------------------------------------------------------|
 | `GET /health`                                                    | none          | `deploy.sh` polls for liveness after a restart.                                                |
-| `POST /api/admin/snapshot?ts=<id>`                               | admin         | Triggers `EXPORT DATABASE`. `ts` must match `[A-Za-z0-9_-]{1,64}`. 409 if the dir exists.      |
+| `POST /api/admin/snapshot?ts=<id>`                               | admin         | Writes `schema.sql` + per-table `.vortex` files. `ts` must match `[A-Za-z0-9_-]{1,64}`. 409 if the dir exists. |
 | `POST /api/admin/sql` (body `{"sql": …}`, `?format=json\|table`) | admin         | Read-only SQL only — `SELECT`/`WITH`/`PRAGMA`/`SHOW`/`DESCRIBE`/`EXPLAIN`.                     |
 | `POST /api/ingest`                                               | ingest        | Used by CI, not by these scripts. Documented under [`crate::ingest`].                          |
 

benchmarks-website/ops/backup.sh

@@ -4,16 +4,16 @@
 #
 # Hourly snapshot to S3, called by vortex-bench-backup.timer.
 #
-# Asks the running server to EXPORT DATABASE via /api/admin/snapshot
-# (so the export uses the same DuckDB process that owns the file — no
-# stop required), `tar czf`s the resulting CSV dir into a single
-# archive, uploads it to $S3_BACKUP_PREFIX/<ts>.tar.gz, and deletes
-# the local copies.
+# Asks the running server to write a per-table Vortex snapshot via
+# /api/admin/snapshot (so the writer uses the same DuckDB process
+# that owns the file — no stop required), `tar czf`s the resulting
+# directory into a single archive, uploads it to
+# $S3_BACKUP_PREFIX/<ts>.tar.gz, and deletes the local copies.
 #
-# We gzip rather than uploading raw CSVs because DuckDB's CSV EXPORT
-# is verbose for our shape (most data lands in BIGINT[] runtime
-# columns that bloat 2–3× as text); gzip typically reclaims 5–7× on
-# this kind of payload.
+# Vortex compresses our shape (mostly BIGINT[] runtime arrays + short
+# strings) far better than gzipped CSV; the additional gzip on the
+# tarball is largely catching schema.sql and tar metadata, not the
+# data files themselves.
 #
 # The instance IAM role must already permit s3:PutObject under
 # $S3_BACKUP_PREFIX. (Same bucket the v2 backup script used.)

benchmarks-website/server/src/admin.rs

@@ -14,19 +14,23 @@
 //!
 //! ### `POST /api/admin/snapshot?ts=<id>`
 //!
-//! Runs `EXPORT DATABASE '<snapshot_dir>/<ts>/' (FORMAT csv)` against the
-//! live DuckDB connection. CSV is the only EXPORT format the
-//! `bundled` libduckdb-sys feature ships with — switching to parquet or
-//! a Vortex layout later means flipping the duckdb feature flag and
-//! changing one literal below. CSV round-trips losslessly through
-//! `IMPORT DATABASE` (a `schema.sql` is written alongside the data so
-//! types and array columns rehydrate correctly).
+//! Writes a snapshot directory `<snapshot_dir>/<ts>/` containing:
+//! - `schema.sql` — verbatim copy of [`crate::schema::SCHEMA_DDL`], so a
+//!   restore knows how to recreate the tables before bulk-loading.
+//! - `<table>.vortex` for every table in [`crate::schema::TABLES`] —
+//!   each produced by a `COPY (SELECT * FROM <table>) TO …
+//!   (FORMAT vortex)`. The vortex DuckDB extension is auto-installed
+//!   from the community repo on first call, then `LOAD`ed.
+//!
+//! Vortex compresses the BIGINT[] runtime arrays and string columns
+//! roughly an order of magnitude better than gzipped CSV on this shape;
+//! it is also the project's own format, which is the obvious dogfood.
 //!
 //! `ts` must match `[A-Za-z0-9_-]{1,64}`; the snapshot script
 //! conventionally passes a UTC timestamp like `20260508T010000Z`. The
-//! target subdirectory must not already exist (409 otherwise). The export
-//! is transactionally consistent: writes during the export queue on the
-//! connection mutex.
+//! target subdirectory must not already exist (409 otherwise). All
+//! per-table COPY statements run on a connection cloned from the
+//! shared handle, so concurrent ingest writes are not blocked.
 //!
 //! ### `POST /api/admin/sql`
 //!
@@ -62,6 +66,7 @@ use thiserror::Error;
 
 use crate::app::AppState;
 use crate::db;
+use crate::schema;
 
 /// Errors surfaced by `/api/admin/*` handlers. Auth (401) is handled by
 /// [`require_admin_bearer`] and never reaches a handler.
@@ -162,8 +167,9 @@ pub struct SnapshotResponse {
     pub snapshot_dir: String,
 }
 
-/// Handler for `POST /api/admin/snapshot?ts=<id>`. Runs `EXPORT DATABASE`
-/// to a fresh subdirectory under [`AppState::snapshot_dir`].
+/// Handler for `POST /api/admin/snapshot?ts=<id>`. Writes
+/// `schema.sql` plus one `<table>.vortex` file per fact/dim table into
+/// a fresh subdirectory under [`AppState::snapshot_dir`].
 pub async fn snapshot(
     State(state): State<AppState>,
     Query(q): Query<SnapshotQuery>,
@@ -176,27 +182,43 @@ pub async fn snapshot(
             target.display()
         )));
     }
-    if let Some(parent) = target.parent() {
-        std::fs::create_dir_all(parent)
-            .with_context(|| format!("creating snapshot parent {}", parent.display()))?;
-    }
-    let target_str = target
-        .to_str()
-        .ok_or_else(|| AdminError::Internal(anyhow::anyhow!("snapshot path is not UTF-8")))?
-        .to_string();
-    let target_for_response = target.clone();
+    std::fs::create_dir_all(&target)
+        .with_context(|| format!("creating snapshot dir {}", target.display()))?;
+
+    // Schema is just our DDL string verbatim; restore reads this with
+    // `duckdb -init schema.sql` (or `.read schema.sql`) before
+    // bulk-loading the per-table vortex files.
+    std::fs::write(target.join("schema.sql"), schema::SCHEMA_DDL)
+        .with_context(|| format!("writing schema.sql under {}", target.display()))?;
+
+    let target_for_db = target.clone();
     db::run_blocking(&state.db, move |conn| {
-        // `target_str` is composed from the configured snapshot dir + a
-        // validated [A-Za-z0-9_-] timestamp, so single-quote escaping is
-        // a non-issue here.
-        let sql = format!("EXPORT DATABASE '{target_str}' (FORMAT csv)");
-        conn.execute_batch(&sql)
-            .with_context(|| format!("EXPORT DATABASE to {target_str}"))
+        // Idempotent — `INSTALL` is a no-op if the extension is already
+        // present, `LOAD` is cheap once the binary is on disk. The
+        // bundled libduckdb-sys has autoload enabled, so the very first
+        // call also auto-fetches the extension from the DuckDB
+        // community repo. Subsequent calls are entirely local.
+        conn.execute_batch("INSTALL vortex FROM community; LOAD vortex;")
+            .context("INSTALL/LOAD vortex extension")?;
+        for table in schema::TABLES {
+            // Single-quote escaping is a non-issue: `target_for_db`
+            // is composed from the operator-configured snapshot dir +
+            // a validated [A-Za-z0-9_-] timestamp, and table names
+            // come from the closed const list in schema.rs.
+            let path = target_for_db.join(format!("{table}.vortex"));
+            let path_str = path
+                .to_str()
+                .ok_or_else(|| anyhow::anyhow!("snapshot path is not UTF-8: {}", path.display()))?;
+            let sql = format!("COPY (SELECT * FROM {table}) TO '{path_str}' (FORMAT vortex)");
+            conn.execute_batch(&sql)
+                .with_context(|| format!("COPY {table} TO {path_str}"))?;
+        }
+        Ok(())
     })
     .await
     .map_err(AdminError::Internal)?;
     Ok(Json(SnapshotResponse {
-        snapshot_dir: target_for_response.display().to_string(),
+        snapshot_dir: target.display().to_string(),
     }))
 }
 

benchmarks-website/server/src/schema.rs

@@ -183,3 +183,16 @@ CREATE TABLE IF NOT EXISTS vector_search_runs (
 /// Schema version expected by the server. The ingest envelope's
 /// `run_meta.schema_version` must match this exactly at alpha.
 pub const SCHEMA_VERSION: i32 = 1;
+
+/// Every table in the schema, in the order a fresh boot creates them.
+/// Used by the snapshot endpoint to drive a per-table `COPY ... TO`
+/// across the whole DB and by the restore docs to document the same
+/// list. `commits` is the dim table; the rest are facts.
+pub const TABLES: &[&str] = &[
+    "commits",
+    "query_measurements",
+    "compression_times",
+    "compression_sizes",
+    "random_access_times",
+    "vector_search_runs",
+];

benchmarks-website/server/tests/admin.rs

@@ -240,7 +240,13 @@ async fn admin_unmounted_when_admin_token_absent() -> Result<()> {
     Ok(())
 }
 
+// The snapshot endpoint INSTALLs and LOADs the vortex DuckDB community
+// extension on first call; that needs outbound network to
+// `community-extensions.duckdb.org` which sandboxed CI environments
+// generally don't allow. Run manually before merge:
+//   cargo test -p vortex-bench-server --test admin -- --ignored
 #[tokio::test]
+#[ignore = "needs network to install the vortex DuckDB community extension"]
 async fn admin_snapshot_creates_export_directory() -> Result<()> {
     let server = Server::start_with_admin().await?;
     let client = reqwest::Client::new();
@@ -260,11 +266,22 @@ async fn admin_snapshot_creates_export_directory() -> Result<()> {
         .context("snapshot_dir field")?;
     let dir_path = std::path::PathBuf::from(dir);
     assert!(dir_path.exists(), "{dir} should exist");
-    // EXPORT DATABASE always writes a schema.sql.
+    // schema.sql is written verbatim from SCHEMA_DDL.
     assert!(
         dir_path.join("schema.sql").exists(),
         "{dir}/schema.sql should exist"
     );
+    // One .vortex file per table — `commits` is the dim table and is
+    // present even when the DB is otherwise empty (the schema was
+    // applied at AppState::open).
+    assert!(
+        dir_path.join("commits.vortex").exists(),
+        "{dir}/commits.vortex should exist"
+    );
+    assert!(
+        dir_path.join("query_measurements.vortex").exists(),
+        "{dir}/query_measurements.vortex should exist"
+    );
     // And the directory should be under the configured snapshot dir.
     assert!(
         dir_path.starts_with(&server.snapshot_dir),
@@ -275,6 +292,7 @@ async fn admin_snapshot_creates_export_directory() -> Result<()> {
 }
 
 #[tokio::test]
+#[ignore = "needs network to install the vortex DuckDB community extension"]
 async fn admin_snapshot_rejects_existing_directory() -> Result<()> {
     let server = Server::start_with_admin().await?;
     let client = reqwest::Client::new();