benchmarks-website: systemd-based v3 deploy + admin endpoints

Replaces the ad-hoc SSH-and-`nohup` deploy of the v3 benchmarks site
with a systemd timer that polls origin/develop every 60s, builds and
atomically swaps the binary, and verifies /health. Adds an hourly
gzipped-snapshot timer and two server-side admin endpoints so backups
and ad-hoc reads no longer need to stop the server.

Two new routes mounted only when ADMIN_BEARER_TOKEN is set:

- POST /api/admin/snapshot?ts=<id>: runs `EXPORT DATABASE … (FORMAT
  csv)` against the live DuckDB connection, into a fresh subdirectory
  under AppState::snapshot_dir. ts must match [A-Za-z0-9_-]{1,64}.
  CSV is the only EXPORT format that ships with libduckdb-sys's
  `bundled` feature; flipping to parquet or a Vortex layout later is
  a one-line change.
- POST /api/admin/sql {sql, ?format=json|table}: runs read-only SQL
  (SELECT/WITH/PRAGMA/SHOW/DESCRIBE/EXPLAIN, anything else 403) and
  renders either JSON or a duckdb-cli-style ASCII table. Uses the
  same connection mutex as ingest, so a slow SELECT briefly delays
  writes.

Auth is independent of the ingest token (separate ADMIN_BEARER_TOKEN
env var) so the two rotate separately. Both use constant-time eq.

Everything an EC2 host needs lives under benchmarks-website/ops/:

- install.sh: idempotent one-time bootstrap (state dirs under
  /var/lib/vortex-bench, sudoers fragment, env-file template, systemd
  units, enable + start the timers). Recommended first-time path is
  "wait for the deploy timer to build, then run migrate.sh"; preserving
  an existing $HOME/bench.duckdb is documented as a side note.
- deploy.sh: called by vortex-bench-deploy.timer every 60s. Cheap fast
  path (sha == stamp → exit 0). Path filter on benchmarks-website/{server,
  migrate}, Cargo.toml, Cargo.lock — vortex-array PRs fast-forward the
  working tree but skip the rebuild. Atomic versioned-binary symlink
  swap, sudo systemctl restart, /health verification with rollback to
  previous binary on failure, stamp updated only on success so failures
  retry on next tick. Keeps last KEEP_BINARIES (default 3) versions.
- migrate.sh: stops server, snapshots current DB to bench.prev-<ts>.duckdb,
  passes args through to `cargo run -p vortex-bench-migrate --`, restarts.
- backup.sh: hourly. Calls /api/admin/snapshot, `tar czf`s the CSV
  directory into <ts>.tar.gz (gzip reclaims ~5–7× on this shape since
  most data lands in BIGINT[] runtime arrays serialised as text), uploads
  with `aws s3 cp` to s3://vortex-ci-benchmark-results/v3-backups/, and
  cleans up both local copies. Logs the compression ratio so a future
  regression shows up in `journalctl -u vortex-bench-backup`.
- inspect.sh: thin wrapper around /api/admin/sql, no server stop.
- systemd/ units: server (Type=simple, Restart=on-failure, hardening
  via ProtectSystem=strict), deploy oneshot + 60s timer, backup
  oneshot + hourly timer (Persistent=true so a missed hour catches up
  after reboot).

A symlink at /var/lib/vortex-bench/ops -> .../benchmarks-website/ops
keeps the systemd ExecStart paths stable as the repo location changes.

- ops/README.md: full operator runbook — first-time install, day-to-day
  ops (push to develop → live in 60s, monitor a deploy, force a deploy,
  re-run migration, ad-hoc SQL, backup/restore, token rotation), and
  failure modes (deploy retry loop, /health stuck, disk filling up,
  backup not running, host migration). Targeted at a fresh admin who
  has never seen the system before.
- benchmarks-website/README.md, benchmarks-website/AGENTS.md: updated
  to point at ops/ and to reflect the systemd-based deploy.
- server/src/{lib.rs, app.rs, main.rs, admin.rs}: module map, route
  table, env-var list, and admin module doc all updated.

The previous v3 docker artifacts are removed:
- benchmarks-website/ec2-init.txt: replaced by ops/README.md.
- benchmarks-website/server/Dockerfile: v3 isn't containerised any more.
- benchmarks-website/server/scripts/backup.sh: replaced by ops/backup.sh.

The v2 React/Vite stack is untouched. docker-compose.yml is left in
place; its v3 service entry is now orphaned but harmless and the v2
service is unaffected.

server/tests/admin.rs (9 tests):
- SQL round-trip (JSON + ASCII table format)
- Read-only allow-list (DELETE/UPDATE/DROP/INSERT/CREATE/ATTACH → 403)
- Allowed verbs (PRAGMA/SHOW/DESCRIBE/EXPLAIN/WITH)
- Bearer enforcement: missing/wrong/ingest-token-on-admin → 401
- Admin router not mounted when ADMIN_BEARER_TOKEN unset → 404
- Snapshot creates the export dir + schema.sql
- Snapshot of an existing dir → 409
- ts validation: empty / "../oops" / "with space" / 65 chars → 400

cargo test -p vortex-bench-server passes (admin: 9, rest: 18 pre-existing).
cargo clippy -p vortex-bench-server --all-targets --all-features clean.

Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Author: claude

benchmarks-website/AGENTS.md

@@ -14,10 +14,12 @@ covers Rust style, test layout, commit conventions.
 Until the cutover PR lands, the top-level v2 files
 (`server.js`, `src/`, `index.html`, `vite.config.js`, `package.json`,
 `package-lock.json`, `public/`, the top-level `Dockerfile`,
-`docker-compose.yml`, `ec2-init.txt`) and the `benchmarks-website` service
-in `docker-compose.yml` and the `publish-benchmarks-website.yml` workflow
+`docker-compose.yml`) and the `publish-benchmarks-website.yml` workflow
 are production. Don't edit them as part of unrelated work.
 
+The v3 deploy lives entirely under `server/`, `migrate/`, and `ops/`.
+The operator runbook is [`ops/README.md`](ops/README.md).
+
 ## v3 specifics
 
 - **Wire shapes are a coordinated change.** [`server/src/records.rs`](server/src/records.rs),

benchmarks-website/README.md

@@ -9,16 +9,16 @@ The website behind `bench.vortex.dev`. The directory currently houses **two
 implementations side by side**, run together until the v3 cutover lands:
 
 - **v2** (top-level files: `server.js`, `src/`, `index.html`, `vite.config.js`,
-  `package.json`, `Dockerfile`, `docker-compose.yml`, `ec2-init.txt`,
-  `public/`). The Node + React stack that has shipped to production for the
-  life of the site. Built and published by
+  `package.json`, `Dockerfile`, `docker-compose.yml`, `public/`). The Node +
+  React stack that has shipped to production for the life of the site. Built
+  and published by
   [`.github/workflows/publish-benchmarks-website.yml`](../.github/workflows/publish-benchmarks-website.yml).
-- **v3** (`server/` + `migrate/`). A single Rust binary —
+- **v3** (`server/` + `migrate/` + `ops/`). A single Rust binary —
   [`vortex-bench-server`](server/) — that owns a DuckDB file on local disk,
   serves the API, and renders the HTML. Compiles all static assets
   (`chart.umd.js`, `chart-init.js`, `style.css`) into the binary so deploys
-  are one file plus a database. Container image at
-  `ghcr.io/vortex-data/vortex/vortex-bench-server:latest`.
+  are one file plus a database. Built directly on the EC2 host by
+  [`ops/deploy.sh`](ops/deploy.sh) — see [`ops/README.md`](ops/README.md).
   [`migrate/`](migrate/) is a one-shot tool that loads v2's S3 dataset into a
   v3 DuckDB; it is throwaway and goes away after cutover.
 
@@ -79,12 +79,22 @@ npm run dev
 
 ## Deployment
 
-`docker-compose.yml` runs both stacks side by side: v2 on `:80` and v3 on
-`:3001`. `watchtower` polls GHCR every 60s so a fresh image push lands
-automatically. v3 reads `INGEST_BEARER_TOKEN` from
-`/etc/vortex-bench/secrets.env`, persists DuckDB to
-`/opt/benchmarks-website/data/bench.duckdb`, and binds `0.0.0.0:3000` so the
-container's `:3001` host port forwards through.
+v3 runs as a systemd service on a single EC2 host. The full operator
+runbook (first-time install, day-to-day, failure modes) is in
+[`ops/README.md`](ops/README.md). Summary:
+
+- A `vortex-bench-deploy.timer` polls `origin/develop` every 60s. If commits
+  in the range touch `benchmarks-website/server/`, `benchmarks-website/migrate/`,
+  `Cargo.toml`, or `Cargo.lock`, it builds and atomically swaps the binary,
+  then verifies `/health`. Otherwise it fast-forwards the working tree and
+  exits silently.
+- A `vortex-bench-backup.timer` fires hourly: it asks the server to
+  `EXPORT DATABASE` via the bearer-gated `/api/admin/snapshot` endpoint,
+  `tar czf`s the CSV output into `<UTC ts>.tar.gz`, uploads it to
+  `s3://vortex-benchmark-results-database/v3-backups/`, and deletes the local
+  copies.
+- For ad-hoc reads against the live DB, `ops/inspect.sh` calls a
+  bearer-gated `/api/admin/sql` endpoint — no server stop required.
 
 The v3 server is throwaway-friendly: every request runs against the local
 DuckDB file, and a fresh boot reapplies the schema DDL idempotently. The
@@ -96,13 +106,12 @@ re-running `vortex-bench-migrate run --output ...` is safe.
 The work to flip `bench.vortex.dev` from v2 to v3 is tracked outside this
 repo. The relevant code-side bits:
 
-- v3 runs alongside v2 on the same EC2 host today (v2 on `:80`, v3 on
-  `:3001`) and is fed by CI's dual-write `--gh-json-v3` path.
+- v3 runs alongside v2 on the same EC2 host today and is fed by CI's
+  dual-write `--gh-json-v3` path.
 - v2 keeps shipping unchanged until DNS flips. **Do not touch the top-level
   v2 files unless you are doing the cleanup PR opened post-flip.**
 - The v2 cleanup PR removes everything top-level under `benchmarks-website/`
   that belongs to v2 (`server.js`, `src/`, `index.html`, `vite.config.js`,
   `package.json`, `package-lock.json`, `public/`, the top-level `Dockerfile`,
-  `docker-compose.yml`, `ec2-init.txt`, and the
-  `publish-benchmarks-website.yml` workflow). The v3 tree under `server/` and
-  `migrate/` is untouched.
+  `docker-compose.yml`, and the `publish-benchmarks-website.yml` workflow).
+  The v3 tree under `server/`, `migrate/`, and `ops/` is untouched.