benchmarks-website/ops: force-rebuild path for "redeploy current branch"

The deploy timer's fast-path exit when `origin/$DEPLOY_BRANCH` hasn't
moved is the right default — but it means there's no way to say "I
want a fresh build of whatever's on the branch I'm tracking right
now, even though origin hasn't moved." Real situations that hit this:

- Flipped DEPLOY_BRANCH and want the new tip within seconds, not on
  the next commit there.
- Edited /etc/vortex-bench.env in a way that needs a rebuild (build
  flags, target paths) rather than just a `systemctl restart`.
- target/ got wedged and you want a clean rebuild from scratch.

deploy.sh now accepts either `FORCE=1` in the environment or a
`.force-rebuild` sentinel file under `$STATE_DIR`. Either path skips
the stamp comparison so the full build → atomic swap → restart →
/health cycle runs against `origin/$DEPLOY_BRANCH`. The sentinel is
consumed on consumption so the very next ordinary timer tick is a
normal no-op again.

ops/force-rebuild.sh is a one-liner wrapper that drops the sentinel
and triggers the deploy service — operator-facing, no env vars or
systemd-override invocations to remember.

ops/README.md grows a "How do I manually restart or redeploy?"
section with three knobs in increasing order of work: restart-only,
deploy-if-moved, and force-rebuild. Drops the earlier "manual cargo
build from a local checkout" path which was over-documented; the
force-rebuild + DEPLOY_BRANCH flip covers the same ground without
encouraging hand-edited build state on the host.

Signed-off-by: Claude <noreply@anthropic.com>

Author: claude

benchmarks-website/ops/README.md

@@ -91,6 +91,7 @@ out-of-tree state — every script and unit lives in
 | [`migrate.sh`](migrate.sh)                 | Manual: stop, snapshot prev DB, run migrate, restart.            |
 | [`backup.sh`](backup.sh)                   | Hourly: trigger `/api/admin/snapshot`, sync to S3, prune local.  |
 | [`inspect.sh`](inspect.sh)                 | Read-only SQL via `/api/admin/sql`, no server stop.              |
+| [`force-rebuild.sh`](force-rebuild.sh)     | Re-run a deploy of `$DEPLOY_BRANCH` even when origin hasn't moved. |
 | [`config/vortex-bench.env.example`](config/vortex-bench.env.example) | Template for `/etc/vortex-bench.env`.       |
 | [`systemd/`](systemd/)                     | Unit files installed into `/etc/systemd/system/`.                |
 
@@ -346,6 +347,51 @@ The timer's next fire (within 60s) will overwrite your manual binary
 with whatever `origin/$DEPLOY_BRANCH` produces, which is usually what
 you want — manual binaries are scratch space, not a long-term state.
 
+### "How do I manually restart or redeploy?"
+
+Three knobs, in increasing order of work done:
+
+**(a) Restart the running binary, no rebuild.** Cheapest restart;
+useful after editing `/etc/vortex-bench.env` or recovering from a hung
+connection. `build_sha` on `/health` will be unchanged afterwards.
+
+```bash
+sudo systemctl restart vortex-bench-server
+journalctl -fu vortex-bench-server               # confirm it came up
+```
+
+**(b) Run a deploy now if origin has moved.** Triggers the same flow
+the 60s timer runs. No-op if `origin/$DEPLOY_BRANCH` hasn't moved
+since the last successful deploy.
+
+```bash
+sudo systemctl start vortex-bench-deploy.service
+journalctl -fu vortex-bench-deploy.service
+```
+
+**(c) Force-rebuild `origin/$DEPLOY_BRANCH` even if origin hasn't
+moved.** Ignores the stamp-file comparison and the path filter and
+runs the full build → atomic swap → restart → `/health` check. Use
+this when you want to redeploy "whatever's on the branch I'm tracking"
+without waiting for a new commit — e.g. after flipping `DEPLOY_BRANCH`
+or recovering from wedged build artefacts:
+
+```bash
+/var/lib/vortex-bench/ops/force-rebuild.sh
+journalctl -fu vortex-bench-deploy.service
+```
+
+Under the hood, `force-rebuild.sh` drops a `.force-rebuild` sentinel
+under `$STATE_DIR` and triggers `vortex-bench-deploy.service`. The
+script consumes the sentinel on its next fire, so the very next
+ordinary timer tick is a normal no-op again.
+
+To test a branch that isn't `$DEPLOY_BRANCH`, edit the env file to
+point `DEPLOY_BRANCH` at it, then call `force-rebuild.sh` (or wait
+60s). The deploy script always builds origin's tip — there is no
+"build whatever I have locally checked out" mode by design. Push to
+a branch first.
+
 ### "A vortex-array PR landed — does the website rebuild?"
 
 No. The path filter ignores anything outside the directories listed

benchmarks-website/ops/deploy.sh

@@ -90,7 +90,26 @@ if ! git fetch --quiet origin "$DEPLOY_BRANCH"; then
 fi
 new_sha="$(git rev-parse "origin/${DEPLOY_BRANCH}")"
 
-if [ "$new_sha" = "$last_sha" ]; then
+# --- Force mode ---
+# A `FORCE=1` env var (or a `.force-rebuild` sentinel under the state
+# dir) bypasses the SHA-unchanged fast path and treats this run as if
+# nothing was ever deployed. Used by `force-rebuild.sh` for the
+# "redeploy current branch right now" path. The sentinel is consumed
+# so the next ordinary timer fire is a no-op again.
+force=0
+if [ "${FORCE:-0}" = "1" ]; then
+    force=1
+fi
+if [ -f "${STATE_DIR}/.force-rebuild" ]; then
+    rm -f "${STATE_DIR}/.force-rebuild"
+    force=1
+fi
+if [ "$force" = "1" ]; then
+    log "force mode: ignoring stamp comparison and path filter"
+    last_sha=""
+fi
+
+if [ "$force" = "0" ] && [ "$new_sha" = "$last_sha" ]; then
     # Common case: nothing new since last fire. Silent on stdout to
     # keep the journal clean.
     exit 0

benchmarks-website/ops/force-rebuild.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright the Vortex contributors
+#
+# Force a rebuild + restart of origin/$DEPLOY_BRANCH right now, even
+# if origin hasn't moved since the last successful deploy. Drops a
+# sentinel that the next deploy.sh fire consumes, then triggers it.
+#
+# Use cases:
+#   - You changed /etc/vortex-bench.env and want a fresh binary build
+#     (e.g. a feature flag baked into config) rather than just a
+#     `systemctl restart` of the existing one.
+#   - You flipped DEPLOY_BRANCH and want the new tip in <60s rather
+#     than waiting for the timer.
+#   - Build artefacts got wedged and you want a clean rebuild.
+#
+# For "build whatever I have locally checked out" rather than fetching
+# origin, edit /etc/vortex-bench.env to point DEPLOY_BRANCH at a
+# branch the local tip is already on, then run this. The deploy
+# script always builds origin's tip — there is no "use local HEAD"
+# mode by design; push to a branch first.
+
+set -euo pipefail
+
+STATE_DIR="${STATE_DIR:-/var/lib/vortex-bench}"
+
+if [ ! -d "$STATE_DIR" ]; then
+    echo "ERROR: ${STATE_DIR} not found — has install.sh run?" >&2
+    exit 2
+fi
+
+# The sentinel file needs to be writable by the user the deploy
+# service runs as. install.sh chowns STATE_DIR to that user, so this
+# works without sudo. If you're running as a different user, sudo.
+if ! touch "${STATE_DIR}/.force-rebuild" 2>/dev/null; then
+    echo "ERROR: cannot write ${STATE_DIR}/.force-rebuild — run as the install user or sudo" >&2
+    exit 2
+fi
+
+echo "[force-rebuild] sentinel dropped; firing deploy service"
+sudo /bin/systemctl start vortex-bench-deploy.service
+echo "[force-rebuild] tail with: journalctl -fu vortex-bench-deploy.service"