Skip to content

Commit fd585cc

Browse files
claudeconnortsui20
claude
authored and
connortsui20
committed
benchmarks-website/ops: restart.sh with visible before/after state
`sudo systemctl restart vortex-bench-server` is silent on success, which is awful UX when you actually want to know the new process came up. Adds ops/restart.sh: snapshots the unit's MainPID + start time + symlink target + /health response, runs systemctl restart, waits up to 30s for the new process to answer /health, prints the same snapshot again, and exits 0/1 with a clear RESTART OK/FAILED line so the operator (or a script) doesn't have to guess. Sample output in the runbook section "How do I restart the server and *see* that it actually restarted?". Signed-off-by: Claude <noreply@anthropic.com>
1 parent 8c61cb0 commit fd585cc

2 files changed

Lines changed: 110 additions & 53 deletions

File tree

‎benchmarks-website/ops/README.md‎

Lines changed: 26 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,7 @@ out-of-tree state — every script and unit lives in
9292
| [`backup.sh`](backup.sh) | Hourly: trigger `/api/admin/snapshot`, sync to S3, prune local. |
9393
| [`inspect.sh`](inspect.sh) | Read-only SQL via `/api/admin/sql`, no server stop. |
9494
| [`force-rebuild.sh`](force-rebuild.sh) | Re-run a deploy of `$DEPLOY_BRANCH` even when origin hasn't moved. |
95+
| [`restart.sh`](restart.sh) | Restart the binary in place with visible before/after state. |
9596
| [`config/vortex-bench.env.example`](config/vortex-bench.env.example) | Template for `/etc/vortex-bench.env`. |
9697
| [`systemd/`](systemd/) | Unit files installed into `/etc/systemd/system/`. |
9798

@@ -293,73 +294,45 @@ saw when it produced the running binary. If it disagrees with
293294
`last-deployed-sha`, the running process is stale (e.g. a manual
294295
binary swap, or systemd is still running an older PID).
295296

296-
### "How do I manually rebuild and restart, outside the timer?"
297+
### "How do I manually restart or redeploy?"
297298

298-
You shouldn't normally need this — the deploy timer covers all
299-
ordinary cases — but it's useful when you want to test an unmerged
300-
branch or recover from a stuck timer. Three knobs:
299+
Three knobs, in increasing order of work done:
301300

302301
**(a) Restart the running binary, no rebuild.** Cheapest restart;
303-
useful after editing `/etc/vortex-bench.env` or recovering from a
304-
hung connection.
302+
useful after editing `/etc/vortex-bench.env` or recovering from a hung
303+
connection. `build_sha` on `/health` will be unchanged afterwards.
304+
`sudo systemctl restart vortex-bench-server` is silent on success;
305+
prefer the wrapper, which prints before/after state so you actually
306+
see the new pid:
305307

306308
```bash
307-
sudo systemctl restart vortex-bench-server
308-
journalctl -fu vortex-bench-server # confirm it came up
309-
curl -fsS http://127.0.0.1:3000/health | jq # build_sha unchanged
309+
/var/lib/vortex-bench/ops/restart.sh
310310
```
311311

312-
**(b) Force a deploy of the configured branch right now.** Triggers
313-
exactly the same flow the timer runs, including build, atomic symlink
314-
swap, and `/health` rollback if anything fails.
312+
Sample output:
315313

316-
```bash
317-
sudo systemctl start vortex-bench-deploy.service
318-
journalctl -fu vortex-bench-deploy.service # watch it
319314
```
315+
BEFORE:
316+
pid: 12345
317+
started: Wed 2026-05-08 14:30:01 UTC
318+
binary: /var/lib/vortex-bench/bin/vortex-bench-server.20260508T143000Z
319+
/health: {"status":"ok",...,"build_sha":"abc123..."}
320320
321-
**(c) Manually build a binary from the current working tree and
322-
install it.** Use this to test a branch that isn't `$DEPLOY_BRANCH`
323-
without flipping the env file. The deploy timer will overwrite your
324-
manual binary on the next tick that sees a relevant change, so you
325-
probably want to pause it first:
326-
327-
```bash
328-
. /etc/vortex-bench.env
329-
sudo systemctl stop vortex-bench-deploy.timer # pause auto-deploy
330-
cd "$REPO_DIR"
331-
git fetch origin
332-
git checkout --force --detach origin/<branch> # pin to whatever you want
333-
cargo build --release -p vortex-bench-server
334-
ts=$(date -u +%Y%m%dT%H%M%SZ)
335-
sudo install -m 0755 -o ec2-user -g ec2-user \
336-
target/release/vortex-bench-server \
337-
"/var/lib/vortex-bench/bin/vortex-bench-server.manual-${ts}"
338-
ln -sfnT "/var/lib/vortex-bench/bin/vortex-bench-server.manual-${ts}" \
339-
/var/lib/vortex-bench/bin/vortex-bench-server
340-
sudo systemctl restart vortex-bench-server
341-
curl -fsS http://127.0.0.1:3000/health | jq .build_sha # verify new SHA
342-
# When done testing:
343-
sudo systemctl start vortex-bench-deploy.timer # resume auto-deploy
344-
```
321+
running: sudo systemctl restart vortex-bench-server
345322
346-
The timer's next fire (within 60s) will overwrite your manual binary
347-
with whatever `origin/$DEPLOY_BRANCH` produces, which is usually what
348-
you want — manual binaries are scratch space, not a long-term state.
323+
AFTER:
324+
pid: 12678 ← different pid proves it restarted
325+
started: Wed 2026-05-08 14:35:42 UTC
326+
binary: /var/lib/vortex-bench/bin/vortex-bench-server.20260508T143000Z
327+
/health: {"status":"ok",...,"build_sha":"abc123..."}
349328
350-
### "How do I manually restart or redeploy?"
351-
352-
Three knobs, in increasing order of work done:
353-
354-
**(a) Restart the running binary, no rebuild.** Cheapest restart;
355-
useful after editing `/etc/vortex-bench.env` or recovering from a hung
356-
connection. `build_sha` on `/health` will be unchanged afterwards.
357-
358-
```bash
359-
sudo systemctl restart vortex-bench-server
360-
journalctl -fu vortex-bench-server # confirm it came up
329+
RESTART OK
361330
```
362331

332+
The binary path and `build_sha` don't change (restart doesn't rebuild);
333+
the **pid** and **started** values do. Exit 0 on success, 1 on failure
334+
(with a pointer to `journalctl`) so the script is usable from automation.
335+
363336
**(b) Run a deploy now if origin has moved.** Triggers the same flow
364337
the 60s timer runs. No-op if `origin/$DEPLOY_BRANCH` hasn't moved
365338
since the last successful deploy.

‎benchmarks-website/ops/restart.sh‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: Apache-2.0
3+
# SPDX-FileCopyrightText: Copyright the Vortex contributors
4+
#
5+
# Restart the vortex-bench-server binary in place (no rebuild), with
6+
# visible before/after state so you don't have to wonder if it worked.
7+
#
8+
# Prints, in order:
9+
# - the running pid + start time + binary path the symlink points at,
10+
# before the restart
11+
# - the systemctl exit code
12+
# - the running pid + start time + /health response after the restart
13+
# - "RESTART OK" / "RESTART FAILED" + non-zero exit on failure
14+
#
15+
# Use this instead of `sudo systemctl restart vortex-bench-server`
16+
# when you want any sign that it actually happened.
17+
18+
set -euo pipefail
19+
20+
ENV_FILE="${ENV_FILE:-/etc/vortex-bench.env}"
21+
SERVER_URL="${SERVER_URL:-http://127.0.0.1:3000}"
22+
if [ -f "$ENV_FILE" ]; then
23+
# shellcheck disable=SC1090
24+
. "$ENV_FILE"
25+
fi
26+
SERVER_URL="${SERVER_URL:-http://127.0.0.1:3000}"
27+
28+
snap() {
29+
# Use systemd as the source of truth for the running pid (matches
30+
# whatever it would restart). Falls back to "?" if the unit is
31+
# already inactive.
32+
local pid started binary health
33+
pid="$(systemctl show -p MainPID --value vortex-bench-server 2>/dev/null || echo 0)"
34+
started="$(systemctl show -p ActiveEnterTimestamp --value vortex-bench-server 2>/dev/null || echo '?')"
35+
if [ -L /var/lib/vortex-bench/bin/vortex-bench-server ]; then
36+
binary="$(readlink /var/lib/vortex-bench/bin/vortex-bench-server)"
37+
else
38+
binary="?"
39+
fi
40+
health="$(curl -fsS --max-time 2 "${SERVER_URL}/health" 2>/dev/null \
41+
| (command -v jq >/dev/null && jq -c . || cat) \
42+
|| echo '(unreachable)')"
43+
printf ' pid: %s\n started: %s\n binary: %s\n /health: %s\n' \
44+
"$pid" "$started" "$binary" "$health"
45+
}
46+
47+
echo "BEFORE:"
48+
snap
49+
50+
echo
51+
echo "running: sudo systemctl restart vortex-bench-server"
52+
if ! sudo /bin/systemctl restart vortex-bench-server; then
53+
echo "ERROR: systemctl restart returned non-zero" >&2
54+
echo
55+
echo "AFTER (restart did not complete):"
56+
snap
57+
exit 1
58+
fi
59+
60+
# Wait up to 30s for the new process to take requests.
61+
deadline=$(( $(date +%s) + 30 ))
62+
ok=0
63+
while [ "$(date +%s)" -lt "$deadline" ]; do
64+
if curl -fsS --max-time 2 "${SERVER_URL}/health" >/dev/null 2>&1; then
65+
ok=1
66+
break
67+
fi
68+
sleep 0.5
69+
done
70+
71+
echo
72+
echo "AFTER:"
73+
snap
74+
75+
if [ "$ok" = "1" ]; then
76+
echo
77+
echo "RESTART OK"
78+
exit 0
79+
else
80+
echo
81+
echo "RESTART FAILED — /health did not respond within 30s" >&2
82+
echo "Inspect with: journalctl -u vortex-bench-server --since '1 min ago' --no-pager" >&2
83+
exit 1
84+
fi

0 commit comments

Comments
 (0)