Skip to content

Commit c956961

Browse files
Spencer Bryngelsonclaude
Spencer Bryngelson
and
claude
committed
misc/frontier: add list-runners, move-runner; improve restart and make-runner
- list-runners.sh: new script using parallel SSH sweep across all 11 nodes simultaneously instead of serial per-runner discovery; flags stale runner.node entries with a warning in the node column - move-runner.sh: new script to relocate a runner between login nodes with one retry on start failure - restart-offline-runners.sh: add retry logic (sleep 5 + second attempt) and runner.node self-healing (detects and corrects stale node entries when a runner is found on a different node than recorded) - make-runner.sh: replace hardcoded RUNNER_VERSION with dynamic GitHub API lookup falling back to pinned version; print selected version at startup - misc/frontier/README.md: document new scripts, update quick reference and troubleshooting sections, note runner.node self-healing behavior - misc/common/README.md: new file documenting site-agnostic shared scripts Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent e704284 commit c956961

6 files changed

Lines changed: 228 additions & 17 deletions

File tree

‎misc/common/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Common Runner Management Scripts
2+
3+
This directory contains site-agnostic scripts shared between the Phoenix and
4+
Frontier runner management setups. Scripts here have no site-specific logic and
5+
can be invoked directly or via thin site-specific wrappers.
6+
7+
## Scripts
8+
9+
| Script | Purpose |
10+
|---|---|
11+
| `rerun-failed.sh` | Rerun failed GitHub Actions workflows on open non-draft MFC PRs and master. Dry-run by default; set `APPLY=1` to actually trigger reruns. |
12+
13+
## Usage
14+
15+
```bash
16+
# Dry run — show which failed workflows would be rerun
17+
bash misc/common/rerun-failed.sh
18+
19+
# Actually rerun failed workflows
20+
APPLY=1 bash misc/common/rerun-failed.sh
21+
```
22+
23+
## Site wrappers
24+
25+
`misc/phoenix/rerun-failed.sh` is a thin wrapper that delegates to this
26+
script, so both `bash misc/phoenix/rerun-failed.sh` and
27+
`bash misc/common/rerun-failed.sh` invoke the same logic.

‎misc/frontier/README.md‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,10 @@ it was last started on. This is used as a fallback hint when restarting offline
1616
runners. The authoritative source of truth for whether a runner is running (and
1717
on which node) is CWD-based process discovery — not any PID file.
1818

19+
`runner.node` is self-healing: `restart-offline-runners.sh` detects when a
20+
runner is actually running on a different node than `runner.node` records (e.g.
21+
after a manual restart) and corrects the file automatically.
22+
1923
Runners occasionally die due to OLCF's firewall/proxy dropping long-lived TCP
2024
connections to GitHub's broker. The `restart-offline-runners.sh` script handles
2125
recovery. Login nodes vary in stability — if a runner keeps dying on a
@@ -24,6 +28,9 @@ particular node, move it to a quieter one (login01 tends to have low load).
2428
## Quick Reference
2529

2630
```bash
31+
# List all runners with GitHub status, node, and memory usage
32+
bash list-runners.sh
33+
2734
# Check runner health across all login nodes
2835
bash check-runners.sh
2936

@@ -36,6 +43,9 @@ bash deploy-runners.sh 23 login01 login02 login03
3643
# Restart all offline runners in place
3744
bash restart-offline-runners.sh
3845

46+
# Move a runner to a different login node
47+
bash move-runner.sh frontier-1 login01
48+
3949
# Stop and deregister a runner
4050
bash stop-runner.sh frontier-12
4151

@@ -50,21 +60,20 @@ APPLY=1 bash ../common/rerun-failed.sh
5060
|---|---|
5161
| `config.sh` | Shared configuration, constants, GitHub API helpers, and CWD-based process management functions. Sourced by all other scripts. |
5262
| `check-runners.sh` | SSH to each login node, show Runner.Listener processes with name, status (idle/BUSY), and RSS memory. |
63+
| `list-runners.sh` | List all runners with GitHub API status, actual node (from parallel SSH sweep), and RSS memory. Flags stale `runner.node` entries. |
5364
| `make-runner.sh` | Download runner binary, register with GitHub via API, start on target node. Usage: `make-runner.sh <num> [login-node]` |
65+
| `move-runner.sh` | Move a runner to a different login node: stop on current node, update `runner.node`, start on target. Usage: `move-runner.sh <runner-name> <target-node>` |
5466
| `deploy-runners.sh` | Deploy multiple runners across login nodes in parallel. Usage: `deploy-runners.sh <start-num> <node1> [node2 ...]` |
55-
| `restart-offline-runners.sh` | Query GitHub for offline frontier runners, locate via CWD-based discovery, stop stale processes, then restart in parallel. Prints final status. |
67+
| `restart-offline-runners.sh` | Query GitHub for offline frontier runners, locate via CWD-based discovery, stop stale processes, then restart in parallel. Self-heals stale `runner.node` files. Prints final status. |
5668
| `stop-runner.sh` | Locate runner via CWD-based discovery, stop the process, and deregister from GitHub. Usage: `stop-runner.sh <runner-name>` |
5769
| `../common/rerun-failed.sh` | Rerun failed GitHub Actions workflows on open PRs and master. No site-specific code. |
5870

5971
## Troubleshooting
6072

6173
**Runner goes OFFLINE repeatedly on the same node** — That login node may have
62-
process culling or high memory pressure. Move it by stopping the runner and
63-
restarting it on a different node:
74+
process culling or high memory pressure. Move it to a different node:
6475
```bash
65-
bash stop-runner.sh frontier-1
66-
echo "login01" > /lustre/orion/cfd154/proj-shared/runners/frontier-1/runner.node
67-
bash restart-offline-runners.sh
76+
bash move-runner.sh frontier-1 login01
6877
```
6978

7079
**Multiple runners OFFLINE at once** — Usually a transient OLCF network blip

‎misc/frontier/list-runners.sh‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#!/usr/bin/env bash
2+
# List all Frontier runners, combining GitHub API status with login-node process info.
3+
#
4+
# Uses a parallel SSH sweep across all 11 login nodes simultaneously to avoid
5+
# the overhead of serial per-runner node discovery. Each node is queried once;
6+
# results are correlated with GitHub API status and the local runner directories.
7+
#
8+
# Usage: bash list-runners.sh
9+
set -euo pipefail
10+
11+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
12+
source "$SCRIPT_DIR/config.sh"
13+
14+
tmpdir=$(mktemp -d)
15+
trap 'rm -rf "$tmpdir"' EXIT
16+
17+
# --- Parallel SSH sweep across all login nodes ---
18+
# Each node prints lines in the format: RUNNER <node> <dir> <rss_mb>
19+
# The RUNNER sentinel prefix allows stripping MOTD noise with grep.
20+
for node in "${NODES[@]}"; do
21+
ssh $SSH_OPTS "$node" '
22+
for p in $(ps aux | grep Runner.Listener | grep -v grep | awk "{print \$2}"); do
23+
cwd=$(readlink -f /proc/$p/cwd 2>/dev/null || true)
24+
rss=$(ps -p $p -o rss= 2>/dev/null | awk "{printf \"%.0f\", \$1/1024}" || echo 0)
25+
[ -n "$cwd" ] && echo "RUNNER '"$node"' $cwd $rss"
26+
done
27+
' 2>/dev/null > "$tmpdir/$node.out" &
28+
done
29+
30+
wait
31+
32+
# --- Build associative arrays from sweep results ---
33+
declare -A runner_node runner_rss
34+
for node in "${NODES[@]}"; do
35+
while IFS= read -r line; do
36+
# Each line: RUNNER <node> <dir> <rss_mb>
37+
read -r _sentinel sweep_node dir rss <<< "$line"
38+
runner_node["$dir"]="$sweep_node"
39+
runner_rss["$dir"]="$rss"
40+
done < <(grep '^RUNNER ' "$tmpdir/$node.out" 2>/dev/null || true)
41+
done
42+
43+
# --- Fetch GitHub API status ---
44+
declare -A gh_status gh_busy
45+
while read -r _id name status busy; do
46+
gh_status["$name"]="$status"
47+
gh_busy["$name"]="$busy"
48+
done < <(gh_list_runners)
49+
50+
# --- Print table ---
51+
printf "%-25s %-8s %-14s %s\n" "NAME" "GITHUB" "NODE" "RSS"
52+
printf "%s\n" "$(printf '%.0s-' {1..60})"
53+
54+
for dir in $(find_runner_dirs); do
55+
name=$(get_runner_name "$dir")
56+
[ -z "$name" ] && continue
57+
58+
# GitHub status column
59+
api_status="${gh_status[$name]:-unknown}"
60+
api_busy="${gh_busy[$name]:-false}"
61+
if [ "$api_busy" = "true" ]; then
62+
gh_col="BUSY"
63+
else
64+
gh_col="$api_status"
65+
fi
66+
67+
# Node and RSS from parallel sweep
68+
actual_node="${runner_node[$dir]:-}"
69+
rss="${runner_rss[$dir]:-}"
70+
71+
if [ -z "$actual_node" ]; then
72+
printf "%-25s %-8s %-14s %s\n" "$name" "$gh_col" "offline" "—"
73+
continue
74+
fi
75+
76+
# Compare sweep result to recorded runner.node; flag stale entries
77+
node_col="$actual_node"
78+
if [ -f "$dir/runner.node" ]; then
79+
recorded=$(cat "$dir/runner.node")
80+
if [ "$actual_node" != "$recorded" ]; then
81+
node_col="${actual_node} *(stale: ${recorded})"
82+
fi
83+
fi
84+
85+
printf "%-25s %-8s %-14s %sMB\n" "$name" "$gh_col" "$node_col" "$rss"
86+
done

‎misc/frontier/make-runner.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ set -euo pipefail
99
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
1010
source "$SCRIPT_DIR/config.sh"
1111

12-
RUNNER_VERSION="2.332.0"
12+
RUNNER_VERSION="${RUNNER_VERSION:-$(gh_latest_runner_version 2>/dev/null || echo "2.332.0")}"
1313

1414
RUNNER_NUM="${1:?Usage: $0 <runner-number> [login-node]}"
1515
TARGET_NODE="${2:-$(hostname -s)}"
@@ -18,6 +18,7 @@ TARBALL="actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz"
1818
RUNNER_NAME="frontier-${RUNNER_NUM}"
1919
RUNNER_DIR="${SHARED_DIR}/${RUNNER_NAME}"
2020

21+
echo "==> Using runner version ${RUNNER_VERSION}"
2122
echo "==> Setting up runner: ${RUNNER_NAME} on ${TARGET_NODE}"
2223

2324
# --- Download tarball once to shared dir ---

‎misc/frontier/move-runner.sh‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
#!/usr/bin/env bash
2+
# Move a Frontier runner to a different login node.
3+
#
4+
# Stops the runner on its current node, updates runner.node, and starts it on
5+
# the target node. Retries the start once after 5 seconds if the first attempt
6+
# fails.
7+
#
8+
# Usage: move-runner.sh <runner-name> <target-node>
9+
# Example: move-runner.sh frontier-1 login01
10+
set -euo pipefail
11+
12+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
13+
source "$SCRIPT_DIR/config.sh"
14+
15+
RUNNER_NAME="${1:?Usage: $0 <runner-name> <target-node>}"
16+
TARGET_NODE="${2:?Usage: $0 <runner-name> <target-node>}"
17+
18+
RUNNER_DIR="${SHARED_DIR}/${RUNNER_NAME}"
19+
20+
# --- Validate runner directory ---
21+
if [ ! -d "$RUNNER_DIR" ]; then
22+
echo "ERROR: Runner directory not found: ${RUNNER_DIR}" >&2
23+
exit 1
24+
fi
25+
26+
# --- Validate target node is in the known node list ---
27+
valid=0
28+
for node in "${NODES[@]}"; do
29+
[ "$node" = "$TARGET_NODE" ] && valid=1 && break
30+
done
31+
if [ "$valid" -eq 0 ]; then
32+
echo "ERROR: '${TARGET_NODE}' is not a valid Frontier login node." >&2
33+
echo " Valid nodes: ${NODES[*]}" >&2
34+
exit 1
35+
fi
36+
37+
# --- Find current node ---
38+
echo "==> Locating ${RUNNER_NAME}..."
39+
current_node=$(find_node "$RUNNER_DIR")
40+
41+
if [ "$current_node" = "$TARGET_NODE" ]; then
42+
echo "==> ${RUNNER_NAME} is already running on ${TARGET_NODE}. Nothing to do."
43+
exit 0
44+
fi
45+
46+
# --- Stop runner on current node (if running) ---
47+
if [ "$current_node" != "offline" ]; then
48+
echo "==> Stopping ${RUNNER_NAME} on ${current_node}..."
49+
stop_runner "$current_node" "$RUNNER_DIR"
50+
fi
51+
52+
# --- Update runner.node ---
53+
echo "$TARGET_NODE" > "${RUNNER_DIR}/runner.node"
54+
55+
# --- Start runner on target node (with one retry) ---
56+
echo "==> Starting ${RUNNER_NAME} on ${TARGET_NODE}..."
57+
if start_runner "$TARGET_NODE" "$RUNNER_DIR"; then
58+
echo "==> ${RUNNER_NAME} is now running on ${TARGET_NODE}."
59+
else
60+
echo " First start attempt failed. Retrying in 5 seconds..."
61+
sleep 5
62+
if start_runner "$TARGET_NODE" "$RUNNER_DIR"; then
63+
echo "==> ${RUNNER_NAME} is now running on ${TARGET_NODE}."
64+
else
65+
echo "ERROR: ${RUNNER_NAME} failed to start on ${TARGET_NODE} after retry." >&2
66+
exit 1
67+
fi
68+
fi

‎misc/frontier/restart-offline-runners.sh‎

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -36,29 +36,49 @@ restart_one() {
3636
return
3737
fi
3838

39-
# Check if it's actually already running somewhere (GitHub may lag)
39+
# Determine the recorded node from runner.node
40+
local recorded_node target_node
41+
if [ -f "${dir}/runner.node" ]; then
42+
recorded_node=$(cat "${dir}/runner.node")
43+
else
44+
echo "WARN: No runner.node for ${runner_name}, skipping."
45+
return
46+
fi
47+
48+
# Check if the runner is actually already running somewhere (GitHub may lag)
4049
local actual_node
4150
actual_node=$(find_node "$dir")
4251

4352
if [ "$actual_node" != "offline" ]; then
53+
# Self-healing: if the runner is on a different node than runner.node records,
54+
# update runner.node to reflect reality before stopping and restarting.
55+
if [ "$actual_node" != "$recorded_node" ]; then
56+
echo "==> ${runner_name}: found on ${actual_node}, runner.node says ${recorded_node} — updating runner.node."
57+
echo "$actual_node" > "${dir}/runner.node"
58+
recorded_node="$actual_node"
59+
fi
4460
echo "==> ${runner_name} appears running on ${actual_node} (GitHub may lag) — stopping first..."
4561
stop_runner "$actual_node" "$dir"
46-
fi
47-
48-
# Determine target node from runner.node fallback
49-
local target_node
50-
if [ -f "${dir}/runner.node" ]; then
51-
target_node=$(cat "${dir}/runner.node")
62+
# Restart where it was actually running
63+
target_node="$actual_node"
5264
else
53-
echo "WARN: No runner.node for ${runner_name}, skipping."
54-
return
65+
# Runner is truly offline; fall back to the last known node
66+
target_node="$recorded_node"
5567
fi
5668

5769
echo "==> Starting ${runner_name} on ${target_node}..."
5870
if start_runner "$target_node" "$dir"; then
71+
echo "$target_node" > "${dir}/runner.node"
5972
echo " ${runner_name}: started on ${target_node}."
6073
else
61-
echo " ${runner_name}: ERROR — failed to start on ${target_node}." >&2
74+
echo " First start attempt failed. Retrying in 5 seconds..."
75+
sleep 5
76+
if start_runner "$target_node" "$dir"; then
77+
echo "$target_node" > "${dir}/runner.node"
78+
echo " ${runner_name}: started on ${target_node}."
79+
else
80+
echo " ${runner_name}: ERROR — failed to start on ${target_node} after retry." >&2
81+
fi
6282
fi
6383
}
6484

0 commit comments

Comments
 (0)