refactor: remove superseded restart-offline-runners.sh, update READMEs

restart-offline-runners.sh is superseded by rebalance-runners.sh, which
handles offline runners as part of its distribution algorithm and also
calls sync_runner_nodes to self-heal stale runner.node files.

Update frontier/README.md: remove restart-offline-runners.sh references,
replace CWD-based discovery language with EXE-based, add rebalance-runners.sh.
Update phoenix/README.md: fix config.sh description (functions now in
runner-lib.sh), fix list-runners.sh description (now parallel sweep),
fix slurm PATH wording (portable grep, not /opt/slurm).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: Spencer Bryngelson

misc/frontier/README.md

@@ -14,35 +14,38 @@ starting it on another — no binary copying needed.
 Each runner directory contains a `runner.node` file recording which login node
 it was last started on. This is used as a fallback hint when restarting offline
 runners. The authoritative source of truth for whether a runner is running (and
-on which node) is CWD-based process discovery — not any PID file.
+on which node) is EXE-based process discovery via `/proc/$pid/exe` — not any
+PID file.
 
-`runner.node` is self-healing: `restart-offline-runners.sh` detects when a
-runner is actually running on a different node than `runner.node` records (e.g.
-after a manual restart) and corrects the file automatically.
+`runner.node` is self-healing: `rebalance-runners.sh` calls `sync_runner_nodes`
+at startup, which sweeps all nodes in parallel and corrects any stale
+`runner.node` files automatically.
 
 Runners occasionally die due to OLCF's firewall/proxy dropping long-lived TCP
-connections to GitHub's broker. The `restart-offline-runners.sh` script handles
-recovery. Login nodes vary in stability — if a runner keeps dying on a
+connections to GitHub's broker. Run `rebalance-runners.sh` to restart and
+redistribute them. Login nodes vary in stability — if a runner keeps dying on a
 particular node, move it to a quieter one (login01 tends to have low load).
 
 ## Quick Reference
 
 ```bash
-# List all runners with GitHub status, node, and memory usage
+# List all runners with GitHub status, node, slurm, and memory usage
 bash list-runners.sh
 
 # Check runner health across all login nodes
 bash check-runners.sh
 
+# Restart offline runners and rebalance across all 11 nodes
+bash rebalance-runners.sh              # dry run
+APPLY=1 bash rebalance-runners.sh      # execute
+APPLY=1 FORCE=1 bash rebalance-runners.sh  # move busy runners too
+
 # Deploy a new runner on a specific node
 bash make-runner.sh 23 login01
 
 # Deploy multiple runners across nodes (e.g. runners 23, 24, 25)
 bash deploy-runners.sh 23 login01 login02 login03
 
-# Restart all offline runners in place
-bash restart-offline-runners.sh
-
 # Move a runner to a different login node
 bash move-runner.sh frontier-1 login01
 
@@ -58,14 +61,14 @@ APPLY=1 bash ../common/rerun-failed.sh
 
 | Script | Purpose |
 |---|---|
-| `config.sh` | Shared configuration, constants, GitHub API helpers, and CWD-based process management functions. Sourced by all other scripts. |
-| `check-runners.sh` | SSH to each login node, show Runner.Listener processes with name, status (idle/BUSY), and RSS memory. |
-| `list-runners.sh` | List all runners with GitHub API status, actual node (from parallel SSH sweep), and RSS memory. Flags stale `runner.node` entries. |
+| `config.sh` | Shared configuration: Frontier constants, `find_runner_dirs()`, and `sync_runner_nodes()`. Sources `../common/runner-lib.sh` for shared functions. |
+| `check-runners.sh` | SSH to each login node, show Runner.Listener processes with name, status (idle/BUSY), slurm PATH, and RSS memory. |
+| `list-runners.sh` | List all runners with GitHub API status, actual node (from parallel SSH sweep), slurm status, and RSS memory. Flags stale `runner.node` entries. |
+| `rebalance-runners.sh` | Sync node locations, compute optimal distribution, and move runners across all 11 login nodes. Handles offline runners. Dry run by default. |
 | `make-runner.sh` | Download runner binary, register with GitHub via API, start on target node. Usage: `make-runner.sh <num> [login-node]` |
 | `move-runner.sh` | Move a runner to a different login node: stop on current node, update `runner.node`, start on target. Usage: `move-runner.sh <runner-name> <target-node>` |
 | `deploy-runners.sh` | Deploy multiple runners across login nodes in parallel. Usage: `deploy-runners.sh <start-num> <node1> [node2 ...]` |
-| `restart-offline-runners.sh` | Query GitHub for offline frontier runners, locate via CWD-based discovery, stop stale processes, then restart in parallel. Self-heals stale `runner.node` files. Prints final status. |
-| `stop-runner.sh` | Locate runner via CWD-based discovery, stop the process, and deregister from GitHub. Usage: `stop-runner.sh <runner-name>` |
+| `stop-runner.sh` | Locate runner via EXE-based discovery, stop the process, and deregister from GitHub. Usage: `stop-runner.sh <runner-name>` |
 | `../common/rerun-failed.sh` | Rerun failed GitHub Actions workflows on open PRs and master. No site-specific code. |
 
 ## Troubleshooting
@@ -77,9 +80,9 @@ bash move-runner.sh frontier-1 login01
 ```
 
 **Multiple runners OFFLINE at once** — Usually a transient OLCF network blip
-to GitHub. Run `restart-offline-runners.sh` to recover all at once.
+to GitHub. Run `rebalance-runners.sh` to recover and redistribute all at once.
 
 **Runner appears offline on GitHub but process is running** — GitHub status can
-lag. `restart-offline-runners.sh` uses CWD-based process discovery first: if a
+lag. `rebalance-runners.sh` uses EXE-based process discovery first: if a
 process is found running, it will stop it before restarting, preventing
 duplicate runner processes.

misc/frontier/restart-offline-runners.sh

@@ -47,9 +47,9 @@ APPLY=1 bash rerun-failed.sh           # execute
 
 | Script | Purpose |
 |---|---|
-| `config.sh` | Shared config: Phoenix constants (`ORG`, `RUNNER_GROUP`, `RUNNER_LABEL`, `NODES`), GitHub API helpers (`gh_list_runners`, `gh_registration_token`), and process management (`start_runner`, `stop_runner`, `find_node`, `find_pids`, `has_slurm`). |
-| `check-runners.sh` | Quick per-node health check. One SSH per node. Shows runner names, idle/BUSY status, slurm PATH, RSS, and total memory. |
-| `list-runners.sh` | Detailed table combining GitHub API status (online/offline/busy) with login-node process info. Slower (one SSH per runner + API call). |
+| `config.sh` | Shared config: Phoenix constants (`ORG`, `RUNNER_GROUP`, `RUNNER_LABEL`, `NODES`, `CGROUP_LIMIT`, `RUNNER_PARENT_DIRS`) and `find_runner_dirs()`. Sources `../common/runner-lib.sh` for shared functions. |
+| `check-runners.sh` | Quick per-node health check. One SSH per node. Shows runner names, idle/BUSY status, slurm PATH, RSS, and total cgroup memory. |
+| `list-runners.sh` | Table combining GitHub API status with live node info from a parallel SSH sweep. Shows slurm status and flags stale `runner.node` entries. |
 | `restart-runner.sh` | Stop and restart one runner on a given node with proper login shell PATH and SSH detachment. |
 | `restart-all.sh` | Restart all runners in place. Skips BUSY runners unless `FORCE=1`. |
 | `rebalance-runners.sh` | Auto-compute optimal distribution and move runners. Prefers idle runners. Also places OFFLINE runners. |
@@ -61,7 +61,7 @@ APPLY=1 bash rerun-failed.sh           # execute
 - **Dry run by default**: `rebalance-runners.sh`, `restart-all.sh`, and
   `rerun-failed.sh` show what they would do unless `APPLY=1` is set.
 - **Busy runner protection**: Scripts skip BUSY runners unless `FORCE=1`.
-- **Slurm PATH verification**: After starting, scripts verify `/opt/slurm` is
+- **Slurm PATH verification**: After starting, scripts verify `slurm` appears
   in the runner's PATH and warn if missing.
 
 ## Configuration