From 784fa103acf6f5630b43bf14a3f71919e5ad30a9 Mon Sep 17 00:00:00 2001
From: Jordan Nanos <jordan.nanos@gmail.com>
Date: Sat, 18 Jul 2026 10:44:33 -0400
Subject: [PATCH 1/4] docs(commands): add /ci-queue-times command for
 per-cluster CI queue latency
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Adds a .claude/commands shortcut with the three gh-API queries for checking
CI queue times across all GPU clusters: live queued jobs with wait-so-far,
historical per-cluster queue latency (started_at - created_at, re-run
artifacts filtered), and per-pool runner availability. Includes the
cluster:* label normalization and SLURM second-queue caveats.

中文：新增 /ci-queue-times 命令，用于查看所有 GPU 集群的 CI 排队时间。包含三个基于
gh API 的查询：当前正在排队的任务及已等待时长、按集群统计的历史排队延迟
（started_at − created_at，已过滤重跑导致的异常值），以及按 runner 池统计的
可用性。命令中已处理 cluster:* 标签归一化，并注明 SLURM 二级队列的注意事项。

No PR opened per request; branch only.
---
 .claude/commands/ci-queue-times.md | 85 ++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100644 .claude/commands/ci-queue-times.md

diff --git a/.claude/commands/ci-queue-times.md b/.claude/commands/ci-queue-times.md
new file mode 100644
index 0000000000..c188be0749
--- /dev/null
+++ b/.claude/commands/ci-queue-times.md
@@ -0,0 +1,85 @@
+---
+description: Check CI queue times across all GPU clusters — live queued jobs with wait so far, historical per-cluster queue latency, and per-pool runner availability
+argument-hint: [history-run-limit]   # optional, default 150 recent runs
+---
+
+GitHub has no queue-metrics API for self-hosted runners, so queue time is derived
+from the Actions API: **queue time per job = `started_at − created_at`** (time spent
+waiting for a free runner), and the cluster is the job's runner label. The pool →
+runner-name mapping lives in `configs/runners.yaml`.
+
+Run all three sections and report them as compact tables. `$ARGUMENTS` (optional)
+overrides the history window in Step 2 (default 150 recent runs).
+
+## Step 1 — Live queue: who is waiting right now, and for how long
+
+A run can be `in_progress` while its matrix jobs are still `queued` waiting for
+runners (the sweep fan-out), so scan both statuses:
+
+```bash
+{ gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=50" --jq '.workflow_runs[].id'
+  gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=50" --jq '.workflow_runs[].id'
+} | sort -u | while read -r RUN; do
+  gh api "repos/SemiAnalysisAI/InferenceX/actions/runs/$RUN/jobs?per_page=100" --paginate \
+    --jq '.jobs[] | select(.status=="queued") | {labels: [.labels[] | sub("^cluster:";"") | select(test("^(self-hosted|Linux|X64|ARM64|slurm)$|_\\d+$") | not)], name, created_at, run_id}'
+done | jq -s -r 'sort_by(.created_at) | .[]
+  | [((now - (.created_at|fromdateiso8601))/60 | floor | tostring) + " min",
+     (.labels | join(",")), (.name[0:60]), (.run_id|tostring)] | @tsv' | column -t -s$'\t'
+```
+
+## Step 2 — Historical queue latency per cluster
+
+Uses the last N runs (default 150; `$ARGUMENTS` overrides). **Filter `.q >= 0` is
+mandatory** — re-run jobs keep `started_at` from the first attempt and produce
+negative values. For a wider window raise the limit or add
+`--created ">YYYY-MM-DD"` to `gh run list`.
+
+```bash
+LIMIT="${ARGUMENTS:-150}"
+gh run list --repo SemiAnalysisAI/InferenceX --limit "$LIMIT" --json databaseId --jq '.[].databaseId' | while read -r RUN; do
+  gh api "repos/SemiAnalysisAI/InferenceX/actions/runs/$RUN/jobs?per_page=100" --paginate \
+    --jq '.jobs[] | select(.started_at != null)
+      | {c: ([.labels[] | select(test("^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x)")) | sub("^cluster:";"") | sub("_\\d+$";"")] | first // empty),
+         q: (((.started_at|fromdateiso8601) - (.created_at|fromdateiso8601))/60)}
+      | select(.c != "" and .q >= 0)'
+done | jq -s -r 'group_by(.c)
+  | map({cluster: .[0].c, n: length, avg: (map(.q)|add/length),
+         p50: (map(.q)|sort|.[(length*0.5|floor)]), max: (map(.q)|max)})
+  | sort_by(-.p50) | .[]
+  | [.cluster, (.n|tostring), ((.avg*10|round)/10|tostring), ((.p50*10|round)/10|tostring), ((.max|round)|tostring)] | @tsv' \
+  | (printf "cluster\tjobs\tavg_min\tp50_min\tmax_min\n"; cat) | column -t -s$'\t'
+```
+
+Note: the sample only covers clusters that actually ran within the window — a
+missing cluster means no recent jobs, not zero queue.
+
+## Step 3 — Runner availability per pool (the "why" behind queue times)
+
+```bash
+gh api "repos/SemiAnalysisAI/InferenceX/actions/runners?per_page=100" --paginate \
+  --jq '.runners[] | . as $r | [.labels[].name | select(test("^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x)")) | sub("^cluster:";"") | sub("_\\d+$";"")] | unique | .[] | {c: ., status: $r.status, busy: $r.busy}' \
+| jq -s -r 'group_by(.c) | map({c: .[0].c, total: length,
+    online: ([.[]|select(.status=="online")]|length),
+    busy: ([.[]|select(.busy==true)]|length)}) | sort_by(.c) | .[]
+  | [.c, (.total|tostring), (.online|tostring), (.busy|tostring)] | @tsv' \
+| (printf "pool\ttotal_runners\tonline\tbusy\n"; cat) | column -t -s$'\t'
+```
+
+A pool with `online == busy` (or many runners `offline`) fully explains long queues
+for jobs targeting it.
+
+## Caveats
+
+- **Label shapes:** single-node jobs carry pool labels (`b200`, `mi355x`);
+  multi-node jobs carry `cluster:*` labels (`cluster:gb200-nv`); runners also carry
+  unique per-runner labels (`h100-dgxc-slurm_00`). The `sub()` calls normalize all
+  of these — do not drop the `cluster:` prefix handling or multi-node jobs silently
+  vanish from the results.
+- **Re-run artifacts:** always keep the `.q >= 0` filter (see Step 2).
+- **SLURM second queue:** for SLURM-backed pools (`*-dgxc`, `*-nv`), once the
+  GitHub job starts there is a second queue inside SLURM that GitHub-side numbers
+  do not capture. Check it on the cluster with `squeue -u <runner-user>` (see
+  `.claude/skills/debug-runs/SKILL.md`).
+- Queue time measured this way excludes time spent behind the sweep canary gate:
+  matrix jobs are only *created* after the gate passes, so their `created_at` marks
+  genuine runner-wait start.

From 93a63c5384d45318551f522d8a6ea16689636071 Mon Sep 17 00:00:00 2001
From: Jordan Nanos <jordan.nanos@gmail.com>
Date: Sat, 18 Jul 2026 11:09:17 -0400
Subject: [PATCH 2/4] docs(commands): rework /ci-queue-times to a live snapshot
 of active + queued jobs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Drop the historical per-cluster latency analysis in favor of a single
live-snapshot pass: per-pool active/queued summary, active jobs with
runtime-so-far, and queued jobs with wait-so-far. Adds an optional pool
filter regex argument, a total_count coverage check, and a caveat for the
0-active-but-queued case (offline/stuck/org-scoped-busy runners).

中文：将 /ci-queue-times 命令改为实时快照模式，聚焦当前正在运行的任务和排队队列，
移除按集群统计的历史排队延迟分析。新流程单次扫描即可输出：按池统计的
活跃/排队任务数、活跃任务及已运行时长、排队任务及已等待时长。新增可选的
池过滤正则参数、total_count 覆盖度校验，以及“活跃为 0 但排队堆积”场景
（runner 离线、状态卡死或被组织内其他仓库占用）的排查说明。
---
 .claude/commands/ci-queue-times.md | 97 +++++++++++++++---------------
 1 file changed, 50 insertions(+), 47 deletions(-)

diff --git a/.claude/commands/ci-queue-times.md b/.claude/commands/ci-queue-times.md
index c188be0749..bcf1aeb7e8 100644
--- a/.claude/commands/ci-queue-times.md
+++ b/.claude/commands/ci-queue-times.md
@@ -1,72 +1,70 @@
 ---
-description: Check CI queue times across all GPU clusters — live queued jobs with wait so far, historical per-cluster queue latency, and per-pool runner availability
-argument-hint: [history-run-limit]   # optional, default 150 recent runs
+description: Show CI cluster pressure right now — active jobs per GPU pool, the live queue with wait-so-far, and a per-pool active/queued summary
+argument-hint: [pool-filter-regex]   # optional, e.g. "h200" or "b200|mi355x" to restrict output
 ---
 
-GitHub has no queue-metrics API for self-hosted runners, so queue time is derived
-from the Actions API: **queue time per job = `started_at − created_at`** (time spent
-waiting for a free runner), and the cluster is the job's runner label. The pool →
-runner-name mapping lives in `configs/runners.yaml`.
+GitHub has no queue-metrics API for self-hosted runners, so this derives cluster
+pressure from the Actions API: a job's **wait so far = now − `created_at`** (queued
+jobs) and **runtime so far = now − `started_at`** (active jobs). The cluster is the
+job's runner label; the pool → runner-name mapping lives in `configs/runners.yaml`.
 
-Run all three sections and report them as compact tables. `$ARGUMENTS` (optional)
-overrides the history window in Step 2 (default 150 recent runs).
+`$ARGUMENTS` (optional) is a regex restricting which pools are shown in Steps 2–4
+(e.g. `h200`). Run all steps and report the tables as-is.
 
-## Step 1 — Live queue: who is waiting right now, and for how long
+## Step 1 — Take one snapshot of all GPU jobs (active + queued)
 
 A run can be `in_progress` while its matrix jobs are still `queued` waiting for
-runners (the sweep fan-out), so scan both statuses:
+runners (the sweep fan-out), so scan both statuses in a single pass:
 
 ```bash
-{ gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=50" --jq '.workflow_runs[].id'
-  gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=50" --jq '.workflow_runs[].id'
+SNAP=$(mktemp /tmp/ci_snapshot.XXXXXX.ndjson)
+{ gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=100" --jq '.workflow_runs[].id'
+  gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=100" --jq '.workflow_runs[].id'
 } | sort -u | while read -r RUN; do
   gh api "repos/SemiAnalysisAI/InferenceX/actions/runs/$RUN/jobs?per_page=100" --paginate \
-    --jq '.jobs[] | select(.status=="queued") | {labels: [.labels[] | sub("^cluster:";"") | select(test("^(self-hosted|Linux|X64|ARM64|slurm)$|_\\d+$") | not)], name, created_at, run_id}'
-done | jq -s -r 'sort_by(.created_at) | .[]
-  | [((now - (.created_at|fromdateiso8601))/60 | floor | tostring) + " min",
-     (.labels | join(",")), (.name[0:60]), (.run_id|tostring)] | @tsv' | column -t -s$'\t'
+    --jq '.jobs[] | select(.status=="in_progress" or .status=="queued")
+      | {c: ([.labels[] | select(test("^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x)")) | sub("^cluster:";"") | sub("_\\d+$";"")] | first // "other"),
+         status, name, created_at, started_at, run_id}
+      | select(.c != "other")'
+done > "$SNAP"
 ```
 
-## Step 2 — Historical queue latency per cluster
+Sanity-check coverage: compare against
+`gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=1" --jq .total_count`
+(and the same for `queued`) — if either exceeds 100, paginate the runs list too.
 
-Uses the last N runs (default 150; `$ARGUMENTS` overrides). **Filter `.q >= 0` is
-mandatory** — re-run jobs keep `started_at` from the first attempt and produce
-negative values. For a wider window raise the limit or add
-`--created ">YYYY-MM-DD"` to `gh run list`.
+## Step 2 — Per-pool summary (active vs queued)
 
 ```bash
-LIMIT="${ARGUMENTS:-150}"
-gh run list --repo SemiAnalysisAI/InferenceX --limit "$LIMIT" --json databaseId --jq '.[].databaseId' | while read -r RUN; do
-  gh api "repos/SemiAnalysisAI/InferenceX/actions/runs/$RUN/jobs?per_page=100" --paginate \
-    --jq '.jobs[] | select(.started_at != null)
-      | {c: ([.labels[] | select(test("^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x)")) | sub("^cluster:";"") | sub("_\\d+$";"")] | first // empty),
-         q: (((.started_at|fromdateiso8601) - (.created_at|fromdateiso8601))/60)}
-      | select(.c != "" and .q >= 0)'
-done | jq -s -r 'group_by(.c)
-  | map({cluster: .[0].c, n: length, avg: (map(.q)|add/length),
-         p50: (map(.q)|sort|.[(length*0.5|floor)]), max: (map(.q)|max)})
-  | sort_by(-.p50) | .[]
-  | [.cluster, (.n|tostring), ((.avg*10|round)/10|tostring), ((.p50*10|round)/10|tostring), ((.max|round)|tostring)] | @tsv' \
-  | (printf "cluster\tjobs\tavg_min\tp50_min\tmax_min\n"; cat) | column -t -s$'\t'
+jq -s -r 'group_by(.c) | map({c: .[0].c,
+    active: ([.[] | select(.status=="in_progress")] | length),
+    queued: ([.[] | select(.status=="queued")] | length)})
+  | sort_by(-.queued, -.active) | .[] | [.c, (.active|tostring), (.queued|tostring)] | @tsv' "$SNAP" \
+  | (printf "pool\tactive\tqueued\n"; cat) | column -t -s$'\t'
 ```
 
-Note: the sample only covers clusters that actually ran within the window — a
-missing cluster means no recent jobs, not zero queue.
+A pool with **0 active but a growing queue** means its runners are offline, stuck
+"busy" (cancelled-job artifact), or busy with another repo's jobs (runners may be
+org-scoped) — investigate the fleet, not the workflow.
+
+## Step 3 — Active jobs (what is running, and for how long)
+
+```bash
+jq -s -r '[.[] | select(.status=="in_progress")] | sort_by(.c, .started_at) | .[]
+  | [.c, (((now - (.started_at|fromdateiso8601))/60 | floor | tostring) + " min"), (.name[0:58]), (.run_id|tostring)] | @tsv' "$SNAP" \
+  | column -t -s$'\t'
+```
 
-## Step 3 — Runner availability per pool (the "why" behind queue times)
+## Step 4 — Queued jobs (what is waiting, and for how long)
 
 ```bash
-gh api "repos/SemiAnalysisAI/InferenceX/actions/runners?per_page=100" --paginate \
-  --jq '.runners[] | . as $r | [.labels[].name | select(test("^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x)")) | sub("^cluster:";"") | sub("_\\d+$";"")] | unique | .[] | {c: ., status: $r.status, busy: $r.busy}' \
-| jq -s -r 'group_by(.c) | map({c: .[0].c, total: length,
-    online: ([.[]|select(.status=="online")]|length),
-    busy: ([.[]|select(.busy==true)]|length)}) | sort_by(.c) | .[]
-  | [.c, (.total|tostring), (.online|tostring), (.busy|tostring)] | @tsv' \
-| (printf "pool\ttotal_runners\tonline\tbusy\n"; cat) | column -t -s$'\t'
+jq -s -r '[.[] | select(.status=="queued")] | sort_by(.c, .created_at) | .[]
+  | [.c, (((now - (.created_at|fromdateiso8601))/60 | floor | tostring) + " min"), (.name[0:58]), (.run_id|tostring)] | @tsv' "$SNAP" \
+  | column -t -s$'\t'
 ```
 
-A pool with `online == busy` (or many runners `offline`) fully explains long queues
-for jobs targeting it.
+To apply `$ARGUMENTS`, pipe Steps 2–4 through `grep -E "<pattern>"` (keep the
+header line of Step 2).
 
 ## Caveats
 
@@ -75,7 +73,9 @@ for jobs targeting it.
   unique per-runner labels (`h100-dgxc-slurm_00`). The `sub()` calls normalize all
   of these — do not drop the `cluster:` prefix handling or multi-node jobs silently
   vanish from the results.
-- **Re-run artifacts:** always keep the `.q >= 0` filter (see Step 2).
+- **Re-run artifacts:** re-run jobs can report `started_at` from an earlier
+  attempt, inflating runtime in Step 3. Cross-check an outlier against the job's
+  page before believing it.
 - **SLURM second queue:** for SLURM-backed pools (`*-dgxc`, `*-nv`), once the
   GitHub job starts there is a second queue inside SLURM that GitHub-side numbers
   do not capture. Check it on the cluster with `squeue -u <runner-user>` (see
@@ -83,3 +83,6 @@ for jobs targeting it.
 - Queue time measured this way excludes time spent behind the sweep canary gate:
   matrix jobs are only *created* after the gate passes, so their `created_at` marks
   genuine runner-wait start.
+- Runner capacity per pool (online/busy/offline) is a separate question — use
+  `gh api repos/SemiAnalysisAI/InferenceX/actions/runners?per_page=100 --paginate`
+  and bucket labels the same way as Step 1 if needed.

From 63ed2ac5492060ffcd9c9be7e639f4c53daf53ff Mon Sep 17 00:00:00 2001
From: Jordan Nanos <jordan.nanos@gmail.com>
Date: Sat, 18 Jul 2026 11:16:47 -0400
Subject: [PATCH 3/4] docs(commands): add runners online/busy columns to
 /ci-queue-times summary
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Join the runners API into the Step 2 summary so it shows runners_online,
runners_busy, jobs_active, jobs_queued per pool. Documents the one-job-per-
runner-agent invariant and the diagnostic patterns it enables (saturated,
pool-down, stuck/org-scoped-busy runners, snapshot skew), plus the
pool-label overlap note. All-zero rows are filtered as noise.

中文：为 /ci-queue-times 的汇总表新增 runner 维度列：将 runners API 的数据并入
第二步汇总，按池展示 runners_online、runners_busy、jobs_active、jobs_queued。
注明“每个 runner agent 同一时刻只运行一个任务”的基本约束，以及由此得出的
诊断模式（池饱和、池宕机、runner 状态卡死或被组织内其他仓库占用、快照误差），
并说明池标签重叠会导致按行求和重复计数。全零行作为噪音过滤。
---
 .claude/commands/ci-queue-times.md | 58 ++++++++++++++++++++++--------
 1 file changed, 44 insertions(+), 14 deletions(-)

diff --git a/.claude/commands/ci-queue-times.md b/.claude/commands/ci-queue-times.md
index bcf1aeb7e8..6ff3613844 100644
--- a/.claude/commands/ci-queue-times.md
+++ b/.claude/commands/ci-queue-times.md
@@ -1,5 +1,5 @@
 ---
-description: Show CI cluster pressure right now — active jobs per GPU pool, the live queue with wait-so-far, and a per-pool active/queued summary
+description: Show CI cluster pressure right now — per-pool runners online/busy vs jobs active/queued, plus the live queue with wait-so-far
 argument-hint: [pool-filter-regex]   # optional, e.g. "h200" or "b200|mi355x" to restrict output
 ---
 
@@ -11,13 +11,21 @@ job's runner label; the pool → runner-name mapping lives in `configs/runners.y
 `$ARGUMENTS` (optional) is a regex restricting which pools are shown in Steps 2–4
 (e.g. `h200`). Run all steps and report the tables as-is.
 
-## Step 1 — Take one snapshot of all GPU jobs (active + queued)
+**Key fact for interpreting the numbers:** a self-hosted runner agent executes
+**exactly one job at a time**, so per pool `jobs_active` should always be ≤
+`runners_busy`. Parallelism comes from multiple registered agents per pool
+(`h100-dgxc-slurm_00` … `_19`). A multi-node SLURM job still occupies exactly one
+agent (the orchestrator) — the nodes it allocates inside SLURM are invisible here.
+
+## Step 1 — Take one snapshot of jobs (active + queued) and runners
 
 A run can be `in_progress` while its matrix jobs are still `queued` waiting for
 runners (the sweep fan-out), so scan both statuses in a single pass:
 
 ```bash
 SNAP=$(mktemp /tmp/ci_snapshot.XXXXXX.ndjson)
+RUNNERS=$(mktemp /tmp/ci_runners.XXXXXX.ndjson)
+
 { gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=100" --jq '.workflow_runs[].id'
   gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=100" --jq '.workflow_runs[].id'
 } | sort -u | while read -r RUN; do
@@ -27,25 +35,48 @@ SNAP=$(mktemp /tmp/ci_snapshot.XXXXXX.ndjson)
          status, name, created_at, started_at, run_id}
       | select(.c != "other")'
 done > "$SNAP"
+
+gh api "repos/SemiAnalysisAI/InferenceX/actions/runners?per_page=100" --paginate \
+  --jq '.runners[] | . as $r | [.labels[].name | select(test("^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x)")) | sub("^cluster:";"") | sub("_\\d+$";"")] | unique | .[] | {c: ., status: $r.status, busy: $r.busy}' \
+  > "$RUNNERS"
 ```
 
 Sanity-check coverage: compare against
 `gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=1" --jq .total_count`
 (and the same for `queued`) — if either exceeds 100, paginate the runs list too.
 
-## Step 2 — Per-pool summary (active vs queued)
+## Step 2 — Per-pool summary: runners online/busy vs jobs active/queued
 
 ```bash
-jq -s -r 'group_by(.c) | map({c: .[0].c,
-    active: ([.[] | select(.status=="in_progress")] | length),
-    queued: ([.[] | select(.status=="queued")] | length)})
-  | sort_by(-.queued, -.active) | .[] | [.c, (.active|tostring), (.queued|tostring)] | @tsv' "$SNAP" \
-  | (printf "pool\tactive\tqueued\n"; cat) | column -t -s$'\t'
+jq -n -r --slurpfile jobs "$SNAP" --slurpfile runners "$RUNNERS" '
+  ($jobs | group_by(.c) | map({key: .[0].c, value: {
+      active: ([.[] | select(.status=="in_progress")] | length),
+      queued: ([.[] | select(.status=="queued")] | length)}}) | from_entries) as $j
+  | ($runners | group_by(.c) | map({key: .[0].c, value: {
+      online: ([.[] | select(.status=="online")] | length),
+      busy:   ([.[] | select(.busy == true)] | length)}}) | from_entries) as $r
+  | [([$j, $r] | map(keys) | add | unique | .[]) as $c
+      | {c: $c, on: ($r[$c].online // 0), busy: ($r[$c].busy // 0),
+         active: ($j[$c].active // 0), queued: ($j[$c].queued // 0)}]
+  | map(select(.on + .busy + .active + .queued > 0))
+  | sort_by(-.queued, -.active)
+  | .[] | [.c, (.on|tostring), (.busy|tostring), (.active|tostring), (.queued|tostring)] | @tsv' \
+  | (printf "pool\trunners_online\trunners_busy\tjobs_active\tjobs_queued\n"; cat) | column -t -s$'\t'
 ```
 
-A pool with **0 active but a growing queue** means its runners are offline, stuck
-"busy" (cancelled-job artifact), or busy with another repo's jobs (runners may be
-org-scoped) — investigate the fleet, not the workflow.
+How to read it (one job per runner agent, so these patterns are diagnostic):
+
+| pattern | meaning |
+|---|---|
+| `busy == active`, `queued == 0` | healthy, pool has spare capacity if `busy < online` |
+| `busy == online`, `queued > 0` | pool saturated — jobs waiting for a free runner (normal queue) |
+| `online == 0`, `queued > 0` | **pool is down** — no live runners; investigate the fleet, not the workflow |
+| `busy > active` | runners stuck "busy" (cancelled-job artifact) or busy with another repo's jobs (runners may be org-scoped) |
+| `active > busy` | impossible in steady state — treat as snapshot skew between the two API calls |
+
+Note a physical runner counts toward **every** pool label it carries (e.g. the
+same machine appears in `h200`, `h200-dgxc`, and `h200-dgxc-slurm`), so summing
+rows over-counts physical machines.
 
 ## Step 3 — Active jobs (what is running, and for how long)
 
@@ -83,6 +114,5 @@ header line of Step 2).
 - Queue time measured this way excludes time spent behind the sweep canary gate:
   matrix jobs are only *created* after the gate passes, so their `created_at` marks
   genuine runner-wait start.
-- Runner capacity per pool (online/busy/offline) is a separate question — use
-  `gh api repos/SemiAnalysisAI/InferenceX/actions/runners?per_page=100 --paginate`
-  and bucket labels the same way as Step 1 if needed.
+- Snapshot skew: jobs and runners are fetched a few seconds apart; a job that
+  starts/ends in between can briefly violate `active <= busy`.

From a355cddf6c94ff87a2ae009ddd5af096fb328a59 Mon Sep 17 00:00:00 2001
From: Jordan Nanos <jordan.nanos@gmail.com>
Date: Sat, 18 Jul 2026 11:31:47 -0400
Subject: [PATCH 4/4] docs(commands): drive pool membership from
 configs/runners.yaml, paginate run lists
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Address review feedback on PR #2270:

- Add --paginate to the runs-listing calls (previously silently capped at
  the first page, dropping the oldest/longest-waiting runs in busy periods).
- Bucket runners by configs/runners.yaml pool membership (matched by runner
  name) instead of fanning out over every matching label: one runner no
  longer counts toward 3-4 pool rows, and bogus rows derived from per-runner
  name labels (e.g. h100-dgxc-slurm) can no longer appear. This also surfaces
  every configured pool (mi300x-tw, mi325x-tw, h100-multinode, *-disagg, ...).
- Document that gh api --jq does not pass jq --arg, the demand-vs-capacity
  bucketing split, and the shared-membership reading of busy > active.

中文：处理 PR #2270 的评审意见：

- 为列出工作流运行的请求补上 --paginate（此前只取第一页，繁忙时段会静默
  丢弃最老、等待最久的运行）。
- runner 侧的池归属改为以 configs/runners.yaml 的成员配置为准（按 runner
  名称匹配），不再按标签扇出：同一 runner 不会被重复计入 3-4 个池行，
  由 runner 名称标签衍生的伪池行（如 h100-dgxc-slurm）也不会再出现。
  同时所有已配置的池（mi300x-tw、mi325x-tw、h100-multinode、*-disagg 等）
  都会显示。
- 补充说明：gh api --jq 不支持透传 jq --arg、需求侧（任务标签）与容量侧
  （runners.yaml 成员）的分桶口径，以及共享成员池下 busy > active 的解读。
---
 .claude/commands/ci-queue-times.md | 78 ++++++++++++++++++------------
 1 file changed, 46 insertions(+), 32 deletions(-)

diff --git a/.claude/commands/ci-queue-times.md b/.claude/commands/ci-queue-times.md
index 6ff3613844..38967b4b26 100644
--- a/.claude/commands/ci-queue-times.md
+++ b/.claude/commands/ci-queue-times.md
@@ -5,29 +5,43 @@ argument-hint: [pool-filter-regex]   # optional, e.g. "h200" or "b200|mi355x" to
 
 GitHub has no queue-metrics API for self-hosted runners, so this derives cluster
 pressure from the Actions API: a job's **wait so far = now − `created_at`** (queued
-jobs) and **runtime so far = now − `started_at`** (active jobs). The cluster is the
-job's runner label; the pool → runner-name mapping lives in `configs/runners.yaml`.
+jobs) and **runtime so far = now − `started_at`** (active jobs). Pool membership
+comes from the repo's own `configs/runners.yaml` (the authoritative pool →
+runner-name mapping), so run this **from the repo root**.
 
 `$ARGUMENTS` (optional) is a regex restricting which pools are shown in Steps 2–4
 (e.g. `h200`). Run all steps and report the tables as-is.
 
-**Key fact for interpreting the numbers:** a self-hosted runner agent executes
-**exactly one job at a time**, so per pool `jobs_active` should always be ≤
-`runners_busy`. Parallelism comes from multiple registered agents per pool
-(`h100-dgxc-slurm_00` … `_19`). A multi-node SLURM job still occupies exactly one
-agent (the orchestrator) — the nodes it allocates inside SLURM are invisible here.
+**Key facts for interpreting the numbers:**
+
+- A self-hosted runner agent executes **exactly one job at a time**, so per pool
+  `jobs_active` should always be ≤ `runners_busy`. Parallelism comes from multiple
+  registered agents per pool (`h100-dgxc-slurm_00` … `_19`).
+- Pools overlap **by design** in `configs/runners.yaml` (e.g. `h100-multinode` ⊂
+  `h100`, and the same `b200-dgxc_*` machines serve `b200`, `b200-dsv4`, and
+  `b200-dgxc` jobs), so one physical runner can count toward several pool rows.
+- A multi-node SLURM job still occupies exactly one agent (the orchestrator) — the
+  nodes it allocates inside SLURM are invisible here.
 
 ## Step 1 — Take one snapshot of jobs (active + queued) and runners
 
 A run can be `in_progress` while its matrix jobs are still `queued` waiting for
-runners (the sweep fan-out), so scan both statuses in a single pass:
+runners (the sweep fan-out), so scan both statuses. Every list call uses
+`--paginate` — without it the runs lists are silently capped at the first page.
 
 ```bash
 SNAP=$(mktemp /tmp/ci_snapshot.XXXXXX.ndjson)
 RUNNERS=$(mktemp /tmp/ci_runners.XXXXXX.ndjson)
+POOLS=$(mktemp /tmp/ci_pools.XXXXXX.tsv)
+
+# Pool → runner-name membership, from the repo's own config (bare pool keys and
+# cluster:* keys are normalized to the same bucket: cluster:b200-dgxc → b200-dgxc).
+awk '/^  [A-Za-z0-9:-]+:$/ { k=$1; sub(":$","",k); sub("^cluster:","",k); pool=k; next }
+     /^    - / { print pool "\t" $2 }' configs/runners.yaml > "$POOLS"
 
-{ gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=100" --jq '.workflow_runs[].id'
-  gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=100" --jq '.workflow_runs[].id'
+# Jobs, bucketed by the pool label they requested (demand side).
+{ gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=queued&per_page=100" --paginate --jq '.workflow_runs[].id'
+  gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=100" --paginate --jq '.workflow_runs[].id'
 } | sort -u | while read -r RUN; do
   gh api "repos/SemiAnalysisAI/InferenceX/actions/runs/$RUN/jobs?per_page=100" --paginate \
     --jq '.jobs[] | select(.status=="in_progress" or .status=="queued")
@@ -36,25 +50,26 @@ RUNNERS=$(mktemp /tmp/ci_runners.XXXXXX.ndjson)
       | select(.c != "other")'
 done > "$SNAP"
 
+# Runners by name (capacity side) — names match configs/runners.yaml entries.
 gh api "repos/SemiAnalysisAI/InferenceX/actions/runners?per_page=100" --paginate \
-  --jq '.runners[] | . as $r | [.labels[].name | select(test("^(cluster:)?(b200|b300|h100|h200|gb200|gb300|mi300x|mi325x|mi355x)")) | sub("^cluster:";"") | sub("_\\d+$";"")] | unique | .[] | {c: ., status: $r.status, busy: $r.busy}' \
-  > "$RUNNERS"
+  --jq '.runners[] | {name, status, busy}' > "$RUNNERS"
 ```
 
-Sanity-check coverage: compare against
-`gh api "repos/SemiAnalysisAI/InferenceX/actions/runs?status=in_progress&per_page=1" --jq .total_count`
-(and the same for `queued`) — if either exceeds 100, paginate the runs list too.
-
 ## Step 2 — Per-pool summary: runners online/busy vs jobs active/queued
 
+Joins demand (job labels) against capacity (`configs/runners.yaml` membership):
+
 ```bash
-jq -n -r --slurpfile jobs "$SNAP" --slurpfile runners "$RUNNERS" '
-  ($jobs | group_by(.c) | map({key: .[0].c, value: {
+jq -n -r --slurpfile jobs "$SNAP" --slurpfile runners "$RUNNERS" --rawfile pools "$POOLS" '
+  ($pools | split("\n") | map(select(contains("\t")) | split("\t"))
+    | group_by(.[1]) | map({key: .[0][1], value: [.[].[0]]}) | from_entries) as $name2pools
+  | ($jobs | group_by(.c) | map({key: .[0].c, value: {
       active: ([.[] | select(.status=="in_progress")] | length),
       queued: ([.[] | select(.status=="queued")] | length)}}) | from_entries) as $j
-  | ($runners | group_by(.c) | map({key: .[0].c, value: {
-      online: ([.[] | select(.status=="online")] | length),
-      busy:   ([.[] | select(.busy == true)] | length)}}) | from_entries) as $r
+  | ([ $runners[] as $r | ($name2pools[$r.name] // [])[] as $p | {c: $p, status: $r.status, busy: $r.busy} ]
+     | group_by(.c) | map({key: .[0].c, value: {
+         online: ([.[] | select(.status=="online")] | length),
+         busy:   ([.[] | select(.busy==true)] | length)}}) | from_entries) as $r
   | [([$j, $r] | map(keys) | add | unique | .[]) as $c
       | {c: $c, on: ($r[$c].online // 0), busy: ($r[$c].busy // 0),
          active: ($j[$c].active // 0), queued: ($j[$c].queued // 0)}]
@@ -71,12 +86,9 @@ How to read it (one job per runner agent, so these patterns are diagnostic):
 | `busy == active`, `queued == 0` | healthy, pool has spare capacity if `busy < online` |
 | `busy == online`, `queued > 0` | pool saturated — jobs waiting for a free runner (normal queue) |
 | `online == 0`, `queued > 0` | **pool is down** — no live runners; investigate the fleet, not the workflow |
-| `busy > active` | runners stuck "busy" (cancelled-job artifact) or busy with another repo's jobs (runners may be org-scoped) |
-| `active > busy` | impossible in steady state — treat as snapshot skew between the two API calls |
-
-Note a physical runner counts toward **every** pool label it carries (e.g. the
-same machine appears in `h200`, `h200-dgxc`, and `h200-dgxc-slurm`), so summing
-rows over-counts physical machines.
+| `busy > active` | the pool's runners are busy serving **another pool's** jobs (shared membership in `configs/runners.yaml`, e.g. `b200` vs `b200-dgxc`), stuck "busy" after a cancellation, or busy with another repo's jobs (runners may be org-scoped) |
+| `active > busy` | impossible in steady state — treat as snapshot skew between the API calls |
+| pool absent entirely | no online runners and no jobs — either idle-and-offline, or (if jobs *do* target it) the pool is not defined in `configs/runners.yaml` |
 
 ## Step 3 — Active jobs (what is running, and for how long)
 
@@ -99,11 +111,13 @@ header line of Step 2).
 
 ## Caveats
 
-- **Label shapes:** single-node jobs carry pool labels (`b200`, `mi355x`);
-  multi-node jobs carry `cluster:*` labels (`cluster:gb200-nv`); runners also carry
-  unique per-runner labels (`h100-dgxc-slurm_00`). The `sub()` calls normalize all
-  of these — do not drop the `cluster:` prefix handling or multi-node jobs silently
-  vanish from the results.
+- **Demand vs capacity bucketing:** jobs are bucketed by the label they *request*
+  (single-node jobs use pool labels like `b200`, multi-node jobs use `cluster:*`
+  labels — the `sub()` normalization merges them). Runners are bucketed strictly by
+  `configs/runners.yaml` membership, so bogus rows from per-runner name labels
+  (e.g. `h100-dgxc-slurm`) cannot appear.
+- **`gh api --jq` does not accept jq `--arg`** — extra arguments are parsed as API
+  parameters. Interpolate values into the `--jq` string via the shell instead.
 - **Re-run artifacts:** re-run jobs can report `started_at` from an earlier
   attempt, inflating runtime in Step 3. Cross-check an outlier against the job's
   page before believing it.