feat(power): multinode measured-power aggregation

[arygupt]

Summary

Extends single-node measured-power aggregation (#1558) to multinode srt-slurm benchmarks. Wires per-node perf_samples_<host>.csv from srt-slurm's PR #35 perfmon through the launcher into process_result.py → aggregate_power.py, which now namespaces local GPU indices per source CSV stem so each node's local indices 0..N-1 don't collapse across nodes.

Backward compatible: aggregate_power() accepts both Path and Iterable[Path]; single-CSV callers (single-node start_gpu_monitor path) are unchanged. csv_path param name preserved.

Pipeline

srt-slurm perfmon (PR #35 on NVIDIA/srt-slurm, layered on
  SemiAnalysisAI/srt-slurm:feat/inferencex-perfmon)
  → perf_samples_<host>.csv in outputs/<job>/logs/ on shared NFS
  → launch_gb300-cw.sh exports GPU_METRICS_CSV_GLOB to $GITHUB_ENV
  → process_result.py expands glob → aggregate_power.run() with list
  → aggregate_power.py emits cluster-wide avg_power_w + joules_per_*_token
  → InferenceX-app ETL auto-captures (no schema change)

Files

• utils/aggregate_power.py — csv_path widened to Path | Iterable[Path]. Per-source GPU-id namespacing only kicks in for 2+ sources so single-node num_gpus is unchanged. CLI adds --csv-glob (mutually exclusive with --csv).
• utils/process_result.py — bridge GPU_METRICS_CSV_GLOB env var. Glob takes precedence over single GPU_METRICS_CSV when both are set.
• runners/launch_gb300-cw.sh — point dynamo-sglang at our srt-slurm fork, append monitoring: block to each recipe post-copy (idempotent), write GPU_METRICS_CSV_GLOB to $GITHUB_ENV after the job.
• utils/test_aggregate_power.py — 8 new multinode cases: per-source namespacing, sub-second clock drift, asymmetric prefill/decode power, missing-CSV silent skip, backward-compat single-path-in-list, Iterable acceptance, E2E with list.
• utils/test_process_result.py — 3 new cases: glob aggregation, precedence over single CSV, empty-match falls through.

Test plan

• 36/36 aggregator tests pass (28 existing + 8 new)
• 28/28 process_result tests pass (25 existing + 3 new)
• nvidia-smi inside sglang container on real gb300-cw emits expected columns (timestamp, gpu, power_w) — verified manually with srun --container-image=...sglang...sqsh nvidia-smi --query-gpu=...
• First E2E multinode sweep produces avg_power_w + joules_per_*_token in the agg JSON (pending perf-changelog.yaml entry + sweep-enabled label)
• num_gpus in agg JSON matches prefill_gpus + decode_gpus from launcher (validates per-source namespacing — without the fix, num_gpus would equal a single node's gpus_per_node)
• Chart at inferencex.semianalysis.com renders the new data via ?unofficialrun=<run_id>

Depends on

SemiAnalysisAI/srt-slurm:feat/inferencex-perfmon (pinned by the launcher). Tracks NVIDIA/srt-slurm PR #35 head; will rebase to upstream main once #35 merges.

---

Note

Medium Risk
Touches benchmark launchers, Slurm job env, and agg JSON semantics (disagg joules fields); failures are mostly best-effort for telemetry but wrong glob or namespacing could publish incorrect power numbers.

Overview
Multinode measured power now flows from per-node GPU CSVs into agg JSONs on both NVIDIA (srt-slurm) and AMD (SGLang/vLLM) paths, with CI/runner fixes so telemetry is not silently dropped.

utils/aggregate_power.py accepts multiple CSVs (and --csv-glob), namespaces GPU indices per source file, parses perf_samples_<role>_w<idx>_<host>.csv for workers[], and adds cluster-wide temp/util/mem plus disagg per-stage fields (joules_per_input_token, stage-specific joules_per_output_token, prefill_avg_power_w / decode_avg_power_w). Bench windows can fall back to srt-slurm date + duration when Unix timestamps are missing.

process_result.py uses GPU_METRICS_CSV_GLOB when set (no stale single-CSV fallback) and passes disagg into aggregation. launch_gb300-cw.sh pins the srt-slurm fork, recursively injects monitoring: into recipe YAMLs via find, and exports the perfmon glob. launch_mi355x-amds.sh collects NFS perfmon CSVs before log cleanup, sets the glob, and hardens SLURM squeue polling.

AMD: start_perf_monitor in benchmark_lib.sh (richer amd-smi -m VRAM, CSV header logging), wired through job.slurm / server_sglang.sh / server_vllm.sh. Workflow pre-cleanup sudo rm -rf benchmark_logs avoids root-owned dirs breaking checkout. Tests and perf-changelog.yaml smoke entries document the new paths.

^{Reviewed by Cursor Bugbot for commit d2cca8a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26534043069
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26534043069

[claude]

🔴 When GPU_METRICS_CSV_GLOB is set but matches no files, _csv_arg stays None and the code falls through to the single-CSV candidate list (GPU_METRICS_CSV, gpu_metrics.csv, /workspace/gpu_metrics.csv) — contradicting the comment at lines 145-148 that the glob 'Takes precedence over the single-CSV fallback'. On a persistent self-hosted runner with a stale /workspace/gpu_metrics.csv from a prior single-node run (or a leaked GPU_METRICS_CSV env var), a multinode run whose perfmon failed on every node would silently patch wrong single-node avg_power_w / joules_per_*_token values into the multinode agg JSON. Fix: when _glob_pattern is truthy, skip the single-CSV fallback regardless of whether the glob matched anything.

Extended reasoning...

The contract violation

The block at utils/process_result.py:142-159 documents the precedence contract clearly:

Takes precedence over the single-CSV fallback — if the launcher set the glob, the run was multinode and there is no single-CSV fallback to make.

But the implementation only honors that contract when the glob actually matches files. On empty match:

_csv_arg = None
_glob_pattern = os.environ.get('GPU_METRICS_CSV_GLOB')
if _glob_pattern:
    _matched = sorted(Path(p) for p in _glob_module.glob(_glob_pattern))
    if _matched:
        _csv_arg = _matched
    else:
        print(..., file=sys.stderr)   # warns but doesn't prevent fallthrough

if _csv_arg is None:                  # still None — falls into single-CSV branch
    _csv_candidates = [
        os.environ.get('GPU_METRICS_CSV'),
        'gpu_metrics.csv',
        '/workspace/gpu_metrics.csv',
    ]
    ...

The else branch just logs; _csv_arg stays None, and the next if _csv_arg is None block consults the single-CSV candidates.

Step-by-step proof on a persistent self-hosted runner

1. Single-node run on gb300-cw_N completes successfully. benchmarks/benchmark_lib.sh exports GPU_METRICS_CSV=/workspace/gpu_metrics.csv (it lives in gpu_metrics.csv in cwd too). The file is left behind because the runner is persistent across jobs.
2. Next job is a multinode dynamo-sglang sweep. runners/launch_gb300-cw.sh (lines 297-318) writes GPU_METRICS_CSV_GLOB=$LOGS_DIR/perf_samples_*.csv to $GITHUB_ENV — but only when perf_csv_count > 0. Suppose perfmon failed to start on every node (srt-slurm PR [NVIDIA] Reduce B200 Runs & add B200 FP4 Docker Script #35 had startup issues, host driver mismatch, etc.) — perf_csv_count would be 0 and the glob env var would not be written. Fine — that path is safe.
3. However, suppose perfmon CSVs were written at the end of the job (so the launcher writes the GLOB), but a downstream cleanup hook between launcher and process_result.py removed them, OR srt-slurm wrote the CSVs to a different path on a subsequent retry, OR a persistent env var (GPU_METRICS_CSV_GLOB from a prior job) leaks in. The glob expansion in process_result.py returns empty.
4. process_result.py enters the else branch on line 155, prints a warning, and falls through. os.environ.get('GPU_METRICS_CSV') from the prior single-node job returns /workspace/gpu_metrics.csv (or gpu_metrics.csv in cwd is still there). Path(p).is_file() is True. _csv_arg = Path('/workspace/gpu_metrics.csv').
5. _aggregate_power_run is called with the stale single-node CSV.

Why the bench-window timestamp filter doesn't always save us

One verifier argued the start_unix <= ts <= end_unix filter at aggregate_power.py:177-178 would reject stale samples. That's true if the window comes from explicit Unix timestamps. But this PR adds two new fallback tiers in _load_bench_window:

• Tier 2: date field parsed as a UTC string (YYYYMMDD-HHMMSS).
• Tier 3: bench_result_path.stat().st_mtime — the bench JSON's own mtime, which is the current run's mtime, used as bench-end with start = end - duration.

The mtime tier is exactly the danger zone: on a persistent runner the bench JSON is freshly written, so its mtime is now. If the stale gpu_metrics.csv was also written recently (within the derived [mtime - duration, mtime] window — possible if the prior single-node run finished a few minutes ago), its samples do fall inside the window. Result: silent wrong avg_power_w and joules_per_*_token patched into the multinode agg JSON, which InferenceX-app's ETL auto-captures into the dashboard.

What the test misses

The accompanying test test_multinode_csv_glob_empty_match_falls_through_silently only asserts the no-stale-file case (asserts 'avg_power_w' not in patched). It does not stage a stale fallback CSV, so it can't catch the precedence violation. test_multinode_csv_glob_takes_precedence_over_single_csv only tests precedence when the glob matches.

Fix

One-line change in the empty-match branch:

if _glob_pattern:
    _matched = sorted(Path(p) for p in _glob_module.glob(_glob_pattern))
    if _matched:
        _csv_arg = _matched
    else:
        _csv_arg = []           # sentinel: glob attempted, fallback forbidden
        print(...)

if not _csv_arg:                # treats [] same as None for the downstream check, but…
    if _glob_pattern:
        pass                    # …skip single-CSV candidates when glob was attempted
    else:
        _csv_candidates = [...]
        _csv_arg = next(...)

Or more cleanly: guard the single-CSV block on not _glob_pattern instead of _csv_arg is None.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26547958720
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26547958720

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26548110246
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26548110246

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26593269421
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26593269421

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26600823211
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26600823211

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26604906530
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26604906530

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26605086756
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26605086756

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26605402511
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26605402511

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26606436666
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26606436666

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26607091549
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=26607091549

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d2cca8a. Configure here.}

[cursor]

Squeue failure stalls job wait

Medium Severity

The new job_alive helper treats any failed squeue as proof the job is still running. After the Slurm job has already finished, repeated controller timeouts leave the launcher in the background poll and log-file wait loops indefinitely instead of proceeding to perfmon collection and result processing.

 

^{Reviewed by Cursor Bugbot for commit d2cca8a. Configure here.}