Skip to content

feat(bench): rocm bench load — GPU-saturating load generator for the dashboard#90

Open
michaelroy-amd wants to merge 13 commits into
mainfrom
feat/bench-load-generator
Open

feat(bench): rocm bench load — GPU-saturating load generator for the dashboard#90
michaelroy-amd wants to merge 13 commits into
mainfrom
feat/bench-load-generator

Conversation

@michaelroy-amd

@michaelroy-amd michaelroy-amd commented Jul 8, 2026

Copy link
Copy Markdown
Member

Summary

Makes the bench section of the dashboard's Observe pane functional. The consumer half (CSV tailer → daemon → BenchRing → bench tab) already shipped; this adds the missing producer: rocm bench load, a minimal GPU-saturating load generator, then layers on server-side latency/peaks and a one-click TUI trigger.

The design was kept deliberately low-blast-radius: the generator appends normalized rows to the CSV a running daemon already tails, so the bench tab lights up live with no daemon/protocol/server/runner changes.

rocm bench load (Phase 1)

Fans out concurrent /chat/completions requests at a local OpenAI-compatible endpoint, measures client-side gen_tps/prompt_tps over a wall makespan, and writes one aggregate BenchmarkRow per concurrency cell.

  • tokio JoinSet + owned-permit Semaphore; per-request Result isolation (one refusal never zeroes a cell); makespan pinned before fan-out.
  • Append-only CSV: O_APPEND, one newline-terminated write_all per row (survives the live tailer), header-once, refuses to append on header mismatch (won't corrupt an external agent-bench CSV).
  • Top-level Command::Bench / bench load (the dash command is untouched). http://-only with a clear https:// rejection. Quality fields default to Unknown, so throughput rows never claim pass/fail.

Server-side latency + peaks (Phase 1.5, vLLM-only)

  • Scrapes the endpoint's /metrics before/after each cell for cumulative TTFT/TPOT histogram deltas, and runs a lightweight mid-cell poller (250 ms, 1 s scrape timeout) to capture true peak running/waiting requests.
  • CSV header extended 14→18 (max_running_reqs, max_waiting_reqs, ttft_ms, tpot_ms); shown in the bench detail panel. Non-vLLM endpoints degrade to None and never stall the sweep.

Auto-ramp + one-click TUI trigger (Phase 2)

  • --auto-ramp: doubling sweep 1→128 that stops on a pure should_stop_ramp (throughput plateau <5%, queue saturation guarded on running > 0, or hard cap); rows append incrementally so the tab fills live.
  • Press b on the Observe tab to open a bench-run form that launches rocm bench load via the existing job system, defaulting --out to the daemon-tailed CSV so results stream into the bench tab. Implemented with SpawnJobno protocol change (an in-process Command::RunBench was evaluated and deliberately dropped for buying only ~1 s over the CSV tailer).

Dependencies

Zero new shipped deps. One new dev-dep: wiremock (HTTP stub for tests).

Test plan

  • cargo test -p rocm-dash-collectors -- --test-threads=1 — generator fields/tps, concurrency-cap enforcement, CSV round-trip + header guard, Prometheus delta + non-vLLM degradation, 18-col round-trip, deterministic should_stop_ramp (incl. an at-rest-peaks regression guard)
  • cargo test -p rocm-dash-tui -- --test-threads=1 — overlay invariants, SpawnJob args, https:// rejection, tailed-path default
  • cargo build -p rocm
  • cargo clippy --all-targets -- -D warnings — clean
  • Manual live smoke against a real rocm served model (#[ignore] in-tree)

Notes for reviewers

  • Full design + rationale (and the deferred items) live in plans/rocm-bench-load-generator.md.
  • TTFT/TPOT/peaks are vLLM-only by design (Prometheus /metrics); throughput works on any OpenAI-compatible endpoint.
  • The tool prints a "local smoke-test — not an official ROCm/AMD benchmark" caveat in --help and stdout.

EAI-7361 addendum — populate the dashboard Bench panel

Later commits on this branch fix the observability defect that left the dashboard's Bench panel permanently empty:

  • Root cause: bench_results_dir (the CSV file the daemon's CsvBenchTailer tails) defaulted to None in both the unified rocm-core config (DashboardDaemonConfig) and the standalone rocm-dash-core config (DaemonConfig), so the daemon tailed nothing unless a user hand-edited a config file — even after running rocm bench load. Additionally, rocm bench load with no --out wrote each run to a distinct timestamped file, so a plain CLI run was never on the tailed path.
  • Fix: both configs now default bench_results_dir to <data_dir>/bench/results.csv, resolved through the same AppPaths::discover() data-dir resolution the producer uses (so the ROCM_CLI_DATA_DIR override and host path normalization are honored identically on both sides). rocm bench load's no---out default now appends to that same file. A cross-binary regression test asserts producer default == daemon-tailed default resolve to the same absolute path.
  • Fixed the Bench empty-state copy (it referenced a nonexistent --bench-csv daemon flag; bench_results_dir is config-only) and reworded "directory" → "results file" since it is a single tailed file.

Intentional behavior change (reviewer note)

With a shared default output path (~/.rocm/bench/results.csv), a pre-existing CSV at that path whose header does not match the normalized benchmark schema now yields a loud HeaderMismatch refusal on the next rocm bench load, instead of the old behavior of silently writing a fresh timestamped file. This is intentional: a single canonical, daemon-tailed file is what makes the panel populate, and a schema mismatch should surface rather than fragment results across files. Users who want an ad-hoc separate file can still pass --out <path>.

Note: an earlier commit message on this branch stated the two configs' defaults were "identical / picked up here too" before they actually were — the extra /data/ segment in the rocm-dash-core path made them diverge. That mismatch is corrected by the alignment commit above; the defaults now genuinely resolve to the same path.

Rewrite the bench load-generator plan after an adversarial review pass
against current main. Tightened to ~360 LOC / one new file: writer lives
in collectors (daemon has no csv dep), top-level 'rocm bench load' verb
(Dash untouched), one aggregate row per cell, distinct opt-in output CSV,
http-only with a clear https rejection, and a fixed not-an-official-
benchmark caveat in --help and stdout. Zero shipped deps; one dev-dep
(wiremock). Supersedes the 2026-06-22 draft.

Signed-off-by: Michael Roy <michael.roy@amd.com>
Phase 1 of the bench load generator: the consumer pipeline (CSV tailer ->
daemon -> BenchRing -> bench tab) already shipped; this adds the missing
producer. 'rocm bench load' fans out concurrent /chat/completions requests
at a local OpenAI-compatible endpoint, measures client-side gen_tps and
prompt_tps over a wall makespan, and appends one aggregate BenchmarkRow per
concurrency cell to a distinct CSV a running daemon tails -- so the bench
tab lights up live with zero daemon/protocol/UI changes.

- collectors/bench_load.rs: LoadSpec, run_cell (JoinSet + owned-permit
  Semaphore, hand-serialized POST like lemonade.rs, per-request Result
  isolation, pinned makespan), run_and_append_csv (O_APPEND single
  newline-terminated write_all per row, header-if-empty, refuses on header
  mismatch). thiserror errors; no new shipped deps.
- top-level Command::Bench/Load (Dash untouched); http-only with a clear
  https rejection; distinct opt-in --out (never the shared results dir);
  raw-throughput-vs-agent-workload caveat in --help and stdout.
- tests T1-T5 (wiremock dev-dep) under --test-threads=1; T2 verifies the
  concurrency cap is actually enforced.

Quality fields default to Unknown so throughput rows never claim pass/fail.

Signed-off-by: Michael Roy <michael.roy@amd.com>
…gger (2)

Phase 1.5 and Phase 2 of 'rocm bench load', making it feature-complete.

Phase 1.5 (collectors, vLLM-only, zero new deps): each cell scrapes the
endpoint's /metrics before/after for cumulative TTFT/TPOT histogram deltas,
and a lightweight mid-cell poller (250ms, 1s scrape timeout, AtomicBool
stop + join) tracks true peak running/waiting reqs. CSV header extended
14->18 (max_running_reqs, max_waiting_reqs, ttft_ms, tpot_ms); server-side
gen_tps discarded in favour of the client-measured value. Non-vLLM
endpoints degrade to None, never stalling the sweep. Latency shows in the
bench detail panel.

Phase 2:
- auto-ramp (--auto-ramp): doubling sweep 1..128 that stops on a pure
  should_stop_ramp (plateau <5% gain, or queue saturation guarded on
  running>0, or hard cap); rows append incrementally so the tab fills live.
- TUI trigger: press 'b' on Observe to open a bench-run form that spawns
  'rocm bench load' via the existing job system, defaulting --out to the
  daemon-tailed CSV so rows stream into the bench tab. No protocol/server/
  runner changes (Command::RunBench deliberately not built).

Tests: Prometheus delta + non-vLLM degradation + 18-col round-trip + header
guard; deterministic should_stop_ramp cases incl. an at-rest-peaks
regression guard; overlay invariants + SpawnJob args + https rejection.
All under --test-threads=1; clippy --all-targets -D warnings clean.

Signed-off-by: Michael Roy <michael.roy@amd.com>
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Fixed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Fixed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Fixed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Fixed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Fixed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Dismissed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Dismissed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Dismissed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Fixed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Fixed
@michaelroy-amd

Copy link
Copy Markdown
Member Author

CodeQL "uncontrolled data in path expression" — assessed and dismissed

All 18 rust/path-injection alerts on this PR have been reviewed and dismissed as not exploitable:

  • 5 production hits (append_one_row / run_and_append_csv) — dismissed false positive. The flagged path is the operator's own --out flag (or a trusted config default) used to write the output CSV. It is invoked only from the interactive rocm bench load command, running with the user's own privileges — no trust boundary is crossed. The daemon/server never invoke this code path (Command::RunBench was intentionally not built), so there is no network- or IPC-reachable route. Writing the results file wherever the user chooses is the intended behavior of a benchmarking CLI; restricting it would be user-hostile with no security benefit.
  • 13 test hits — dismissed used in tests. These are tempdir() paths constructed inside #[cfg(test)] unit tests, not reachable in production.

Each alert carries this rationale in its dismissal note. Happy to revisit if a future change routes an untrusted/remote value into the output path.

…pulates

bench_results_dir (the file the daemon's CsvBenchTailer tails for
normalized benchmark rows) defaulted to None in both the unified
rocm-core config and the standalone rocm-dash config, so the Bench
panel stayed empty even after running rocm bench load, unless a user
hand-edited a config file to point at a CSV.

Separately, rocm bench load with no --out wrote each run to a distinct
self-labeled ~/.rocm/data/bench/rocm-bench-<timestamp>.csv file rather
than the shared tailed path, so a plain CLI invocation was never
visible to the daemon even when bench_results_dir was set.

- Default bench_results_dir to <data_dir>/bench/results.csv in both
  DashboardDaemonConfig (rocm-core) and DaemonConfig (rocm-dash-core).
- Default rocm bench load's --out (when omitted) to that same file so
  a plain CLI run appends to what the daemon already tails.
- Fix the Bench tab's empty-state copy, which referenced a
  nonexistent --bench-csv daemon flag (bench_results_dir is
  config-only; there is no such CLI flag on any binary).

Relates to EAI-7361

Signed-off-by: Michael Roy <michael.roy@amd.com>
…dir override

Adversarial-review follow-up for EAI-7361:

- Fix cross-binary path mismatch: rocm-dash-core's default resolved to
  $HOME/.rocm/data/bench/results.csv (extra /data/), but rocm-core's
  default_data_dir() is $HOME/.rocm itself, so the producer writes to
  $HOME/.rocm/bench/results.csv. Remove the stray /data/ segment so both
  configs resolve to the same absolute path the producer uses.
- Honor ROCM_CLI_DATA_DIR on the daemon side: rocm-core's default now
  resolves through the same AppPaths::discover() the producer uses (not
  bare default_data_dir(), which ignores the override), and
  rocm-dash-core's default honors the override too. Previously, with the
  override set, a plain `rocm bench load` wrote a file the daemon never
  tailed.
- Add a cross-binary regression test asserting the producer default
  (default_bench_csv_path) and the daemon-tailed default
  (DashboardDaemonConfig::bench_results_dir) resolve to the same path,
  plus an override-precedence test in rocm-dash-core.
- Reword the Bench empty-state copy: "bench directory" -> "bench results
  file" (bench_results_dir is a single tailed file, not a directory).
- Fix the --out help text's default path (drop the stray /data/).

Relates to EAI-7361

Signed-off-by: Michael Roy <michael.roy@amd.com>
CI follow-up for EAI-7361:

- Regenerate MANIFEST.md for the dependency additions on this branch
  (wiremock dev-dep tree: assert-json-diff, deadpool, num_cpus, etc.).
  Verified with `cargo xtask manifest --check`.
- Harden CsvBenchTailer::drain against the CodeQL rust/path-injection
  alert at the File::open sink: the tailed path is derived from config /
  $ROCM_CLI_DATA_DIR, so route it through validated_read_path(), which
  canonicalizes the path (resolving symlinks and `..`) and rejects
  traversal before opening. The bench results file is intentionally
  user-configurable to any location, so canonicalization — not a
  fixed-root containment check — is the correct barrier. Canonicalization
  fails on an absent file exactly as the previous bare File::open did, so
  the daemon tick loop keeps tolerating a not-yet-created CSV.

Relates to EAI-7361

Signed-off-by: Michael Roy <michael.roy@amd.com>

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, well-tested PR overall: clean copyright/SPDX headers, inline tests, wiremock correctly scoped as a dev-dep, and the CSV append/header-guard design is careful. The cross-binary default-path test is a nice touch. A few issues worth addressing before merge, ranked. Line numbers are on the PR head.

🔴 Correctness

1. --concurrency 0 hangs the process forevercrates/rocm-dash-collectors/src/bench_load.rs:126
Semaphore::new(concurrency as usize) with concurrency == 0 has zero permits, so every spawned task blocks on sem.acquire_owned().await (line 184) and no permit is ever released. js.join_next() never returns → run_cell hangs with no output and no timeout (the 5-min timeout is on the HTTP client, which is never reached). clap accepts 0 (and --concurrency 1,0,8) since there's no value_parser range. Reject non-positive concurrency at parse time, or skip/clamp 0 cells.

2. Unbounded "x".repeat(isl) allocation → OOMbench_load.rs:188
"x".repeat(input_len as usize) is driven directly by the --isl u32 with no upper bound, allocated once per in-flight request. A fat-fingered --isl 2000000000 at concurrency 8 attempts ~16 GB simultaneously and aborts the process instead of returning a clean argument error. Bound --isl (and realistically --osl, --requests) to a sane maximum.

3. Auto-ramp saturation check compares independently-sampled peaksbench_load.rs:443
should_stop_ramp stops when max_waiting_reqs >= max_running_reqs, but the poller (lines 150–163) tracks the two peaks as independent per-field maxima across the whole cell, not from one sample. A transient early queue spike (waiting peaks at 5 during dispatch) plus a later steady state (running peaks at 4) yields 5 >= 4 → premature stop, reporting saturation that never held at any single instant. Capture running+waiting from the same scrape to compare them.

4. A fully-failed cell permanently disables the plateau stopbench_load.rs:474
In run_auto_ramp, if one cell has every request fail (n_success == 0gen_tps == None), prev_gen_tps = row.gen_tps sets it back to None; thereafter the plateau branch (Some(prev), Some(cur)) never matches, so the ramp keeps doubling concurrency up to 128, hammering an already-degraded endpoint. Only overwrite prev_gen_tps when the new value is Some.

🟠 Config / behavior change

5. Daemon now logs bench tailer drain failed every tick on a fresh installcrates/rocm-core/src/lib.rs (default_bench_results_dir) + crates/rocm-dash-collectors/src/bench_tail.rs
Flipping the default from None to Some(<data_dir>/bench/results.csv) means runner_options (apps/rocm/src/dash.rs:46) always builds a CsvBenchTailer, and the tick loop calls drain() every iteration (runner.rs:572). AppPaths::ensure() does not create bench/, so until the first rocm bench load the file is absent, canonicalize returns NotFound, and runner.rs:589 emits warn!("bench tailer drain failed") on every tick. Previously this was silent (no tailer). Suggest treating a not-yet-existing tailed file as an empty drain (return Ok(vec![]) on NotFound) rather than a warning.

6. The two default_bench_results_dir impls genuinely diverge despite the parity claimcrates/rocm-dash-core/src/config.rs:195
rocm-core's default goes through AppPaths::discover(), which — when ROCM_CLI_DATA_DIR is unset — picks up a config-driven managed root (configured_managed_root_from_config, e.g. setup.therock_venv) and the OS project-dirs fallback when $HOME is unset. rocm-dash-core's bench_results_dir_from_env only consults $ROCM_CLI_DATA_DIR/$HOME. So for a managed-root user (on Linux), rocm bench load writes <managed_root>/bench/results.csv but the standalone rocm-dash daemon tails $HOME/.rocm/bench/results.csv — the exact empty-panel bug this PR sets out to fix, reintroduced for the standalone daemon. The unified rocm binary path is consistent; the divergence is confined to standalone rocm-dash. Worth either sharing the resolution or documenting the limitation honestly (the current doc comment claims they "must resolve to the same absolute path").

🟡 Cleanup / altitude

7. Dead traversal guardbench_tail.rs:76
The .components().any(|c| matches!(c, Component::ParentDir)) check runs after std::fs::canonicalize, which already resolves and removes all ... The rejection branch is unreachable, so the "path-injection barrier" the doc comment advertises never fires — the only real protection is that canonicalize resolves (not blocks) the path. Either drop the misleading check/comment or enforce containment under a known root.

8. Redundant header-mismatch pre-checkbench_load.rs:391
run_and_append_csv re-implements the header check (lines 391–403) that append_one_row already performs per row (349–363); run_auto_ramp omits it and works fine. ~13 duplicated lines that must stay byte-identical. Drop the outer copy.

9. Two reqwest::Clients rebuilt per cellbench_load.rs:120
run_cell builds post_client + metrics_client on every call; auto-ramp calls it up to 8× → 16 client builds, discarding connection pools between cells. Build them once and pass them in.

Minor nits (non-blocking): metrics_url hand-rolls scheme/host parsing that rocm_dash_tui::llm / vllm_prom already do; the 250 ms poller 404-storms non-vLLM endpoints for the whole cell even though peaks stay None; and the field name bench_results_dir is a single file, forcing "despite the name" caveats in three places (a rename to bench_results_csv would delete all three).

📋 Repo hygiene — please remove plans/

10. plans/rocm-bench-load-generator.md (269 lines) should not be committed. It's self-described scratch/planning material (Owner: TBD, Supersedes the 2026-06-22 draft, "Phase 1 only") that goes stale immediately. The repo already .gitignores equivalent agent artifacts (active-implementation-log.md, current-user-asks.md, /.rocm-work/, /.claude), and AGENTS.md directs docs to README.md / docs/ / skills/plans/ is not a sanctioned location. There is no existing rule that explicitly forbids it, so this is a hygiene request plus a proposal:

  • Delete plans/ from this PR (move the design content into the PR description or docs/ if worth keeping).
  • Add plans/ to .gitignore.
  • Add a rule to AGENTS.md, e.g.:

    Do not commit planning/scratch docs. Agent- or design-planning notes (e.g. plans/, implementation logs) must stay out of the repo — they go stale and bloat the tree. Durable design docs belong in docs/.

Validate load dimensions, keep queue saturation samples coherent, preserve ramp throughput across failed cells, and avoid warnings before the results CSV exists. Reuse HTTP clients across cells and remove transient planning material.

Signed-off-by: Michael Roy <michael.roy@amd.com>
Comment thread crates/rocm-dash-collectors/src/bench_tail.rs Fixed
Keep the CodeQL path-flow barrier while treating a missing results file as an empty drain.

Signed-off-by: Michael Roy <michael.roy@amd.com>
Signed-off-by: Michael Roy <michael.roy@amd.com>
@michaelroy-amd michaelroy-amd enabled auto-merge July 14, 2026 02:37
…ator

Signed-off-by: Michael Roy <michael.roy@amd.com>
@volen-silo

Copy link
Copy Markdown
Collaborator

🔴 Automated review · pr-review-watcher · 689a07c

Summary

Adds rocm bench load (a concurrency-sweep GPU load generator), vLLM-only server-side latency/peaks, a one-click TUI trigger, and (EAI-7361) defaults the daemon-tailed bench CSV so the dashboard's Bench panel populates. Verdict: Needs work — one blocking durable-config regression; the load generator, TUI wiring, CSV append/header-guard, and path-alignment logic are otherwise sound and well-tested. Verified: read all 18 changed files + collaborators (AppPaths/config serde, state StartJob, vllm_prom); confirmed producer (dash.rs::default_bench_csv_path) and unified-daemon (rocm-core::default_bench_results_dir) defaults both go through AppPaths::discover() so they resolve identically (incl. managed-root); confirmed the bit-packed peak poller, TTFT/TPOT delta math, and header-mismatch guard are correct; confirmed concurrency=0 is unreachable via the clap-validated CLI path. Did not run cargo build/cargo test (no toolchain in review env) — relied on source-level verification. Blocking: 1 · Non-blocking: 6.

🚫 Blocking (must fix before merge)

crates/rocm-core/src/lib.rs:3941 (default_bench_results_dir + the DashboardDaemonConfig.bench_results_dir field, ~3965) — a machine-specific absolute path is baked into the default and then persisted into config.json, silently reintroducing the exact bug EAI-7361 fixes. The field now defaults to Some(<data_dir>/bench/results.csv) and is serialized with skip_serializing_if = "Option::is_none" — which only skips None, so a Some(path) is always written. RocmCliConfig::save() (line 4133, to_vec_pretty(self)) is called from many mutation sites (therock, providers, automations, main). Flow: an existing pre-field config.json loads with the key absent → #[serde(default)] fills in the current machine's absolute path → the next save() pins that concrete path into config.json. Because serde default only fires on a missing key, the pinned path never re-resolves afterward. If $HOME/ROCM_CLI_DATA_DIR/the managed-root then differ (managed-root users, containers, config ported between machines), the daemon tails a stale path while rocm bench load writes elsewhere → Bench panel empty again, with no error. Fix: do not persist a derived per-machine path — resolve the default at read time in the consumer (e.g. in dash.rs::runner_options, fall back to default_bench_csv_path(paths) when the config value is None) and keep the serialized field None, mirroring how the standalone rocm-dash-core config resolves from env rather than storing an absolute path.

Non-blocking

  • apps/rocm/src/dash.rs:391 — explicit --out <path> does not create_dir_all the parent (the default-path branch does), so --out /new/dir/x.csv fails with an opaque os error 2; create the parent in both branches.
  • crates/rocm-dash-collectors/src/bench_load.rs:167run_cell/run_cell_with_clients hang forever on concurrency == 0 (Semaphore::new(0)); not reachable via the clap-validated CLI today, but the fn is pub — guard >= 1 and return BenchLoadError.
  • crates/rocm-dash-collectors/src/bench_load.rs:374 (append_one_row) — three separate opens (metadata → header read → append) are not atomic; two concurrent rocm bench load runs against the shared default path can both write the header (duplicate header parsed as a data row breaks CsvBenchTailer::drain). Open once and gate the header on file.metadata()?.len()==0; add an advisory lock if concurrent runs are supported.
  • crates/rocm-dash-tui/src/ui/bench_run.rs:206try_submit sets *br = None even when StartJob returns empty (a "bench-run" job already running is deduped in state.rs:197), so a re-submit silently closes the form with no job and no message; check fx.is_empty() and keep the form open with an error, like command_screen.rs.
  • crates/rocm-dash-tui/src/ui/bench_run.rs:69 — empty-Out hint claims the CLI default is ~/.rocm/bench/rocm-bench-<ts>.csv, but it is now the fixed <data_dir>/bench/results.csv; the hint is stale/misleading. Also BenchLoadError::HeaderMismatch gives no actionable "pass --out or delete the file" guidance for a user stuck on a pre-existing mismatched results.csv.
  • crates/rocm-dash-collectors/src/bench_load.rs:1087t11_auto_ramp_cap_reaches_128 asserts only non-empty + members-of-RAMP_SEQUENCE; it passes even if the ramp stops at concurrency=1, so it does not test its named claim. Also the cross-binary default-alignment tests silently skip when AppPaths::discover() fails rather than failing.

Signed-off-by: Michael Roy <michael.roy@amd.com>
…ator

Signed-off-by: Michael Roy <michael.roy@amd.com>
Comment thread apps/rocm/src/dash.rs Dismissed
Comment thread apps/rocm/src/dash.rs Dismissed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Dismissed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Dismissed
Comment thread crates/rocm-dash-collectors/src/bench_load.rs Dismissed
@michaelroy-amd

Copy link
Copy Markdown
Member Author

Blocking: derived bench_results_dir was serialized as a machine-specific absolute path.

Addressed in 8905e41: the config now keeps this field unset by default, and runner_options derives <data_dir>/bench/results.csv from the current AppPaths while preserving explicit overrides.

Also addressed the actionable non-blocking findings:

  • create parent directories for explicit and default --out paths;
  • reject run_cell(..., 0) at the public API boundary;
  • serialize cooperating CSV appenders with an advisory file lock so first writers emit one header;
  • keep the bench form open with an error when bench-run is already active;
  • correct the stale output hint and make header mismatch recovery actionable;
  • replace the nondeterministic auto-ramp cap test and remove skip-prone default-path tests.

The branch also includes the latest main at 53a56cc.

Validation:

  • cargo test --workspace --all-targets — 1478 passed, 6 ignored
  • cargo clippy --workspace --all-targets -- -D warnings — passed
  • python3 scripts/smoke_local.py — passed

Signed-off-by: Michael Roy <michael.roy@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants