Commit 5fab56c
authored
test: add BDD E2E tests with cucumber-rs (#69)
* test: add BDD E2E tests with cucumber-rs
Behavioral end-to-end suite exercising the real `rocm` binary through its CLI
and HTTP endpoints (black-box, no crate imports). Gherkin .feature files backed
by Rust step functions, across chat, model serving, examine, and runtime setup.
The suite gates CI: the runner exits non-zero on any failed step, parsing
error, or hook error. Three selections via `cargo xtask e2e`: a blocking
mock-tier job (`not @gpu and not @expected-failure`), a non-blocking known-bugs
job (`@expected-failure`), and a non-blocking GPU job (`@gpu`) on self-hosted
hardware. Known-bug scenarios carry a bare `@expected-failure` tag plus an
`@expected-failure-EAI-NNNN` tag for traceability.
Mock-based scenarios plant a managed-service record (plain JSON matching the
on-disk schema) so `rocm services list` and the local chat provider discover
the mock, exercising the real binary rather than the test's own helper. The
HTML report is rendered with maud (compile-checked, auto-escaped markup); its
stats/status/date logic is unit-tested.
Adds a `cargo xtask e2e` subcommand as the cross-platform entry point.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): suppress false-positive CodeQL path-injection in temp dir setup
CodeQL flags std::env::temp_dir() as a tainted path source in the E2E World
setup. This is test-only code and every path segment joined onto the OS temp
dir is program-controlled (pid, an atomic counter, fixed names), so there is no
attacker-controlled traversal. Add inline codeql[rust/path-injection]
suppressions with justification.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): use tempfile::TempDir for per-scenario isolation
Replaces the manual std::env::temp_dir() + create_dir_all + Drop cleanup with a
tempfile::TempDir. Each World gets a unique isolated config/data/cache root that
auto-removes on drop. This also resolves the CodeQL rust/path-injection findings:
the OS temp-dir lookup now happens inside the tempfile crate rather than our
source, so there is no tainted env value flowing into a path in this code.
(Inline codeql[...] suppressions were ineffective under this repo's CodeQL
setup; breaking the taint flow is the reliable fix.)
tempfile was already in the dependency tree; now a direct dev-facing dep.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* fix(xtask): use if-let over single-arm match in e2e binary resolution
Clippy's single_match fires on the ROCM_CLI_BINARY match; rewrite as
if-let/else. No behavior change.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): make the suite CI-correct on all runners
Three fixes so the E2E suite passes the checks it only started exercising once
clippy went green (the heavy jobs were previously skipped behind the lint gate):
- Exclude the cucumber `e2e` integration test from the default test set
(`test = false`). It uses a custom harness (`harness = false`) and is meant to
run only via `cargo xtask e2e` on Linux. Nextest could not `--list` the custom
harness (breaking "Test (affected crates)"), and the Windows job's
`cargo test --all-targets` ran every scenario unfiltered with no mock, doing
real installs/GPU probes (10/17 failures). `xtask e2e`'s explicit `--test e2e`
still runs it.
- Add xfail inversion for the known-bugs job. `cargo xtask e2e --expect-failures`
sets E2E_EXPECT_FAILURES; the harness then treats an @expected-failure scenario
failing as the expected (green) outcome and fails only on XPASS (a known bug
now passes -> drop its tag), an untagged failure, or a parse/hook error. The
job drops continue-on-error so it genuinely gates instead of showing a
permanently-red, ignored check. Adds report::evaluate_xfail + unit tests.
- Harden the cross-engine expansion check. It used unwrap_or_default(), so two
engines that both errored before printing "resolved model:" compared "" == ""
and passed vacuously; now a missing line fails loudly, mirroring the
full-model-name step.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): split GPU job into expect-pass and known-bugs tiers
The GPU job ran `-t "@gpu"` with normal (non-inverted) semantics, so it swept
up both hardware scenarios expected to pass and `@gpu @expected-failure`
known-bug scenarios. Any unfixed known bug made the whole job red, so a red
result couldn't distinguish a real GPU regression from a known bug still
reproducing — and a fixed known bug (XPASS) went unnoticed.
Mirror the mock tier's split:
- e2e-gpu now runs `@gpu and not @expected-failure` (expect-pass only), so a
red here always means a real GPU regression.
- e2e-gpu-known-bugs runs `@gpu and @expected-failure` under `--expect-failures`
(xfail inversion): green while each known GPU bug still fails, red only on
XPASS (bug fixed -> drop the tag), an untagged failure, or a parse/hook error.
Both remain non-blocking (continue-on-error) given the ephemeral GPU runner.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): run the @gpu tier on the Strix Halo runners too
Adds GPU e2e coverage on the two Strix Halo (gfx1151) self-hosted runners
alongside the existing amd-gpu runner, each with the same expect-pass /
known-bugs split:
- e2e-gpu-strix-ubuntu(-known-bugs): Linux gfx1151, a second AMD arch for the
@gpu scenarios. Targeted by [self-hosted, linux, strix-halo].
- e2e-gpu-strix-windows(-known-bugs): first real Windows GPU coverage — the
existing windows-build-and-test runs on GitHub-hosted windows-latest with no
GPU, so @gpu scenarios have never run on Windows hardware. Targeted by
[self-hosted, windows, strix-halo], runs via pwsh.
Label selectors route each job to exactly one runner (strix-halo vs amd-gpu,
and linux vs windows), so there is no cross-routing. All non-blocking
(continue-on-error) while the new hardware is proven out.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): consolidate per-platform reports into one cross-platform report
Each E2E job (4 environments x expect-pass/known-bugs = 8) uploaded its own
report artifact, with no single view across platforms and no platform label on
any report. Add a consolidated report published once per CI run.
- Extract report generation into a lean `e2e-report` crate (only maud + serde),
moved from e2e-cucumber's report.rs. This lets xtask reuse it without pulling
the harness's cucumber/axum/reqwest/tokio tree. e2e-cucumber re-exports it as
`report` so existing call sites are unchanged; its 10 unit tests move along.
- Add generate_consolidated() + consolidated_summary_markdown() to e2e-report:
a platform x tier matrix (total/pass/fail/skip/xfail/status) plus collapsible
per-platform details, with known-bugs tiers evaluated under xfail inversion
(green while bugs reproduce; a platform is flagged on XPASS or regression).
- Add `cargo xtask e2e-report --artifacts-dir <d> --html-out <f>`: auto-discovers
platforms by scanning artifact subdirs for report.json, derives labels from
the artifact names, writes the merged HTML, and prints the summary matrix to
stdout for $GITHUB_STEP_SUMMARY.
- Add the e2e-report CI job: downloads all `*-report` artifacts (glob, so new
platforms auto-appear in the report), runs the aggregator into the step
summary, and uploads one e2e-consolidated-report HTML. `if: always()` so a
non-blocking/failing platform still shows up.
Adding a platform needs no aggregator change beyond appending its job names to
the report job's `needs:` (ordering only; Actions has no wildcard needs).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): isolate serve preconditions, cover both engines, default-engine + readiness
Make serve-dependent scenarios fail only for the behavior they test, and add
coverage for engine selection and readiness.
- Isolate serve preconditions: the shared GPU serve setup now uses the canonical
model id and an explicit engine, so scenarios that test inference/chat no
longer fail on the vLLM alias-resolution bug. The `qwen2.5` alias now appears
only in the two scenarios that test alias resolution.
- Make the serve readiness timeout configurable via E2E_SERVE_TIMEOUT_SECS
(default 600s) for real-hardware first-serve on a cold cache.
- Strengthen the auto-engine assertion to parse the chosen engine and require it
to be one of the supported backends (lemonade/vllm), catching selection of a
removed engine.
- Add explicit lemonade serve+inference coverage (GGUF model), tagged
expected-failure (EAI-7052: lemonade falls back to its Vulkan backend on
Instinct and inference stalls until it uses system ROCm).
- Add a scenario asserting a service the CLI reports ready can immediately serve
inference (readiness contract), and a rocm-core unit test pinning that the
/v1/models signal alone does not imply inference-readiness (EAI-7333).
- Add a scenario and rocm-core unit tests asserting vLLM is the default serving
engine on Instinct data-center GPUs (gfx*-dcgpu) for vLLM-capable models.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): cap inference requests with a timeout
send_chat used a reqwest client with no timeout, so a hung backend (e.g. a
model that never returns a completion) blocked the HTTP call indefinitely and
let a scenario run until the CI job limit. Build the client with a 10s timeout
(override via E2E_INFERENCE_TIMEOUT_SECS) so a stalled inference fails promptly
instead of stretching the job to tens of minutes.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): keep the Strix Halo toolchain off the runner's full root disk
The self-hosted Strix Halo Ubuntu runner has a full, non-persistent root disk,
so setup-rust-toolchain's rustup bootstrap fails writing rustup-init to /tmp
(curl exit 23). Point CARGO_HOME, RUSTUP_HOME, and TMPDIR at the persistent
actions-runner directory and pre-create the temp dir, so the toolchain installs
successfully and is reused by later jobs.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): support manual dispatch and bootstrap the Windows runner
Add a workflow_dispatch trigger with platform and tier inputs so the E2E jobs
can be run on demand against this ref (dispatch skips build-and-test for a fast
loop; each E2E job builds its own binary via `cargo xtask e2e`, so the skip is
safe). Every E2E job's guard is extended to honor the inputs on dispatch while
preserving the exact push/PR behavior otherwise, and the consolidated report
runs on dispatch too.
Also bootstrap Rust on the Strix Halo Windows runner with a PowerShell step:
setup-rust-toolchain runs an internal bash script the runner lacks, so install
rustup via win.rustup.rs (--default-toolchain none; rust-toolchain.toml pins the
version). Idempotent, so it no-ops once the runner is provisioned.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): fix self-hosted runners and speed up manual dispatch
Several fixes so the self-hosted GPU jobs work and manual dispatches stay fast:
- Strix Halo Windows: the runner has neither bash nor PowerShell 7, so the
toolchain bootstrap and E2E run steps use `powershell` (Windows PowerShell 5.1)
instead of `pwsh`.
- Strix Halo Ubuntu: only the nvme volume has free space (/ and /home/ubuntu are
full), so point HOME at a writable dir on that volume — rustup amends
$HOME/.profile and otherwise fails with ENOSPC.
- workflow_dispatch: skip the non-E2E jobs (build/test/lint/coverage/...) so a
manual E2E run doesn't drag the whole gate along.
- Known-bug GPU tiers: cap serve readiness at 90s (E2E_SERVE_TIMEOUT_SECS) so an
expected failure that manifests as a never-ready serve fails fast instead of
waiting the full 600s.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): stop managed services on scenario teardown
A scenario that ran `rocm serve --managed` left a detached supervisor + engine
process (vLLM / llama-server) alive after the harness finished: the World's Drop
only stopped the mock and removed the temp dir, never the detached engines. On a
persistent self-hosted runner these accumulated across runs, holding the GPU and
piling up temp roots until the box was saturated.
Drop now reads each managed-service record in the scenario's isolated root and
runs `rocm services stop <id> --yes` before the temp dir is removed. Best-effort
(teardown never panics) and black-box (reads the on-disk record, stops via the
CLI). The planted e2e-mock record has no real process and is skipped.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): bootstrap Strix Ubuntu rustup without overriding HOME
The previous fix set HOME to a writable nvme dir so rustup's $HOME/.profile write
wouldn't hit ENOSPC on the full root disk, but that made rustup warn about an
euid/HOME mismatch on every run. Instead keep HOME as the real home and bootstrap
rustup with --no-modify-path, so nothing is written to .profile at all; the
toolchain and installer temp still live on the nvme volume via CARGO_HOME /
RUSTUP_HOME / TMPDIR.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): persist the build cache across GPU jobs
E2E GPU jobs took ~22min, almost all of it a full ~15min `cargo build` of the
rocm binary + test harness (519 crates from scratch) — only ~14s was actual
scenario execution. Cause: `actions/checkout` runs `git clean -ffdx`, which
deletes the gitignored `target/` inside the repo every job, so cargo can never
build incrementally.
Point CARGO_TARGET_DIR at `$RUNNER_WORKSPACE/e2e-target` — a sibling of the
checkout that git clean does not touch and that persists between jobs on the
self-hosted runner. Cargo then rebuilds incrementally: unchanged sources reuse
the cache, changed sources rebuild only the diff. Applied to both app-dev GPU
jobs first; Strix jobs to follow once validated.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* report(e2e): split platform/os/tier columns, add totals, legend, run metadata
Parse the artifact name into distinct Platform / OS / Tier fields (gpu -> MI300X,
strix -> Strix Halo Ubuntu/Windows, mock -> Mock) and render them as separate
columns, sorted so each platform's rows group together. Add a totals row, a
legend explaining tier semantics and what "Mock" means, and a run-metadata header
(commit / branch / run / event from CI env). Fixes a parser bug where the bare
e2e-report artifact rendered as "E2e Report" instead of "Mock".
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* report(e2e): add per-command coverage table (command x model x engine x platform)
Surface exactly which rocm commands the suite exercises, with which model and
engine, on which platform, tied to results. Unify the four duplicated run_rocm
copies into one crate::run_rocm choke point that appends each invocation
(scenario, argv, subcommand, model, engine, rc) to a commands.jsonl sidecar next
to report.json; a before-hook tags each with its scenario. e2e-report joins each
command to its scenario's pass/fail and renders a Command coverage table in the
step-summary markdown — "works" follows the scenario result, not raw rc, so a
command meant to be rejected still reads as covered.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): disable rust-cache on self-hosted GPU jobs
The app-dev GPU jobs persist the build cache themselves via CARGO_TARGET_DIR, so
setup-rust-toolchain's built-in Swatinem/rust-cache is redundant and actively
harmful here: its post-step tries to upload the large e2e-target to GitHub's
cache service (slow — observed ~15min hang keeping a job in_progress after its
work finished), and its cleanup wipes the local target. Set cache: false so the
CARGO_TARGET_DIR persistence is the sole cache mechanism.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): share heavy immutable caches across scenarios on GPU runners
Each scenario runs in an isolated data/cache root, which on GPU runners forced
every serving scenario to re-download the ~3.3GB TheRock runtime and the HF model
weights from scratch. Share the immutable, read-only artifacts across scenarios
while keeping service/config state isolated:
- symlink each scenario's <data>/runtimes at a shared dir (runtime wheels +
registry manifests are write-once),
- set HF_HOME to a shared dir (engines honour it for download + discovery;
weights are content-addressed and immutable),
- set ROCM_CLI_ENGINE_ENVS_ROOT to a shared dir (redirects only the envs/ leaf,
so per-service engine state stays isolated).
Gated on E2E_SHARED_CACHE_DIR: unset (local/mock) keeps full isolation; the
app-dev GPU jobs set it to a persistent path under $RUNNER_WORKSPACE. services/,
config/, cache/, and engine state/locks remain per-scenario. Strix jobs to follow.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): keep all Strix Ubuntu writes on the nvme (root disk is full)
On the Strix Halo Ubuntu runner, /, /home/ubuntu, and /tmp are all on a small
root partition that is full from provisioning; only /home/ubuntu/actions-runner
(1.7T nvme) has space. The earlier fix moved the toolchain/temp there, but the
ROCm SDK install still wrote pip's cache to ~/.cache/pip on the full root and
failed with ENOSPC before any scenario ran.
Point HOME, PIP_CACHE_DIR, CARGO_TARGET_DIR and E2E_SHARED_CACHE_DIR at the nvme
so nothing touches root. HOME on the nvme also catches any other $HOME writer;
the rustup bootstrap keeps --no-modify-path. The shared cache additionally
de-duplicates the multi-GB runtime + model downloads across scenarios (they were
installed per-scenario before), cutting nvme usage too.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): drop now-unused Command import in serving_steps
Unifying run_rocm into crate::run_rocm left serving_steps.rs importing
std::process::Command without using it. CI builds with -D warnings, so the dead
import failed the harness build and thus every E2E job (including mock). Remove it.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): wait for the expected model, not just any 200, on the shared serve port
On the single serial Strix Windows runner a serve leaked from a prior scenario
kept answering on the shared port 11435; because scenarios run in isolated data
dirs, the next scenario's rocm has no record of that managed service and can't
stop it. The old readiness check only required /v1/models to return 200, so the
inference scenario proceeded against the WRONG model (got Qwen3-0.6B-Q4_0.gguf
from the lemonade scenario while serving Qwen2.5-1.5B), failing "the response
identifies the correct model".
Make serve readiness model-aware: wait until /v1/models actually lists the model
this scenario served (distinctive substring) before proceeding, so a stale
endpoint no longer satisfies readiness. wait_for_endpoint keeps its old
any-200 behaviour for the mock scenarios.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): only share state-free caches (HF, pip), not runtimes/engine-envs
Sharing <data>/runtimes across scenarios regressed the app-dev GPU tier (4/7 →
1/7): the runtimes registry is state the suite asserts on — "Installing the SDK"
requires `a machine with no CLI-managed runtimes`, which a shared registry
populated by other scenarios breaks, cascading into serve failures. Engine envs
are similarly state-adjacent.
Narrow the sharing to genuinely state-free, content-addressed caches only: HF
model weights (HF_HOME) and the pip cache. Drop the runtimes symlink and
ROCM_CLI_ENGINE_ENVS_ROOT. Runtimes re-install per scenario into the isolated
data root (on the nvme via TMPDIR on Strix), trading some dedup for correctness.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): unique concurrency per dispatch + 15-min job timeouts
Two robustness fixes exposed by an offline self-hosted runner:
- Concurrency: a workflow_dispatch run whose job is queued on an offline runner
cannot be cancelled by GitHub, so under the shared per-ref concurrency group it
deadlocked — blocking every later dispatch from starting. Give manual dispatches
a unique group (github.run_id); push/PR/merge_group keep the shared per-ref group
so new commits still supersede in-flight runs.
- timeout-minutes: 15 on all E2E jobs, so a hung/slow job self-fails at the 15-min
ceiling instead of running for hours — also enforces the per-platform time budget.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): reclaim the GPU from stray serves before each GPU job
The app-dev GPU expect-pass tier timed out at 15 min because a lemonade Vulkan
llama-server leaked by an earlier run (whose Drop teardown never ran after a
kill/timeout) kept spinning on the GPU at ~96% CPU, starving this run's vLLM
serves. The suite itself never serves that model, confirming it was an orphan.
Add a best-effort pre-job step to every self-hosted GPU job that kills stray E2E
serve processes (scoped to /tmp/rocm-e2e-*, e2e-target, e2e-shared — never the
runner or /workload manual processes) and clears leftover temp roots, so each job
starts with a free GPU. bash on Linux, PowerShell on Windows.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): free the shared serve port before each GPU serve
GPU expect-pass timed out because scenarios share port 11435 on one serial box
with per-scenario isolated data dirs: a prior scenario's managed serve detaches
and its record lives in a different dir, so the next scenario's rocm can't stop
it. Multiple vLLM servers then accumulated, oversubscribing GPU memory until the
15-min timeout.
Before each GPU serve, ensure the shared port is free: if something still answers
on 11435, kill the listener (fuser/lsof on Linux, Get-NetTCPConnection on Windows)
and wait for it to close. Best-effort — the serve can still replace it — so it
never hard-fails a scenario. Removes the cross-scenario server pile-up.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): free serve port even while a prior server is still loading
The first port-free pass only killed a server that already answered /v1/models,
so a prior scenario's vLLM still LOADING (holding the port + 0.80 GPU mem but not
yet HTTP-ready) slipped through; the next serve then started a second vLLM,
overcommitted GPU memory, and the collision crashed a server — the following chat
POST failed with "error sending request". Kill by port unconditionally (catches
the starting server) and wait on a raw TCP connect for the socket to close.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): kill the auto-started lemonade assistant + bound expect-pass serve wait
Even with vLLM accumulation fixed, the app-dev GPU expect-pass still exceeded the
15-min cap: the CLI auto-starts its built-in lemonade assistant (Qwen3-4B) on a
Vulkan llama-server (port 8001) that pins a GPU core (EAI-7052) and starves the
vLLM serve under test, so serve readiness never arrives within 600s.
- ensure_serve_port_free now also frees the assistant port 8001 (no scenario
needs the built-in assistant), and the GPU reclaim step kills vulkan/llama-server.
- app-dev expect-pass gets E2E_SERVE_TIMEOUT_SECS=300 so a starved serve fails the
scenario with a real error well under the 15-min job cap instead of being cancelled.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): move vLLM inference scenarios to known-bugs (EAI-7333)
Scenarios 5, 8, and the inference half of 6 assert that a vLLM service which
reports ready can serve inference. On the self-hosted MI300X CI runner the
service reports ready (/v1/models 200) but POST /v1/chat/completions is refused —
the readiness signal is a false positive (EAI-7333). This reproduces identically
on the pre-change baseline (run 29104869493) and in an expect-pass-only run (no
GPU contention), so it is a genuine standing product bug, not a harness/flake.
Tag 5 and 8 @expected-failure-EAI-7333; split 6 so engine-selection + reachable
endpoint stay expect-pass (they pass) and the inference assertion moves to a new
known-bug scenario 6b. The expect-pass GPU tier now contains only scenarios that
actually pass; the inference bug is tracked honestly in the known-bugs tier until
EAI-7333 is fixed.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): move scenario 6 (default-engine working endpoint) to known-bugs (EAI-7333)
Serving Qwen2.5-1.5B without --engine does not reach a ready /v1/models endpoint
within 300s on the MI300X CI runner — the same readiness gap as scenarios 5/6b/8
(EAI-7333). The engine-SELECTION contract stays covered as expect-pass by scenario
9 (serves 0.5B, checks only that vLLM is selected, no readiness wait). This leaves
the expect-pass GPU tier with only reliably-passing scenarios: examine 3/4,
serving 9, runtime 1.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): make examine scenario 4 hermetic by planting an unmanaged ROCm install
Scenario 4 ("distinguishes CLI-managed from pre-existing ROCm") had a no-op Given
step that silently depended on the box already having a non-CLI ROCm install. That
holds on the MI300X runner (ambient /opt/rocm) but not on the Strix Halo Windows
runner (iGPU, display driver only), where `rocm examine` correctly reports "No ROCm
installs saved yet" and the scenario failed.
Plant a fake pre-existing ROCm install in the scenario's isolated tree
(legacy-rocm/.info/version) and export it via ROCM_PATH, which detect_legacy_rocm_summary
honors. examine then reports detected_unmanaged and emits the "rocm install sdk"
adopt guidance on every platform, independent of any ambient system ROCm. No
assertion change and no product change; this also hardens the previously-incidental
MI300X pass.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): per-scenario expectation resolution + reconciled report grid
Replace the global @expected-failure / E2E_EXPECT_FAILURES tag model with a
per-(scenario × host) resolution to expected-pass / expected-fail (xfail) /
not-applicable (skip), derived from a host capability probe plus a declarative
xfail matrix, and render a reconciled (scenario × platform) grid in the report.
Harness (tests/e2e-cucumber):
- capability.rs: probe `rocm examine --json` + `engines list` once; derive the
effective serve engine (re-implements the product family/OS rule, drift-guard
tests). Distinguishes engine "adapter present" from "can start here".
- expectation.rs: resolve(tags, capability, matrix) -> pass/xfail/skip.
- expectations.toml: EAI-7333 conditions use effective_engine = "vllm", fixing
the Strix-Windows XPASS from run #543.
- features: stable @id per scenario; @gpu -> @requires-gpu; @requires-engine pins.
- harness: filter_run skips N/A; per-scenario reconciliation; platform.json sidecar.
Report (crates/e2e-report):
- parse platform.json; reconcile actual vs expected by @id into a CellOutcome;
render the (scenario × platform) grid in markdown + HTML with a Needs-attention
list; add scenario_results_by_id().
Verified: 19 e2e-cucumber + 24 e2e-report lib tests pass; mock run resolves
8 pass / 2 xfail / 11 skip, exit 0; grid renders from real artifacts.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* style(e2e): satisfy clippy on the new expectation modules
Collapse nested `if let`/`if` into let-chains (Rust 1.96), make label()/is_empty()
const, use Self in the Expectation::label match, and shorten two doc first
paragraphs. No behaviour change; 19 e2e-cucumber + 24 e2e-report lib tests still
pass, both crates clippy-clean under -D warnings.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): collapse to one E2E job per platform
With the harness resolving pass/xfail/skip per scenario, the expect-pass vs
known-bugs tier split is redundant — membership is decided at runtime, not by a
tag filter. Collapse the 8 E2E jobs (4 platforms × 2 tiers) to 4 (one per
platform), keeping the mock(hosted, blocking) vs GPU(self-hosted, non-blocking)
boundary.
- xtask: drop the `--expect-failures` flag and the E2E_EXPECT_FAILURES env; each
job just runs `cargo xtask e2e` (no tag filter). Ad-hoc `-- <args>` forwarding
stays for local use.
- ci.yml: remove the four `*-known-bugs` jobs and all `-t "@gpu and ..."` /
`--expect-failures` args; e2e-report `needs:` shrinks 8→4; drop the now-obsolete
`tier` workflow_dispatch input (keep `platform`). Per-scenario serve timeouts
come from expectations.toml; the job-level E2E_SERVE_TIMEOUT_SECS stays as the
outer bound.
Verified: xtask/e2e-report/e2e-cucumber build clippy-clean under -D warnings;
43 lib tests pass; `cargo xtask e2e` runs end-to-end (8 pass / 2 xfail / 11 skip,
exit 0); ci.yml parses to exactly 4 E2E jobs + consolidator with no stale refs.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* fix(e2e): probe GPU from examine text, disambiguate strix slug by OS
Full platform=all run #544 exposed two probe bugs:
1. The capability probe parsed `examine --json`, whose `Examination` struct
reported has_amd_gpu:false / no gfx on a real MI300X — so the probe derived
platform_slug=mock and effective_serve_engine=lemonade on GPU boxes, making
every @requires-gpu scenario resolve to skip and the box masquerade as mock.
Parse the HUMAN `examine` text instead (os:/detected_gfx_target:/wsl:), the
same signal the scenarios themselves trust — GPU is present iff a real
detected_gfx_target is reported.
2. Strix Halo Ubuntu and Windows share gfx1151, so both derived slug
"strix-halo" and collided into a single report-grid column (one overwrote the
other). Append the OS for multi-OS families → "strix-halo-linux" /
"strix-halo-windows", giving each its own column.
Tests updated to the text parser and per-OS slugs; 19 lib tests pass, clippy
clean under -D warnings; Mac probe still resolves mock/lemonade.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* docs(e2e): correct capability doc comments to reference examine text
Follow-up to 99d5890: the probe now parses human `examine` output, not
`--json`; update the HostCapability field docs and module doc accordingly. No
code change.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* fix(e2e): per-scenario serve timeout + raise GPU job cap to 25min
Full run #545 cancelled the MI300X job at the 15-min cap: with tiers collapsed,
one job runs every serve scenario, and the EAI-7333 known-bug serves each waited
the full 300s for a readiness that never comes. A cancelled job writes no
platform.json, so the grid lost that column entirely.
- Wire the per-scenario `serve_timeout_secs` from expectations.toml: the `before`
hook resolves it (matching conditions on this host) onto the World, and the
serve steps prefer it over the global default. So an xfail serve that a bug
keeps from becoming ready fails fast (90s) instead of burning 300s, while real
expect-pass serves keep the full window. `Expectations::serve_timeout_for` +
unit test.
- Raise the three non-blocking GPU jobs' timeout-minutes 15 → 25 as a safety net
so a stuck serve can never cancel a job before it writes its report. Mock stays
15 (fast + blocking).
20 e2e-cucumber lib tests pass; clippy clean under -D warnings; ci.yml parses.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): fix Windows lemonade model-name match + gate adopt to Linux
Two Strix-Windows "failures" from run #545 were test bugs, not product bugs:
1. serve-lemonade-inference: lemonade inference WORKS on Windows (serve + chat +
reply all pass) — only the final model-name assertion failed, because lemonade
reports the concrete GGUF artifact (`Qwen3-0.6B-Q4_0.gguf`) while the scenario
served the catalog name (`Qwen3-0.6B-GGUF`). Match on a normalized base
(strip `.gguf`, the `-gguf` catalog marker, and quantization suffixes) so the
two reduce to the same id; vLLM's exact-id echo still matches.
2. runtime-adopt-preexisting-rejected: the step adopts `/opt/rocm` with
`--python /usr/bin/python3` — Unix paths that on Windows resolve to a bogus
`C:/usr/bin/python3`, so the CLI errors on the missing path before emitting
the install-type guidance the assertion checks. The scenario's premise is
Linux-only. Add a general `@requires-os:<os>` capability gate (mirrors
@requires-gpu/@requires-engine) and tag the scenario `@requires-os:linux`, so
it runs on Linux hosts and skips on Windows instead of failing spuriously.
21 e2e-cucumber lib tests pass (incl. requires-os gate); clippy clean under
-D warnings.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): raise GPU job cap to 35min so the run completes and writes its report
A full MI300X run measured ~28min and was cancelled at the 25min cap, so it
never wrote platform.json → no grid column. The dominant cost is `install sdk`
re-running per scenario (isolated per-scenario data dirs) plus vLLM cold-starts,
not the serve-readiness waits the per-scenario timeout already bounds. Raise the
three non-blocking GPU jobs to 35min so they complete and produce their report;
mock stays 15min (blocking, fast). Sharing the runtime across scenarios to cut
the redundant installs is tracked as a follow-up.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* feat(e2e-report): add CLI command-coverage denominator + coverage %
The command-coverage table showed which commands WERE exercised but had no notion
of the full surface, so it couldn't report what's missing. Add a curated
KNOWN_COMMAND_SURFACE (the `rocm <base>` catalog from the CLI's --help tree) as
the denominator, compute covered/total, and render a "CLI surface coverage:
N/M (P%)" header plus a <details> fold listing every uncovered command. Matching
normalizes the recorded signatures' --engine / (default engine) suffixes to the
base command. New command_base + command_coverage_summary with unit tests.
26 e2e-report tests pass; clippy clean under -D warnings; verified against a real
artifact (5/43 mock-only, 38 uncovered listed).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 0fc67d12b3963679343682d3e75fed6a45f9e32f)
* test(dash): add user-journey coverage for the dash TUI
The black-box cucumber suite drives the `rocm` CLI and can't exercise the
interactive `dash` TUI, leaving that user journey uncovered. Add journey tests at
the seam the dash crate already uses (public AppState API + ratatui TestBackend):
navigate every tab on live telemetry, apply a wire Snapshot event and see it
reflected, drive the theme picker, and hold a chat exchange (submit → reply, plus
an error turn). Assertions key on state transitions (the reliable signal), with
rendering as a no-panic check — complementing the render-only characterization in
dash_characterization.rs. 5 tests, clippy clean under -D warnings; existing dash
tests still green.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit f315ba0f45ebe7685ce8be5b781f997909816296)
* fix(e2e-report): reconcile summary status with the expectation grid
The consolidated report's per-platform Status column derived from raw junit
pass/fail counts, so a platform whose only failures were known-bug xfails
(e.g. mock's two EAI-7219 scenarios) was shown FAIL while the reconciled
scenario-by-platform grid correctly showed it clean. A report that
contradicts itself is not a trustworthy capability map.
Status and the summary Pass/Fail/Skip/Xfail counts now derive from the same
id-keyed reconciliation the grid uses: a known bug that fails is xfail, not a
failure; only unexpected-fail, XPASS, or ran-when-N/A red a platform.
Artifacts predating the expectation system (no platform.json) fall back to
the legacy junit status unchanged.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit afbabc8e563e392a0f7a79e815e4869d5659f8d7)
* fix(e2e-report): drop vestigial Tier column, clarify legend, use n/a
The consolidated report's Tier column always read "expect-pass" after the
collapse to one job per platform (no artifact carries a known-bugs tier
anymore), so it conveyed nothing — removed it from both the markdown summary
and the HTML matrix. Rewrote the platform caption to explain what "gates the
PR" and "non-blocking" mean and to describe the actual columns, and clarified
that Mock fakes the inference backend (not the CLI). Replaced every
"not applicable" dash with n/a across the grid, summary, and command-coverage
tables.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit d9d3adbf1806db14ac2a1715cb7c79f57c9dc63f)
* test(e2e): resolve engine-agnostic scenarios to the host's native engine
Several scenarios captured engine-agnostic behaviour (end-to-end chat, tool
definitions, the readiness contract) but were pinned @requires-engine:vllm, so
they showed n/a on every platform except MI300X even though the behaviour
exists on lemonade hosts too. The shared serve precondition
(`a model is being served on GPU`) now serves the model matching the host's
effective engine — safetensors via vLLM on Instinct, GGUF via lemonade on
Strix — so those scenarios drop the vLLM pin and run on each GPU platform with
its native engine.
Renamed serve-inference-response -> serve-vllm-inference (it is the deliberate
vLLM half of a per-engine pair with serve-lemonade-inference, so it keeps the
pin — the slug now names the engine) and serve-ready-implies-inference ->
serve-readiness-contract (the behaviour is the trustworthiness of the ready
signal, not a specific engine). Added EAI-7052 lemonade+linux xfail conditions
for the broadened inference scenarios so they xfail on Strix Ubuntu (where
lemonade inference hangs) rather than reading as unexpected failures.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit f63ca2c0d4da2eee8097eb9f606ad5b34cdf47e1)
* feat(e2e-report): show full command + resolved engine in coverage table
The command-coverage table recorded a stripped signature and only the
--engine flag value, so the Command column hid the flag's value, a
default-engine serve showed no engine at all, and a redundant Model column
just repeated what the command already contained.
The harness now records the full command as executed (argv) and, for a serve
with no --engine, the engine the CLI resolved itself (parsed from the serve
plan's `engine:` line, restricted to serve so another command's output is
never misattributed). The report shows the full command, an Engine column
that marks a CLI-chosen engine as "<engine> (default)", and drops the Model
column. Older artifacts without the new fields fall back to the signature.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 6bd0933ef8ff4e30d3cd5bd6b458bdc7db0f4a93)
* fix(e2e-report): count serve as covered + clarify blank-cell legend
The CLI-surface coverage matcher required a recorded command's base to equal a
KNOWN_COMMAND_SURFACE entry exactly, but a serve command embeds its model
(`rocm serve Qwen/...`), which never equals the bare `rocm serve` entry — so
serve was reported uncovered despite being exercised (and appeared in BOTH the
table and the Uncovered list). Match by the longest surface entry that is a
word-boundary prefix of the base, so `rocm serve <model>` maps to `rocm serve`
and `rocm install sdk` still wins over any shorter prefix.
Also reworded the command-coverage legend to explain that a blank cell means
the command was not run on that platform — usually because its scenario is not
applicable there, or the command is platform-specific by construction.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit b67897e4c72836732b77a66c309ce839d681d44e)
* test(e2e): cover the rocm chat CLI with a one-shot prompt scenario
The chat scenarios sent inference by calling the model endpoint directly over
HTTP, so the `rocm chat` command itself was never exercised and showed as
uncovered in the command-surface table. Add a mock-tier scenario that runs the
real one-shot CLI (`rocm chat --provider local --model <served> --prompt`)
against the planted mock service and asserts the CLI prints the assistant's
reply — covering arg parsing, local-provider service discovery, and output
rendering. No GPU needed, so it runs on the blocking tier and every platform.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit e58f3657e2b345d19c608dd59d6f971bb8eb1271)
* test(e2e): cover help alphabetization (EAI-7383) and runtime path nesting (EAI-7384)
Two black-box scenarios surfaced by tester dogfounding, each tracked as xfail
until fixed:
- help-lists-subcommands-alphabetically (examine.feature, mock-tier): asserts
`rocm help` lists subcommands alphabetically. Currently declaration order →
xfail EAI-7383.
- runtime-path-not-nested (runtime_setup.feature, GPU-gated): asserts the active
managed runtime folder path has no recursive `runtimes/wheel/.../runtimes/wheel/`
segment. xfail EAI-7384 (awaiting first GPU run to confirm).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 81304b9423a2c3d0d4b71564369ee597b1106629)
* test(e2e): add large-model vLLM serve+inference coverage
Serve a big model end-to-end at least once (dogfooding W9): new scenario
serve-large-model-inference serves Qwen/Qwen3.6-27B (BF16, ~54 GiB, the
catalog's Instinct recommendation) on vLLM and asserts real inference. Pinned
@requires-engine:vllm so in our fleet it runs only on MI300X (fits 1 GPU; Strix
Halo is lemonade / too little VRAM). Confirmed working on app-dev MI300X.
Adds a `@serve-timeout:<secs>` scenario tag so an expected-pass scenario with a
genuinely slow serve (a cold ~54 GiB load far exceeds the 600s default) can
lengthen its readiness window. Precedence in the before hook: tag →
expectations.toml xfail serve_timeout_secs → default.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 50a756005de86f606dd277bc3b1ceb0789e91589)
* chore: ignore workspace and report artifacts
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): raise CLI vLLM readiness cap for the large-model serve
The large-model scenario's @serve-timeout tag widened only the harness poll, but
`rocm serve --managed` has its own vLLM readiness cap (5-min default) and
SIGTERM-kills the server when a big model's cold load exceeds it — so the serve
died before the poll could ever see it ready. isolate_cmd now also exports
ROCM_CLI_VLLM_READY_TIMEOUT_SECS from serve_timeout_override, keeping the CLI's
cap in lockstep with the harness window.
Verified on app-dev MI300X with Qwen/Qwen3.6-27B (weights pre-seeded in the
shared HF cache): default 300s aborts mid-load; with the raised cap the model
reaches ready and returns inference. The too-low product default is filed as
EAI-7393.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 07be420b82fff594ba0525b95033649ae46e05c5)
* fix(e2e): validate E2E_SHARED_CACHE_DIR before use (path-injection)
CodeQL flagged shared_cache_dir() as a high-severity path-injection: the
CI-provided E2E_SHARED_CACHE_DIR flowed into create_dir_all unchecked. Require
the override to be an absolute path with no `..` traversal components before it
reaches the filesystem sink — rejects a malformed/relative value and clears the
taint flow. Behaviour is unchanged for the well-formed absolute path CI sets.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): scope EAI-7052 lemonade-inference xfail to any OS, not just Linux
The lemonade inference-failure xfail conditions were scoped os="linux", but the
Strix Halo Windows run shows the same failure there — lemonade reports the
service ready, then inference drops the connection ("CURL error: Failure when
receiving data from the peer" / "error sending request"). So the four broadened
inference scenarios (serve-lemonade-inference, serve-readiness-contract,
chat-tool-definitions-accepted, chat-end-to-end-local-model) read as unexpected
failures on Strix Windows. Drop the os constraint — EAI-7052 is engine-specific
(lemonade llama.cpp backend), not OS-specific — so it xfails on any lemonade host.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): wait for GPU VRAM to drain before starting a new serve
ensure_serve_port_free freed the shared port but not device memory: a killed
vLLM releases its VRAM only as the process fully exits, which lags the socket
close. After a large model (Qwen3.6-27B, ~54 GiB) the residue dropped free VRAM
below the next serve's gpu-memory-utilization request, so the following scenario
died with "Free memory on device cuda:0 ... less than desired GPU memory
utilization" (engine core init failed) — observed on app-dev where the 27B
serve starved the next Qwen2.5-1.5B serve.
Poll amd-smi (then rocm-smi) after the port frees and wait until enough VRAM is
free before returning. Best-effort: if no smi tool exists (mock/local, no ROCm)
it returns immediately, so non-GPU runs are unaffected.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 4f39d2b481da358f633d7dcf101e4175140d43be)
* test(e2e): address PR review — clippy gate, real assertions, CI hardening
Resolves the review findings on PR #69:
Blocking:
- Wire clippy for the e2e harness + step files: the `test = false` e2e target is
skipped by `clippy --all-targets`, leaving ~1.6k lines unlinted. Add a CI step
`cargo clippy -p e2e-cucumber --test e2e -- -D warnings`, and fix the 9 lints it
surfaced (map/unwrap_or_else→let-else, collapsible-if, Duration::from_mins,
operator-precedence, doc-paragraph, items-after-statements).
- Two vacuous scenarios no longer pass while testing nothing:
chat-privacy-notice-accurate now fails (TUI-only, unverifiable black-box) and is
tracked xfail (EAI-7222); serve-vllm-default-on-instinct asserts rc == 0.
Should-fix:
- Unify the report.json pass predicate: the CI gate (scenario_results_by_id) and
the grid (id_pass_map/tally) disagreed on "skipped" status; both now route
through scenario_passed (all-steps-passed), so the same artifact can't fail the
job yet render green.
- Enforce serial GPU serves: cap cucumber concurrency to 1 when a GPU is present
(shared port 11435 / one card); mock keeps default parallelism.
- Mock e2e job runs at job level (not skipped on non-heavy PRs, which stalled the
merge queue for a required check); heavy/build-and-test gating moved to a step.
- Remove the unanchored `pkill -f 'vulkan/llama-server'` (killed manual serves on
the shared runner); the anchored /tmp/rocm-e2e line already covers it.
- scenario_status treats a failed before/after hook as failed (was scored passed).
- Default-engine serve uses a model-aware readiness wait + asserts the reply model
id, so it can't pass against a leaked prior serve on the shared port.
- Update the stale README (abandoned tag model / 3-tier CI → @requires-* +
expectations.toml + one-job-per-platform) and fix the expectations.toml path ref.
Nits: bar_widths skip remainder, e2e-report crate doc (serde_json), duplicate-@id
assertion, stale 15-min comment.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit c67cb693edaebbc02db8b2927a15ef8d3b065985)
* ci(e2e): raise app-dev GPU job cap to 90min for the large-model serve
The collapsed GPU job was cancelled at the 35min cap once the Qwen3.6-27B
large-model scenario was added (~54 GiB load + post-serve VRAM-drain wait on top
of the existing per-scenario install-sdk + vLLM cold-starts). Raise to 90min so
the job completes and writes platform.json. Follow-up: tag the heaviest scenarios
nightly-only so the per-PR GPU run stays short.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 36be4db58df7c4b2c5826b9dac9f708a96cc8dd7)
* test(e2e): add @nightly tag so expensive scenarios run only nightly
The large-model (Qwen3.6-27B) serve dominates the GPU job's wall-clock (~54 GiB
load + VRAM-drain). Tag it @nightly: the resolver skips @nightly scenarios unless
E2E_INCLUDE_NIGHTLY is set, so per-PR and on-demand GPU runs stay fast, while a
new e2e-gpu-nightly job in nightly.yml runs the full suite (flag set) once a day
on MI300X.
- expectation.rs: parse @nightly; resolve() skips it (cheapest gate, first) unless
include_nightly; unit test covers skip-without / run-with / GPU-gate-still-applies.
- e2e.rs harness: read E2E_INCLUDE_NIGHTLY and pass it to resolve().
- model_serving.feature: tag serve-large-model-inference @nightly.
- nightly.yml: e2e-gpu-nightly job (self-hosted amd-gpu, 90min, E2E_INCLUDE_NIGHTLY=1),
independent of the nightly-build skip.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 31f38b12ea89b5ae97567f7a67955ad42aa814e3)
* perf(e2e): share uv download cache across scenarios so install sdk is warm
Each GPU scenario runs `rocm install sdk`, which fetches multi-GB TheRock
runtime wheels with uv. In isolated data dirs every scenario re-downloaded
them cold (~160s each), which — multiplied across the suite — starved the
90min GPU cap before the large-model serve could run.
Share uv's content-addressed download/build cache via a new
E2E_SHARED_UV_CACHE_DIR env var, wired to UV_CACHE_DIR in isolate_cmd. Only
the first scenario pays the cold download; the rest reuse it (~34s warm,
measured on MI300X — ~5x). No mutable state is shared: the runtimes registry
the suite asserts on still lives in each scenario's isolated <data>/runtimes.
Kept as its own env var (not derived from E2E_SHARED_CACHE_DIR) so CI can
host the ~23GB uv cache on the roomy / overlay rather than the near-full
model-weights PVC. Refactored the absolute/no-`..` path validation shared by
both caches into validated_shared_dir(). No-op locally (var unset).
Set E2E_SHARED_UV_CACHE_DIR=/var/tmp/rocm-e2e-uv-cache in the ci.yml e2e-gpu
and nightly e2e-gpu-nightly jobs.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 6c6231b271f82838da729d771561aa2e96453503)
* perf(e2e): share one installed runtime across GPU scenarios (install once)
Each GPU scenario's `a managed runtime is active` precondition ran its own
`rocm install sdk` in an isolated data dir — a multi-GiB TheRock install whose
post-install probe also unpacks an ~8.8 GiB devel tarball. Multiplied across
~10 serve/chat scenarios this blew the GPU time cap.
Install the runtime ONCE per runner and share it: `use_shared_runtimes()`
symlinks a scenario's `data/runtimes` at `E2E_SHARED_RUNTIMES_DIR`, so the
first scenario to need a runtime finds the pre-warmed tree already present
instead of re-installing. Opt-in from the `a managed runtime is active`
precondition only; the clean-slate scenarios (`runtime-install-sdk-active`,
`a machine with no CLI-managed runtimes`) stay isolated and still see an empty
slate. A shared runtime resolves via `single_ready_runtime` and reports
`status=ready`, so both the precondition and serve are satisfied without any
active-key wiring. No-op locally (env var unset) and validated via
`validated_shared_dir` (absolute, no `..`).
CI (e2e-gpu + nightly e2e-gpu-nightly): build the rocm binary once, reuse it
for both a serial pre-warm and the suite via `ROCM_CLI_BINARY` (no redundant
`cargo run --release`); pre-warm the shared runtime once and persist it on
`$RUNNER_WORKSPACE` across runs, so after the first run ever the pre-warm is a
no-op.
Verified on MI300X: a 2-scenario probe (shared-runtime serve + inference, and
the isolated clean-slate install) both pass; mock gate unaffected (env unset).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit ebc00b1ee294f7d8e2aac3dfd0a491aa6b30b5f9)
* fix(e2e): pre-warm shared runtime in place — never mv (breaks baked paths)
The share-one-runtime pre-warm installed into a scratch dir then `mv`d the tree
to the shared location. `install sdk` bakes ABSOLUTE paths (install_root,
python_executable) into the runtime manifest, so after the mv those pointed at
a deleted directory and every serve failed instantly — the precondition passed
(runtime listed ready) but `rocm serve` errored before starting the engine
(observed on run 29320025393: both serve scenarios failed in ~1s, the isolated
install scenario passed).
Install the pre-warm directly into its persistent data/runtimes and point
E2E_SHARED_RUNTIMES_DIR at that same dir — no mv — so the baked manifest paths
stay valid and each scenario's data/runtimes symlink resolves to the real tree.
Verified by hand on MI300X: serve READY in 75s + real chat completion through a
scenario symlink to an in-place-installed shared runtime.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit 021b14c60c0f116125edd04f68aa01c2a320732b)
* test(e2e): drop mismapped EAI-7221 vLLM xfail from chat-end-to-end-local-model
EAI-7221 is a TUI-only bug (TUI chat hardcodes 'local-model' instead of
querying /v1/models). This scenario exercises the CLI `rocm chat --prompt`
path, which discovers the model correctly and works — it never touched the TUI
defect, so the vLLM-path xfail was mismapped. It surfaced as an XPASS on the
full MI300X run (29322186691); confirmed expect-pass with 3/3 green re-runs and
a clean 0-xfail/0-XPASS reconciliation after removing the block. The lemonade
path stays xfail (real inference → EAI-7052 backend bug).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
(cherry picked from commit fc4687becae8e8c9b3b02921b4a05182a05ee48a)
* test(e2e): make default-engine serve recipe-aware; re-key Instinct xfail (#23)
`rocm serve <model>` with no --engine is recipe-driven, not platform-driven: it
resolves the request to the recipe's preferred model+engine, which may differ
from what was requested (e.g. a safetensors request resolves to a GGUF recipe on
lemonade). The default-engine scenarios hardcoded the requested model in the
readiness wait, so they timed out whenever the recipe resolved to a different
model — on a lemonade-default host the safetensors request isn't even servable.
Wait on the model the CLI actually resolved (parsed from the serve plan) instead
of the requested id, via new resolved_model() + ready_substr_for() helpers; also
dedup two existing `resolved model:` parses onto resolved_model().
Re-key the Instinct (effective_engine=vllm) xfail from EAI-7333 to EAI-7052:
verified on MI300X that the default serve resolves to a GGUF recipe on lemonade,
whose Vulkan backend hangs on Instinct (EAI-7052) — the default path doesn't use
vLLM at all here, so EAI-7333 was the wrong bug.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* tmp(e2e): name_filter dispatch input wired to strix-ubuntu for #23 probe
TEMPORARY: scoped-probe input forwarded to the cucumber harness on the
strix-ubuntu job so a dispatch can run just the two serve-default-engine-*
scenarios and validate the #23 recipe-aware fix on the lemonade-native Strix
path. Empty = full suite. Remove after validation.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): address volen-silo review — expectations, report, CI pins, #23 xfail
- expectation.rs: `#[serde(deny_unknown_fields)]` on `Condition`/`XfailEntry` so
a typo'd key can't parse to an all-None (unconditionally matching) xfail.
- e2e-report: route `scenario_pass_map` through canonical `scenario_passed`; split
reconcile's `Missing` — an expected pass/xfail with NO result is now `Absent`
(a problem), so a lost-results run reds the platform instead of passing.
- model_serving.feature: gate `serve-vllm-default-on-instinct` with
`@requires-engine:vllm` (false failure on lemonade-default hosts).
- expectations.toml: xfail both `serve-default-engine-*` on effective_engine=
lemonade (EAI-7423: first serve runs a backend install that hides the plan
line; endpoint dies pre-inference via lemonade Vulkan instability).
- ci.yml/nightly.yml: SHA-pin the newly-added actions/upload-artifact@v4 (v4.6.2)
and download-artifact@v4 (v4.3.0) per AGENTS.md §6.
- chat_steps.rs / mock_server.rs: remove dead step + never-read received_models.
- dash_journeys.rs: assert real transitions (theme changes; chat error surfaced
as an Error-role turn), not just a non-empty render.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* fix(e2e): drop now-unused mock_server state after removing received_models
Removing the never-read `received_models` recording left the `handle_chat`
`State(state)` extractor and the `MockServer.state` field unused. CI builds with
`-D warnings`, so these tripped `unused-variables` / `field never read` and failed
the "Unit tests (e2e-cucumber lib)" step (a plain local `cargo test` without
`-D warnings` did not catch it). Drop the state extractor from `handle_chat` and
the dead `state` field from `MockServer`; the router keeps its own `with_state`
clone for `handle_models`.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): report component versions per platform + scope lemonade xfails to Linux
- Report OS/ROCm/vLLM/lemonade versions per platform: the harness collects them
from the installed runtime (examine distro, runtimes-list version, vLLM
dist-info, lemond --version) into platform.json; the report renders them in each
column heading.
- Scope the lemonade xfails (serve-default-engine-*, serve-readiness-contract,
serve-lemonade-inference, chat-end-to-end-local-model) to os=linux: run
29338475133 showed they PASS on native Windows lemonade and only fail on the
Linux llama.cpp Vulkan backend, so the any-OS xfails XPASS'd on Strix-Windows.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): tag runtime-install-sdk-active @nightly (slow cold devel install)
This is the one GPU scenario that must do a real cold `install sdk` (its
precondition is a clean slate, so it can't reuse the shared pre-warmed runtime).
The post-install probe unpacks + scans the 8.8GB->13GB devel tree on the runner
overlay, which takes ~35min and nearly caps the 90min per-PR GPU job. Move it to
@nightly so the fast per-PR GPU run stays ~25min; it still runs in nightly where
there's time. Follow-up: make the cold devel install/probe faster or install-once.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): scope chat-tool-definitions-accepted lemonade xfail to os=linux
Run 29354305288 (Strix-Windows) XPASS'd chat-tool-definitions-accepted: lemonade
inference works on native Windows, so like the other lemonade inference scenarios
this EAI-7052 xfail applies only on Linux (llama.cpp Vulkan hang). Scope it to
os=linux — the last remaining Strix-Windows XPASS.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): collect OS version on every platform + wire Strix runtime versions
The report only showed versions on MI300X because collect_versions() was gated
entirely on E2E_SHARED_RUNTIMES_DIR (set only by the app-dev-gpu job), so mock and
both Strix jobs got no versions — not even OS. Fix:
- collect_versions now ALWAYS reads OS (from examine) on every platform, incl.
mock; the runtimes dir is an Option, and ROCm/vLLM/lemonade are read only when a
persistent runtime dir is available.
- ci.yml: set E2E_SHARED_RUNTIMES_DIR on both Strix jobs (Ubuntu + Windows) so
their runtime's ROCm/vLLM/lemonade versions are readable at end-of-run (and so
serve/chat scenarios install once). Windows engine subpaths differ, so vLLM/
lemonade there degrade to n/a gracefully; OS + ROCm still show.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): grey-x xfail grid cells, n/a coverage blanks, OS-aware version probe
- Expectation grid: render xfail as a grey cross (status-xfail) instead of the
word "xfail", with a legend line; fix a latent maud bug that emitted two
class attributes so the status colour was dropped.
- Command-coverage: render not-run cells as n/a instead of blank, updated legend.
- collect_versions: locate site-packages across OS layouts (Windows Lib/,
non-pinned python3.x on Unix) and find lemond/lemond.exe, so vLLM/lemonade
versions populate on Strix, not just MI300X.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): resolve runtime root from shared tree so Strix reports vllm/lemonade
collect_versions trusted the manifest's install_root, which on Strix is a
per-scenario temp dir that no longer exists by report time — so vLLM/lemonade
probed a dead path and came back None (only os+rocm populated). Derive the root
as <runtimes_dir>/wheel/<runtime_key> from the shared tree we were handed,
falling back to the manifest path only when the derived one is absent. On MI300X
the two coincide (prewarm installs in place), so this doesn't change it there.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): add include_nightly dispatch input for scoped nightly probes
The app-dev-gpu job never set E2E_INCLUDE_NIGHTLY, so the @nightly scenarios
(large-model serve, cold install) could only run via the full ~90min nightly
workflow. Add an off-by-default boolean dispatch input that sets
E2E_INCLUDE_NIGHTLY, so a manual dispatch can confirm a single nightly scenario
in-suite (combined with name_filter) without the whole nightly run.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): floor inference timeout at serve override so 27B chat doesn't time out
serve-large-model-inference failed in-suite (CI run 29369965558): the 27B served
and became ready, but the chat POST aborted with "error sending request" — the
inference client timeout defaulted to 10s, far too short for a 54 GiB BF16 model's
first token in --enforce-eager mode. A manual `curl -m 60` passed, which is why
the manual proof missed it. inference_timeout_for(world) now floors the client
timeout at the scenario's @serve-timeout override (2400s for the 27B), so a
heavy-model scenario gets inference headroom while normal/known-bug scenarios keep
the 10s fail-fast the EAI-7052 hang-detection relies on.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): honor name_filter in the app-dev-gpu job too
name_filter was only wired into the strix-ubuntu job, so a scoped dispatch
targeting app-dev ran the full 25-scenario suite instead of the named scenario.
Apply the same --name filter to the app-dev-gpu step so a single scenario (e.g.
the 27B nightly) can be probed there without the whole suite.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): pre-warm the shared runtime in place on strix-ubuntu
The strix-ubuntu job had no pre-warm — it relied on the first "a managed runtime
is active" scenario to install into the shared dir. But `install sdk` bakes
absolute paths (install_root, python_executable, rocm_sdk.*, .rocm-cli-runtime.json)
into the runtime manifest pointing at that scenario's isolated /tmp/rocm-e2e-XXXX
data dir, which is deleted when the scenario ends. Every later serve then saw
`status=unusable (install root is missing)` and failed — diagnosed directly on
the box. Add the same in-place pre-warm the app-dev e2e-gpu job uses: build rocm
once, install sdk into $RUNNER_WORKSPACE/e2e-prewarm/data/runtimes with no mv, so
the baked paths stay valid for all scenarios and the version probe. No-op after
the first run (persists across runs).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* test(e2e): re-key Strix lemonade serve xfails to EAI-7423 (ready-then-shutdown)
Box investigation (2026-07-15) disproved the earlier reasons for the lemonade
LINUX serve/inference xfails: the serve does NOT hide the engine plan line, and
it is NOT the EAI-7052 Vulkan hang. The real cause is a single product bug —
the lemonade managed serve reaches ready on :8001 then is shut down ~0.08s later
(now tracked in EAI-7423). Re-key the six lemonade+os=linux entries
(serve-default-engine-working-endpoint, serve-default-engine-inference,
serve-readiness-contract, serve-lemonade-inference, chat-tool-definitions-accepted,
chat-end-to-end-local-model) from EAI-7052/old-EAI-7423-reason to EAI-7423 with
the corrected ready-then-shutdown reason. The two vLLM/Instinct default-serve
entries keep EAI-7052 (a genuinely different failure mode). Scoping unchanged
(os=linux only — these pass on native Windows lemonade).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): pre-warm the shared runtime in place on strix-windows
Same fix as strix-ubuntu (cf3f9d5), in PowerShell. The strix-windows job had no
pre-warm — the first serve scenario installed the runtime, triggering a cold
216MB backend + 4.6GB therock dist download that failed `rocm serve`
(serve-readiness-contract scenario 8, run 29357303454). `install sdk` bakes
absolute paths into the runtime manifest, so a per-scenario temp install leaves
later serves pointing at a deleted dir. Pre-warm in place (build rocm once,
install sdk into RUNNER_WORKSPACE\e2e-prewarm\data\runtimes, no move, no-op after
first run) + honor name_filter like the other jobs. Needs a CI dispatch to
verify on the Windows box (no local console access).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
* ci(e2e): bounded GPU preflight so GPU jobs fail fast instead of hanging
Add a GPU-availability preflight to the three GPU E2E jobs (MI300X, strix-ubuntu,
strix-windows), right after the reclaim step. It's a boun…1 parent 5b32c86 commit 5fab56c
34 files changed
Lines changed: 8820 additions & 10 deletions
File tree
- .github/workflows
- crates
- e2e-report
- src
- rocm-core/src
- rocm-dash-tui/tests
- tests/e2e-cucumber
- features
- src
- tests
- e2e
- xtask
- src
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
309 | 309 | | |
310 | 310 | | |
311 | 311 | | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
111 | | - | |
| 111 | + | |
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
116 | 116 | | |
117 | 117 | | |
118 | 118 | | |
119 | | - | |
| 119 | + | |
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
| |||
0 commit comments