You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- Add license headers to all Python and shell files
- Fix ruff import ordering and formatting
- Fix shellcheck warnings (brace variable references)
- Remove hard line wraps in README and scenarios prose
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Copy file name to clipboardExpand all lines: tests/e2e/README.md
+21-60Lines changed: 21 additions & 60 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -34,60 +34,37 @@ Behavioral end-to-end tests for rocm-cli using Robot Framework.
34
34
└─────────────────┘
35
35
```
36
36
37
-
Tests run the real `rocm` binary via subprocess, the same way a user would.
38
-
Keyword libraries parse CLI output and make HTTP requests to served endpoints.
39
-
No imports from the rocm-cli codebase.
37
+
Tests run the real `rocm` binary via subprocess, the same way a user would. Keyword libraries parse CLI output and make HTTP requests to served endpoints. No imports from the rocm-cli codebase.
40
38
41
39
## Why a separate test layer
42
40
43
41
The existing test infrastructure covers two levels:
44
42
45
-
-**Rust unit and integration tests** (`cargo test --workspace`) — pure,
46
-
deterministic, no GPU or runtime needed. These validate individual components
`scripts/*_therock_gpu_test.py`) — zero-dependency, stdlib-only scripts that
50
-
verify CLI behavior against real or fake binaries. These run as linear scripts
51
-
with basic substring assertions.
43
+
-**Rust unit and integration tests** (`cargo test --workspace`) — pure, deterministic, no GPU or runtime needed. These validate individual components in isolation with fixtures and mocks.
44
+
-**Python smoke scripts** (`scripts/smoke_local.py`, `scripts/*_therock_gpu_test.py`) — zero-dependency, stdlib-only scripts that verify CLI behavior against real or fake binaries. These run as linear scripts with basic substring assertions.
52
45
53
-
Both layers are valuable but leave a gap: **cross-component user journeys are
54
-
not tested.** Each component works individually, but the seams between them
55
-
break in practice. For example:
46
+
Both layers are valuable but leave a gap: **cross-component user journeys are not tested.** Each component works individually, but the seams between them break in practice. For example:
56
47
57
-
- Model alias resolution works in the recipe registry but is not forwarded
58
-
correctly to all engine adapters.
59
-
-`rocm serve` binds to port 11435 but the TUI chat detection probes ports
60
-
8000 and 13305.
61
-
- The managed-services registry tracks a running server with its real model
62
-
name, but the TUI chat falls back to a hardcoded `local-model` default.
48
+
- Model alias resolution works in the recipe registry but is not forwarded correctly to all engine adapters.
49
+
-`rocm serve` binds to port 11435 but the TUI chat detection probes ports 8000 and 13305.
50
+
- The managed-services registry tracks a running server with its real model name, but the TUI chat falls back to a hardcoded `local-model` default.
63
51
64
-
These are integration failures that only surface when exercising the full user
These are integration failures that only surface when exercising the full user flow: install → examine → configure runtime → serve model → detect → chat.
66
53
67
54
## Why Robot Framework
68
55
69
-
The product scripts (`scripts/*.py`) are deliberately zero-dependency to run on
70
-
user machines and in minimal CI environments. E2E tests have a different
71
-
audience — they run in developer and CI environments where installing packages
72
-
is expected. Keeping the two concerns separate avoids compromising either:
56
+
The product scripts (`scripts/*.py`) are deliberately zero-dependency to run on user machines and in minimal CI environments. E2E tests have a different audience — they run in developer and CI environments where installing packages is expected. Keeping the two concerns separate avoids compromising either:
73
57
74
58
- Product scripts stay zero-dependency and stdlib-only.
75
-
- E2E tests get proper test infrastructure without reinventing scenario
76
-
discovery, tagging, parameterization, setup/teardown lifecycle, and
77
-
structured reporting.
59
+
- E2E tests get proper test infrastructure without reinventing scenario discovery, tagging, parameterization, setup/teardown lifecycle, and structured reporting.
78
60
79
61
Robot Framework was chosen because:
80
62
81
-
-**Keyword-driven syntax** — `.robot` files are readable without knowing
82
-
Python, making scenarios accessible to anyone reviewing test coverage.
83
-
-**Built-in tagging** — scenarios can be tagged `gpu`, `mock`, `smoke`, etc.
84
-
and selectively run in different CI environments.
85
-
-**Setup/teardown lifecycle** — suite and test-level setup and teardown handle
86
-
isolated config directories, server startup/shutdown, and cleanup.
87
-
-**Structured reporting** — HTML reports show which scenario failed, at which
88
-
step, with full output, without custom reporting code.
89
-
-**Step reuse** — shared keywords across scenarios without building a custom
90
-
framework.
63
+
-**Keyword-driven syntax** — `.robot` files are readable without knowing Python, making scenarios accessible to anyone reviewing test coverage.
64
+
-**Built-in tagging** — scenarios can be tagged `gpu`, `mock`, `smoke`, etc. and selectively run in different CI environments.
65
+
-**Setup/teardown lifecycle** — suite and test-level setup and teardown handle isolated config directories, server startup/shutdown, and cleanup.
66
+
-**Structured reporting** — HTML reports show which scenario failed, at which step, with full output, without custom reporting code.
67
+
-**Step reuse** — shared keywords across scenarios without building a custom framework.
Scenarios are written in BDD style (Given/When/Then). The `.robot` file is a
179
-
near-verbatim copy of the behavioral scenario. The keyword library translates
180
-
the behavioral language into technical actions.
155
+
Scenarios are written in BDD style (Given/When/Then). The `.robot` file is a near-verbatim copy of the behavioral scenario. The keyword library translates the behavioral language into technical actions.
-**Platform-agnostic scenarios.** The same behavioral expectations apply
230
-
across MI300X, RDNA 3, and WSL. Hardware determines which GPU is detected or
231
-
which engine is auto-selected, but the contract is the same: "examine detects
232
-
a GPU," "serve makes a model reachable," "chat connects and gets a response."
233
-
234
-
-**Real binary, real hardware.** Tests run the actual `rocm` binary, not a
235
-
mock. On GPU runners they exercise real serving and inference. On CPU-only
236
-
runners they verify graceful failure and non-GPU paths.
237
-
238
-
-**Isolated state.** Each suite uses isolated `ROCM_CLI_CONFIG_DIR`,
239
-
`ROCM_CLI_DATA_DIR`, and `ROCM_CLI_CACHE_DIR` directories (following the
240
-
pattern established by `smoke_local.py`). Tests never touch the user's real
241
-
`~/.rocm` state.
242
-
243
-
-**Black-box only.** Keywords interact with `rocm` through its CLI and HTTP
244
-
endpoints. No imports from the rocm-cli Rust codebase or internal Python
245
-
modules. Tests verify observable behavior, not implementation details.
204
+
-**Platform-agnostic scenarios.** The same behavioral expectations apply across MI300X, RDNA 3, and WSL. Hardware determines which GPU is detected or which engine is auto-selected, but the contract is the same: "examine detects a GPU," "serve makes a model reachable," "chat connects and gets a response."
205
+
-**Real binary, real hardware.** Tests run the actual `rocm` binary, not a mock. On GPU runners they exercise real serving and inference. On CPU-only runners they verify graceful failure and non-GPU paths.
206
+
-**Isolated state.** Each suite uses isolated `ROCM_CLI_CONFIG_DIR`, `ROCM_CLI_DATA_DIR`, and `ROCM_CLI_CACHE_DIR` directories (following the pattern established by `smoke_local.py`). Tests never touch the user's real `~/.rocm` state.
207
+
-**Black-box only.** Keywords interact with `rocm` through its CLI and HTTP endpoints. No imports from the rocm-cli Rust codebase or internal Python modules. Tests verify observable behavior, not implementation details.
246
208
247
209
## Relationship to existing tests
248
210
@@ -255,8 +217,7 @@ This layer does **not** replace existing tests:
255
217
|`*_therock_gpu_test.py`| Per-engine GPU validation | Hardware CI (planned) |
256
218
|**`tests/e2e/`**| Cross-component user journeys | Hardware CI + mock subset on hosted CI |
257
219
258
-
The e2e suites are the only layer that tests the seams between components —
259
-
where most user-facing bugs live.
220
+
The e2e suites are the only layer that tests the seams between components — where most user-facing bugs live.
0 commit comments