Skip to content

Commit c16c049

Browse files
docs: document skip tag support for overlay E2E tests
Add documentation for the new tag-based test skipping feature that auto-derives --grep-invert from JOB_NAME in run-e2e.sh. Updated run-e2e reference, CI pipeline tutorial, and spec files guide. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 6da74af commit c16c049

3 files changed

Lines changed: 106 additions & 1 deletion

File tree

docs/overlay/reference/run-e2e.md

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,54 @@ A workspace is discovered when it has a `workspaces/<name>/e2e-tests/` directory
8282
|----------|-------------|---------|
8383
| `E2E_NIGHTLY_MODE` | When `true`, uses released OCI images from metadata; defaults `E2E_TEST_UTILS_VERSION` to `latest` | `false` |
8484
| `GIT_PR_NUMBER` | PR number for OCI URL generation (uses PR-built images) | - |
85-
| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection | - |
85+
| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection. Also used to [auto-derive skip tags](#skip-tags). | - |
86+
87+
## Skip Tags
88+
89+
When `JOB_NAME` is set (by OpenShift CI), the script auto-derives a Playwright tag and injects `--grep-invert` to exclude tests tagged with it. This lets test authors skip specific tests in specific CI jobs using standard Playwright tags.
90+
91+
### How It Works
92+
93+
The job suffix is extracted from `JOB_NAME` by stripping everything up to and including `-e2e-`:
94+
95+
| JOB_NAME (suffix shown) | Derived tag | `--grep-invert` |
96+
|--------------------------|-------------|-----------------|
97+
| `...-e2e-ocp-helm` | `@skip-ocp-helm` | `@skip-ocp-helm` |
98+
| `...-e2e-ocp-helm-nightly` | `@skip-ocp-helm-nightly` | `@skip-ocp-helm-nightly` |
99+
| `...-e2e-ocp-operator` | `@skip-ocp-operator` | `@skip-ocp-operator` |
100+
| `...-e2e-ocp-operator-nightly` | `@skip-ocp-operator-nightly` | `@skip-ocp-operator-nightly` |
101+
102+
If `JOB_NAME` doesn't contain `-e2e-`, no tag is derived and no filtering is applied.
103+
104+
### Tagging Tests
105+
106+
Add Playwright tags to `test.describe` or individual `test` calls:
107+
108+
```typescript
109+
// Skip in ocp-helm-nightly job
110+
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { ... });
111+
112+
// Skip in all ocp-helm jobs (PR + nightly, since regex matches as substring)
113+
test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... });
114+
115+
// Multiple tags — skip in both helm and operator nightly
116+
test.describe("Suite", {
117+
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
118+
}, () => { ... });
119+
```
120+
121+
### From a Workspace Directory
122+
123+
Skip tags work the same way when running tests from a workspace — pass `--grep-invert` manually:
124+
125+
```bash
126+
cd workspaces/tech-radar/e2e-tests
127+
yarn test -- --grep-invert "@skip-ocp-helm"
128+
```
129+
130+
::: info Precedence
131+
The auto-derived `--grep-invert` is prepended to the Playwright arguments. If you also pass `--grep-invert` on the command line, Playwright uses the last value (last wins), so your manual flag takes precedence.
132+
:::
86133

87134
## How It Works
88135

docs/overlay/test-structure/spec-files.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,33 @@ test.describe("Test <plugin>", () => {
5252
In CI, namespaces are automatically deleted after all tests complete via the built-in teardown reporter. No manual cleanup code is needed. See [Namespace Cleanup](/guide/core-concepts/playwright-fixtures#namespace-cleanup-teardown) for details.
5353
:::
5454

55+
## Tagging Tests
56+
57+
Use Playwright tags to control which tests run in which CI jobs. Tags are added via the second argument to `test.describe` or `test`:
58+
59+
```typescript
60+
// Skip this suite in the ocp-helm-nightly CI job
61+
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => {
62+
test("test case", async () => { ... });
63+
});
64+
65+
// Skip a single test in ocp-helm jobs (both PR and nightly)
66+
test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... });
67+
68+
// Multiple tags
69+
test.describe("Suite", {
70+
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
71+
}, () => { ... });
72+
```
73+
74+
When running in CI, `run-e2e.sh` auto-derives the skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright. From a workspace directory, pass it manually:
75+
76+
```bash
77+
yarn test -- --grep-invert "@skip-ocp-helm"
78+
```
79+
80+
See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full list of tags and derivation logic.
81+
5582
## Imports
5683

5784
Import test utilities from `@red-hat-developer-hub/e2e-test-utils`:

docs/overlay/tutorials/ci-pipeline.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,37 @@ To test nightly job changes via the release repo:
9393
2. Comment `/test e2e-ocp-helm-nightly` on the PR
9494
3. The rehearse job runs with `JOB_MODE=nightly`, testing all workspaces
9595

96+
## Skipping Tests by CI Job
97+
98+
Tests can opt out of specific CI jobs using Playwright tags. The `run-e2e.sh` script auto-derives a skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright.
99+
100+
For example, to skip a test in the nightly helm job:
101+
102+
```typescript
103+
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => {
104+
// This entire suite is excluded when JOB_NAME contains "e2e-ocp-helm-nightly"
105+
});
106+
```
107+
108+
### Available Tags
109+
110+
| Tag | Skipped in |
111+
|-----|-----------|
112+
| `@skip-ocp-helm` | `e2e-ocp-helm` (PR check) |
113+
| `@skip-ocp-helm-nightly` | `e2e-ocp-helm-nightly` |
114+
| `@skip-ocp-operator` | `e2e-ocp-operator` (PR check) |
115+
| `@skip-ocp-operator-nightly` | `e2e-ocp-operator-nightly` |
116+
117+
Multiple tags can be combined on a single test or describe block:
118+
119+
```typescript
120+
test.describe("Suite", {
121+
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
122+
}, () => { ... });
123+
```
124+
125+
See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full derivation logic and how to use tags from a workspace directory.
126+
96127
## PR OCI Image Builds
97128

98129
When testing a PR, the plugins need to be built into OCI images that RHDH can use. This is handled through the `/publish` command.

0 commit comments

Comments
 (0)