Skip to content

Commit 922463d

Browse files
subhashkhilericlaude
subhashkhileri
and
claude
authored
docs: document skip tag support for overlay E2E tests (#88)
* docs: document skip tag support for overlay E2E tests Add documentation for the new tag-based test skipping feature that auto-derives --grep-invert from JOB_NAME in run-e2e.sh. Updated run-e2e reference, CI pipeline tutorial, and spec files guide. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update skip tag docs to reflect exact matching with lookahead Tags now match exactly — @skip-ocp-helm won't filter @skip-ocp-helm-nightly. Updated tables and examples to clarify per-job isolation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add skipping and disabling tests guidance --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 6da74af commit 922463d

3 files changed

Lines changed: 116 additions & 1 deletion

File tree

‎docs/overlay/reference/run-e2e.md‎

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,54 @@ A workspace is discovered when it has a `workspaces/<name>/e2e-tests/` directory
8282
|----------|-------------|---------|
8383
| `E2E_NIGHTLY_MODE` | When `true`, uses released OCI images from metadata; defaults `E2E_TEST_UTILS_VERSION` to `latest` | `false` |
8484
| `GIT_PR_NUMBER` | PR number for OCI URL generation (uses PR-built images) | - |
85-
| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection | - |
85+
| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection. Also used to [auto-derive skip tags](#skip-tags). | - |
86+
87+
## Skip Tags
88+
89+
When `JOB_NAME` is set (by OpenShift CI), the script auto-derives a Playwright tag and injects `--grep-invert` to exclude tests tagged with it. This lets test authors skip specific tests in specific CI jobs using standard Playwright tags.
90+
91+
### How It Works
92+
93+
The job suffix is extracted from `JOB_NAME` by stripping everything up to and including `-e2e-`. A negative lookahead `(?!-)` is appended so each tag matches exactly — `@skip-ocp-helm` won't accidentally filter `@skip-ocp-helm-nightly`:
94+
95+
| JOB_NAME (suffix shown) | `--grep-invert` pattern |
96+
|--------------------------|-------------------------|
97+
| `...-e2e-ocp-helm` | `@skip-ocp-helm(?!-)` |
98+
| `...-e2e-ocp-helm-nightly` | `@skip-ocp-helm-nightly(?!-)` |
99+
| `...-e2e-ocp-operator` | `@skip-ocp-operator(?!-)` |
100+
| `...-e2e-ocp-operator-nightly` | `@skip-ocp-operator-nightly(?!-)` |
101+
102+
If `JOB_NAME` doesn't contain `-e2e-`, no tag is derived and no filtering is applied.
103+
104+
### Tagging Tests
105+
106+
Add Playwright tags to `test.describe` or individual `test` calls:
107+
108+
```typescript
109+
// Skip only in ocp-helm-nightly job
110+
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { ... });
111+
112+
// Skip only in ocp-helm PR check job (won't affect nightly)
113+
test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... });
114+
115+
// Multiple tags — skip in both helm and operator nightly
116+
test.describe("Suite", {
117+
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
118+
}, () => { ... });
119+
```
120+
121+
### From a Workspace Directory
122+
123+
Skip tags work the same way when running tests from a workspace — pass `--grep-invert` manually:
124+
125+
```bash
126+
cd workspaces/tech-radar/e2e-tests
127+
yarn test -- --grep-invert "@skip-ocp-helm"
128+
```
129+
130+
::: info Precedence
131+
The auto-derived `--grep-invert` is prepended to the Playwright arguments. If you also pass `--grep-invert` on the command line, Playwright uses the last value (last wins), so your manual flag takes precedence.
132+
:::
86133

87134
## How It Works
88135

‎docs/overlay/test-structure/spec-files.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,41 @@ test.describe("Test <plugin>", () => {
5252
In CI, namespaces are automatically deleted after all tests complete via the built-in teardown reporter. No manual cleanup code is needed. See [Namespace Cleanup](/guide/core-concepts/playwright-fixtures#namespace-cleanup-teardown) for details.
5353
:::
5454

55+
## Skipping and Disabling Tests
56+
57+
| Mechanism | When to use | Report |
58+
|-----------|-------------|--------|
59+
| `{ tag: "@skip-ocp-helm" }` | Test works, but skip in a specific CI job | Hidden from that job and report |
60+
| `test.fixme()` | Test is broken or flaky, needs fixing | Shows as "skipped" everywhere |
61+
| `test.skip(condition, reason)` | Skip based on runtime (e.g., missing secret) | Shows as "skipped" with reason |
62+
63+
### Tagging Tests
64+
65+
Use Playwright tags to control which tests run in which CI jobs. Tags are added via the second argument to `test.describe` or `test`:
66+
67+
```typescript
68+
// Skip this suite in the ocp-helm-nightly CI job
69+
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => {
70+
test("test case", async () => { ... });
71+
});
72+
73+
// Skip a single test in ocp-helm jobs (both PR and nightly)
74+
test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... });
75+
76+
// Multiple tags
77+
test.describe("Suite", {
78+
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
79+
}, () => { ... });
80+
```
81+
82+
When running in CI, `run-e2e.sh` auto-derives the skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright. From a workspace directory, pass it manually:
83+
84+
```bash
85+
yarn test -- --grep-invert "@skip-ocp-helm"
86+
```
87+
88+
See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full list of tags and derivation logic.
89+
5590
## Imports
5691

5792
Import test utilities from `@red-hat-developer-hub/e2e-test-utils`:

‎docs/overlay/tutorials/ci-pipeline.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,39 @@ To test nightly job changes via the release repo:
9393
2. Comment `/test e2e-ocp-helm-nightly` on the PR
9494
3. The rehearse job runs with `JOB_MODE=nightly`, testing all workspaces
9595

96+
## Skipping Tests by CI Job
97+
98+
Tests can opt out of specific CI jobs using Playwright tags. The `run-e2e.sh` script auto-derives a skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright.
99+
100+
For example, to skip a test in the nightly helm job:
101+
102+
```typescript
103+
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => {
104+
// This entire suite is excluded when JOB_NAME contains "e2e-ocp-helm-nightly"
105+
});
106+
```
107+
108+
### Available Tags
109+
110+
| Tag | Skipped in | Not skipped in |
111+
|-----|-----------|----------------|
112+
| `@skip-ocp-helm` | `e2e-ocp-helm` (PR check) | `e2e-ocp-helm-nightly` |
113+
| `@skip-ocp-helm-nightly` | `e2e-ocp-helm-nightly` | `e2e-ocp-helm` (PR check) |
114+
| `@skip-ocp-operator` | `e2e-ocp-operator` (PR check) | `e2e-ocp-operator-nightly` |
115+
| `@skip-ocp-operator-nightly` | `e2e-ocp-operator-nightly` | `e2e-ocp-operator` (PR check) |
116+
117+
Each tag matches exactly — `@skip-ocp-helm` won't accidentally skip tests in the nightly job.
118+
119+
Multiple tags can be combined on a single test or describe block:
120+
121+
```typescript
122+
test.describe("Suite", {
123+
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
124+
}, () => { ... });
125+
```
126+
127+
See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full derivation logic and how to use tags from a workspace directory.
128+
96129
## PR OCI Image Builds
97130

98131
When testing a PR, the plugins need to be built into OCI images that RHDH can use. This is handled through the `/publish` command.

0 commit comments

Comments
 (0)