Skip to content

Commit 25df36b

Browse files
openshift-merge-bot[bot]openshift-merge-bot[bot]
authored
Merge pull request #5864 from HarshwardhanPatil07/automate-test-skill
MCO-2222: /automate-test skill for generating e2e tests from specifications
2 parents 384e6fa + 053ba38 commit 25df36b

2 files changed

Lines changed: 190 additions & 0 deletions

File tree

‎.claude/commands/automate-test.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
description: Automate MCO test creation from test specifications, learning from previous code reviews
3+
argument-hint: <spec-source> "<suite-name>" [longduration|disruptive]
4+
---
5+
6+
Generate new MCO e2e test cases in `test/extended-priv/` from test specifications.
7+
8+
Use the conventions in `.claude/skills/mco-automate-test-workflow.md` for all test generation rules.
9+
10+
**Input modes** (determined by `<spec-source>` format):
11+
- **Text file**: a file path containing the test specification
12+
- **Polarion ID** (future, MCO-2220): a numeric ID like `OCP-12345`
13+
- **Jira ID** (future, MCO-2221): a Jira key like `MCO-1234`
14+
15+
**Workflow:**
16+
1. Parse arguments. Read the spec file and extract test case ID, title, preconditions, steps, expected results, and tags. Derive target file from suite name (`"mco security"` -> `mco_security.go`).
17+
2. Learn from previous code reviews — check memory (`review_patterns_mco.md`), fetch new patterns from merged PRs via `gh api` on `openshift/machine-config-operator`.
18+
3. Read existing tests and utilities in `test/extended-priv/` to understand available helpers and patterns.
19+
4. Generate the test code following all conventions from the workflow skill.
20+
5. Build with `make machine-config-tests-ext` and verify the test appears in listing. Fix any build errors.

‎.claude/skills/mco-automate-test-workflow.md‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
---
2+
name: MCO Automate Test Workflow
3+
description: Conventions and workflow for generating new MCO e2e tests from test specifications
4+
---
5+
6+
# MCO Test Automation Rules
7+
8+
Generate new e2e test code in `test/extended-priv/` from test specifications.
9+
10+
## Input Detection
11+
12+
- **Text file**: path exists on disk -> read and parse it for test case ID, title, preconditions, steps, expected results, tags
13+
- **Polarion ID**: matches `OCP-\d+` or purely numeric -> future (MCO-2220)
14+
- **Jira ID**: matches `MCO-\d+` -> future (MCO-2221)
15+
16+
Only text file input is supported now.
17+
18+
## Target File Resolution
19+
20+
Derive from `<suite-name>`: `"mco security"` -> `mco_security.go`
21+
- If `test/extended-priv/<filename>` exists: add new `g.It()` block to it
22+
- If it doesn't exist: create new file with full test structure
23+
24+
## Code Review Learning
25+
26+
Learn from previous code reviews to produce review-ready code. This is cumulative — patterns are stored in memory (`review_patterns_mco.md`) and grow with every run.
27+
28+
- Load existing patterns from memory, note the `Last Analyzed PR` as high-water mark
29+
- Fetch merged PRs that modified `test/extended-priv/` via `gh`, only newer than last analyzed
30+
- For each PR, fetch inline review comments and PR-level reviews via `gh api`
31+
- Extract actionable coding standards (naming, resource handling, error handling, code structure, common mistakes)
32+
- Deduplicate against existing patterns. Newer patterns win on contradiction
33+
- Save updated patterns to `review_patterns_mco.md` memory
34+
- Fall back to built-in conventions below if `gh` is unavailable
35+
36+
## Codebase Context
37+
38+
Before generating, read existing code to understand available utilities and patterns:
39+
40+
- The target file (if existing) — its Describe block, JustBeforeEach, variables, existing tests
41+
- 2-3 similar test files based on topic
42+
- Resource wrappers: `machineconfig.go`, `machineconfigpool.go`, `node.go`, etc.
43+
- Helpers: `util.go`, `resource.go`, `remotefile.go`, `gomega_matchers.go`
44+
- Testdata templates in `test/extended-priv/testdata/files/`
45+
46+
## Conventions
47+
48+
1. **Use GetCompactCompatiblePool** for MCP selection, unless the test requires the master pool
49+
2. **Call GetCurrentTestPolarionIDNumber()** once per test
50+
3. **Use Generic MC template** when possible
51+
4. **Base64 encode** file configuration in MachineConfigs
52+
5. **Use test number** in file paths, content, and resource names for uniqueness
53+
6. **Node selection**: `GetSortedNodesOrFail()[0]`
54+
7. **Step logging**: `exutil.By("<step>")` for each step, `logger.Infof("OK!\n")` after success
55+
8. **Declare variables** in the `var` block when possible
56+
9. **Error messages**: log the full resource with `%s`, not just `.GetName()`
57+
10. **State recovery**: always recover initial state via defer (e.g., `defer resource.DeleteWithWait()`)
58+
11. **Prefer Resource struct methods** over `oc.AsAdmin().Run`
59+
12. **RemoteFile** with gomega checkers for verifying files inside nodes
60+
13. **Concise comments** — one or two lines maximum
61+
14. **Skip on SNO/Compact** when the test requires multiple nodes or a dedicated worker pool: `if IsCompactOrSNOCluster(oc.AsAdmin()) { g.Skip("...") }` or `exutil.SkipOnSingleNodeTopology(oc.AsAdmin())`
62+
63+
## New File Structure
64+
65+
```go
66+
package extended
67+
68+
import (
69+
g "github.com/onsi/ginkgo/v2"
70+
o "github.com/onsi/gomega"
71+
exutil "github.com/openshift/machine-config-operator/test/extended-priv/util"
72+
logger "github.com/openshift/machine-config-operator/test/extended-priv/util/logext"
73+
)
74+
75+
var _ = g.Describe("[sig-mco][Suite:openshift/machine-config-operator/<suite-type>][Serial][Disruptive] MCO <suite-name>", func() {
76+
defer g.GinkgoRecover()
77+
78+
var oc = exutil.NewCLI("mco-<cli-name>", exutil.KubeConfigPath())
79+
80+
g.JustBeforeEach(func() {
81+
PreChecks(oc)
82+
})
83+
84+
g.It("[PolarionID:<id>][OTP] <description> [Disruptive]", func() {
85+
// test body
86+
})
87+
})
88+
```
89+
90+
- `[Serial][Disruptive]` is **always required** on Describe blocks
91+
- `<cli-name>`: suite name hyphenated (e.g., `mco-security`)
92+
93+
## Existing File Insertion
94+
95+
Insert new `g.It` block inside the existing `g.Describe`, after the last test. Add new imports as needed. Do not modify existing code.
96+
97+
If the test requires new helper functions, append them **after all existing functions** at the end of the file — never insert between existing functions. Helper functions belong outside the `g.Describe` block, after its closing `})`.
98+
99+
## Test Body Pattern
100+
101+
```go
102+
g.It("[PolarionID:<id>][OTP] <description> [Disruptive]", g.Label("Platform:aws"), func() {
103+
testID := GetCurrentTestPolarionIDNumber()
104+
mcp := GetCompactCompatiblePool(oc.AsAdmin())
105+
node := mcp.GetSortedNodesOrFail()[0]
106+
107+
exutil.By("<step 1 from spec>")
108+
// implementation
109+
logger.Infof("OK!\n")
110+
111+
exutil.By("<step 2 from spec>")
112+
// implementation
113+
logger.Infof("OK!\n")
114+
})
115+
```
116+
117+
## Generation Rules
118+
119+
- Each spec step becomes an `exutil.By()` block
120+
- Use existing helpers — prefer utility functions over raw oc commands
121+
- Defer cleanup immediately after resource creation
122+
- Resource names include test ID: `fmt.Sprintf("test-%s-mc", testID)`
123+
- Node verification: `NewRemoteFile(node, path)` with gomega matchers
124+
- Pool waiting after changes: `mcp.waitForComplete()`
125+
- Platform labels: `g.Label("Platform:aws", "Platform:gce")` based on spec tags
126+
- Feature gate labels: `g.Label("OCPFeatureGate:XXX")` if the spec requires it
127+
- Skip functions for platform/architecture: `skipTestIfSupportedPlatformNotMatched(oc, AWSPlatform, GCPPlatform)`, `architecture.SkipNonAmd64SingleArch(oc)`
128+
- Skip on SNO/Compact: if the test needs more than one node or a dedicated worker pool, add `if IsCompactOrSNOCluster(oc.AsAdmin()) { g.Skip("This test requires multiple nodes and cannot run on SNO/Compact clusters") }` at the start of the test body
129+
- If the test requires new YAML templates, create them in `test/extended-priv/testdata/files/` and reference with `SetMCOTemplate("<name>.yaml")`
130+
131+
## Post-Generation Verification
132+
133+
After generating the test code, verify completeness:
134+
135+
1. Extract every numbered step, precondition check, and expected result from the specification
136+
2. Map each to an `exutil.By()` block in the generated code
137+
3. Confirm 1:1 coverage — every spec step must have a corresponding `exutil.By()` block
138+
4. Report any missing steps and add the missing `exutil.By()` blocks before finalizing
139+
140+
## Platform Constants
141+
142+
| Platform | Constant | g.Label Value |
143+
|---|---|---|
144+
| AWS | `AWSPlatform` | `"Platform:aws"` |
145+
| GCP | `GCPPlatform` | `"Platform:gce"` |
146+
| Azure | `AzurePlatform` | `"Platform:azure"` |
147+
| vSphere | `VspherePlatform` | `"Platform:vsphere"` |
148+
149+
## Suite Types
150+
151+
| Type | Describe Annotation |
152+
|---|---|
153+
| longduration | `[sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive]` |
154+
| disruptive | `[sig-mco][Suite:openshift/machine-config-operator/disruptive][Serial][Disruptive]` |
155+
156+
## Path Mappings
157+
158+
| What | Path |
159+
|---|---|
160+
| Test files | `test/extended-priv/mco_<feature>.go` |
161+
| Testdata/templates | `test/extended-priv/testdata/files/` |
162+
| Utility functions | `test/extended-priv/util/` |
163+
| Resource wrappers | `test/extended-priv/*.go` |
164+
165+
## Build & Verify
166+
167+
```bash
168+
make machine-config-tests-ext
169+
./_output/linux/amd64/machine-config-tests-ext list | grep <PolarionID>
170+
```

0 commit comments

Comments
 (0)