|
| 1 | +# /oape.api-generate-tests - Generate integration tests for API types |
| 2 | + |
| 3 | +## Purpose |
| 4 | + |
| 5 | +Generate `.testsuite.yaml` integration test files for OpenShift API type |
| 6 | +definitions. Reads Go type definitions, CRD manifests, and validation markers |
| 7 | +to produce comprehensive test suites covering create, update, validation, and |
| 8 | +error scenarios. |
| 9 | + |
| 10 | +This command should be run AFTER API types and CRD manifests have been generated. |
| 11 | + |
| 12 | +## Arguments |
| 13 | + |
| 14 | +- `$ARGUMENTS`: `<path-to-types-file-or-api-directory>` |
| 15 | + |
| 16 | +## Process |
| 17 | + |
| 18 | +### Phase 0: Prechecks |
| 19 | + |
| 20 | +All prechecks must pass before proceeding. If ANY fails, STOP immediately. |
| 21 | + |
| 22 | +#### Precheck 1 -- Verify Repository and Tools |
| 23 | + |
| 24 | +```bash |
| 25 | +if ! git rev-parse --is-inside-work-tree &> /dev/null 2>&1; then |
| 26 | + echo "PRECHECK FAILED: Not inside a git repository." |
| 27 | + exit 1 |
| 28 | +fi |
| 29 | + |
| 30 | +REPO_ROOT=$(git rev-parse --show-toplevel) |
| 31 | + |
| 32 | +if [ ! -f "$REPO_ROOT/go.mod" ]; then |
| 33 | + echo "PRECHECK FAILED: No go.mod found at repository root." |
| 34 | + exit 1 |
| 35 | +fi |
| 36 | + |
| 37 | +GO_MODULE=$(head -1 "$REPO_ROOT/go.mod" | awk '{print $2}') |
| 38 | +echo "Repository root: $REPO_ROOT" |
| 39 | +echo "Go module: $GO_MODULE" |
| 40 | +``` |
| 41 | + |
| 42 | +#### Precheck 2 -- Identify Target API Types |
| 43 | + |
| 44 | +```bash |
| 45 | +TARGET_PATH="$ARGUMENTS" |
| 46 | + |
| 47 | +if [ -z "$TARGET_PATH" ]; then |
| 48 | + echo "PRECHECK FAILED: No target path provided." |
| 49 | + echo "Usage: /oape.api-generate-tests <path-to-types-file-or-api-directory>" |
| 50 | + exit 1 |
| 51 | +fi |
| 52 | +``` |
| 53 | + |
| 54 | +```thinking |
| 55 | +Determine which API types to generate tests for: |
| 56 | +1. User provided a specific types file path -> use it directly |
| 57 | +2. User provided an API directory -> find all types files in it |
| 58 | +
|
| 59 | +Extract: API group, version, kind, resource plural, all fields with types, |
| 60 | +validation markers, and godoc. |
| 61 | +``` |
| 62 | + |
| 63 | +#### Precheck 3 -- Verify CRD Manifests Exist |
| 64 | + |
| 65 | +```bash |
| 66 | +# For openshift/api repos |
| 67 | +find "$REPO_ROOT" -type d -name 'zz_generated.crd-manifests' -not -path '*/vendor/*' | head -5 |
| 68 | + |
| 69 | +# For operator repos |
| 70 | +find "$REPO_ROOT" -type d -name 'bases' -path '*/crd/*' -not -path '*/vendor/*' | head -5 |
| 71 | +``` |
| 72 | + |
| 73 | +If no CRD manifests found: |
| 74 | + |
| 75 | +```text |
| 76 | +WARNING: No CRD manifests found. Run 'make update' or 'make manifests' first. |
| 77 | +Test suites reference CRD manifests -- tests will fail without them. |
| 78 | +``` |
| 79 | + |
| 80 | +--- |
| 81 | + |
| 82 | +### Phase 1: Read API Types and CRD Manifests |
| 83 | + |
| 84 | +Read the target Go types file(s) and extract all information needed for test |
| 85 | +generation: |
| 86 | + |
| 87 | +```thinking |
| 88 | +From the Go types, extract every field, type, marker, and validation rule: |
| 89 | +1. Top-level CRD types (structs with +kubebuilder:object:root=true or +genclient) |
| 90 | +2. For each CRD type: kind, API group, version, resource plural, scope, singleton |
| 91 | +3. Every spec and status field: name, type, optional/required, pointer semantics |
| 92 | +4. All validation markers: enums, min/max, minLength/maxLength, minItems/maxItems, |
| 93 | + pattern, format, XValidation CEL rules |
| 94 | +5. Enum types and allowed values |
| 95 | +6. Discriminated unions: discriminator field, member types |
| 96 | +7. Immutable fields (XValidation rules referencing oldSelf) |
| 97 | +8. Default values |
| 98 | +9. Feature gate annotations (+openshift:enable:FeatureGate) |
| 99 | +10. Nested object validation |
| 100 | +11. Map key/value constraints |
| 101 | +12. Any other kubebuilder or OpenShift marker |
| 102 | +
|
| 103 | +The list above is guidance, not exhaustive. Extract ALL markers found. |
| 104 | +``` |
| 105 | + |
| 106 | +Also read CRD manifest(s) to get: full CRD name, OpenAPI v3 schema, feature |
| 107 | +set annotations. |
| 108 | + |
| 109 | +### Phase 2: Identify Test Directory and Existing Tests |
| 110 | + |
| 111 | +Determine where test files should be placed: |
| 112 | + |
| 113 | +**openshift/api:** |
| 114 | + |
| 115 | +```text |
| 116 | +<group>/<version>/tests/<plural>.<group>/ |
| 117 | +``` |
| 118 | + |
| 119 | +**Operator repos:** |
| 120 | + |
| 121 | +```text |
| 122 | +api/<version>/tests/<plural>.<group>/ |
| 123 | +``` |
| 124 | + |
| 125 | +or |
| 126 | + |
| 127 | +```text |
| 128 | +api/<group>/<version>/tests/<plural>.<group>/ |
| 129 | +``` |
| 130 | + |
| 131 | +Check for existing test files to avoid duplicating tests. |
| 132 | + |
| 133 | +### Phase 3: Generate Test Suites |
| 134 | + |
| 135 | +Generate `.testsuite.yaml` files covering these categories. Derive specific test |
| 136 | +cases from the types and validation rules read in Phase 1. |
| 137 | + |
| 138 | +#### Category 1 -- Minimal Valid Create |
| 139 | + |
| 140 | +Every test suite MUST include at least one test that creates a minimal valid |
| 141 | +instance with only required fields populated. |
| 142 | + |
| 143 | +#### Category 2 -- Valid Field Values |
| 144 | + |
| 145 | +For each field in the spec: |
| 146 | + |
| 147 | +- Test that valid values are accepted and persisted correctly |
| 148 | +- For enum fields: test each allowed enum value |
| 149 | +- For optional fields: test with and without the field |
| 150 | +- For fields with defaults: verify the default is applied |
| 151 | + |
| 152 | +#### Category 3 -- Invalid Field Values (Validation Failures) |
| 153 | + |
| 154 | +For each field with validation rules: |
| 155 | + |
| 156 | +- Enum fields: test a value not in the allowed set -> `expectedError` |
| 157 | +- Pattern fields: test a value that doesn't match -> `expectedError` |
| 158 | +- Min/max constraints: test values at and beyond boundaries -> `expectedError` |
| 159 | +- Required fields: test omission -> `expectedError` |
| 160 | +- CEL validation rules: test inputs that violate each rule -> `expectedError` |
| 161 | + |
| 162 | +#### Category 4 -- Update Scenarios |
| 163 | + |
| 164 | +For fields that can be updated: |
| 165 | + |
| 166 | +- Test valid updates (change field value) -> `expected` |
| 167 | +- For immutable fields: test that updates are rejected -> `expectedError` |
| 168 | +- For fields with update-specific validation: test boundary cases |
| 169 | + |
| 170 | +#### Category 5 -- Singleton Name Validation |
| 171 | + |
| 172 | +If the CRD is a cluster-scoped singleton (name must be "cluster"): |
| 173 | + |
| 174 | +- Test creation with `resourceName: cluster` -> success |
| 175 | +- Test creation with `resourceName: not-cluster` -> `expectedError` |
| 176 | + |
| 177 | +#### Category 6 -- Discriminated Unions |
| 178 | + |
| 179 | +If the type uses discriminated unions: |
| 180 | + |
| 181 | +- Test each valid discriminator + member combination -> `expected` |
| 182 | +- Test mismatched discriminator + member -> `expectedError` |
| 183 | +- Test missing required member -> `expectedError` |
| 184 | + |
| 185 | +#### Category 7 -- Feature-Gated Fields |
| 186 | + |
| 187 | +If fields are gated behind a FeatureGate: |
| 188 | + |
| 189 | +- Stable/default test suite: setting gated field is rejected -> `expectedError` |
| 190 | +- TechPreview test suite: gated field is accepted -> `expected` |
| 191 | + |
| 192 | +#### Category 8 -- Status Subresource |
| 193 | + |
| 194 | +If the type has a status subresource: |
| 195 | + |
| 196 | +- Test valid status updates |
| 197 | +- Test invalid status updates -> `expectedStatusError` |
| 198 | + |
| 199 | +#### Category 9 -- Additional Coverage |
| 200 | + |
| 201 | +```thinking |
| 202 | +Re-examine every marker, annotation, CEL rule, godoc comment, and structural |
| 203 | +detail. Ask: is there any validation behavior or edge case NOT already covered? |
| 204 | +
|
| 205 | +Examples: cross-field dependencies, mutually exclusive fields, nested object |
| 206 | +validation, list item uniqueness, map key/value constraints, string format |
| 207 | +validations, complex CEL rules, defaulting interactions, zero values vs nil. |
| 208 | +``` |
| 209 | + |
| 210 | +### Phase 4: Write Test Suite Files |
| 211 | + |
| 212 | +Write `.testsuite.yaml` file(s) following this format: |
| 213 | + |
| 214 | +```yaml |
| 215 | +apiVersion: apiextensions.k8s.io/v1 |
| 216 | +name: "<DisplayName>" |
| 217 | +crdName: <plural>.<group> |
| 218 | +tests: |
| 219 | + onCreate: |
| 220 | + - name: Should be able to create a minimal <Kind> |
| 221 | + initial: | |
| 222 | + apiVersion: <group>/<version> |
| 223 | + kind: <Kind> |
| 224 | + spec: {} |
| 225 | + expected: | |
| 226 | + apiVersion: <group>/<version> |
| 227 | + kind: <Kind> |
| 228 | + spec: {} |
| 229 | + - name: Should reject <Kind> with invalid <fieldName> |
| 230 | + initial: | |
| 231 | + apiVersion: <group>/<version> |
| 232 | + kind: <Kind> |
| 233 | + spec: |
| 234 | + <fieldName>: <invalidValue> |
| 235 | + expectedError: "<expected error substring>" |
| 236 | + onUpdate: |
| 237 | + - name: Should not allow changing immutable field <fieldName> |
| 238 | + initial: | |
| 239 | + apiVersion: <group>/<version> |
| 240 | + kind: <Kind> |
| 241 | + spec: |
| 242 | + <fieldName>: <value1> |
| 243 | + updated: | |
| 244 | + apiVersion: <group>/<version> |
| 245 | + kind: <Kind> |
| 246 | + spec: |
| 247 | + <fieldName>: <value2> |
| 248 | + expectedError: "<expected error substring>" |
| 249 | +``` |
| 250 | +
|
| 251 | +#### File Naming Conventions |
| 252 | +
|
| 253 | +Derive from existing patterns: |
| 254 | +
|
| 255 | +**openshift/api repos:** |
| 256 | +
|
| 257 | +- `stable.<kind>.testsuite.yaml` |
| 258 | +- `techpreview.<kind>.testsuite.yaml` |
| 259 | +- `stable.<kind>.<context>.testsuite.yaml` |
| 260 | + |
| 261 | +**Operator repos:** |
| 262 | + |
| 263 | +- `<kind>.testsuite.yaml` |
| 264 | + |
| 265 | +### Phase 5: Output Summary |
| 266 | + |
| 267 | +Write `artifacts/operator-feature-dev/api/test-generation-summary.md`: |
| 268 | + |
| 269 | +```text |
| 270 | +=== API Test Generation Summary === |
| 271 | +
|
| 272 | +Target API: <group>/<version> <Kind> |
| 273 | +CRD Name: <plural>.<group> |
| 274 | +
|
| 275 | +Generated Test Files: |
| 276 | + - <path/to/testsuite.yaml> -- <N> onCreate tests, <N> onUpdate tests |
| 277 | +
|
| 278 | +Test Coverage: |
| 279 | + onCreate: |
| 280 | + - Minimal valid create |
| 281 | + - <field>: valid values (<count> tests) |
| 282 | + - <field>: invalid values (<count> tests) |
| 283 | + - Singleton name validation |
| 284 | + onUpdate: |
| 285 | + - Immutable field <field>: rejected |
| 286 | + - Valid update for <field> |
| 287 | +
|
| 288 | +Next Steps: |
| 289 | + 1. Review the generated test suites |
| 290 | + 2. Run the integration tests |
| 291 | + 3. Verify coverage: make verify |
| 292 | + 4. Add additional edge-case tests as needed |
| 293 | +``` |
| 294 | + |
| 295 | +## Behavioral Rules |
| 296 | + |
| 297 | +1. **Derive from source**: All test values and expectations MUST come from |
| 298 | + actual Go types, validation markers, and CRD manifests. |
| 299 | +2. **Match existing style**: If the repo has test suites, match their naming, |
| 300 | + formatting, and detail level exactly. |
| 301 | +3. **Comprehensive but focused**: Test every field and validation rule found, |
| 302 | + but don't invent scenarios not supported by the schema. |
| 303 | +4. **Error messages**: For `expectedError` fields, use substrings from the |
| 304 | + actual CRD validation rules. |
| 305 | +5. **Minimal YAML**: Include only fields relevant to each specific test case. |
| 306 | +6. **Surgical additions**: When adding to an existing suite, preserve all |
| 307 | + existing tests and append new ones. |
| 308 | + |
| 309 | +## Output |
| 310 | + |
| 311 | +- Generated `.testsuite.yaml` files in the appropriate test directory |
| 312 | +- Summary saved to `artifacts/operator-feature-dev/api/test-generation-summary.md` |
0 commit comments