|
| 1 | +--- |
| 2 | +description: Run systematic QA testing against the implemented feature, validating acceptance criteria through browser-driven or CLI-based testing. |
| 3 | +scripts: |
| 4 | + sh: scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks |
| 5 | + ps: scripts/powershell/check-prerequisites.ps1 -Json -RequireTasks -IncludeTasks |
| 6 | +--- |
| 7 | + |
| 8 | +## User Input |
| 9 | + |
| 10 | +```text |
| 11 | +$ARGUMENTS |
| 12 | +``` |
| 13 | + |
| 14 | +You **MUST** consider the user input before proceeding (if not empty). |
| 15 | + |
| 16 | +## Pre-Execution Checks |
| 17 | + |
| 18 | +**Check for extension hooks (before QA)**: |
| 19 | +- Check if `.specify/extensions.yml` exists in the project root. |
| 20 | +- If it exists, read it and look for entries under the `hooks.before_qa` key |
| 21 | +- If the YAML cannot be parsed or is invalid, skip hook checking silently and continue normally |
| 22 | +- Filter out hooks where `enabled` is explicitly `false`. Treat hooks without an `enabled` field as enabled by default. |
| 23 | +- For each remaining hook, do **not** attempt to interpret or evaluate hook `condition` expressions: |
| 24 | + - If the hook has no `condition` field, or it is null/empty, treat the hook as executable |
| 25 | + - If the hook defines a non-empty `condition`, skip the hook and leave condition evaluation to the HookExecutor implementation |
| 26 | +- For each executable hook, output the following based on its `optional` flag: |
| 27 | + - **Optional hook** (`optional: true`): |
| 28 | + ``` |
| 29 | + ## Extension Hooks |
| 30 | +
|
| 31 | + **Optional Pre-Hook**: {extension} |
| 32 | + Command: `/{command}` |
| 33 | + Description: {description} |
| 34 | +
|
| 35 | + Prompt: {prompt} |
| 36 | + To execute: `/{command}` |
| 37 | + ``` |
| 38 | + - **Mandatory hook** (`optional: false`): |
| 39 | + ``` |
| 40 | + ## Extension Hooks |
| 41 | +
|
| 42 | + **Automatic Pre-Hook**: {extension} |
| 43 | + Executing: `/{command}` |
| 44 | + EXECUTE_COMMAND: {command} |
| 45 | +
|
| 46 | + Wait for the result of the hook command before proceeding to the Outline. |
| 47 | + ``` |
| 48 | +- If no hooks are registered or `.specify/extensions.yml` does not exist, skip silently |
| 49 | +
|
| 50 | +## Goal |
| 51 | +
|
| 52 | +Perform systematic quality assurance testing of the implemented feature by validating acceptance criteria from the specification against actual application behavior. Supports two modes: **Browser QA** for web applications (using Playwright or similar browser automation) and **CLI QA** for non-web applications (using test runners, API calls, and command-line validation). |
| 53 | +
|
| 54 | +## Operating Constraints |
| 55 | +
|
| 56 | +**NON-DESTRUCTIVE**: QA testing should not corrupt production data or leave the application in a broken state. Use test databases, test accounts, and cleanup procedures where applicable. |
| 57 | +
|
| 58 | +**Evidence-Based**: Every pass/fail determination must include evidence (screenshots, response payloads, console output, or test results). |
| 59 | +
|
| 60 | +## Outline |
| 61 | +
|
| 62 | +1. Run `{SCRIPT}` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot"). |
| 63 | +
|
| 64 | +2. **Load QA Context**: |
| 65 | + - **REQUIRED**: Read `spec.md` for acceptance criteria, user stories, and success criteria |
| 66 | + - **REQUIRED**: Read `tasks.md` to identify implemented features and affected areas |
| 67 | + - **IF EXISTS**: Read `plan.md` for technical details, routes, and API endpoints |
| 68 | + - **IF EXISTS**: Read review reports in FEATURE_DIR/reviews/ for known issues to verify |
| 69 | + - **IF EXISTS**: Read `/memory/constitution.md` for quality standards |
| 70 | +
|
| 71 | +3. **Extract Test Scenarios**: |
| 72 | + From the loaded artifacts, build a structured test plan: |
| 73 | + - Map each user story to one or more test scenarios |
| 74 | + - Map each acceptance criterion to a verifiable test case |
| 75 | + - Identify happy paths, error paths, and edge cases |
| 76 | + - Prioritize scenarios: critical user flows → error handling → edge cases → performance |
| 77 | +
|
| 78 | + Output the test plan as a numbered list: |
| 79 | + ``` |
| 80 | + QA Test Plan: |
| 81 | + TC-001: [User Story X] - [Scenario description] - [Expected outcome] |
| 82 | + TC-002: [User Story Y] - [Scenario description] - [Expected outcome] |
| 83 | + ... |
| 84 | + ``` |
| 85 | +
|
| 86 | +4. **Detect QA Mode**: |
| 87 | + Determine the appropriate testing approach based on the project: |
| 88 | +
|
| 89 | + **Browser QA Mode** (for web applications): |
| 90 | + - Detect if the project is a web application (check for: package.json with dev/start scripts, index.html, web framework in plan.md) |
| 91 | + - Check for browser automation tools: Playwright, Puppeteer, Cypress, Selenium |
| 92 | + - If available, use browser automation for UI testing |
| 93 | + - If not available but project is a web app, use `curl`/`fetch` for API-level testing |
| 94 | +
|
| 95 | + **CLI QA Mode** (for non-web applications): |
| 96 | + - Use the project's existing test runner (npm test, pytest, go test, cargo test, etc.) |
| 97 | + - Execute CLI commands and validate output |
| 98 | + - Use API calls for service validation |
| 99 | + - Check database state for data integrity |
| 100 | +
|
| 101 | +5. **Environment Setup**: |
| 102 | + - Attempt to start the application if it's not already running: |
| 103 | + - Check for common start commands: `npm run dev`, `npm start`, `python manage.py runserver`, `go run .`, `cargo run`, etc. |
| 104 | + - Use the dev/start command from `plan.md` if specified |
| 105 | + - Wait for the application to be responsive (health check endpoint or port availability) |
| 106 | + - If the application cannot be started, fall back to running the existing test suite |
| 107 | + - Create the QA output directories: |
| 108 | + - `FEATURE_DIR/qa/` for reports |
| 109 | + - `FEATURE_DIR/qa/screenshots/` for visual evidence (browser mode) |
| 110 | + - `FEATURE_DIR/qa/responses/` for API response captures (CLI mode) |
| 111 | +
|
| 112 | +6. **Execute Test Scenarios — Browser QA Mode**: |
| 113 | + For each test scenario in the plan: |
| 114 | + - Navigate to the relevant route/page |
| 115 | + - Perform the user actions described in the scenario |
| 116 | + - Capture a screenshot at each key state transition |
| 117 | + - Validate the expected outcome: |
| 118 | + - UI element presence/absence |
| 119 | + - Text content verification |
| 120 | + - Form submission results |
| 121 | + - Navigation behavior |
| 122 | + - Error message display |
| 123 | + - Record the result: ✅ PASS, ❌ FAIL, ⚠️ PARTIAL, ⏭️ SKIPPED |
| 124 | + - For failures: capture the screenshot, console errors, and network errors |
| 125 | + - For partial passes: document what worked and what didn't |
| 126 | +
|
| 127 | +7. **Execute Test Scenarios — CLI QA Mode**: |
| 128 | + For each test scenario in the plan: |
| 129 | + - Run the appropriate command or API call |
| 130 | + - Capture stdout, stderr, and exit codes |
| 131 | + - Validate the expected outcome: |
| 132 | + - Command output matches expected patterns |
| 133 | + - Exit codes are correct (0 for success, non-zero for expected errors) |
| 134 | + - API responses match expected schemas and status codes |
| 135 | + - Database state reflects expected changes |
| 136 | + - File system changes are correct |
| 137 | + - Record the result: ✅ PASS, ❌ FAIL, ⚠️ PARTIAL, ⏭️ SKIPPED |
| 138 | + - For failures: capture full output, error messages, and stack traces |
| 139 | +
|
| 140 | +8. **Run Existing Test Suites**: |
| 141 | + In addition to scenario-based testing, run the project's existing test suites: |
| 142 | + - Detect test runner: `npm test`, `pytest`, `go test ./...`, `cargo test`, `dotnet test`, `mvn test`, etc. |
| 143 | + - Run the full test suite and capture results |
| 144 | + - Report: total tests, passed, failed, skipped, coverage percentage (if available) |
| 145 | + - Flag any pre-existing test failures vs. new failures from implementation changes |
| 146 | +
|
| 147 | +9. **Generate QA Report**: |
| 148 | + Create the QA report at `FEATURE_DIR/qa/qa-{timestamp}.md` using the QA report template. The report must include: |
| 149 | +
|
| 150 | + - **QA Summary**: Overall verdict (✅ ALL PASSED / ⚠️ PARTIAL PASS / ❌ FAILURES FOUND) |
| 151 | + - **Test Results Table**: Each scenario with ID, description, mode, result, evidence link |
| 152 | + - **Acceptance Criteria Coverage**: Matrix of criteria vs. test status |
| 153 | + - **Test Suite Results**: Existing test suite pass/fail summary |
| 154 | + - **Failures Detail**: For each failed scenario — steps to reproduce, expected vs. actual, evidence |
| 155 | + - **Environment Info**: OS, browser (if applicable), runtime versions, application URL |
| 156 | + - **Metrics**: Total scenarios, passed, failed, partial, skipped, coverage percentage |
| 157 | +
|
| 158 | +10. **Provide QA Verdict**: |
| 159 | + Based on results, provide one of: |
| 160 | + - ✅ **QA PASSED**: All critical scenarios pass, no blockers. Safe to proceed to `/speckit.ship` |
| 161 | + - ⚠️ **QA PASSED WITH NOTES**: Critical paths pass but some edge cases or non-critical scenarios failed. List items. |
| 162 | + - ❌ **QA FAILED**: Critical user flows or acceptance criteria are not met. Must fix and re-test. |
| 163 | +
|
| 164 | +## Post-QA Actions |
| 165 | +
|
| 166 | +Suggest next steps based on verdict: |
| 167 | +- If QA PASSED: "Run `/speckit.ship` to prepare the release" |
| 168 | +- If QA PASSED WITH NOTES: "Address noted items if possible, then run `/speckit.ship`" |
| 169 | +- If QA FAILED: "Fix failing scenarios, then run `/speckit.qa` again to re-test" |
| 170 | +
|
| 171 | +**Check for extension hooks (after QA)**: |
| 172 | +- Check if `.specify/extensions.yml` exists in the project root. |
| 173 | +- If it exists, read it and look for entries under the `hooks.after_qa` key |
| 174 | +- If the YAML cannot be parsed or is invalid, skip hook checking silently and continue normally |
| 175 | +- Filter out hooks where `enabled` is explicitly `false`. Treat hooks without an `enabled` field as enabled by default. |
| 176 | +- For each remaining hook, do **not** attempt to interpret or evaluate hook `condition` expressions: |
| 177 | + - If the hook has no `condition` field, or it is null/empty, treat the hook as executable |
| 178 | + - If the hook defines a non-empty `condition`, skip the hook and leave condition evaluation to the HookExecutor implementation |
| 179 | +- For each executable hook, output the following based on its `optional` flag: |
| 180 | + - **Optional hook** (`optional: true`): |
| 181 | + ``` |
| 182 | + ## Extension Hooks |
| 183 | +
|
| 184 | + **Optional Hook**: {extension} |
| 185 | + Command: `/{command}` |
| 186 | + Description: {description} |
| 187 | +
|
| 188 | + Prompt: {prompt} |
| 189 | + To execute: `/{command}` |
| 190 | + ``` |
| 191 | + - **Mandatory hook** (`optional: false`): |
| 192 | + ``` |
| 193 | + ## Extension Hooks |
| 194 | +
|
| 195 | + **Automatic Hook**: {extension} |
| 196 | + Executing: `/{command}` |
| 197 | + EXECUTE_COMMAND: {command} |
| 198 | + ``` |
| 199 | +- If no hooks are registered or `.specify/extensions.yml` does not exist, skip silently |
0 commit comments