Skip to content

Commit 87700e7

Browse files
committed
feat(qa): manifest-driven coverage gate (Phases A, B, C, E)
Active QA infrastructure that treats audit/manifest.json as the canonical inventory and fails commits / releases when new manifest entries land without test coverage. Same baseline-shrink discipline phpstan-baseline uses: uncovered_total can drop, never grow. Phase A: bin/qa-coverage-check.php - Reads audit/manifest.json + test source files (includes/qa/, includes/cli/journeys/, includes/cli/scenarios/, tests/). - Categories: rest endpoints, ajax handlers, hooks_fired with consumers, cron hooks, wp_cli commands. - REST coverage detected via `$this->rest( METHOD, /path )` calls AND `'B7: METHOD /path'` style test labels (covers both direct calls and helper-wrapped tests). - Writes audit/qa-coverage.json with per-item covered/uncovered map + drift summary (current vs previous uncovered_total). - Plugin-agnostic: works on any plugin's manifest, no hard-coded names. - Initial baseline: 36/149 covered, 113 uncovered, 104 hooks skipped (intentional extensibility surface). Phase B: bin/qa-stub-gen.php - Generates starter test stubs for any uncovered manifest entry. - Categories: rest, ajax, hook, cron. - Appends a TODO test method to includes/qa/class-rest-tests.php with context pulled from the manifest (handler, nonce, capability, args). - Each uncovered entry in qa-coverage.json carries the exact stub_command to scaffold its test in seconds. Phase C: pre-commit hook integration - .githooks/pre-commit runs qa-coverage-check.php after PHPStan + WPCS. - Blocks commits where uncovered_total grew. Bypass: COVERAGE_SKIP=1. Phase E: bin/build-release.sh integration - New step 6b runs qa-coverage-check.php between lint and smoke. - Blocks the dist zip on coverage regression. Bypass: COVERAGE_SKIP=1. Documentation: audit/QA-COVERAGE.md - Architecture, schema, exit codes, gate behavior. - Adopt-into-new-plugin recipe — Jetonomy is the reference; the scripts are plug-and-play for other Wbcom plugins. Phases D (bug-fix-needs-journey discipline) and A.2 (broader signals: parse live qa-actions output, detect journey-class REST coverage) are follow-ups; both extend the existing gate contract additively. Also pulls in grunt-regenerated assets (.min.css, .pot) from the build-release run that exercised the new gate.
1 parent 8651490 commit 87700e7

9 files changed

Lines changed: 2004 additions & 86 deletions

File tree

.githooks/pre-commit

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,5 +107,26 @@ if [ -n "$STAGED_JS" ]; then
107107
fi
108108
fi
109109

110+
# ── Gate: manifest-driven QA coverage drift (Phase C) ──
111+
# Reason: every new REST endpoint / AJAX handler / cron / hook with a
112+
# consumer should carry a corresponding test stub. The qa-coverage-check
113+
# script reads audit/manifest.json + test source and writes
114+
# audit/qa-coverage.json — exit 1 when the uncovered count grew vs the
115+
# previous run. Same baseline-shrink discipline phpstan-baseline uses:
116+
# the gate can drop, never grow.
117+
#
118+
# Skip: COVERAGE_SKIP=1 git commit (emergency only — fix it next commit).
119+
if [ -f bin/qa-coverage-check.php ] && [ -z "${COVERAGE_SKIP:-}" ]; then
120+
echo " -> qa-coverage drift"
121+
if ! php bin/qa-coverage-check.php --quiet; then
122+
echo
123+
echo "pre-commit FAILED: QA coverage regressed (audit/qa-coverage.json shows new uncovered manifest entries)."
124+
echo "Each uncovered entry has a stub_command in audit/qa-coverage.json — run those commands"
125+
echo "to scaffold a test, fill in the assertions, retry the commit."
126+
echo "Bypass (emergency only): COVERAGE_SKIP=1 git commit ..."
127+
exit 1
128+
fi
129+
fi
130+
110131
echo "pre-commit: OK"
111132
exit 0

assets/css/jetonomy-rtl.min.css

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

assets/css/jetonomy.min.css

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

audit/QA-COVERAGE.md

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# QA Coverage — manifest-driven QA gate
2+
3+
This plugin treats `audit/manifest.json` as the canonical inventory and runs
4+
an automated gate that fails when new manifest entries land without
5+
corresponding test coverage. The discipline is the same as `phpstan-baseline.neon`:
6+
the uncovered count can drop, never grow.
7+
8+
---
9+
10+
## How it works
11+
12+
```
13+
┌─────────────────────────────┐
14+
│ audit/manifest.json │
15+
│ (canonical inventory) │
16+
└──────────────┬──────────────┘
17+
18+
19+
┌───────────────────────────────────────────┐
20+
│ bin/qa-coverage-check.php │
21+
│ Reads manifest, scans test source, │
22+
│ writes audit/qa-coverage.json │
23+
└────────┬─────────────────────┬────────────┘
24+
│ │
25+
┌────────────▼──────────┐ ┌──────▼─────────────┐
26+
│ Pre-commit hook │ │ bin/build-release │
27+
│ blocks commits that │ │ blocks the dist │
28+
│ grow uncovered_total │ │ zip on regression │
29+
└───────────────────────┘ └────────────────────┘
30+
31+
When uncovered:
32+
┌────────────────────────────────────────────────┐
33+
│ bin/qa-stub-gen.php rest POST /foo │
34+
│ → appends a TODO test stub to class-rest-tests │
35+
│ → human fills assertions → run again → covered │
36+
└────────────────────────────────────────────────┘
37+
```
38+
39+
## Categories tracked (Phase A)
40+
41+
| Category | Coverage signal |
42+
|---|---|
43+
| `rest` | `$this->rest( 'METHOD', '/path' )` calls in QA test source AND `'B7: METHOD /path'` style test labels |
44+
| `ajax` | Action name appears as a literal string in any test source file |
45+
| `hooks_fired` | Only entries with non-empty `consumed_by[]`. Covered when a `do_action`/`apply_filters` for that hook is found in CLI journeys / scenarios |
46+
| `cron` | Cron hook name appears as a literal string in any test source file |
47+
| `wp_cli` | A `class-{slug}-journey.php` file exists for the command |
48+
49+
Categories deferred to Phase A.2: `blocks`, `shortcodes`, `admin_pages`.
50+
51+
## Scripts
52+
53+
### `bin/qa-coverage-check.php` — the gate
54+
55+
```bash
56+
# Default usage — runs against current plugin directory
57+
php bin/qa-coverage-check.php
58+
59+
# Run against a specific plugin
60+
php bin/qa-coverage-check.php --plugin=/path/to/plugin
61+
62+
# JSON summary on stdout (for CI integration)
63+
php bin/qa-coverage-check.php --json
64+
65+
# Strict mode — exit 1 on ANY uncovered (not just regression)
66+
php bin/qa-coverage-check.php --strict
67+
68+
# Quiet — suppress human-readable summary (for hooks)
69+
php bin/qa-coverage-check.php --quiet
70+
```
71+
72+
Exit codes:
73+
- `0` — coverage equal or improved vs prior run (or first run)
74+
- `1` — coverage regressed (`uncovered_total` grew)
75+
- `2` — manifest missing or malformed
76+
77+
Output: `audit/qa-coverage.json`. Schema:
78+
79+
```json
80+
{
81+
"schema_version": "v1",
82+
"plugin": { "slug": "...", "version": "..." },
83+
"generated": { "at": "ISO", "manifest_at": "...", "manifest_branch": "..." },
84+
"categories": {
85+
"rest": { "total": 64, "covered": 31, "uncovered": 33, "skipped": 0,
86+
"covered_items": [...], "uncovered_items": [...] },
87+
"ajax": { ... },
88+
"hooks_fired": { ... },
89+
"cron": { ... },
90+
"wp_cli": { ... }
91+
},
92+
"summary": {
93+
"categories_checked": 5,
94+
"items_total": 149,
95+
"items_covered": 36,
96+
"items_uncovered": 113,
97+
"items_skipped": 104,
98+
"coverage_percent": 24.2
99+
},
100+
"drift": {
101+
"previous_uncovered": 122,
102+
"current_uncovered": 113,
103+
"delta": -9
104+
}
105+
}
106+
```
107+
108+
### `bin/qa-stub-gen.php` — fast scaffolding
109+
110+
Generates a starter test stub for any uncovered manifest entry. The stub
111+
appends to `includes/qa/class-rest-tests.php` with TODO markers — humans
112+
fill in fixture-specific assertions, then the entry moves uncovered →
113+
covered on the next coverage check.
114+
115+
```bash
116+
# REST endpoint
117+
php bin/qa-stub-gen.php rest POST "/posts/(?P<id>\d+)/idea-status"
118+
119+
# AJAX handler
120+
php bin/qa-stub-gen.php ajax jetonomy_ban_user
121+
122+
# Hook (with consumer)
123+
php bin/qa-stub-gen.php hook jetonomy_user_left_space
124+
125+
# Cron hook
126+
php bin/qa-stub-gen.php cron jetonomy_prune_activity
127+
```
128+
129+
The `stub_command` field in `qa-coverage.json#/categories/<cat>/uncovered_items[]`
130+
is the exact command to run for that uncovered entry.
131+
132+
## Gates
133+
134+
### Pre-commit (`.githooks/pre-commit`)
135+
136+
Runs `bin/qa-coverage-check.php --quiet`. Blocks commits where
137+
`uncovered_total` grew vs the previous run. Bypass (emergencies only):
138+
`COVERAGE_SKIP=1 git commit ...`
139+
140+
### Release (`bin/build-release.sh`)
141+
142+
Runs `bin/qa-coverage-check.php` between the lint pass and the smoke test
143+
(step 6b in the build pipeline). Blocks the dist zip if coverage regressed.
144+
Bypass: `COVERAGE_SKIP=1 bin/build-release.sh`
145+
146+
## Adding the gate to a new plugin
147+
148+
The scripts are plugin-agnostic. To adopt:
149+
150+
1. Copy `bin/qa-coverage-check.php` and `bin/qa-stub-gen.php` to the new
151+
plugin's `bin/` directory.
152+
2. Make both executable: `chmod +x bin/qa-coverage-check.php bin/qa-stub-gen.php`.
153+
3. Add the pre-commit gate block to `.githooks/pre-commit` (see Jetonomy's
154+
hook for the exact snippet).
155+
4. Add the build-release gate block to `bin/build-release.sh` (step 6b
156+
between `php -l` and the smoke test).
157+
5. Run `php bin/qa-coverage-check.php` once to establish the baseline
158+
`audit/qa-coverage.json`. Commit it.
159+
6. Adjust `$test_globs` in `qa-coverage-check.php` if the plugin's test
160+
files live in a non-standard location (default: `includes/qa/*.php`,
161+
`includes/cli/journeys/*.php`, `includes/cli/scenarios/*.php`,
162+
`tests/**/*.php`).
163+
164+
That's it. The gate is now active.
165+
166+
## When to skip
167+
168+
Skipping the gate is for genuine emergencies: a customer-reported P0 fix
169+
that needs to ship before the next test scaffolding cycle. Use
170+
`COVERAGE_SKIP=1` and FILE THE STUB IN THE NEXT COMMIT. If you find
171+
yourself skipping more than once a quarter, the gate is mis-tuned (too
172+
strict) or the codebase is shipping untested features — fix one or the
173+
other.
174+
175+
## Why we count things this way
176+
177+
The Phase A detection is intentionally conservative — it only counts
178+
"covered" when there's a clear static signal in the test source. This
179+
means real coverage may be higher than what the gate reports (e.g., a
180+
test calling a wrapper method instead of `$this->rest()` directly is
181+
not detected). The trade-off: false negatives are easy to fix (write a
182+
stub or annotate the manifest); false positives let bugs ship.
183+
184+
The next phases improve detection signals (Phase A.2: parse
185+
`wp jetonomy qa-actions` output for live test results; Phase A.3: detect
186+
journey-class coverage of REST endpoints) without changing the gate
187+
contract: still uncovered_total can shrink, never grow.
188+
189+
## Out-of-scope (handled by other gates)
190+
191+
- **Live test PASS/FAIL** — that's `wp jetonomy qa-actions`, run as part
192+
of the pre-release smoke. Coverage check answers "is there a test?";
193+
qa-actions answers "does the test pass?".
194+
- **Browser-side assertions** — handled by the smoke skill (`/jetonomy-smoke`
195+
with FREE / COMBO modes).
196+
- **Cross-plugin wiring** — handled by `/action-audit`.
197+
- **Code style / static analysis** — PHPCS + PHPStan in the same hook.
198+
199+
The coverage gate fills the gap between "the code exists" and "the code
200+
is exercised by a test." It's a documentation-of-intent layer that the
201+
other gates don't carry.

0 commit comments

Comments
 (0)