Skip to content

Commit 6a17fa8

Browse files
docs: add issue and PR templates for failure triage
1 parent 9cdf2c9 commit 6a17fa8

2 files changed

Lines changed: 117 additions & 0 deletions

File tree

.github/ISSUE_failure-triage.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# feat: `test failure triage` — group batch failures by root cause
2+
3+
## Problem
4+
5+
When many tests fail in the same project run (batch `test run --all`, regression reruns, MCP execute), agents and humans see a flat list of 20–50 separate failures. In practice, those failures often share one underlying root cause:
6+
7+
- authentication / token expired
8+
- staging environment down (`network_timeout`, `infra`)
9+
- broken navigation (`routing_404`)
10+
- shared code defect (`recommendedFixTarget.reference` points to the same file)
11+
- corrupted test data / producer failure cascading to consumers
12+
13+
Today the CLI only supports **per-test** analysis (`test failure get`, `test failure summary`). An agent must download many bundles or guess which test to investigate first.
14+
15+
## Proposed solution (CLI Phase-0)
16+
17+
Add `testsprite test failure triage --project <id>`:
18+
19+
1. List all failed tests in the project (`GET /tests?status=failed`)
20+
2. Fetch lightweight `failure/summary` per test (no screenshots/video)
21+
3. Group client-side using deterministic heuristics over existing M2.1 fields:
22+
- shared `recommendedFixTarget.reference`
23+
- env-wide `failureKind` (`infra`, `network`, `network_timeout`, `routing_404`)
24+
- normalized `rootCauseHypothesis` prefix
25+
- singleton fallback
26+
4. Return clusters with:
27+
- `representativeTestId`
28+
- `memberTestIds`
29+
- `confidence`
30+
- `fixPriority` (lower = fix first)
31+
32+
## Why CLI-first (Phase-0)
33+
34+
- Uses only existing public APIs — no backend changes required
35+
- Immediately reduces duplicate investigation, bundle downloads, and token usage
36+
- Becomes the read surface when native backend clustering ships later
37+
38+
## Acceptance criteria
39+
40+
- [ ] `testsprite test failure triage --project <id> --output json` returns clustered output
41+
- [ ] `--type`, `--filter`, `--max-concurrency` supported
42+
- [ ] `--dry-run` returns canned sample
43+
- [ ] Unit tests for grouping logic + integration tests for command
44+
- [ ] DOCUMENTATION.md, README, CHANGELOG, agent skill updated
45+
46+
## Future (backend)
47+
48+
Native clustering API with semantic embeddings, wave/cascade graph, and `--rerun-representatives` orchestration can replace/augment the client-side heuristics.
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
## Summary
2+
3+
Adds **`testsprite test failure triage --project <id>`** — groups all failed tests in a project into root-cause clusters using existing M2.1 analysis fields, without downloading failure bundles.
4+
5+
Closes #[ISSUE_NUMBER]
6+
7+
## Motivation
8+
9+
When a batch run produces many failures, agents currently see a flat list and must investigate each test independently. This command collapses duplicate root causes into a few clusters with a representative test, confidence score, and fix priority — reducing bundle downloads, AI tokens, and debug time.
10+
11+
Mentor-approved as CLI Phase-0 triage ahead of native backend clustering.
12+
13+
## What changed
14+
15+
### New command
16+
```bash
17+
testsprite test failure triage --project proj_xxx --output json
18+
testsprite test failure triage --project proj_xxx --type backend --filter checkout --dry-run
19+
```
20+
21+
### Grouping heuristics (deterministic, no LLM)
22+
1. Shared `recommendedFixTarget.reference`
23+
2. Env-wide `failureKind` (`infra`, `network`, `network_timeout`, `routing_404`)
24+
3. Normalized `rootCauseHypothesis` prefix
25+
4. Singleton fallback
26+
27+
### Files
28+
| File | Change |
29+
|------|--------|
30+
| `src/lib/failure-triage.ts` | Grouping logic, confidence/priority scoring, text renderer |
31+
| `src/lib/failure-triage.test.ts` | 11 unit tests |
32+
| `src/commands/test.ts` | `runFailureTriage`, Commander wiring |
33+
| `src/commands/test.test.ts` | 7 integration tests |
34+
| `DOCUMENTATION.md`, `README.md`, `CHANGELOG.md` | Docs |
35+
| `skills/testsprite-verify.skill.md` | Agent triage playbook |
36+
| `test/help.snapshot.test.ts.snap` | Help snapshot |
37+
38+
## Example JSON output
39+
```json
40+
{
41+
"projectId": "proj_abc",
42+
"clusters": [
43+
{
44+
"clusterId": "cluster_kind_network_timeout",
45+
"label": "Environment issue (network_timeout)",
46+
"groupReason": "failure_kind",
47+
"representativeTestId": "test_a",
48+
"memberTestIds": ["test_a", "test_b"],
49+
"confidence": 0.88,
50+
"fixPriority": 1
51+
}
52+
],
53+
"summary": { "totalFailed": 2, "clusterCount": 1, "skipped": 0 }
54+
}
55+
```
56+
57+
## Test plan
58+
59+
- [x] `npm run typecheck` — pass
60+
- [x] `npm run lint` — pass
61+
- [x] `npx vitest run src/lib/failure-triage.test.ts` — 11/11 pass
62+
- [x] `npx vitest run src/commands/test.test.ts -t runFailureTriage` — 7/7 pass
63+
- [x] `npm run build` — pass
64+
- [x] Help snapshot added for `test failure triage --help`
65+
66+
## Notes
67+
68+
- Client-side Phase-0 only; when backend ships native clustering, this command can become a thin wrapper over a new read API.
69+
- Skips tests listed as `failed` but with no `failure/summary` (stale status race) — reported in `summary.skipped`.

0 commit comments

Comments
 (0)