Skip to content

Commit 12b6df5

Browse files
SahilRakhaiya05SahilRakhaiya05
committed
docs: update PR template with live API verification results
1 parent 6a17fa8 commit 12b6df5

1 file changed

Lines changed: 131 additions & 50 deletions

File tree

‎.github/PULL_REQUEST_failure-triage.md‎

Lines changed: 131 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -1,69 +1,150 @@
11
## Summary
22

3-
Adds **`testsprite test failure triage --project <id>`** — groups all failed tests in a project into root-cause clusters using existing M2.1 analysis fields, without downloading failure bundles.
3+
Adds **`testsprite test failure triage --project <id>`** — after a batch run with many failures, groups them into root-cause clusters by calling the **real TestSprite API** (`GET /tests?status=failed` + `GET /tests/{id}/failure/summary` per test). Returns a representative test per cluster so agents investigate one bundle, not thirty.
44

5-
Closes #[ISSUE_NUMBER]
5+
Closes #<ISSUE_NUMBER>
66

7-
## Motivation
7+
---
88

9-
When a batch run produces many failures, agents currently see a flat list and must investigate each test independently. This command collapses duplicate root causes into a few clusters with a representative test, confidence score, and fix priority — reducing bundle downloads, AI tokens, and debug time.
9+
## Real-world problem this solves
1010

11-
Mentor-approved as CLI Phase-0 triage ahead of native backend clustering.
11+
| Before | After |
12+
|--------|-------|
13+
| Batch run: 30 tests fail | Same 30 failures |
14+
| Agent sees 30 flat red rows | `failure triage` → 3 clusters |
15+
| Downloads 30 failure bundles | Downloads 1 bundle (representative) |
16+
| Fixes same auth bug 5 times | Fixes root once, reruns representative |
1217

13-
## What changed
18+
---
1419

15-
### New command
20+
## Real production usage
21+
22+
Requires configured API key (`testsprite setup` or `TESTSPRITE_API_KEY`).
23+
24+
```bash
25+
# 1. Run full project suite
26+
testsprite test run --all --project <project-id> --wait --output json
27+
28+
# 2. Triage — real API calls against your project
29+
testsprite test failure triage --project <project-id> --output json
30+
31+
# 3. Investigate highest-priority cluster only
32+
testsprite test failure get <representativeTestId> --out ./.testsprite/failure
33+
34+
# 4. Fix code, verify representative, then full regression
35+
testsprite test rerun <representativeTestId> --wait
36+
testsprite test rerun --all --project <project-id> --wait
37+
```
38+
39+
**With filters:**
1640
```bash
17-
testsprite test failure triage --project proj_xxx --output json
18-
testsprite test failure triage --project proj_xxx --type backend --filter checkout --dry-run
41+
testsprite test failure triage --project <project-id> --type backend --filter checkout --output json
1942
```
2043

21-
### Grouping heuristics (deterministic, no LLM)
22-
1. Shared `recommendedFixTarget.reference`
23-
2. Env-wide `failureKind` (`infra`, `network`, `network_timeout`, `routing_404`)
24-
3. Normalized `rootCauseHypothesis` prefix
25-
4. Singleton fallback
44+
**What hits the wire:**
45+
46+
| Step | API | Purpose |
47+
|------|-----|---------|
48+
| 1 | `GET /api/cli/v1/tests?projectId=...&status=failed` | List all failed tests (paginated) |
49+
| 2 | `GET /api/cli/v1/tests/{testId}/failure/summary` × N | Real M2.1 analysis per test (parallel, default 5) |
50+
51+
No bundle download. No screenshots. No video.
52+
53+
> **Note:** If your `~/.testsprite/credentials` points at a local dev proxy (`127.0.0.1`), pass `--endpoint-url https://api.testsprite.com` or fix the profile URL.
54+
55+
---
56+
57+
## Grouping logic (deterministic over real API fields)
58+
59+
| Priority | Groups by | Real scenario |
60+
|----------|-----------|---------------|
61+
| 1 | `recommendedFixTarget.reference` | 8 tests point to same file:line → 1 cluster |
62+
| 2 | `failureKind` ∈ infra/network/network_timeout/routing_404 | 15 tests all `network_timeout` → env outage |
63+
| 3 | Normalized `rootCauseHypothesis` prefix | Similar LLM hypothesis text |
64+
| 4 | Singleton | Independent failures, 1 test per cluster |
65+
66+
Each cluster: `representativeTestId`, `memberTestIds`, `confidence`, `fixPriority` (lower = fix first).
67+
68+
---
69+
70+
## Real test coverage (18 automated tests)
71+
72+
### Unit — `src/lib/failure-triage.test.ts` (11)
73+
74+
| Test | Validates |
75+
|------|-----------|
76+
| `normalizeHypothesis` | Real LLM text with varied spacing groups correctly |
77+
| `computeGroupKey` — fix_target | Same `recommendedFixTarget.reference` → same key |
78+
| `computeGroupKey` — failure_kind | `network_timeout` → env outage cluster |
79+
| `computeGroupKey` — hypothesis | Similar root-cause text groups |
80+
| `computeGroupKey` — singleton | No signal → independent cluster |
81+
| `pickRepresentativeTestId` | Richest hypothesis + freshest `updatedAt` wins |
82+
| `computeClusterConfidence` | Multi fix_target = 0.92, singleton = 0.40 |
83+
| `computeFixPriority` | Infra before assertion bugs |
84+
| `buildFailureClusters` | 3 failures → 2 clusters (shared code + env) |
2685

27-
### Files
28-
| File | Change |
29-
|------|--------|
30-
| `src/lib/failure-triage.ts` | Grouping logic, confidence/priority scoring, text renderer |
86+
### Integration — `src/commands/test.test.ts` (7)
87+
88+
Full command against **real HTTP wire shapes** (same mock harness as `runFailureSummary` / `runList`):
89+
90+
| Test | Real scenario |
91+
|------|---------------|
92+
| JSON clusters by shared fix target | 3 failed tests → 2 clusters, correct representative |
93+
| Text mode output | Human triage card |
94+
| Empty project | `GET /tests?status=failed` → `[]`, exit 0 |
95+
| Stale failed row | Summary 404 `no_failing_run` → `summary.skipped` |
96+
| Missing projectId | `VALIDATION_ERROR` exit 5, no API call |
97+
| Command surface + help snapshot | `triage` subcommand registered |
98+
99+
---
100+
101+
## Live API verification (maintainer / reviewer)
102+
103+
Verified against **production** `https://api.testsprite.com` with a real API key (not committed):
104+
105+
```bash
106+
git checkout feat/failure-triage && npm run build
107+
108+
# Auth — all scopes present including read:tests, run:tests
109+
testsprite auth status --output json --endpoint-url https://api.testsprite.com
110+
# → exit 0, scopes: read:projects, read:tests, write:tests, run:tests, ...
111+
112+
# Real project, zero failed tests
113+
testsprite test failure triage --project <project-id> --output json --endpoint-url https://api.testsprite.com
114+
# → exit 0
115+
# → { "clusters": [], "summary": { "totalFailed": 0, "clusterCount": 0, "skipped": 0 } }
116+
117+
# Invalid project id
118+
testsprite test failure triage --project proj_does_not_exist --output json --endpoint-url https://api.testsprite.com
119+
# → exit 4 NOT_FOUND (project validation from API)
120+
121+
# Command exists on built binary
122+
node dist/index.js test failure triage --help
123+
# → shows --project, --type, --filter, --max-concurrency
124+
```
125+
126+
Automated: `npx vitest run src/lib/failure-triage.test.ts` (11/11), `npx vitest run src/commands/test.test.ts -t runFailureTriage` (7/7), `npm run typecheck`, `npm run lint`, `npm run build` — all pass.
127+
128+
---
129+
130+
## Files changed
131+
132+
| File | Purpose |
133+
|------|---------|
134+
| `src/lib/failure-triage.ts` | Grouping, confidence, priority, text renderer |
31135
| `src/lib/failure-triage.test.ts` | 11 unit tests |
32-
| `src/commands/test.ts` | `runFailureTriage`, Commander wiring |
136+
| `src/commands/test.ts` | `runFailureTriage()`, API fan-out, Commander |
33137
| `src/commands/test.test.ts` | 7 integration tests |
34138
| `DOCUMENTATION.md`, `README.md`, `CHANGELOG.md` | Docs |
35139
| `skills/testsprite-verify.skill.md` | Agent triage playbook |
36-
| `test/help.snapshot.test.ts.snap` | Help snapshot |
37-
38-
## Example JSON output
39-
```json
40-
{
41-
"projectId": "proj_abc",
42-
"clusters": [
43-
{
44-
"clusterId": "cluster_kind_network_timeout",
45-
"label": "Environment issue (network_timeout)",
46-
"groupReason": "failure_kind",
47-
"representativeTestId": "test_a",
48-
"memberTestIds": ["test_a", "test_b"],
49-
"confidence": 0.88,
50-
"fixPriority": 1
51-
}
52-
],
53-
"summary": { "totalFailed": 2, "clusterCount": 1, "skipped": 0 }
54-
}
55-
```
56-
57-
## Test plan
140+
| `test/help.snapshot.test.ts.snap` | Help regression guard |
58141

59-
- [x] `npm run typecheck` — pass
60-
- [x] `npm run lint` — pass
61-
- [x] `npx vitest run src/lib/failure-triage.test.ts` — 11/11 pass
62-
- [x] `npx vitest run src/commands/test.test.ts -t runFailureTriage` — 7/7 pass
63-
- [x] `npm run build` — pass
64-
- [x] Help snapshot added for `test failure triage --help`
142+
---
65143

66-
## Notes
144+
## CI gates
67145

68-
- Client-side Phase-0 only; when backend ships native clustering, this command can become a thin wrapper over a new read API.
69-
- Skips tests listed as `failed` but with no `failure/summary` (stale status race) — reported in `summary.skipped`.
146+
- [x] `npm run typecheck`
147+
- [x] `npm run lint`
148+
- [x] `npm run build`
149+
- [x] 18 new tests pass
150+
- [x] Live production API smoke test (auth + triage + NOT_FOUND path)

0 commit comments

Comments
 (0)