Skip to content

Commit 4b72461

Browse files
authored
feat(cli): add 'test flaky' repeat-run flaky-test detector (#132)
* feat(cli): add 'test flaky' repeat-run flaky-test detector * fix(flaky): cap --runs at 10 per maintainer scope (#115) Rescope the flaky detector's --runs bound from 1-100 to 1-10 as requested in the #115 triage: uncapped FE replays amplify free executions. Updates the MAX_FLAKY_RUNS constant (which drives the validation, error message, and --runs help text), docs, changelog, and the runs-bound tests. Regenerates the help snapshot, which also adds the previously-missing 'test flaky' entry. * test(snapshot): refresh flaky help snapshot after rebase onto main Rebasing onto current main (which added the global --request-timeout option, #17) changes the 'Global options' line rendered in the test flaky --help output. Regenerate the snapshot so the help snapshot test stays green on CI. * docs(changelog): resolve leftover merge-conflict markers (keep JUnit + flaky 1-10)
1 parent c6f946e commit 4b72461

9 files changed

Lines changed: 928 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ All notable changes to `@testsprite/testsprite-cli` are documented here. The for
77
### Added
88

99
- **JUnit XML report export for batch `--wait` runs.** `test run --all` and batch `test rerun` (`--all` or multiple test ids) accept `--report junit --report-file <path>` to write a CI-friendly XML sidecar after polling completes. `--output json` is unchanged; the report is written even when the batch exits non-zero. `--dry-run` writes a canned sample without network calls.
10+
- **`testsprite test flaky <test-id>`** — repeat-run flaky-test detector. Replays a test N times (`--runs <n>`, default 5), aggregates the outcomes, and reports a stability verdict (`stable` / `flaky` / `failing`) plus the `runId` and `failureKind` of every attempt that did not pass. Replays run with auto-heal OFF (strict verbatim) so a healed drift can't mask a nondeterministic pass/fail. Exit code is 0 only when every attempt passed, so CI can gate a merge on flakiness (`testsprite test flaky <id> --runs 5 || exit 1`). Flags: `--runs <n>` (1–10), `--until-fail` (stop at the first non-passing attempt), `--timeout <s>` (per-attempt), and `--output json` for a machine-readable stability report. Frontend replays are free verbatim script replays; a one-line advisory is printed for backend tests, whose closure reruns may cost credits.
1011

1112
## [0.2.0] - 2026-06-29
1213

DOCUMENTATION.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,29 @@ Flags:
403403

404404
A batch rerun returns `accepted[]` (one `runId` per dispatched test) plus `deferred[]` for any test shed by the per-key run-rate limit; under `--wait`, a non-empty `deferred[]` exits 7 with a `nextAction` you can retry with a fresh idempotency key.
405405

406+
#### `testsprite test flaky <test-id>`
407+
408+
Detect a **flaky** test by replaying it several times and reporting how often it passes. Each attempt is a rerun with auto-heal **off** (a strict verbatim replay), so healed drift can't disguise a nondeterministic pass/fail — this measures the replay stability of the saved script against the configured URL. Frontend replays are free verbatim script replays; backend tests re-run their dependency closure and may cost credits (a one-line stderr advisory is printed before the run).
409+
410+
```bash
411+
# Replay 10 times and print a stability score
412+
testsprite test flaky test_xxxxxxxx --runs 10
413+
414+
# Fast "is it flaky at all?" — stop at the first non-passing attempt
415+
testsprite test flaky test_xxxxxxxx --runs 10 --until-fail
416+
417+
# Machine-readable stability report for CI
418+
testsprite test flaky test_xxxxxxxx --runs 10 --output json
419+
```
420+
421+
Flags:
422+
423+
- `--runs <n>` — number of replays (1–10, default 5).
424+
- `--until-fail` — stop at the first attempt that does not pass.
425+
- `--timeout <s>` — per-attempt polling deadline (same semantics as `test wait`).
426+
427+
`--output json` emits `{ testId, runs, passed, failed, stableRatio, verdict, failures: [{ attempt, runId, outcome, failureKind }] }`. Exit codes: **0** when every observed attempt passed (`stable`); **1** when any attempt did not pass (`flaky` or `failing`); **4** when the test has no replayable run (trigger `testsprite test run <id>` first); **5** on a validation error.
428+
406429
#### `testsprite test wait <run-id>`
407430

408431
Block until a run reaches a terminal status. Same exit-code matrix as `test run --wait`. Used to resume polling after a timed-out `test run --wait`, or when an agent already has a `runId` from a previous invocation.

0 commit comments

Comments
 (0)