Skip to content

Commit b63c890

Browse files
doc(proposal): flaky (#4)
1 parent 86ed945 commit b63c890

File tree

1 file changed

+61
-0
lines changed

1 file changed

+61
-0
lines changed
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Feature proposal: `flaky`
2+
3+
This flag causes a test-case or suite to be re-run a number of times until it either passes or has not passed after the final re-try.
4+
5+
## API
6+
7+
```js
8+
describe.flaky('some temperamental thing', () => {
9+
it('may take several times', () => {…});
10+
11+
it('may also take several times', () => {…});
12+
});
13+
14+
it.flaky('should do something', () => {…});
15+
```
16+
17+
```js
18+
describe('some temperamental thing', { flaky: true }, () => {
19+
it('may take several times', () => {…});
20+
21+
it('may also take several times', () => {…});
22+
});
23+
24+
it('may take several times', { flaky: true }, () => {…});
25+
it('may also take several times', { flaky: 100 }, () => {…});
26+
```
27+
28+
When `flaky` is `true`, the test harness re-tries the suite or test-case up to the default number of times (eg 20), inclusive.
29+
30+
When `flaky` is falsy (the default), the test harness does not re-try the suite or test-case.
31+
32+
When `flaky` is a positive integer, the test harness re-tries the suite or test-case up to the specified number of times, inclusive.
33+
34+
When `flaky` is a non-positive integers or non-boolean, the test harness emits/throws an error.
35+
36+
When both the suite and an included test-case specify the `flaky` flag, the test-case's `flaky` value wins.
37+
38+
## Output
39+
40+
When a suite or test-case is marked `flaky`, the number of re-tries taken to achieve a passing result is reported with that suite or test-case.
41+
42+
The summary also lists the total number of _test-cases_ marked flaky; a "flaky" suite containing 10 test-cases increases the total flaky count by 10.
43+
44+
### Example: TAP
45+
46+
```
47+
ok 1 - may take several times # FLAKY 42 re-tries
48+
49+
tests 1
50+
suites 1
51+
pass 1
52+
fail 0
53+
flaky 1
54+
cancelled 0
55+
skipped 0
56+
todo 0
57+
```
58+
59+
## Notes
60+
61+
Feature documentation (possibly in a Learn article rather than API documentation) warns users of common caveats related to re-tries (e.g. [idempotence](https://en.wikipedia.org/wiki/Idempotence), statefulness).

0 commit comments

Comments
 (0)