|
| 1 | +# Feature proposal: `flaky` |
| 2 | + |
| 3 | +This flag causes a test-case or suite to be re-run a number of times until it either passes or has not passed after the final re-try. |
| 4 | + |
| 5 | +## API |
| 6 | + |
| 7 | +```js |
| 8 | +describe.flaky('some temperamental thing', () => { |
| 9 | + it('may take several times', () => {…}); |
| 10 | + |
| 11 | + it('may also take several times', () => {…}); |
| 12 | +}); |
| 13 | + |
| 14 | +it.flaky('should do something', () => {…}); |
| 15 | +``` |
| 16 | + |
| 17 | +```js |
| 18 | +describe('some temperamental thing', { flaky: true }, () => { |
| 19 | + it('may take several times', () => {…}); |
| 20 | + |
| 21 | + it('may also take several times', () => {…}); |
| 22 | +}); |
| 23 | + |
| 24 | +it('may take several times', { flaky: true }, () => {…}); |
| 25 | +it('may also take several times', { flaky: 100 }, () => {…}); |
| 26 | +``` |
| 27 | + |
| 28 | +When `flaky` is `true`, the test harness re-tries the suite or test-case up to the default number of times (eg 20), inclusive. |
| 29 | + |
| 30 | +When `flaky` is falsy (the default), the test harness does not re-try the suite or test-case. |
| 31 | + |
| 32 | +When `flaky` is a positive integer, the test harness re-tries the suite or test-case up to the specified number of times, inclusive. |
| 33 | + |
| 34 | +When `flaky` is a non-positive integers or non-boolean, the test harness emits/throws an error. |
| 35 | + |
| 36 | +When both the suite and an included test-case specify the `flaky` flag, the test-case's `flaky` value wins. |
| 37 | + |
| 38 | +## Output |
| 39 | + |
| 40 | +When a suite or test-case is marked `flaky`, the number of re-tries taken to achieve a passing result is reported with that suite or test-case. |
| 41 | + |
| 42 | +The summary also lists the total number of _test-cases_ marked flaky; a "flaky" suite containing 10 test-cases increases the total flaky count by 10. |
| 43 | + |
| 44 | +### Example: TAP |
| 45 | + |
| 46 | +``` |
| 47 | +ok 1 - may take several times # FLAKY 42 re-tries |
| 48 | +
|
| 49 | +tests 1 |
| 50 | +suites 1 |
| 51 | +pass 1 |
| 52 | +fail 0 |
| 53 | +flaky 1 |
| 54 | +cancelled 0 |
| 55 | +skipped 0 |
| 56 | +todo 0 |
| 57 | +``` |
| 58 | + |
| 59 | +## Notes |
| 60 | + |
| 61 | +Feature documentation (possibly in a Learn article rather than API documentation) warns users of common caveats related to re-tries (e.g. [idempotence](https://en.wikipedia.org/wiki/Idempotence), statefulness). |
0 commit comments