Skip to content

Commit 5a7de44

Browse files
authored
chore(bugbot): Add rules to flag test-flake-provoking patterns (#20192)
We need to get a grip on our test flake situation again. Currently, CI flakes on almost every initial run, which is especially painful when cutting releases. This PR adds a few rules for bug bot to look out for anti patterns that are likely to introduce new test flakes.
1 parent 499f042 commit 5a7de44

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

.cursor/BUGBOT.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,11 @@ Do not flag the issues below if they appear in tests.
5858
- Flag usage of `expect.objectContaining` and other relaxed assertions, when a test expects something NOT to be included in a payload but there's no respective assertion.
5959
- Flag usage of conditionals in one test and recommend splitting up the test for the different paths.
6060
- Flag usage of loops testing multiple scenarios in one test and recommend using `(it)|(test).each` instead.
61+
- Flag tests that are likely to introduce flakes. In our case this usually means we wait for some telemetry requests sent from an SDK. Patterns to look out for:
62+
- Only waiting for a request, after an action is performed. Instead, start waiting, perform action, await request promise.
63+
- Race conditions when waiting on multiple requests. Ensure that waiting checks are unique enough and don't depend on a hard order when there's a chance that telemetry can be sent in arbitrary order.
64+
- Timeouts or sleeps in tests. Instead suggest concrete events or other signals to wait on.
65+
- Flag usage of `getFirstEnvelope*`, `getMultipleEnvelope*` or related test helpers. These are NOT reliable anymore. Instead suggest helpers like `waitForTransaction`, `waitForError`, `waitForSpans`, etc.
6166

6267
## Platform-safe code
6368

0 commit comments

Comments
 (0)