Skip to content

Commit 84e9055

Browse files
TovRudyyclaudedevindforddrichards-87
authored
Flaky Tests Management: document attempt-to-fix detection across batched commits (#37554)
* Flaky Tests Management: document attempt-to-fix detection across batched commits Clarify that the attempt-to-fix remediation flow triggers when the unique test key appears in any commit included in the triggering push, not only the most recent commit (batched or squashed pushes, or CI providers that report only the head commit). Note that CI retries of the same push are not double-processed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Flaky Tests Management: correct attempt-to-fix batched-commit accuracy Align the docs with the implementation after an impl-vs-docs audit: - State the real 10-commit lookback limit (head commit + recent history), instead of implying every commit in the push is scanned. - Fix the squash & merge wording: only the squash commit's message is read (the original commits no longer exist individually). - Drop the unsupported 'fix is not processed more than once' claim. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Flaky Tests Management: move attempt-to-fix limits into Troubleshooting Keep the main remediation-flow steps concise; move the 10-commit lookback limit and squash-merge behavior into a new Troubleshooting entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update content/en/tests/flaky_management/_index.md Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com> * Update content/en/tests/flaky_management/_index.md Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com> * Update content/en/tests/flaky_management/_index.md Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Devin Ford <devindford@gmail.com> Co-authored-by: DeForest Richards <56796055+drichards-87@users.noreply.github.com>
1 parent fb54a59 commit 84e9055

1 file changed

Lines changed: 9 additions & 1 deletion

File tree

content/en/tests/flaky_management/_index.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ When you fix a flaky test, Test Optimization's remediation flow can confirm the
122122
1. For the test you are fixing, click {{< ui >}}Link commit to Flaky Test fix{{< /ui >}} in the Flaky Tests Management UI.
123123
1. Copy the unique flaky test key that is displayed (for example, `DD_ABC123`).
124124
1. Include the test key in your Git commit title or message for the fix (for example, `git commit -m "DD_ABC123"`).
125-
1. When Datadog detects the test key in your commit, it automatically triggers the remediation flow for that test:
125+
1. When Datadog detects the test key, it automatically triggers the remediation flow for that test. The key does not need to be in the most recent commit. Datadog also checks recent preceding commits, so the flow still triggers when you push several commits together or when your CI provider reports only the most recent commit. The remediation flow:
126126
- Retries any tests you're attempting to fix 20 times (or the number of retries you specified in your [Flaky Test Policies configuration](#configure-policies-to-automate-the-flaky-test-lifecycle)).
127127
- Tags every retry with `@test.test_management.is_attempt_to_fix:true` in test run events.
128128
- Runs tests even if they are marked as `Disabled`.
@@ -250,6 +250,14 @@ To find your account name, go to the [Slack integration tile][5] and check the
250250

251251
The weekly digest summary notification does not have a self-service opt-out. To disable it, contact [Datadog Support][14].
252252

253+
### Attempt-to-fix remediation does not trigger after linking a fix
254+
255+
After you include the test key (for example, `DD_ABC123`) in a commit, Datadog scans the commit that triggered the test run and up to the 10 most recent commits before it. If the remediation flow does not start, check the following:
256+
257+
- **The key is in an older commit.** Datadog scans only the triggering commit and the 10 most recent commits in its history. If you push more than 10 commits at once, include the key in the triggering commit or one of the 10 most recent commits.
258+
- **The fix was squash-merged.** A squash merge combines the original commits into a single commit, so Datadog reads only the squash commit's message; the individual pre-squash commits are no longer scanned. Most providers include the original commit messages in the squash commit, so a key in any of them is preserved. If your provider omits them, add the key to the squash commit message.
259+
- **The key does not match the expected format.** Use the exact key shown in the Flaky Tests Management UI (for example, `DD_ABC123`) in the commit title or message.
260+
253261
## Further reading
254262

255263
{{< partial name="whats-next/whats-next.html" >}}

0 commit comments

Comments
 (0)