feat(flaky-detection): Add mark to exclude tests from flaky-detection reruns by remyduthu · Pull Request #441 · Mergifyio/pytest-mergify

remyduthu · 2026-07-07T09:42:26Z

Some tests do not benefit from rerun-based flaky detection: migration tests,
expensive integration tests, or tests whose nodeid embeds a volatile value
(e.g. a commit SHA in a parametrize id) that makes new mode flag them as new
on every run and rerun them pointlessly.

Provide @pytest.mark.mergify(reruns=False) so authors can opt a test out of
the rerun loop in every mode, without disabling the plugin or altering the
test's identity.

mergify · 2026-07-07T09:42:46Z

Copilot

Pull request overview

This PR adds an opt-out mechanism for flaky-detection reruns by introducing a mergify pytest mark option (flaky_detection=False), allowing authors to exclude specific tests (e.g., expensive or volatile-nodeid tests) from the rerun loop while keeping the plugin enabled.

Changes:

Exclude tests marked with @pytest.mark.mergify(flaky_detection=False) from flaky-detection processing in both new and unhealthy modes.
Register the mergify(flaky_detection=bool) marker with pytest to avoid unknown-marker warnings and document the option.
Add an end-to-end test asserting opted-out tests are not rerun and do not appear in the flaky-detection report.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
tests/test_ci_insights.py	Adds an end-to-end test ensuring opted-out tests don’t enter flaky-detection reruns/reporting.
pytest_mergify/flaky_detection.py	Implements opt-out filtering during session preparation and adds a helper to detect the disabling mark.
pytest_mergify/init.py	Registers the new `mergify` marker and documents the `flaky_detection` option.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

remyduthu · 2026-07-07T10:27:21Z

Revision history

#	Type	Changes	Date
1	initial	`199e931`	2026-07-07 10:27 UTC
2	content	`199e931 → 39ead0b` (raw)	2026-07-07 10:27 UTC
3	content	`39ead0b → 1260e86` (raw)	2026-07-07 10:37 UTC

… reruns Some tests do not benefit from rerun-based flaky detection: migration tests, expensive integration tests, or tests whose nodeid embeds a volatile value (e.g. a commit SHA in a parametrize id) that makes `new` mode flag them as new on every run and rerun them pointlessly. Provide `@pytest.mark.mergify(reruns=False)` so authors can opt a test out of the rerun loop in every mode, without disabling the plugin or altering the test's identity. Change-Id: Iafe9ba67112c1647c4977c1a9093e998042f7c71

JulianMaurin

LGTM 👍

One naming thought, totally non-blocking: reruns=False borrows the pytest-rerunfailures kwarg, but there it's an int count (@pytest.mark.flaky(reruns=5)), so a plural noun as a bool reads a bit odd. And since the mark opts the test out of flaky detection entirely (no cicd.test.* flaky attributes at all), not just the rerun loop, maybe mergify(flaky_detection=False) would describe the behavior more accurately — and leaves room for other per-test options later. reruns=0 could be an alternative if you'd rather stay in the rerunfailures vocabulary. Feel free to ignore!

mergify · 2026-07-07T12:17:23Z

…ction` The per-test opt-out shipped as `@pytest.mark.mergify(reruns=False)` in #441. Rename it to `flaky_detection=False`: the option governs Mergify's flaky-detection reruns specifically, so a `flaky_detection` name makes that scope explicit and avoids confusion with generic rerun plugins such as pytest-rerunfailures. Change-Id: Ib3e7f9601127fe5d3e8ae16de452dedfc288f231

…ction` (#442) The per-test opt-out shipped as `@pytest.mark.mergify(reruns=False)` in #441. Rename it to `flaky_detection=False`: the option governs Mergify's flaky-detection reruns specifically, so a `flaky_detection` name makes that scope explicit and avoids confusion with generic rerun plugins such as pytest-rerunfailures.

Copilot AI review requested due to automatic review settings July 7, 2026 09:42

mergify Bot had a problem deploying to Mergify Merge Protections July 7, 2026 09:42 Failure

Copilot started reviewing on behalf of remyduthu July 7, 2026 09:42 View session

mergify Bot requested a review from a team July 7, 2026 09:43

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread tests/test_ci_insights.py

Comment thread tests/test_ci_insights.py Outdated

Comment thread tests/test_ci_insights.py

Comment thread pytest_mergify/flaky_detection.py

mergify Bot added the review threads unresolved label Jul 7, 2026

remyduthu self-assigned this Jul 7, 2026