feat: run only the reduced test selection served by Mergify#448
Open
AlexandreGaubert wants to merge 1 commit into
Open
feat: run only the reduced test selection served by Mergify#448AlexandreGaubert wants to merge 1 commit into
AlexandreGaubert wants to merge 1 commit into
Conversation
On a merge-queue rerun (max_checks_retries attempt or bisection step), Mergify already knows which tests failed on the previous attempt. At session start, ask the new test-selection endpoint with the run's own identity (head branch + head SHA + pipeline + job — the same values this plugin uploads with every test); when the answer is a subset, filter the collection down to it (exact nodeid match, standard pytest_deselected reporting) and say so in the terminal summary. Full-suite is the only fallback: any HTTP error, timeout, missing endpoint (404), disabled feature, or a subset matching none of the collected tests (renamed tests) runs everything. The reduced run can only remove work, never correctness. Disable with MERGIFY_TEST_SELECTION_DISABLE=true. Related to MRGFY-7854 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Merge Protections🔴 3 of 6 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 ApprovalWaiting for
This rule is failing.
🔴 Continuous IntegrationWaiting for
This rule is failing.
🔴 🔎 ReviewsWaiting for
This rule is failing.
Show 3 satisfied protections🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 📕 PR description
🟢 🚦 Auto-queueWhen all merge protections are satisfied, this pull request will be queued automatically. |
There was a problem hiding this comment.
Pull request overview
This PR introduces support for Mergify “test selection” so that merge-queue reruns (e.g., retries/bisection) can execute only the subset of tests that previously failed, while safely degrading to running the full suite on any error/unknown condition.
Changes:
- Add a
TestSelectionclient that queries the Mergify test-selection endpoint and filters collected pytest items accordingly. - Wire test selection loading into
MergifyCIInsightsand apply it viapytest_collection_modifyitems(trylast). - Add unit tests validating subset application and safe fallbacks for common failure modes.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
pytest_mergify/test_selection.py |
New TestSelection dataclass to query the API and filter collected items for reduced reruns. |
pytest_mergify/ci_insights.py |
Loads TestSelection at startup (with kill switch + xdist-worker guard) using CI resource attributes. |
pytest_mergify/__init__.py |
Applies selection during collection and reports selection status/errors in terminal summary. |
tests/test_test_selection.py |
Adds tests for subset filtering and fallback behavior on errors/missing endpoint. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+67
to
+75
| try: | ||
| response.raise_for_status() | ||
| payload = response.json() | ||
| selection = payload["selection"] | ||
| reason = payload["reason"] | ||
| tests = payload["tests"] | ||
| except (requests.HTTPError, requests.exceptions.JSONDecodeError, KeyError) as exc: | ||
| self.init_error_msg = f"Error when querying Mergify's API, the full test suite will run. Error: {str(exc)}" | ||
| return |
Comment on lines
+196
to
+205
| if ( | ||
| self.token is None | ||
| or self.repo_name is None | ||
| or utils.strtobool( | ||
| os.environ.get("MERGIFY_TEST_SELECTION_DISABLE", "false") | ||
| ) | ||
| # On xdist workers the controller already filtered the collection. | ||
| or os.environ.get("PYTEST_XDIST_WORKER") is not None | ||
| ): | ||
| return |
Comment on lines
+134
to
+140
| @responses.activate | ||
| def test_connection_error_degrades_to_full() -> None: | ||
| # No responses registered → ConnectionError raised by the responses lib. | ||
| selection = _make_selection() | ||
|
|
||
| assert selection.selection == "full" | ||
| assert selection.init_error_msg is not None |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a merge-queue batch fails and Mergify reruns it (bisection or
max_checks_retries), the rerun does not need to execute the whole suite: only the tests that failed on the predecessor run are informative. This PR makes pytest-mergify ask the new Mergify test selection endpoint whether the current run is such a reduced rerun, and if so deselects everything but the previously-failing tests.How it works
GET /v1/ci/{owner}/repositories/{repo}/test-selectionwith the run's branch, head SHA, pipeline name and job name — the exact identity keys the uploaded test results are stored under.selection: subsetwith a list of test ids, orselection: full. The server owns the whole policy ("is this a merge-queue rerun?", "do I have the predecessor's results?"); the plugin just obeys. Anything other than a cleansubsetanswer — HTTP error, timeout, feature disabled, unknown response — falls back to running the full suite, so the plugin can never lose test coverage.subset,pytest_collection_modifyitemskeeps only the selected node ids (properly reported as deselected), and the terminal summary prints a✂️ Test selectionsection stating what was selected and why.Safety rails
MERGIFY_TEST_SELECTION_DISABLE=1skips the endpoint call entirely.pytest-xdistworkers never query the endpoint (only the controller would).full.Server side: Mergifyio/monorepo#36661.
Validated end-to-end on a real GitHub Actions runner against a live engine: a merge-queue retry draft PR collected 4 tests, deselected 2 and ran only the 2 previously-failing ones (
Selection: subset (reason: reduced_rerun)), with the full suite correctly restored on normal runs.Related to MRGFY-7854
🤖 Generated with Claude Code