ci: reduce PR test matrix, run full matrix in merge queue#830
Merged
Conversation
PRs now run a minimal CI matrix to save time and resources: - Python tests: only Python 3.12 on Ubuntu (was 3 versions x 2 OSes) - E2E tests: only amd64 (was amd64 + arm64) The merge queue runs the full matrix before merging to main: - Python tests: 3.11, 3.12, 3.13 on both Ubuntu and macOS - E2E tests: both amd64 and arm64 This reduces PR CI from ~12 parallel jobs to ~4, while maintaining full coverage before merge.
Contributor
📝 WalkthroughWalkthroughTwo GitHub Actions workflows gain event-driven matrix computation. In ChangesDynamic CI Matrix Computation and Execution Control
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Move e2e-compat-old-controller and e2e-compat-old-client to only run on merge queue, push to main, and workflow_dispatch. These cross-version compatibility tests are important but don't need to gate every PR.
Add a rollup job (like pytest already has) so GitHub branch protection can use a single 'e2e' required check that works for both PRs and merge queue, regardless of which matrix jobs ran. The rollup tolerates intentionally-skipped compat jobs on PRs but fails if e2e-tests itself was skipped when changes were detected.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
.github/workflows/e2e.yaml (1)
19-23: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winGuard matrix consumers when
changescan be skipped.Line 19 can skip
changes, but Lines 61/100/185 still parseneeds.changes.outputs.e2e-matrix. On forkworkflow_dispatch/pushruns, this can becomefromJson('')and fail before jobs start.Suggested fix
build-controller-image: needs: changes - if: needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch' || (github.event_name == 'push' && github.ref == 'refs/heads/main') + if: github.repository_owner == 'jumpstarter-dev' && (needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch' || (github.event_name == 'push' && github.ref == 'refs/heads/main')) strategy: matrix: include: ${{ fromJson(needs.changes.outputs.e2e-matrix) }} build-operator-image: needs: changes - if: needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch' || (github.event_name == 'push' && github.ref == 'refs/heads/main') + if: github.repository_owner == 'jumpstarter-dev' && (needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch' || (github.event_name == 'push' && github.ref == 'refs/heads/main')) strategy: matrix: include: ${{ fromJson(needs.changes.outputs.e2e-matrix) }}Also applies to: 61-61, 100-100, 182-185
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/e2e.yaml around lines 19 - 23, The jobs at lines 61, 100, and 185 that consume the `needs.changes.outputs.e2e-matrix` output lack guards to handle when the "changes" job is skipped on fork repositories. Add conditional checks to these consuming jobs (typically using `if: needs.changes.result == 'success'` or similar) to ensure they only attempt to parse the e2e-matrix output when the upstream changes job has actually executed, preventing fromJson('') failures on fork workflow_dispatch and push runs..github/workflows/python-tests.yaml (1)
18-24: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winAlign
pytest-matrixgating withchangesoutput dependency.Line 18 may skip
changes, but Lines 61/67 always parse its outputs. With Line 57 allowingworkflow_dispatch, this can fail asfromJson('')instead of cleanly skipping.Suggested fix
pytest-matrix: needs: changes - if: needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch' + if: github.repository_owner == 'jumpstarter-dev' && (needs.changes.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch') runs-on: ${{ matrix.runs-on }} strategy: matrix: runs-on: ${{ fromJson(needs.changes.outputs.runners) }} python-version: ${{ fromJson(needs.changes.outputs.python-versions) }}Also applies to: 55-67
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/python-tests.yaml around lines 18 - 24, The pytest-matrix job references outputs from the changes job (steps.filter.outputs.python, steps.filter.outputs.renode_driver) on lines 61/67, but the changes job may be skipped due to the conditional on line 18 when github.repository_owner is not 'jumpstarter-dev'. Since workflow_dispatch can trigger the workflow regardless of this condition, the pytest-matrix job will fail trying to parse empty outputs with fromJson(''). Add an explicit dependency by including needs: changes in the pytest-matrix job definition and apply the same repository owner conditional (if: github.repository_owner == 'jumpstarter-dev') to ensure pytest-matrix only runs when changes has successfully produced outputs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/e2e.yaml:
- Around line 19-23: The jobs at lines 61, 100, and 185 that consume the
`needs.changes.outputs.e2e-matrix` output lack guards to handle when the
"changes" job is skipped on fork repositories. Add conditional checks to these
consuming jobs (typically using `if: needs.changes.result == 'success'` or
similar) to ensure they only attempt to parse the e2e-matrix output when the
upstream changes job has actually executed, preventing fromJson('') failures on
fork workflow_dispatch and push runs.
In @.github/workflows/python-tests.yaml:
- Around line 18-24: The pytest-matrix job references outputs from the changes
job (steps.filter.outputs.python, steps.filter.outputs.renode_driver) on lines
61/67, but the changes job may be skipped due to the conditional on line 18 when
github.repository_owner is not 'jumpstarter-dev'. Since workflow_dispatch can
trigger the workflow regardless of this condition, the pytest-matrix job will
fail trying to parse empty outputs with fromJson(''). Add an explicit dependency
by including needs: changes in the pytest-matrix job definition and apply the
same repository owner conditional (if: github.repository_owner ==
'jumpstarter-dev') to ensure pytest-matrix only runs when changes has
successfully produced outputs.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4d411be1-2ad9-460b-bc17-50fadc4a377c
📒 Files selected for processing (2)
.github/workflows/e2e.yaml.github/workflows/python-tests.yaml
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
.github/workflows/e2e.yaml (1)
338-338: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuePin the rollup runner for consistency.
The rest of this workflow pins runners (e.g.
ubuntu-24.04), but the aggregator uses the floatingubuntu-latest. Pinning avoids surprise behavior changes when GitHub rolls thelatestlabel.♻️ Optional consistency tweak
e2e: - runs-on: ubuntu-latest + runs-on: ubuntu-24.04🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/e2e.yaml at line 338, The aggregator job at line 338 uses the floating `ubuntu-latest` runner while other jobs in the workflow use pinned versions like `ubuntu-24.04`. Replace the `runs-on: ubuntu-latest` value with a specific pinned version (e.g., `ubuntu-24.04`) to match the consistency of other jobs in the workflow and prevent unexpected behavior changes when GitHub updates what the `latest` tag points to.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/workflows/e2e.yaml:
- Line 338: The aggregator job at line 338 uses the floating `ubuntu-latest`
runner while other jobs in the workflow use pinned versions like `ubuntu-24.04`.
Replace the `runs-on: ubuntu-latest` value with a specific pinned version (e.g.,
`ubuntu-24.04`) to match the consistency of other jobs in the workflow and
prevent unexpected behavior changes when GitHub updates what the `latest` tag
points to.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c16c1cef-0f0b-4bae-a98d-c58619351ce2
📒 Files selected for processing (1)
.github/workflows/e2e.yaml
kirkbrauer
approved these changes
Jun 23, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PR CI runs the full test matrix (3 Python versions x 2 OSes = 6 pytest jobs, plus 2 architectures for e2e), which is slow and expensive. Most of this coverage is redundant for validating a PR — the full matrix only needs to pass before merging.
Changes
Python tests (
python-tests.yaml)E2E tests (
e2e.yaml)Implementation
Both workflows compute the matrix dynamically in the
changesjob based ongithub.event_name, outputting JSON arrays consumed byfromJson()in the matrix strategy.Impact
No reduction in pre-merge coverage — the merge queue runs everything.