Skip to content

Commit c9f23c2

Browse files
eduardoboucaseduardoboucas
authored
chore: optimize CI (#8199)
Currently, shipping a single feature ends up triggering the full cross-platform test matrix roughly four times — once on the feature PR, once on the `main` merge, once on the release-please PR, and once more on the release merge. That's ~200+ jobs per released feature, most of them duplicating work that had already passed on a parent commit. Combined with the slower and flakier Windows runners, the feedback loop on PRs had grown long and noisy, and the release workflow was spending a lot of CI time re-verifying code that was already green. This PR restructures when and where tests run across the development lifecycle so that we only pay for the coverage that actually tells us something new. ## Problems this fixes - **Redundant test runs across the release lifecycle.** The same commit was being tested on the PR, on the `main` push after merge, on the release PR, on the release merge, and again inside the publish job. We now test each commit at most twice (feature PR + release PR), and the publish job trusts the green release PR instead of re-running E2E. - **Slow feedback on feature PRs.** Every PR had to wait for the full 3 OS × 3 Node matrix across unit, integration (sharded 4 ways), and E2E — while most issues are catchable on Linux alone. Feature PRs now run Linux-only, trading a small amount of platform coverage at PR time for dramatically faster iteration. - **Flaky Windows E2E gating every PR.** Windows-specific flakiness was a common reason PRs got stuck. Windows (and macOS) now only run on release PRs and on the nightly schedule — they're still part of our safety net, just moved to where their cost is acceptable. - **A broken release-PR filter.** The existing `!release-please--**` filter in the test workflows was targeting the PR base branch and therefore never actually skipped anything. Fixed by detecting release-please PRs via `github.head_ref` and using that to switch matrix strategies. - **Docs-only PRs running the full test suite.** Added a `paths-filter` step so changes that only touch markdown, LICENSE, and template files skip the test jobs entirely. - **Hardcoded Node versions scattered across eleven workflow files.** Bumping Node versions used to mean edits in many places with inconsistencies creeping in (e.g. `22.x` vs `22`). Consolidated into a single config file (see below). ## When each workflow now runs ### Unit, Integration, E2E tests | Trigger | Matrix | |---|---| | Feature PR | Ubuntu × 3 Node versions | | Release-please PR | Ubuntu × 3 Node versions + macOS/Windows on primary Node | | Nightly schedule (E2E only) | Full matrix (same as release PR) | | Push to `main` | **Not run** — the PR already tested this commit | | Publish job | **Not run** — the release PR already tested this commit | If a PR only changes documentation (markdown, LICENSE, issue/PR templates, Vale config), the test jobs are skipped and a lightweight status check passes, so required checks still resolve. ### Release pipeline - **Feature PR merges to `main`** → no test run. Release-please opens or updates the release PR. - **Release-please PR** → runs the *full* cross-platform matrix as the last line of defense. - **Release-please PR merges** → publish job builds and publishes without re-running E2E. It trusts the release PR's checks. ### Single-version workflows (lint, typecheck, format, verify-docs, benchmark, pre-release, release publish) Same triggers as before. Each now reads its Node version from the shared config rather than hardcoding `'24'`. ## Practical implications - **Faster PR feedback.** Feature PRs now run roughly one-third of the jobs they used to. The mean wait-for-CI should drop significantly, and Windows flakes no longer block unrelated work. - **A new tradeoff.** Because Windows/macOS aren't tested on every feature PR, a platform-specific regression can slip through to the release PR. The expectation is that this is rare and cheap to fix — but the first few times it happens, the follow-up is a small PR rather than a blocked release. - **Release PR is now genuinely the final gate.** If its checks are green, the publish job will ship without re-testing. Do not merge the release PR if anything in that matrix is failing or was manually bypassed. ## Centralized source of truth: `.github/test-matrix.yml` All Node versions and cross-platform OS lists are now declared once, in a short YAML config file at `.github/test-matrix.yml`. Each key has a comment explaining what it drives and which workflows consume it. The three fields are: - `node_versions` — the list of Node versions that run on Linux in every PR. - `extra_oses_on_release` — the additional operating systems added on release PRs and on the nightly E2E schedule. - `primary_node_version` — the single "primary" Node version used both for the extra OSes in the release matrix and as the pinned version in every single-version workflow (lint, typecheck, publish, etc.). Each workflow reads this file at run time via `yq` (pre-installed on GitHub-hosted Ubuntu runners). To add or drop a Node version, or to rotate the primary version, edit this one file — no workflow changes required.
1 parent 901060d commit c9f23c2

12 files changed

Lines changed: 295 additions & 101 deletions

‎.github/test-matrix.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
# Node.js versions tested on Linux in every PR's test matrix.
3+
# The minimum version should stay aligned with package.json `engines.node`.
4+
# Used by: unit-tests.yml, integration-tests.yml, e2e-tests.yml
5+
node_versions: ['20.12.2', '22', '24']
6+
7+
# Additional operating systems added to the full test matrix on release-please
8+
# PRs and scheduled (cron) E2E runs. Only the primary Node version (see below)
9+
# is tested on these platforms to keep the job count manageable.
10+
# Do NOT include ubuntu here — Linux is always covered by `node_versions` above.
11+
# Used by: unit-tests.yml, integration-tests.yml, e2e-tests.yml
12+
extra_oses_on_release: ['macOS-latest', 'windows-2025']
13+
14+
# The "primary" Node.js version. Two uses:
15+
# 1. Paired with each of `extra_oses_on_release` in the full matrix.
16+
# 2. The pinned version for all single-version workflows.
17+
# Must be present in `node_versions` above.
18+
# Used by: unit-tests.yml, integration-tests.yml, e2e-tests.yml,
19+
# release-please.yml, pre-release.yml, pre-release-sha.yml,
20+
# benchmark.yml, lint.yml, typecheck.yml, format.yml, verify-docs.yml
21+
primary_node_version: '24'

‎.github/workflows/benchmark.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,14 @@ jobs:
1414
- name: Git checkout
1515
uses: actions/checkout@v6
1616

17+
- name: Read primary Node version
18+
id: node
19+
run: echo "version=$(yq eval '.primary_node_version' .github/test-matrix.yml)" >> "$GITHUB_OUTPUT"
20+
1721
- name: Use Node.js
1822
uses: actions/setup-node@v6
1923
with:
20-
node-version: '24'
24+
node-version: ${{ steps.node.outputs.version }}
2125
cache: npm
2226

2327
- name: Install dependencies

‎.github/workflows/e2e-tests.yml‎

Lines changed: 84 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -2,30 +2,80 @@
22
name: E2E Tests
33

44
on:
5-
push:
6-
branches: [main]
75
pull_request:
8-
branches:
9-
- '**'
10-
- '!release-please--**'
6+
branches: ['**']
117
schedule:
128
# check daily to notice potential package manager issues
139
- cron: '0 1 * * *' # At 01:00 daily
1410

1511
jobs:
16-
e2e-windows:
17-
name: E2E Windows tests
12+
setup:
13+
name: Setup
14+
runs-on: ubuntu-latest
15+
outputs:
16+
matrix: ${{ steps.matrix.outputs.value }}
17+
has-code-changes: ${{ steps.filter.outputs.code || 'true' }}
18+
steps:
19+
- name: Git checkout
20+
uses: actions/checkout@v6
21+
22+
- name: Detect code changes
23+
uses: dorny/paths-filter@v3
24+
if: github.event_name == 'pull_request'
25+
id: filter
26+
with:
27+
filters: |
28+
code:
29+
- '**'
30+
- '!**/*.md'
31+
- '!LICENSE'
32+
- '!docs/**'
33+
- '!.vale.ini'
34+
- '!.github/ISSUE_TEMPLATE/**'
35+
- '!.github/PULL_REQUEST_TEMPLATE.md'
36+
37+
- name: Compute matrix
38+
id: matrix
39+
env:
40+
HEAD_REF: ${{ github.head_ref }}
41+
EVENT_NAME: ${{ github.event_name }}
42+
run: |
43+
node_versions=$(yq eval '.node_versions | join(" ")' .github/test-matrix.yml)
44+
extra_oses=$(yq eval '.extra_oses_on_release | join(" ")' .github/test-matrix.yml)
45+
primary=$(yq eval '.primary_node_version' .github/test-matrix.yml)
46+
47+
OS_NODE=()
48+
for node in $node_versions; do
49+
OS_NODE+=("ubuntu-latest:$node")
50+
done
51+
if [[ "$EVENT_NAME" == "schedule" ]] || [[ "$HEAD_REF" == release-please--* ]]; then
52+
for os in $extra_oses; do
53+
OS_NODE+=("$os:$primary")
54+
done
55+
fi
56+
57+
entries=()
58+
for combo in "${OS_NODE[@]}"; do
59+
os="${combo%:*}"
60+
node="${combo##*:}"
61+
entries+=("{\"os\":\"$os\",\"node-version\":\"$node\"}")
62+
done
63+
joined=$(IFS=,; echo "${entries[*]}")
64+
echo "value={\"include\":[$joined]}" >> "$GITHUB_OUTPUT"
65+
66+
e2e:
67+
name: E2E
68+
needs: setup
69+
if: needs.setup.outputs.has-code-changes == 'true'
1870
runs-on: ${{ matrix.os }}
1971
timeout-minutes: 30
2072
strategy:
21-
matrix:
22-
os: [windows-2025]
23-
# Pinning 20.x version as a temporary workaround due to this https://github.com/nodejs/node/issues/52884
24-
node-version: ['20.12.2', '22', '24']
2573
fail-fast: false
74+
matrix: ${{ fromJson(needs.setup.outputs.matrix) }}
2675
steps:
2776
# This improves Windows network performance. We need this since we open many ports in our tests.
2877
- name: Increase Windows port limit and reduce time wait delay
78+
if: matrix.os == 'windows-2025'
2979
run: |
3080
netsh int ipv4 set dynamicport tcp start=1025 num=64511
3181
REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TCPIP\Parameters /v TcpTimedWaitDelay /t REG_DWORD /d 30 /f
@@ -42,59 +92,14 @@ jobs:
4292
cache: npm
4393
check-latest: true
4494

45-
# Let corepack manage npm, pnpm, and yarn versions for E2E tests
46-
- name: Enable corepack
95+
# Let corepack manage npm, pnpm, and yarn versions on Windows.
96+
- name: Enable corepack (Windows)
97+
if: matrix.os == 'windows-2025'
4798
run: corepack enable
4899

49-
- name: Install dependencies
50-
run: npm ci --no-audit
51-
52-
- name: Build project
53-
run: npm run build
54-
55-
- name: Run E2E tests
56-
run: npm run test:e2e
57-
58-
- name: Notify Slack
59-
uses: 8398a7/action-slack@v3
60-
if: ${{ github.event_name == 'schedule' && failure() }}
61-
with:
62-
status: custom
63-
fields: workflow
64-
custom_payload: |
65-
{
66-
attachments: [{
67-
title: 'E2E Test Failed! (${{ matrix.os }},node-${{ matrix.node-version }}',
68-
color: 'danger',
69-
text: `${process.env.AS_WORKFLOW}\n\nEither something broke or a test is flaky.\n\nConsider doing something about it :fire_engine:`,
70-
}]
71-
}
72-
env:
73-
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
74-
75-
e2e-linux-based-os:
76-
name: E2E
77-
runs-on: ${{ matrix.os }}
78-
timeout-minutes: 30
79-
strategy:
80-
matrix:
81-
os: [ubuntu-latest, macOS-latest]
82-
node-version: ['20.12.2', '22', '24']
83-
fail-fast: false
84-
steps:
85-
- name: Git checkout
86-
uses: actions/checkout@v6
87-
with:
88-
fetch-depth: 0
89-
90-
- name: Setup node
91-
uses: actions/setup-node@v6
92-
with:
93-
node-version: ${{ matrix.node-version }}
94-
cache: npm
95-
check-latest: true
96-
97-
- uses: pnpm/action-setup@v4
100+
- name: Setup pnpm (non-Windows)
101+
if: matrix.os != 'windows-2025'
102+
uses: pnpm/action-setup@v4
98103
with:
99104
version: 10
100105

@@ -123,3 +128,20 @@ jobs:
123128
}
124129
env:
125130
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
131+
132+
status:
133+
name: E2E Tests Status
134+
needs: [setup, e2e]
135+
if: always()
136+
runs-on: ubuntu-latest
137+
steps:
138+
- name: Verify result
139+
run: |
140+
if [[ "${{ needs.setup.result }}" != "success" ]]; then
141+
echo "Setup job failed"
142+
exit 1
143+
fi
144+
if [[ "${{ needs.e2e.result }}" == "failure" || "${{ needs.e2e.result }}" == "cancelled" ]]; then
145+
echo "E2E tests failed"
146+
exit 1
147+
fi

‎.github/workflows/format.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,13 @@ jobs:
1717
steps:
1818
- uses: actions/checkout@v6
1919

20+
- name: Read primary Node version
21+
id: node
22+
run: echo "version=$(yq eval '.primary_node_version' .github/test-matrix.yml)" >> "$GITHUB_OUTPUT"
23+
2024
- uses: actions/setup-node@v6
2125
with:
22-
node-version: '24'
26+
node-version: ${{ steps.node.outputs.version }}
2327
cache: npm
2428

2529
- name: Install dependencies

‎.github/workflows/integration-tests.yml‎

Lines changed: 76 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,26 +2,74 @@
22
name: Integration Tests
33

44
on:
5-
push:
6-
branches:
7-
- main
85
pull_request:
9-
branches:
10-
- '**'
11-
- '!release-please--**'
6+
branches: ['**']
127

138
jobs:
9+
setup:
10+
name: Setup
11+
runs-on: ubuntu-latest
12+
outputs:
13+
matrix: ${{ steps.matrix.outputs.value }}
14+
has-code-changes: ${{ steps.filter.outputs.code || 'true' }}
15+
steps:
16+
- name: Git checkout
17+
uses: actions/checkout@v6
18+
19+
- name: Detect code changes
20+
uses: dorny/paths-filter@v3
21+
if: github.event_name == 'pull_request'
22+
id: filter
23+
with:
24+
filters: |
25+
code:
26+
- '**'
27+
- '!**/*.md'
28+
- '!LICENSE'
29+
- '!docs/**'
30+
- '!.vale.ini'
31+
- '!.github/ISSUE_TEMPLATE/**'
32+
- '!.github/PULL_REQUEST_TEMPLATE.md'
33+
34+
- name: Compute matrix
35+
id: matrix
36+
env:
37+
HEAD_REF: ${{ github.head_ref }}
38+
run: |
39+
node_versions=$(yq eval '.node_versions | join(" ")' .github/test-matrix.yml)
40+
extra_oses=$(yq eval '.extra_oses_on_release | join(" ")' .github/test-matrix.yml)
41+
primary=$(yq eval '.primary_node_version' .github/test-matrix.yml)
42+
43+
OS_NODE=()
44+
for node in $node_versions; do
45+
OS_NODE+=("ubuntu-latest:$node")
46+
done
47+
if [[ "$HEAD_REF" == release-please--* ]]; then
48+
for os in $extra_oses; do
49+
OS_NODE+=("$os:$primary")
50+
done
51+
fi
52+
53+
entries=()
54+
for combo in "${OS_NODE[@]}"; do
55+
os="${combo%:*}"
56+
node="${combo##*:}"
57+
for shard in "1/4" "2/4" "3/4" "4/4"; do
58+
entries+=("{\"os\":\"$os\",\"node-version\":\"$node\",\"shard\":\"$shard\"}")
59+
done
60+
done
61+
joined=$(IFS=,; echo "${entries[*]}")
62+
echo "value={\"include\":[$joined]}" >> "$GITHUB_OUTPUT"
63+
1464
integration:
1565
name: Integration
66+
needs: setup
67+
if: needs.setup.outputs.has-code-changes == 'true'
1668
runs-on: ${{ matrix.os }}
1769
timeout-minutes: 40
1870
strategy:
19-
matrix:
20-
os: [ubuntu-latest, macOS-latest, windows-2025]
21-
# Pinning 20.x version as a temporary workaround due to this https://github.com/nodejs/node/issues/52884
22-
node-version: ['20.12.2', '22', '24']
23-
shard: ['1/4', '2/4', '3/4', '4/4']
2471
fail-fast: false
72+
matrix: ${{ fromJson(needs.setup.outputs.matrix) }}
2573
steps:
2674
# This improves Windows network performance, we need this since we open many ports in our tests
2775
- name: Increase Windows port limit and reduce time wait delay
@@ -104,3 +152,20 @@ jobs:
104152
with:
105153
flags: ${{ steps.test-coverage-flags.outputs.os }},${{ steps.test-coverage-flags.outputs.node }}
106154
token: ${{ secrets.CODECOV_TOKEN }}
155+
156+
status:
157+
name: Integration Tests Status
158+
needs: [setup, integration]
159+
if: always()
160+
runs-on: ubuntu-latest
161+
steps:
162+
- name: Verify result
163+
run: |
164+
if [[ "${{ needs.setup.result }}" != "success" ]]; then
165+
echo "Setup job failed"
166+
exit 1
167+
fi
168+
if [[ "${{ needs.integration.result }}" == "failure" || "${{ needs.integration.result }}" == "cancelled" ]]; then
169+
echo "Integration tests failed"
170+
exit 1
171+
fi

‎.github/workflows/lint.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,13 @@ jobs:
1717
steps:
1818
- uses: actions/checkout@v6
1919

20+
- name: Read primary Node version
21+
id: node
22+
run: echo "version=$(yq eval '.primary_node_version' .github/test-matrix.yml)" >> "$GITHUB_OUTPUT"
23+
2024
- uses: actions/setup-node@v6
2125
with:
22-
node-version: '24'
26+
node-version: ${{ steps.node.outputs.version }}
2327
cache: npm
2428

2529
- name: Install dependencies

‎.github/workflows/pre-release-sha.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,13 @@ jobs:
2020
with:
2121
ref: ${{ inputs.ref }}
2222

23+
- name: Read primary Node version
24+
id: node
25+
run: echo "version=$(yq eval '.primary_node_version' .github/test-matrix.yml)" >> "$GITHUB_OUTPUT"
26+
2327
- uses: actions/setup-node@v6
2428
with:
25-
node-version: '24'
29+
node-version: ${{ steps.node.outputs.version }}
2630
cache: npm
2731
registry-url: 'https://registry.npmjs.org'
2832

‎.github/workflows/pre-release.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,13 @@ jobs:
1414
steps:
1515
- uses: actions/checkout@v6
1616

17+
- name: Read primary Node version
18+
id: node
19+
run: echo "version=$(yq eval '.primary_node_version' .github/test-matrix.yml)" >> "$GITHUB_OUTPUT"
20+
1721
- uses: actions/setup-node@v6
1822
with:
19-
node-version: '24'
23+
node-version: ${{ steps.node.outputs.version }}
2024
cache: npm
2125
registry-url: 'https://registry.npmjs.org'
2226

0 commit comments

Comments
 (0)