Skip to content

Commit 6a82aae

Browse files
lucarliglucarlig
authored
Use cargo-nextest for Rust tests (#68)
* Use cargo-nextest for Rust tests Signed-off-by: lucarlig <luca.carlig@ibm.com> * Address PR #68 review follow-ups for CI tooling and validation. Use binary installs for cargo-nextest/cargo-mutants, extract CI selection validation into a maintainable Python script, and document why Python::initialize() is required in the rate_limiter test under nextest process isolation. Signed-off-by: lucarlig <luca.carlig@ibm.com> * Remove taiki-e action usage from CI workflows. Install cargo-nextest via the official nexte.st binary script on Linux and keep a cargo-install fallback for non-Linux matrix jobs so IBM org action allowlists are no longer required. Signed-off-by: lucarlig <luca.carlig@ibm.com> * Fix CI workflow contract expected by plugin-catalog tests. Keep the IBM-allowed nextest installer path while restoring canonical step names and single-step install blocks so workflow-shape tests pass without reintroducing disallowed actions. Signed-off-by: lucarlig <luca.carlig@ibm.com> * Use nextest binary installer on macOS CI Install cargo-nextest from official macOS binaries to avoid source builds on macOS runners while keeping the existing Windows fallback. Signed-off-by: lucarlig <luca.carlig@ibm.com> * Fix nextest macOS installer URL Signed-off-by: lucarlig <luca.carlig@ibm.com> * Fix PR mutation diff scope Signed-off-by: lucarlig <luca.carlig@ibm.com> --------- Signed-off-by: lucarlig <luca.carlig@ibm.com>
1 parent 9c3c326 commit 6a82aae

19 files changed

Lines changed: 925 additions & 68 deletions

File tree

‎.cargo/mutants.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
3+
test_tool = "nextest"
4+
additional_cargo_test_args = ["--profile=mutants"]
5+
cap_lints = false

‎.config/nextest.toml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
3+
nextest-version = "0.9.133"
4+
5+
[profile.ci]
6+
fail-fast = false
7+
failure-output = "immediate-final"
8+
9+
[profile.mutants]
10+
fail-fast = true
11+
failure-output = "final"

‎.github/workflows/ci-rust-python-package.yaml‎

Lines changed: 124 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ on:
77
- "Makefile"
88
- "Cargo.toml"
99
- "Cargo.lock"
10+
- ".cargo/**"
11+
- ".config/nextest.toml"
1012
- "deny.toml"
1113
- "crates/**"
1214
- "README.md"
@@ -23,6 +25,8 @@ on:
2325
- "Makefile"
2426
- "Cargo.toml"
2527
- "Cargo.lock"
28+
- ".cargo/**"
29+
- ".config/nextest.toml"
2630
- "deny.toml"
2731
- "crates/**"
2832
- "README.md"
@@ -51,6 +55,9 @@ jobs:
5155
has_plugins: ${{ steps.detect.outputs.has_plugins }}
5256
plugin_count: ${{ steps.detect.outputs.plugin_count }}
5357
cargo_packages: ${{ steps.detect.outputs.cargo_packages }}
58+
mutation_cargo_packages: ${{ steps.detect.outputs.mutation_cargo_packages }}
59+
mutation_jobs: ${{ steps.detect.outputs.mutation_jobs }}
60+
has_mutation_cargo_packages: ${{ steps.detect.outputs.has_mutation_cargo_packages }}
5461
steps:
5562
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
5663
with:
@@ -65,7 +72,7 @@ jobs:
6572
run: |
6673
set -euo pipefail
6774
if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then
68-
selection="$(python3 tools/plugin_catalog.py ci-selection . diff "${{ github.event.pull_request.base.sha }}" "${{ github.sha }}")"
75+
selection="$(python3 tools/plugin_catalog.py ci-selection . diff "${{ github.event.pull_request.base.sha }}" "${{ github.event.pull_request.head.sha }}")"
6976
elif [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" ]]; then
7077
selection="$(python3 tools/plugin_catalog.py ci-selection . all '' '')"
7178
elif [[ "${{ github.event.before }}" == "0000000000000000000000000000000000000000" ]]; then
@@ -74,21 +81,28 @@ jobs:
7481
selection="$(python3 tools/plugin_catalog.py ci-selection . diff "${{ github.event.before }}" "${{ github.sha }}")"
7582
fi
7683
77-
selection="$(printf '%s' "${selection}" | python3 -c 'import json, re, sys; payload = json.load(sys.stdin); slug_re = re.compile(r"^[a-z0-9_]+$"); plugins = payload.get("plugins"); cargo_packages = payload.get("cargo_packages"); has_plugins = payload.get("has_plugins"); plugin_count = payload.get("plugin_count"); assert isinstance(plugins, list) and all(isinstance(item, str) and slug_re.fullmatch(item) for item in plugins); assert isinstance(cargo_packages, list) and all(isinstance(item, str) and slug_re.fullmatch(item) for item in cargo_packages); assert isinstance(has_plugins, bool); assert isinstance(plugin_count, int) and plugin_count == len(plugins); print(json.dumps({"plugins": plugins, "has_plugins": has_plugins, "plugin_count": plugin_count, "cargo_packages": cargo_packages}))')"
84+
selection="$(printf '%s' "${selection}" | python3 tools/validate_ci_selection.py)"
7885
plugins="$(printf '%s' "${selection}" | python3 -c 'import json, sys; print(json.dumps(json.load(sys.stdin)["plugins"]))')"
7986
has_plugins="$(printf '%s' "${selection}" | python3 -c 'import json, sys; print(str(json.load(sys.stdin)["has_plugins"]).lower())')"
8087
plugin_count="$(printf '%s' "${selection}" | python3 -c 'import json, sys; print(json.load(sys.stdin)["plugin_count"])')"
8188
cargo_packages="$(printf '%s' "${selection}" | python3 -c 'import json, sys; print(json.dumps(json.load(sys.stdin)["cargo_packages"]))')"
89+
mutation_cargo_packages="$(printf '%s' "${selection}" | python3 -c 'import json, sys; print(json.dumps(json.load(sys.stdin)["mutation_cargo_packages"]))')"
90+
mutation_jobs="$(printf '%s' "${selection}" | python3 -c 'import json, sys; print(json.dumps(json.load(sys.stdin)["mutation_jobs"]))')"
91+
has_mutation_cargo_packages="$(printf '%s' "${selection}" | python3 -c 'import json, sys; print(str(json.load(sys.stdin)["has_mutation_cargo_packages"]).lower())')"
8292
if [[ "${has_plugins}" == "false" ]]; then
8393
has_plugins_output="false"
8494
else
8595
has_plugins_output="true"
8696
fi
97+
has_mutation_cargo_packages_output="${has_mutation_cargo_packages}"
8798
{
8899
echo "plugins=${plugins}"
89100
echo "plugin_count=${plugin_count}"
90101
echo "cargo_packages=${cargo_packages}"
91102
echo "has_plugins=${has_plugins_output}"
103+
echo "mutation_cargo_packages=${mutation_cargo_packages}"
104+
echo "mutation_jobs=${mutation_jobs}"
105+
echo "has_mutation_cargo_packages=${has_mutation_cargo_packages_output}"
92106
} >> "$GITHUB_OUTPUT"
93107
94108
build-test:
@@ -115,6 +129,23 @@ jobs:
115129
rustc --version
116130
cargo --version
117131
132+
- name: Install cargo-nextest
133+
run: |
134+
if [[ "${RUNNER_OS}" == "Linux" ]]; then
135+
mkdir -p "${CARGO_HOME:-$HOME/.cargo}/bin"
136+
curl -LsSf https://get.nexte.st/0.9.133/linux | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
137+
elif [[ "${RUNNER_OS}" == "macOS" ]]; then
138+
mkdir -p "${CARGO_HOME:-$HOME/.cargo}/bin"
139+
if [[ "$(uname -m)" == "arm64" ]]; then
140+
curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
141+
else
142+
curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
143+
fi
144+
else
145+
cargo install cargo-nextest --version 0.9.133 --locked
146+
fi
147+
cargo nextest --version
148+
118149
- name: Install uv
119150
run: python -m pip install uv==0.9.30 maturin==1.12.6
120151

@@ -125,11 +156,15 @@ jobs:
125156
- name: Plugin CI build verification
126157
if: matrix.os == 'ubuntu-latest'
127158
working-directory: plugins/rust/python-package/${{ matrix.plugin }}
159+
env:
160+
NEXTEST_PROFILE: ci
128161
run: make ci-build
129162

130163
- name: Plugin CI verification
131164
if: matrix.os != 'ubuntu-latest'
132165
working-directory: plugins/rust/python-package/${{ matrix.plugin }}
166+
env:
167+
NEXTEST_PROFILE: ci
133168
run: make ci
134169

135170
security-policy:
@@ -150,6 +185,73 @@ jobs:
150185
- name: Run cargo deny
151186
run: cargo deny check --config deny.toml
152187

188+
mutation-testing:
189+
needs: validate-and-detect
190+
if: github.event_name == 'pull_request' && needs.validate-and-detect.outputs.has_mutation_cargo_packages == 'true'
191+
strategy:
192+
fail-fast: false
193+
matrix:
194+
mutation_job: ${{ fromJson(needs.validate-and-detect.outputs.mutation_jobs) }}
195+
runs-on: ubuntu-latest
196+
defaults:
197+
run:
198+
shell: bash
199+
steps:
200+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
201+
with:
202+
fetch-depth: 0
203+
204+
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405
205+
with:
206+
python-version: "3.12"
207+
208+
- name: Verify Rust toolchain
209+
run: |
210+
rustc --version
211+
cargo --version
212+
213+
- name: Install Rust mutation testing tooling
214+
run: |
215+
if [[ "${RUNNER_OS}" == "Linux" ]]; then
216+
mkdir -p "${CARGO_HOME:-$HOME/.cargo}/bin"
217+
curl -LsSf https://get.nexte.st/0.9.133/linux | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
218+
elif [[ "${RUNNER_OS}" == "macOS" ]]; then
219+
mkdir -p "${CARGO_HOME:-$HOME/.cargo}/bin"
220+
if [[ "$(uname -m)" == "arm64" ]]; then
221+
curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
222+
else
223+
curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
224+
fi
225+
else
226+
cargo install cargo-nextest --version 0.9.133 --locked
227+
fi
228+
cargo nextest --version
229+
cargo install cargo-mutants --version 27.0.0 --locked
230+
cargo mutants --version
231+
232+
- name: Create mutation diff
233+
env:
234+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
235+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
236+
run: git diff "${BASE_SHA}..${HEAD_SHA}" -- '*.rs' > cargo-mutants.diff
237+
238+
- name: Run cargo-mutants with nextest
239+
env:
240+
CARGO_PACKAGE: ${{ matrix.mutation_job.cargo_package }}
241+
IN_DIFF: ${{ matrix.mutation_job.in_diff }}
242+
PYO3_PYTHON: python
243+
TEST_PACKAGES: ${{ toJson(matrix.mutation_job.test_packages) }}
244+
run: |
245+
mapfile -t test_packages < <(python3 -c 'import json, os; [print(package) for package in json.loads(os.environ["TEST_PACKAGES"])]')
246+
cargo_args=("-p" "${CARGO_PACKAGE}")
247+
if [[ "${IN_DIFF}" == "true" ]]; then
248+
cargo_args+=("--in-diff" "cargo-mutants.diff")
249+
fi
250+
for package in "${test_packages[@]}"; do
251+
cargo_args+=("--test-package" "${package}")
252+
done
253+
cargo mutants "${cargo_args[@]}"
254+
153255
coverage:
154256
needs: validate-and-detect
155257
if: needs.validate-and-detect.outputs.has_plugins == 'true'
@@ -169,12 +271,30 @@ jobs:
169271
rustup component add llvm-tools-preview
170272
cargo install cargo-llvm-cov --version 0.8.4 --locked
171273
274+
- name: Install cargo-nextest
275+
run: |
276+
if [[ "${RUNNER_OS}" == "Linux" ]]; then
277+
mkdir -p "${CARGO_HOME:-$HOME/.cargo}/bin"
278+
curl -LsSf https://get.nexte.st/0.9.133/linux | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
279+
elif [[ "${RUNNER_OS}" == "macOS" ]]; then
280+
mkdir -p "${CARGO_HOME:-$HOME/.cargo}/bin"
281+
if [[ "$(uname -m)" == "arm64" ]]; then
282+
curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
283+
else
284+
curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
285+
fi
286+
else
287+
cargo install cargo-nextest --version 0.9.133 --locked
288+
fi
289+
cargo nextest --version
290+
172291
- name: Install Python build tooling
173292
run: python -m pip install uv==0.9.30 maturin==1.12.6
174293

175294
- name: Generate Rust coverage report
176295
env:
177296
CARGO_PACKAGES: ${{ needs.validate-and-detect.outputs.cargo_packages }}
297+
NEXTEST_PROFILE: ci
178298
PLUGINS: ${{ needs.validate-and-detect.outputs.plugins }}
179299
PYO3_PYTHON: python
180300
run: |
@@ -186,6 +306,8 @@ jobs:
186306
cargo_args+=("-p" "${package}")
187307
done
188308
cargo llvm-cov clean --workspace
309+
mkdir -p coverage
310+
cargo llvm-cov nextest --no-report "${cargo_args[@]}" -P "${NEXTEST_PROFILE}"
189311
eval "$(cargo llvm-cov show-env --sh)"
190312
export CARGO_TARGET_DIR="${CARGO_LLVM_COV_TARGET_DIR}/llvm-cov-target"
191313
export CARGO_LLVM_COV_BUILD_DIR="${CARGO_TARGET_DIR}"
@@ -194,7 +316,6 @@ jobs:
194316
for plugin in "${plugins[@]}"; do
195317
(cd "plugins/rust/python-package/${plugin}" && make sync && uv run maturin develop)
196318
done
197-
cargo test "${cargo_args[@]}"
198319
for plugin in "${plugins[@]}"; do
199320
(cd "plugins/rust/python-package/${plugin}" && make test-integration)
200321
done

‎.github/workflows/ci-scaffold-generator.yaml‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,9 @@ jobs:
2727
test:
2828
runs-on: ubuntu-latest
2929
steps:
30-
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
30+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
3131

32-
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
32+
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405
3333
with:
3434
python-version: "3.12"
3535

@@ -52,6 +52,23 @@ jobs:
5252
rustc --version
5353
cargo --version
5454
55+
- name: Install cargo-nextest
56+
run: |
57+
if [[ "${RUNNER_OS}" == "Linux" ]]; then
58+
mkdir -p "${CARGO_HOME:-$HOME/.cargo}/bin"
59+
curl -LsSf https://get.nexte.st/0.9.133/linux | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
60+
elif [[ "${RUNNER_OS}" == "macOS" ]]; then
61+
mkdir -p "${CARGO_HOME:-$HOME/.cargo}/bin"
62+
if [[ "$(uname -m)" == "arm64" ]]; then
63+
curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
64+
else
65+
curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf - -C "${CARGO_HOME:-$HOME/.cargo}/bin"
66+
fi
67+
else
68+
cargo install cargo-nextest --version 0.9.133 --locked
69+
fi
70+
cargo nextest --version
71+
5572
- name: Generate default plugin (tool_pre_invoke)
5673
run: |
5774
python3 tools/scaffold_plugin.py --non-interactive \

‎Makefile‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
.PHONY: help plugins-list plugins-validate plugin-test plugin-scaffold plugin-scaffold-help
1+
.PHONY: help plugins-list plugins-validate plugin-test plugin-mutants plugin-mutants-list plugin-scaffold plugin-scaffold-help
22

33
help:
4-
@printf "plugins-list\nplugins-validate\nplugin-test PLUGIN=<slug>\nplugin-scaffold\nplugin-scaffold-help\n"
4+
@printf "plugins-list\nplugins-validate\nplugin-test PLUGIN=<slug>\nplugin-mutants PLUGIN=<slug>\nplugin-mutants-list PLUGIN=<slug>\nplugin-scaffold\nplugin-scaffold-help\n"
55

66
plugins-list:
77
@python3 tools/plugin_catalog.py list .
@@ -14,6 +14,14 @@ plugin-test:
1414
@test -n "$(PLUGIN)" || (echo "Set PLUGIN=<slug>" && exit 1)
1515
@cd plugins/rust/python-package/$(PLUGIN) && make sync && make ci
1616

17+
plugin-mutants:
18+
@test -n "$(PLUGIN)" || (echo "Set PLUGIN=<slug>" && exit 1)
19+
cargo mutants -p "$(PLUGIN)"
20+
21+
plugin-mutants-list:
22+
@test -n "$(PLUGIN)" || (echo "Set PLUGIN=<slug>" && exit 1)
23+
cargo mutants --list -p "$(PLUGIN)"
24+
1725
plugin-scaffold:
1826
@python3 -m pip install --quiet jinja2 2>/dev/null || pip install --quiet jinja2 2>/dev/null || true
1927
@python3 tools/scaffold_plugin.py

‎TESTING.md‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,10 +30,13 @@ Each plugin has its own Rust and Python test suite.
3030
```bash
3131
cd plugins/rust/python-package/rate_limiter
3232
uv sync --dev
33+
cargo install cargo-nextest --version 0.9.133 --locked
3334
make install
3435
make test-all
3536
```
3637

38+
Set `NEXTEST_PROFILE=ci` to use the repository CI profile locally. The CI profile is defined in `.config/nextest.toml`; it disables fail-fast so all Rust test failures are reported in one run.
39+
3740
Equivalent repo-level helper:
3841

3942
```bash
@@ -51,6 +54,7 @@ To run the same coverage check locally for all managed Rust plugins:
5154
```bash
5255
rustup component add llvm-tools-preview
5356
cargo install cargo-llvm-cov --version 0.8.4 --locked
57+
cargo install cargo-nextest --version 0.9.133 --locked
5458
mkdir -p coverage
5559
CARGO_PACKAGES="$(python3 tools/plugin_catalog.py ci-selection-field . all '' '' cargo_packages)"
5660
PLUGINS="$(python3 tools/plugin_catalog.py ci-selection-field . all '' '' plugins)"
@@ -61,6 +65,7 @@ for package in "${cargo_packages[@]}"; do
6165
cargo_args+=("-p" "${package}")
6266
done
6367
cargo llvm-cov clean --workspace
68+
cargo llvm-cov nextest --no-report "${cargo_args[@]}" -P ci
6469
eval "$(cargo llvm-cov show-env --sh)"
6570
export CARGO_TARGET_DIR="${CARGO_LLVM_COV_TARGET_DIR}/llvm-cov-target"
6671
export CARGO_LLVM_COV_BUILD_DIR="${CARGO_TARGET_DIR}"
@@ -69,14 +74,30 @@ mkdir -p "${CARGO_TARGET_DIR}"
6974
for plugin in "${plugins[@]}"; do
7075
(cd "plugins/rust/python-package/${plugin}" && make sync && uv run maturin develop)
7176
done
72-
cargo test "${cargo_args[@]}"
7377
for plugin in "${plugins[@]}"; do
7478
(cd "plugins/rust/python-package/${plugin}" && make test-integration)
7579
done
7680
env -u CARGO_TARGET_DIR -u CARGO_LLVM_COV_BUILD_DIR -u CARGO_LLVM_COV_TARGET_DIR -u LLVM_PROFILE_FILE cargo llvm-cov report "${cargo_args[@]}" --cobertura --output-path coverage/cobertura.xml
7781
python3 tools/plugin_catalog.py coverage-check . coverage/cobertura.xml 90.00 "${PLUGINS}"
7882
```
7983

84+
Rust unit tests use `cargo nextest run`. Coverage uses `cargo llvm-cov nextest --no-report` for the Rust test phase, then runs pytest before generating the final report so PyO3 paths stay covered. CI uses the `ci` nextest profile, which disables fail-fast and prints failure output immediately and again at the end. Nextest does not run Rust doctests; this repo currently has no Rust doctest code blocks, so there is no separate doctest step.
85+
86+
Criterion benchmarks are verified in CI with `cargo nextest run --benches -E 'kind(bench)' --no-run`, which compiles benchmark test targets without rerunning normal unit tests or collecting noisy performance measurements on shared CI runners.
87+
88+
## 4. Mutation Testing
89+
90+
Mutation testing runs in PR CI on Ubuntu for Rust code touched by the pull request diff. It is also available locally through cargo-mutants and runs Rust tests with nextest.
91+
92+
```bash
93+
cargo install cargo-nextest --version 0.9.133 --locked
94+
cargo install cargo-mutants --version 27.0.0 --locked
95+
make plugin-mutants-list PLUGIN=retry_with_backoff
96+
make plugin-mutants PLUGIN=retry_with_backoff
97+
```
98+
99+
`.cargo/mutants.toml` sets `test_tool = "nextest"`, selects the `mutants` nextest profile, and keeps `cap_lints = false` so Rust warnings are not downgraded during mutant builds. The `mutants` profile keeps fail-fast enabled because cargo-mutants only needs one failing test to mark a mutant as caught. CI installs `cargo-mutants` with `cargo install cargo-mutants --version 27.0.0 --locked` and runs `cargo mutants "${cargo_args[@]}"`, using `--in-diff cargo-mutants.diff` for Rust source changes and full-package mutation for mutation-tooling config changes.
100+
80101
## CI Behavior
81102

82103
Repo contract tests run in their own CI workflow. The Rust plugin CI workflow uses the same plugin catalog to select affected plugin build, integration, and coverage jobs.

0 commit comments

Comments
 (0)