Skip to content

Commit b44bc29

Browse files
danielmeppielCopilot
danielmeppiel
and
Copilot
authored
test(architecture): add bounded mutation pilot (#2241)
* test: add bounded mutation pilot Add an exact-function mutmut lane for four stable canonical owners, ratchet reviewed survivors, and schedule advisory nightly evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 42596632-8392-4c21-a8e4-ea562251e900 * test: harden mutation pilot contracts Fold review findings for fail-closed outcome handling, cache completeness, direct runner tests, and contributor mode guidance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 42596632-8392-4c21-a8e4-ea562251e900 * fix: classify killed mutation processes accurately Track SIGKILL outcomes separately from segmentation faults and keep both fail-closed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 42596632-8392-4c21-a8e4-ea562251e900 * test(mutation-pilot): pin tie-break contract and add runner boundary tests Folds two coordinator-accepted follow-ups from panel review: 1. Add a regression test pinning the exact tie-break contract in _closest_target: 'coade' ties at edit distance 2 with both real KNOWN_TARGETS 'claude' and 'codex' (no other target is closer), and production must deterministically pick the lexicographically- first match encountered during the scan ('claude'). This kills the two previously-accepted survivors _closest_target__mutmut_12 (strict < swapped to <=, which would pick the last tied candidate) and _mutmut_13 (distance compared against the candidate name instead of the running-best distance). 2. Add direct tests for the mutation-pilot runner's real metadata and report boundary: _load_exit_codes against corrupt JSON, a missing exit_code_by_key map, wrong-type exit code values, no matching mutants, a malformed canonical name (real fnmatch-reachable via the trailing wildcard), and the defensive duplicate-canonical-name guard; plus _write_report's atomic, deterministic, ASCII-safe output and its failure-cleanup path. Ran a fresh (non-cached) mutmut execution: 480 mutants total, 32 survivors (down from 34), with both target mutants now killed and zero unexpected survivors or fatal outcomes. Updated tests/mutation/baseline.json to drop exactly the two resolved survivors. No production source changed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eef94cf3-d677-4802-8391-978b41797444 * test: close mutation tie-break gaps Pin deterministic target suggestion ties and harden the mutation metadata/report trust boundary with direct parser and atomic-write tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 42596632-8392-4c21-a8e4-ea562251e900 * test: consolidate mutation follow-up coverage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 42596632-8392-4c21-a8e4-ea562251e900 --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 90333e1 commit b44bc29

14 files changed

Lines changed: 1697 additions & 4 deletions

File tree

‎.github/workflows/mutation-pilot.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
name: Mutation Testing Pilot
2+
3+
on:
4+
schedule:
5+
- cron: '17 3 * * *'
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
11+
concurrency:
12+
group: mutation-testing-pilot
13+
cancel-in-progress: true
14+
15+
env:
16+
PYTHON_VERSION: '3.12'
17+
18+
jobs:
19+
mutation-pilot:
20+
name: Bounded Mutation Pilot
21+
runs-on: ubuntu-24.04
22+
timeout-minutes: 20
23+
24+
steps:
25+
- uses: actions/checkout@v4
26+
with:
27+
persist-credentials: false
28+
29+
- name: Set up Python
30+
uses: actions/setup-python@v5
31+
with:
32+
python-version: ${{ env.PYTHON_VERSION }}
33+
34+
- name: Install uv
35+
uses: astral-sh/setup-uv@v6
36+
with:
37+
enable-cache: true
38+
39+
- name: Install frozen development dependencies
40+
run: uv sync --frozen --extra dev
41+
42+
- name: Restore exact-input mutation cache
43+
id: mutation-cache
44+
uses: actions/cache@v4
45+
with:
46+
path: mutants
47+
key: >-
48+
mutation-pilot-v1-${{ runner.os }}-${{
49+
hashFiles(
50+
'uv.lock',
51+
'pyproject.toml',
52+
'scripts/ratchet_baseline.py',
53+
'scripts/run_mutation_pilot.py',
54+
'tests/mutation/baseline.json',
55+
'tests/conftest.py',
56+
'src/apm_cli/models/dependency/subsets.py',
57+
'src/apm_cli/install/plan.py',
58+
'src/apm_cli/policy/discovery.py',
59+
'src/apm_cli/compilation/link_resolver.py',
60+
'tests/unit/test_skill_subset_persistence.py',
61+
'tests/unit/test_dep_targets_persistence.py',
62+
'tests/unit/install/conftest.py',
63+
'tests/unit/install/test_plan.py',
64+
'tests/unit/policy/test_cache_merged_effective.py',
65+
'tests/unit/compilation/test_link_resolver.py',
66+
'tests/unit/compilation/test_link_resolver_phase3.py',
67+
'tests/unit/compilation/test_link_resolver_resolution.py'
68+
) }}
69+
70+
- name: Run bounded mutation pilot
71+
shell: bash
72+
run: |
73+
args=()
74+
if [[ "${{ steps.mutation-cache.outputs.cache-hit }}" == "true" ]]; then
75+
args+=(--reuse-cache)
76+
fi
77+
uv run --frozen --extra dev python scripts/run_mutation_pilot.py \
78+
--max-children 2 \
79+
--output mutation-pilot-report.json \
80+
"${args[@]}"
81+
82+
- name: Upload deterministic mutation report
83+
if: always()
84+
uses: actions/upload-artifact@v4
85+
with:
86+
name: mutation-pilot-report
87+
path: mutation-pilot-report.json
88+
if-no-files-found: error
89+
retention-days: 14

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ ENV/
5353
.coverage_*.json
5454
coverage*.json
5555
htmlcov/
56+
mutants/
5657
.tox/
5758
.pytest_cache/
5859
.mypy_cache/

‎CONTRIBUTING.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,49 @@ uv run pytest tests/unit -x -v
199199

200200
Tests run in parallel automatically (`-n auto` is configured in `pyproject.toml`). To force serial execution, add `-n0`.
201201

202+
### Running the bounded mutation pilot
203+
204+
The advisory mutation pilot checks four stable owners: dependency subset
205+
selection, update-plan construction, cached-policy serialization, and canonical
206+
in-package link projection. It is intentionally separate from required PR CI
207+
and runs nightly or by manual workflow dispatch with a 20-minute job budget.
208+
209+
Run the same exact-function allowlist locally:
210+
211+
```bash
212+
uv run --frozen --extra dev python scripts/run_mutation_pilot.py \
213+
--output mutation-pilot-report.json
214+
```
215+
216+
The command exits nonzero for a new survivor, timeout, suspicious result,
217+
unchecked mutant, or any other incomplete outcome. It writes a sorted,
218+
timestamp-free JSON report even when the survivor comparison fails. Reuse the
219+
`mutants/` cache only when the source, test seams, configuration, runner, and
220+
lockfile are unchanged:
221+
222+
```bash
223+
uv run --frozen --extra dev python scripts/run_mutation_pilot.py \
224+
--reuse-cache --output mutation-pilot-report.json
225+
```
226+
227+
To inspect existing mutmut metadata without executing mutants:
228+
229+
```bash
230+
uv run --frozen --extra dev python scripts/run_mutation_pilot.py \
231+
--report-only --output mutation-pilot-report.json
232+
```
233+
234+
The accepted survivor allowlist is
235+
[`tests/mutation/baseline.json`](tests/mutation/baseline.json). Do not update it
236+
to make a run green. Review every surviving diff with `mutmut show`, add a
237+
behavioral test for real contract gaps, and use `--update-baseline` only for a
238+
reviewed baseline change.
239+
240+
```bash
241+
uv run --frozen --extra dev python scripts/run_mutation_pilot.py \
242+
--update-baseline --output mutation-pilot-report.json
243+
```
244+
202245
If you don't have `uv` available, you can use a standard Python venv and pip:
203246

204247
```bash

‎docs/src/content/docs/contributing/development-guide.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,47 @@ pip install -e .[dev]
100100
pytest -q
101101
```
102102

103+
### Running the bounded mutation pilot
104+
105+
The advisory mutation pilot covers four stable owners: dependency subset
106+
selection, update-plan construction, cached-policy serialization, and canonical
107+
in-package link projection. It runs nightly or by manual workflow dispatch,
108+
not as required PR CI, and has a 20-minute hosted job budget.
109+
110+
Run the exact-function allowlist locally:
111+
112+
```bash
113+
uv run --frozen --extra dev python scripts/run_mutation_pilot.py \
114+
--output mutation-pilot-report.json
115+
```
116+
117+
The command fails on new survivors, timeouts, suspicious results, unchecked
118+
mutants, and incomplete outcomes. Its timestamp-free JSON report is suitable
119+
for comparing runs. Pass `--reuse-cache` only when the allowlisted source,
120+
tests, configuration, runner, and lockfile are unchanged.
121+
122+
To inspect existing mutmut metadata without executing mutants:
123+
124+
```bash
125+
uv run --frozen --extra dev python scripts/run_mutation_pilot.py \
126+
--report-only --output mutation-pilot-report.json
127+
```
128+
129+
The reviewed survivor allowlist lives in
130+
[`tests/mutation/baseline.json`](https://github.com/microsoft/apm/blob/main/tests/mutation/baseline.json).
131+
Do not update it to make a run green. Inspect surviving diffs with `mutmut show`
132+
and add behavioral tests for real contract gaps. Use `--update-baseline` only
133+
when the baseline change itself has been reviewed:
134+
135+
```bash
136+
uv run --frozen --extra dev python scripts/run_mutation_pilot.py \
137+
--update-baseline --output mutation-pilot-report.json
138+
```
139+
140+
The repository
141+
[`CONTRIBUTING.md`](https://github.com/microsoft/apm/blob/main/CONTRIBUTING.md#running-the-bounded-mutation-pilot)
142+
owns the review policy for baseline changes.
143+
103144
## Coding Style
104145

105146
This project follows:

‎pyproject.toml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ dev = [
5353
"jsonschema>=4.0.0",
5454
"pylint>=3.0.0",
5555
"hypothesis>=6.0.0",
56+
"mutmut==3.6.0",
5657
]
5758
build = [
5859
"pyinstaller>=6.0.0",
@@ -160,8 +161,39 @@ exclude_lines = [
160161
"@overload",
161162
]
162163

164+
[tool.mutmut]
165+
source_paths = ["src/"]
166+
only_mutate = [
167+
"src/apm_cli/models/dependency/subsets.py",
168+
"src/apm_cli/install/plan.py",
169+
"src/apm_cli/policy/discovery.py",
170+
"src/apm_cli/compilation/link_resolver.py",
171+
]
172+
pytest_add_cli_args = [
173+
"-n0",
174+
"-p",
175+
"no:cacheprovider",
176+
"-o",
177+
"addopts=",
178+
]
179+
pytest_add_cli_args_test_selection = [
180+
"tests/unit/test_skill_subset_persistence.py",
181+
"tests/unit/test_dep_targets_persistence.py",
182+
"tests/unit/install/test_plan.py",
183+
"tests/unit/policy/test_cache_merged_effective.py::test_policy_serializer_covers_every_dataclass_leaf",
184+
"tests/unit/policy/test_cache_merged_effective.py::test_policy_serializer_preserves_sparse_bin_deploy_shapes",
185+
"tests/unit/policy/test_cache_merged_effective.py::TestPolicyRoundTrip::test_all_effective_policy_fields_survive_cache_round_trip",
186+
"tests/unit/compilation/test_link_resolver.py",
187+
"tests/unit/compilation/test_link_resolver_phase3.py",
188+
"tests/unit/compilation/test_link_resolver_resolution.py",
189+
]
190+
timeout_constant = 1.0
191+
timeout_multiplier = 5.0
192+
use_setproctitle = false
193+
163194
[tool.pytest.ini_options]
164195
addopts = "-m 'not benchmark and not live'"
196+
testpaths = ["tests"]
165197
markers = [
166198
"unit: pure logic with no filesystem and no CLI",
167199
"component: in-process behavior that touches a filesystem or one command boundary",

0 commit comments

Comments
 (0)