test(architecture): add bounded mutation pilot

[danielmeppiel]

add(test-architecture): establish a bounded mutation pilot

TL;DR

This adds an advisory mutmut 3.6.0 lane for four stable canonical owners, with exact-function execution, an explicit survivor ratchet, deterministic JSON evidence, and a nightly/manual workflow. Survivor analysis strengthened only observed behavioral gaps, reducing the measured baseline from 101 to 32 reviewed survivors without changing production owner behavior. The final cold run checks 480 mutants in 41.3 seconds and has no timeout, suspicious, skipped, type-checked, unchecked, or no-test outcomes.

Important

This workflow is intentionally not required PR CI. It runs nightly or by manual dispatch, has a 20-minute hosted budget, and must not be added to the merge gate during the pilot.

Problem (WHY)

• Test-architecture audit section 8.8 had no reproducible mutation evidence for the four selected owner seams.
• The first measurement exposed 101 survivors and a slow property-test seam that timed out under mutation instrumentation; neither condition was suitable for a truthful baseline.
• [!] A broad module mutator would cross into unrelated policy and resolver behavior, while a required PR job would spend CI minutes on every push.
• [!] A raw mutation score would encourage percentage chasing instead of preserving explicit, reviewed behavioral contracts.

Why these matter:

• "Grounding outputs in deterministic tool execution transforms probabilistic generation into verifiable action."
• "agents pattern-match well against concrete structures" -- the checked-in owner/test map and survivor allowlist make the pilot reviewable rather than implicit.

Approach (WHAT)

#	Fix
1	Pin mutmut 3.6.0 and allowlist only the four source modules plus their exact function patterns.
2	Make one runner own execution, metadata normalization, report rendering, and survivor comparison.
3	Replace the slow policy property seam with deterministic field-complete and round-trip tests.
4	Add only survivor-proven behavioral tests; retain reviewed equivalent/diagnostic variants in an explicit baseline.
5	Run on nightly/manual Linux CI with read-only permissions, no persisted checkout credentials, an exact-input cache, and a 20-minute timeout.

Implementation (HOW)

Surface	Change
.github/workflows/mutation-pilot.yml	Adds the advisory schedule/manual job, fixture-complete exact-input mutants/ cache, frozen install, 20-minute cap, and 14-day report artifact.
scripts/run_mutation_pilot.py	Defines the four owners once for runtime execution, strips credentials, normalizes names to ASCII, rejects malformed metadata and non-integer outcomes, and writes sorted JSON atomically.
pyproject.toml / uv.lock	Pins the maintained tool and its graph, scopes source/test seams, forces serial deterministic pytest execution, and keeps default pytest collection inside tests/.
tests/mutation/baseline.json	Records the schema/tool/owner contract and 32 sorted accepted survivors; policy serialization and link projection have empty survivor lists.
tests/unit/scripts/test_run_mutation_pilot.py	Guards exact scope, credentials, execution modes, signal states, corrupt/partial/wrong-type metadata, canonical-name failures/collisions, and deterministic atomic reports.
tests/unit/test_skill_subset_persistence.py / tests/unit/test_dep_targets_persistence.py	Add blank-entry, invalid-type, edit-distance, no-suggestion, deletion-distance, and deterministic tied-suggestion contracts.
tests/unit/install/test_plan.py	Asserts complete records and canonical action order, non-default-host display identity, and unresolved branch behavior.
tests/unit/policy/test_cache_merged_effective.py	Proves sparse bin_deploy shapes survive canonical policy serialization.
tests/unit/compilation/test_link_resolver_phase3.py	Covers directory rejection, preserved bundle-local links, and external source-package projection into the deployment frame.
CONTRIBUTING.md / development guide	Documents fresh/cached/report-only runs, failure semantics, report location, and the reviewed baseline-update rule.
.gitignore	Excludes mutmut's generated cache tree.

Production owner files are mutation inputs only; this PR does not modify them. deps/lockfile.py, ADO surfaces, critical_suite.toml, TM002, README, and CHANGELOG remain untouched.

Diagrams

Legend: The yellow block is the new advisory execution path; review the hard fail boundary between normalized metadata and the uploaded report.

sequenceDiagram
    participant W as Nightly or manual workflow
    participant C as Exact-input cache
    participant R as run_mutation_pilot.py
    participant M as mutmut 3.6.0
    participant A as Report artifact

    rect rgb(255, 247, 200)
        W->>C: restore mutants cache
        C-->>W: hit or miss
        W->>R: run with optional --reuse-cache
        R->>M: exact owner function patterns
        M-->>R: per-mutant metadata
        R->>R: normalize and compare survivor allowlist
        R-->>A: sorted timestamp-free JSON
    end
    Note over R,A: New survivors or incomplete outcomes fail the advisory job

Loading

Trade-offs

• mutmut over Cosmic Ray. mutmut's function-pattern execution keeps the large policy and link-resolver modules bounded; Cosmic Ray's broader module filtering was rejected for this pilot.
• Reviewed baseline over a 100% mutation score. The 32 remaining survivors are reviewed equivalent/default-argument or diagnostic-string variants; forcing them closed would test implementation trivia rather than behavior.
• Nightly/manual over required PR CI. The pilot provides drift signal without adding mutation cost to every contributor push or changing merge-gate semantics.
• Pinned tool graph over minimal dependency count. mutmut brings Textual and updates Rich in the dev lock graph; exact pinning, frozen install, and a no-secret read-only workflow contain that trade-off.
• Deterministic seams over the broad Hypothesis property test. The property test remains in the normal suite, but deterministic field-complete/round-trip policy tests avoid instrumentation-amplified timeouts in mutation runs.

Benefits

1. Exactly 480 mutants are checked across four stable owners, with a measured 41.3-second cold local runtime under a 20-minute hosted cap.
2. Survivor-proven tests increased killed mutants from 379 to 448 and reduced reviewed survivors from 101 to 32.
3. Policy serialization kills 139/139 mutants and link projection kills 44/44, both with zero fatal outcomes.
4. Any new survivor, timeout, suspicious result, skipped or type-checked result, no-test result, unchecked mutant, segfault, or unknown status exits nonzero and remains visible in deterministic JSON.
5. Identical source/test/config/runner/lock inputs can reuse the cached mutant tree; the cached run reproduced the report byte-for-byte.

Validation

scripts/run_mutation_pilot.py:

[+] Mutation pilot report: 480 mutants, 32 survivors, 41.3s runtime
[i] subset-selection: total=144 killed=121 survived=23 timeout=0 suspicious=0
[i] update-plan: total=153 killed=144 survived=9 timeout=0 suspicious=0
[i] policy-serialization: total=139 killed=139 survived=0 timeout=0 suspicious=0
[i] link-projection: total=44 killed=44 survived=0 timeout=0 suspicious=0

Repository validation evidence

uv run --extra dev pytest -q tests/unit tests/test_console.py -x:

18928 passed, 2 skipped, 21 xfailed, 19 warnings, 86 subtests passed in 126.44s

uv run --extra dev pytest -p no:cacheprovider -q tests/quality:

25 passed in 17.77s
[+] assertion-quality ratchet clean: AQ001=5, AQ002=12
[+] exact test duplicate ratchet clean: 1053 files, 8 allowed duplicate group(s)

uv run --extra dev pytest tests/spec_conformance -p no:randomly --no-header -q:

[+] orphan_check OK: 100 requirements aligned across anchors / manifest / Appendix C / pytest markers
140 passed, 2 skipped in 10.59s

uv run apm audit --ci:

[+] Replayed 7 package(s)
[+] No drift detected
[*] All 9 check(s) passed

Canonical lint and owner boundaries:

All checks passed!
Your code has been rated at 10.00/10
[+] auth-signal lint clean
[+] architecture boundary lint clean

Exact-head hosted checks at c2d27eccac0a7780666552b6e68c5ac3cb88b178:

Build & Test Shard 1 (Linux)  pass
Build & Test Shard 2 (Linux)  pass
Coverage Combine (Linux)      pass
Lint                          pass
Test Architecture Ratchets    pass
Spec conformance gate         pass

Scenario Evidence

#	Scenario (user promise)	Principle(s)	Test(s) proving it	Type
1	Dependency subset names reject invalid values, preserve later valid entries, and choose the first known target deterministically when suggestions tie.	Portability by manifest, DevX (pragmatic as npm)	tests/unit/test_skill_subset_persistence.py::TestSkillSubsetFilterTokens::test_blank_name_does_not_hide_later_names tests/unit/test_dep_targets_persistence.py::test_parse_targets_suggests_name_at_edit_distance_boundary tests/unit/test_dep_targets_persistence.py::test_parse_targets_tie_break_prefers_lexicographically_first_match	unit
2	Update previews keep every before/after fact, canonical action order, and readable identity for non-default hosts.	Governed by policy, Vendor-neutral, DevX (pragmatic as npm)	tests/unit/install/test_plan.py::TestBuildUpdatePlan::test_entries_preserve_full_state_and_canonical_action_order tests/unit/install/test_plan.py::TestBuildUpdatePlan::test_non_default_host_display_names_preserve_action_context	unit
3	Sparse executable deployment policy survives cache serialization without losing enforcement.	Governed by policy	tests/unit/policy/test_cache_merged_effective.py::test_policy_serializer_preserves_sparse_bin_deploy_shapes	unit
4	Bundle-local links stay portable while external source packages project into the deployment frame without rewriting directories.	Portability by manifest, Secure by default	tests/unit/compilation/test_link_resolver_phase3.py::TestResolveInPackageAssetLink::test_link_inside_preserved_source_root_stays_bundle_relative tests/unit/compilation/test_link_resolver_phase3.py::TestResolveInPackageAssetLink::test_external_source_package_projects_into_base_dir tests/unit/compilation/test_link_resolver_phase3.py::TestResolveInPackageAssetLink::test_directory_candidate_returns_none	unit
5	Contributors get a nonzero result for malformed metadata or incomplete mutation outcomes, while reviewed survivors remain an explicit ratchet.	Governed by policy, OSS / community-driven, DevX (pragmatic as npm)	tests/unit/scripts/test_run_mutation_pilot.py::test_load_exit_codes_raises_on_non_object_root tests/unit/scripts/test_run_mutation_pilot.py::test_load_exit_codes_raises_on_wrong_type_exit_code_value tests/unit/scripts/test_run_mutation_pilot.py::test_write_report_preserves_existing_file_when_atomic_replace_fails	unit

How to test

• Run uv sync --frozen --extra dev -- the locked mutmut 3.6.0 environment installs without resolving new versions.
• Run uv run --frozen --extra dev python scripts/run_mutation_pilot.py --output mutation-pilot-report.json -- it reports 480 total mutants, 32 reviewed survivors, and zero fatal outcomes.
• Inspect mutation-pilot-report.json -- status is accepted, owner names are sorted, and no timestamp or machine-specific path appears.
• Re-run with --reuse-cache and compare reports -- the JSON is byte-identical.
• Run uv run --extra dev pytest -q -n0 tests/unit/scripts/test_run_mutation_pilot.py -- all runner contract tests pass.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

Establishes a bounded, advisory mutation-testing pilot (mutmut pinned to 3.6.0) targeting four “canonical owner” seams, with a checked-in survivor baseline, deterministic JSON reporting, and a nightly/manual GitHub Actions workflow. This adds guardrail tests around the pilot runner plus several new unit tests that close survivor-proven behavioral gaps.

Changes:

• Add scripts/run_mutation_pilot.py runner to execute allowlisted mutmut patterns, normalize outcomes to deterministic ASCII JSON, and enforce a reviewed survivor baseline.
• Add an advisory nightly/manual workflow with an exact-input cache and artifact upload for the JSON report.
• Pin mutmut in dev dependencies and add baseline + unit tests to ratchet survivor behavior.

Show a summary per file

File	Description
uv.lock	Adds mutmut and its dependency graph (e.g., Textual/Rich) to the locked dev environment.
pyproject.toml	Pins mutmut==3.6.0, adds [tool.mutmut] allowlist config, and sets pytest default testpaths = ["tests"].
scripts/run_mutation_pilot.py	New mutation pilot runner: executes mutmut for exact patterns, reads metadata, compares to baseline, writes deterministic JSON.
tests/mutation/baseline.json	New checked-in baseline schema with explicit owner contract and accepted survivors list.
tests/unit/scripts/test_run_mutation_pilot.py	New unit tests validating runner behavior (pattern generation, name normalization, baseline validation/ratchet semantics).
tests/unit/test_skill_subset_persistence.py	Adds subset-selection behavior tests to kill survivors (blank entries, invalid skill names).
tests/unit/test_dep_targets_persistence.py	Adds target suggestion/validation tests and edit-distance contract assertions.
tests/unit/install/test_plan.py	Adds update-plan behavioral assertions for completeness, ordering, display identity, and unresolved-branch behavior.
tests/unit/policy/test_cache_merged_effective.py	Adds policy serialization coverage for sparse bin_deploy shapes.
tests/unit/compilation/test_link_resolver_phase3.py	Adds link-resolver projection edge case tests (directories, preserved roots, external source package projection).
docs/src/content/docs/contributing/development-guide.md	Documents how to run the pilot, how caching works, and how to treat the reviewed baseline.
CONTRIBUTING.md	Adds contributor-facing instructions for running and interpreting the bounded mutation pilot.
.gitignore	Ignores mutants/ cache directory generated by mutmut.
.github/workflows/mutation-pilot.yml	New scheduled/manual workflow to run the pilot with cache + report artifact upload.

Review details

• Files reviewed: 12/14 changed files
• Comments generated: 1
• Review effort level: Low

[danielmeppiel]

APM Review Panel: ship_with_followups

Adds a bounded, deterministic mutation-testing advisory lane with zero product-source changes, sound architecture, and an explicit 32-survivor allowlist; nine panelists found no correctness regression.

cc @sergio-sisternes-epam -- a fresh advisory pass is ready for your review.

Final author update: All five curated panel follow-ups and the legitimate
Copilot signal-classification finding remain folded. The bounded coordinator
follow-up adds the real coade tie-break regression, kills
_closest_target__mutmut_12 and _mutmut_13, and directly defends the
runner's metadata/report trust boundary. Fresh evidence is 480 total,
448 killed, and 32 reviewed survivors with zero fatal outcomes. Exact-head
hosted CI is green at c2d27eccac; the PR is frozen unmerged for maintainer
review.

Nine specialist panelists converge on the same signal: this PR lands a well-scoped quality-infrastructure primitive. No product owner code changes, no lockfile/auth/README surface touched, and the architecture (single canonical-owner runner, deterministic JSON baseline, explicit survivor allowlist) is supported by passing tests across five scenario rows. The recommended findings cluster around cache-key correctness, fail-closed status semantics, runner contract tests, and contributor-mode docs/logs.

The highest-signal items are the missing unit proof for credential stripping and the cache-key omission of conftest fixtures. The former protects the secure-by-default subprocess boundary; the latter protects the deterministic cache claim. Treating skipped and type_checked as incomplete outcomes closes the remaining silent-shrink path.

Dissent. No material dissent. The mutable setup-uv action tag matches the existing repository pattern and is contained by a read-only advisory workflow with persisted credentials disabled.

Aligned with: secure by default through child-process credential stripping; governed by policy through an explicit 32-survivor ratchet; OSS community-driven through reproducible contributor instructions and an advisory schedule.

Growth signal. The measured 480-mutant evidence and 101-to-32 survivor reduction are a credible quality-by-evidence story for a later release cycle, outside this focused PR.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	2	3	Sound single-runner design; keep the runtime owner map and mutmut config parity guard load-bearing.
CLI Logging Expert	0	1	2	Output is consistent; cached and fresh execution should be distinguishable in logs.
DevX UX Expert	0	0	3	Contributor flow is pragmatic; improve recovery hints and mode discoverability.
Supply Chain Security Expert	0	1	1	Read-only, secret-free workflow is bounded; action pinning is a repo-wide follow-up.
OSS Growth Hacker	0	0	1	Contributor onboarding is solid; mirror the anti-baseline-gaming warning in the docs site.
Doc Writer	0	2	2	Document --report-only and reduce future drift between the two contributor surfaces.
Test Coverage Expert	0	2	1	Scenario tests pass; credential stripping and execution modes need direct runner tests.
Performance Expert	0	3	1	Add fixture inputs to the cache key and fail closed on every incomplete mutmut status.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 5 follow-ups

1. [Test Coverage Expert] Add a unit test for _sanitized_environment -- focused proof prevents a credential-strip regression from silently reaching mutant subprocesses.
2. [Performance Expert] Hash tests/conftest.py and tests/unit/install/conftest.py -- fixture-only changes must invalidate cached mutation metadata.
3. [Performance Expert] Treat mutmut skipped and type_checked results as incomplete/fatal -- the checked population must not silently shrink.
4. [Test Coverage Expert] Add runner tests for --report-only and --reuse-cache -- both documented modes need direct branch-level regression traps.
5. [CLI Logging Expert + Doc Writer] Document --report-only and announce cached versus fresh execution -- the contributor and hosted logs should state which mode ran.

Architecture

classDiagram
    direction LR
    class Owner {
        <<ValueObject>>
        +key str
        +source str
        +functions tuple
        +test_seams tuple
        +module str
        +patterns tuple
    }
    class RunMutationPilot {
        <<Module>>
        +OWNERS tuple
        +FATAL_STATUSES frozenset
        +_run_mutmut() float
        +_owner_report() dict
        +_validate_baseline() dict
        +_compare_with_baseline() tuple
        +main() int
    }
    class RatchetBaseline {
        <<SharedModule>>
        +load_baseline()
        +write_baseline()
    }
    class PilotError
    class BaselineError
    RunMutationPilot *-- Owner : owns four scopes
    RunMutationPilot ..> RatchetBaseline : delegates baseline IO
    RunMutationPilot ..> PilotError : fails execution
    RatchetBaseline ..> BaselineError : rejects corrupt baseline
    class Owner:::touched
    class RunMutationPilot:::touched
    classDef touched fill:#fff3b0,stroke:#d47600

Loading

flowchart TD
    A["Nightly, manual, or local invocation"] --> B{"Report only?"}
    B -- No --> C["mutmut run with exact function patterns"]
    B -- Yes --> D["Reuse existing metadata"]
    C --> E["Read per-mutant metadata"]
    D --> E
    E --> F["Normalize names and group by owner"]
    F --> G["Validate explicit baseline contract"]
    G --> H{"New survivor or incomplete result?"}
    H -- Yes --> I["Write deterministic regression report and exit nonzero"]
    H -- No --> J["Write deterministic accepted report and exit zero"]

Loading

Recommendation

All five bounded panel follow-ups, the Copilot signal fix, and the coordinator's tie-break plus metadata/report boundary follow-up are folded. Keep the exact-head-green PR frozen for maintainer review; do not merge.

---

Full per-persona findings

Python Architect

• [recommended] The runtime OWNERS tuple and pyproject mutmut lists are two required representations of scope.
  The passing parity test is load-bearing. A static boundary check would provide defense in depth, but is not necessary for this bounded lane.
• [recommended] --update-baseline writes the reviewed allowlist without a dry-run preview.
  The explicit flag and documented review rule bound the risk; a future preview mode could improve governance without adding interactivity.
• [nit] Pass --max-children 2 explicitly in the workflow so its cost decision is visible.
• [nit] Include scripts/ratchet_baseline.py in the cache key.
• [nit] Add a tool-version mismatch unit test beside the schema-version check.

CLI Logging Expert

• [recommended] --reuse-cache has no script-level announcement.
  Cached and fresh runs currently produce the same summary, so an operator cannot tell which path ran at the point of use.
• [nit] Use [i] rather than [!] when accepted survivors become killed.
• [nit] Echo the workflow cache decision beside the runner invocation.

DevX UX Expert

• [nit] The mutmut show <id> recovery hint does not map directly from the report's canonical mutant name.
• [nit] The default report path differs from every documented explicit --output example; always printing the report path would remove ambiguity.
• [nit] --report-only is discoverable only through --help.

Supply Chain Security Expert

• [recommended] astral-sh/setup-uv@v6 is a mutable third-party action tag.
  This matches existing workflows and the new lane is read-only, scheduled/manual, secret-free, and uses persist-credentials: false; repo-wide SHA pinning is the coherent follow-up.
• [nit] The explicit credential-name strip can miss future secret variable names.
  Keep the known-name list documented and guard it with a focused unit test rather than broadly stripping environment suffixes that tests may require.

OSS Growth Hacker

• [nit] The docs-site guide omits CONTRIBUTING's "Do not update it to make a run green" warning.

Auth Expert -- inactive

The PR touches workflow/test tooling and contributor docs only; no AuthResolver, token precedence, host classification, downloader, authorization, or remote-fallback behavior changed.

Doc Writer

• [recommended] --report-only is implemented but absent from both contributor docs.
• [recommended] The two contributor sections are near-duplicates and can drift.
  A concise cross-reference should clarify the canonical detailed procedure without removing either required contributor surface.
• [nit] Show the explicit --update-baseline command.
• [nit] Align the baseline reference/link style across both files.

Test Coverage Expert

• [recommended] _sanitized_environment has no focused unit test.
  Proof (missing at): tests/unit/scripts/test_run_mutation_pilot.py::test_sanitized_environment_removes_all_sensitive_names -- would prove mutation subprocesses do not receive live credentials. [secure-by-default]
• [recommended] --report-only and --reuse-cache have no direct branch tests.
  Proof (missing at): tests/unit/scripts/test_run_mutation_pilot.py::test_report_only_skips_mutmut_execution -- would prove metadata-only mode does not launch mutmut. [devx,oss]
• [nit] The tool-version mismatch validation branch lacks a sibling test.
  Proof (missing at): tests/unit/scripts/test_run_mutation_pilot.py::test_validate_baseline_rejects_tool_version_mismatch -- would prove stale tool baselines fail closed. [governed-by-policy]

Performance Expert

• [recommended] Captured mutmut output leaves little live evidence before the hosted timeout.
  A concise start/mode announcement is a lower-noise bounded improvement; full streaming can remain a later operational decision.
• [recommended] The cache key omits tests/conftest.py and tests/unit/install/conftest.py.
  Both affect selected tests, so cached metadata can become stale after fixture-only changes.
• [recommended] skipped and type_checked are not fatal statuses.
  A killed mutant can move into either bucket without becoming a new survivor, reducing checked behavior while the report remains accepted.
• [nit] timeout_constant = 1.0 may be tight on cold shared runners.
  Keep the measured zero-timeout value for the pilot and tune only if hosted evidence shows noise.

_{This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.}

[danielmeppiel]

Docs sync advisory

Verdict: no_change * Pages affected: 0 * LLM calls: 1/15 * Took: not recorded

Summary

No documentation changes are needed beyond the contributor guidance already included in this PR.

The L0 path gate found no product user-surface changes: the PR changes mutation tooling, tests, workflow/configuration, and contributor documentation only. The Starlight contributor page is a meta/contributor surface outside the consume/produce/govern narrative, and the PR already updates it alongside CONTRIBUTING.md.

---

How this advisory was produced

• Classifier verdict: no_change (confidence: high, source: L0)
• Panel composition: classifier only; no-change short-circuit
• Tool-verified claims: 0 (0 verified, 0 refuted, 0 inconclusive)
• CDO redraft rounds: 0/3

This is an advisory comment from the docs-sync skill. It does not gate merge. The maintainer ships.

Re-run by removing and re-applying the docs-sync label.