Coverage-based test selection (done right) — shadow mode (#1461)

* docs: design spec for coverage-based test selection (done right)

* docs: implementation plan for coverage-based test selection

* feat(test): add stable param_hash key for coverage selection

* feat(test): coverage map load/save with freshness metadata

* feat(test): ALWAYS_RUN_ALL classification for coverage selection

* feat(test): conservative coverage selection ladder

* feat(test): TestCase.coverage_key() = param_hash(params)

* feat(test): robust changed-file detection (CI list + self-healing git)

* feat(test): coverage selection summary line

* feat(test): coverage map health check (loud anti-rot)

* fix(test): key coverage_key on full params, not mods (soundness)

* feat(test): gcov coverage-map collector, keyed by param_hash

* feat(test): wire --only-changes (shadow) and --build-coverage-map

* ci: coverage map refresh + health workflows

* fix(test): coverage_key on TestCaseBuilder + treat empty coverage as run (soundness)

* feat(test): accept space/comma/newline separated --changed-files (CI paths-filter)

* fix(test): force full run on src/**/include/ changes (gcov can't attribute includes)

* Delete docs/superpowers/plans/2026-05-29-coverage-test-selection.md

* Delete docs/superpowers directory

* fix(test): empty --changed-files is uncertainty (run all), not skip-all

* fix(test): ALWAYS_RUN_ALL covers test/run infra; cases.py runs new tests via rung 5 (soundness)

* fix(coverage): exclude failed/partial-coverage tests from coverage map

Fix 1: in build_coverage_map Phase 2, a test with non-empty failures produced
only partial .gcda files (crash mid-pipeline). Previously those tests were
still added to test_results, and their truncated coverage was cached. A later
.fpp change that ran only in the missing stage would be silently skipped.
Fix: when failures is non-empty, record in all_failures and continue without
adding to test_results — absent entries are conservatively included by
select_tests (rung 5).

Fix 2: in _parse_gcov_json_output, a mid-stream json.JSONDecodeError returned
the partial result set, which is untrustworthy (the truncated JSON stream may
be missing coverage for .fpp files that were not yet serialised). Fix: return
None on that error path so the caller omits the test from the map entirely.

Fix 6 (comment): correct the post_process comment (~line 347) — the regular
suite runs post_process only under --test-all (which CI sets), not never.

* fix(coverage): broaden always-run-all and rung3 Fortran extension matching

Fix 3: ALWAYS_RUN_ALL_EXACT + prefixes enumerated only a handful of toolchain
files, missing case.py, build.py, common.py, state.py, sched.py, etc. Any
toolchain/mfc/*.py change (except cases.py) affects every test's generation
or execution, so under-enumeration was unsound. Replace with a catch-all:
any(f.startswith('toolchain/mfc/') and f.endswith('.py') and f != CASES_PY).
Drop the now-redundant individual file entries and toolchain/mfc/params/ and
toolchain/mfc/run/ prefixes (all subsumed). Keep CMakeLists.txt,
toolchain/cmake/, toolchain/bootstrap/, and src include rules.

Fix 4: rung 3 matched only .f90 and .f, missing .F90, .F95, .F03, .F08, .FOR
and all other uppercase/mixed variants. Changed files ending in those extensions
under src/ would fall through to per-test selection against a coverage map that
only tracks .fpp, causing silent under-inclusion. Fix: case-insensitive match
against the full tuple (.f90, .f, .f95, .f03, .f08, .for).

Tests: add three new unit tests covering the above fixes.

* ci(coverage-refresh): add contents:write permission and document push caveat

Add 'permissions: contents: write' at the workflow top level so the
coverage-refresh job is authorized to commit and push the updated
coverage_map.json.gz back to master. Without this, the GITHUB_TOKEN has
only read permissions in newer default permission settings.

Also add a comment on the git push step noting that branch protection
may still reject the default GITHUB_TOKEN and that a PAT or GitHub App
with bypass-branch-protection permission may be needed.

* ci: coverage selection shadow mode on self-hosted jobs (git-detected changed files)

* fix(coverage): invert ALWAYS_RUN_ALL to unattributable-catch-all and validate map shape

Fix 1: Replace the allowlist of run-all files with an inverted design: define a
small, conservative allowlist of provably test-irrelevant files (docs/*.md, LICENSE,
etc.) and treat any changed file that is not .fpp, not cases.py, and not in that
allowlist as unattributable -> run all.  This closes the gap where mfc.sh,
.github/**, tests/**, toolchain/pyproject.toml, and similar files would silently
fall through to rung-7 (skip-all) under --select-enforce.

Fix 2: In load_map, validate that every entry is str -> list[str] after popping
_meta.  A malformed entry returns (None, None) so the caller runs the full suite
rather than silently misrouting tests.

Tests: expand test_docs_only_still_skips_all to cover docs/, LICENSE, .claude/;
add test_unattributable_nonsource_change_runs_all for mfc.sh / pyproject.toml /
tests/** / .github/workflows; add test_load_map_rejects_malformed_entry.

* fix(cli): clarify --changed-files help to mention space-separated input

* fix(ci): gate --changed-files with --only-changes via bash array (PR-only)

* fix(coverage-build): add mpiexec as fallback MPI launcher between mpirun and srun

Author: sbryngelson

.github/scripts/check_coverage_map_health.py

@@ -0,0 +1,26 @@
+"""Fail loudly if the committed coverage map is stale or under-covers. Used by coverage-health.yml."""
+import datetime
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "toolchain"))
+from mfc.test.coverage import COVERAGE_MAP_PATH, load_map, map_health  # noqa: E402
+from mfc.test.cases import list_cases  # noqa: E402  (returns the current test list)
+
+MAX_AGE_DAYS = 10
+MIN_FRACTION = 0.80
+
+entries, meta = load_map(COVERAGE_MAP_PATH)
+if entries is None:
+    sys.exit("Coverage map missing or corrupt.")
+current_keys = {b.to_case().coverage_key() for b in list_cases()}
+ok, msg = map_health(
+    meta=meta,
+    current_keys=current_keys,
+    mapped_keys=set(entries),
+    now=datetime.datetime.now(datetime.timezone.utc).isoformat(),
+    max_age_days=MAX_AGE_DAYS,
+    min_fraction=MIN_FRACTION,
+)
+print(msg)
+sys.exit(0 if ok else 1)

.github/workflows/common/coverage-refresh.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -e
+NJOBS="${SLURM_CPUS_ON_NODE:-24}"; [ "$NJOBS" -gt 64 ] && NJOBS=64
+./mfc.sh clean
+source .github/scripts/retry-build.sh
+retry_build ./mfc.sh build --gcov -j 8
+./mfc.sh test --build-coverage-map --gcov -j "$NJOBS"

.github/workflows/common/test.sh

@@ -57,4 +57,12 @@ if [ -n "${job_shard:-}" ]; then
     shard_opts="--shard $job_shard"
 fi
 
-./mfc.sh test -v --max-attempts 3 --no-build -a -j $n_test_threads $rdma_opts $device_opts $build_opts $shard_opts -- -c $job_cluster
+# Coverage-based test selection in SHADOW mode on PRs: prints what it WOULD select but the
+# full suite still runs (no --select-enforce). Changed files come from git detection
+# (self-healing deepen) since the SLURM job doesn't receive the paths-filter list.
+select_opts=""
+if [ "${GITHUB_EVENT_NAME:-}" = "pull_request" ]; then
+    select_opts="--only-changes"
+fi
+
+./mfc.sh test -v --max-attempts 3 --no-build $select_opts -a -j $n_test_threads $rdma_opts $device_opts $build_opts $shard_opts -- -c $job_cluster

.github/workflows/coverage-health.yml

@@ -0,0 +1,18 @@
+# .github/workflows/coverage-health.yml
+name: 'Coverage Map Health'
+on:
+  schedule:
+    - cron: '0 7 * * *'      # daily; loud if the refresh stopped working
+  workflow_dispatch:
+jobs:
+  health:
+    if: github.repository == 'MFlowCode/MFC'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: '3.12' }
+      - name: Initialize MFC
+        run: ./mfc.sh init
+      - name: Check coverage map freshness
+        run: build/venv/bin/python3 .github/scripts/check_coverage_map_health.py

.github/workflows/coverage-refresh.yml

@@ -0,0 +1,44 @@
+# .github/workflows/coverage-refresh.yml
+name: 'Coverage Map Refresh'
+on:
+  schedule:
+    - cron: '0 6 * * 1'      # weekly floor
+  push:
+    branches: [master]
+    paths:
+      - 'toolchain/mfc/test/cases.py'
+      - 'src/**/*.fpp'
+  workflow_dispatch:
+permissions:
+  contents: write
+concurrency:
+  group: coverage-refresh
+  cancel-in-progress: true
+jobs:
+  refresh:
+    if: github.repository == 'MFlowCode/MFC'
+    timeout-minutes: 240
+    runs-on:
+      group:  phoenix
+      labels: gt
+    steps:
+      - uses: actions/checkout@v4
+        with: { clean: false }
+      - name: Build + collect coverage map (SLURM)
+        run: bash .github/scripts/submit-slurm-job.sh .github/workflows/common/coverage-refresh.sh cpu none phoenix
+      - name: Commit refreshed map
+        run: |
+          if ! git diff --quiet tests/coverage_map.json.gz; then
+            git config user.name  "mfc-bot"
+            git config user.email "mfc-bot@users.noreply.github.com"
+            git add tests/coverage_map.json.gz
+            git commit -m "test: refresh coverage map [skip ci]"
+            # NOTE: pushing to a protected default branch requires a token or
+            # GitHub App with bypass-branch-protection permission.  The default
+            # GITHUB_TOKEN may be rejected by branch protection rules; if so,
+            # configure a PAT or App token with the `contents: write` scope and
+            # pass it as `GITHUB_TOKEN` in the environment for this step.
+            git push origin HEAD:master
+          else
+            echo "Coverage map unchanged."
+          fi

.github/workflows/test.yml

@@ -53,6 +53,7 @@ jobs:
     runs-on: 'ubuntu-latest'
     outputs:
       checkall: ${{ steps.changes.outputs.checkall }}
+      changed_files: ${{ steps.changes.outputs.checkall_files }}
     steps:
       - name: Clone
         uses: actions/checkout@v4
@@ -62,6 +63,7 @@ jobs:
         id: changes
         with:
           filters: ".github/file-filter.yml"
+          list-files: shell
 
   github:
     name: ${{ matrix.nvhpc && format('NVHPC {0} ({1})', matrix.nvhpc, matrix.target) || format('Github ({0}, {1}, {2}, {3})', matrix.os, matrix.mpi, matrix.debug, matrix.intel && 'intel' || 'GNU') }}
@@ -262,11 +264,17 @@ jobs:
       - name: Test
         if:   '!matrix.nvhpc'
         run:  |
-          /bin/bash mfc.sh test -v --max-attempts 3 -j $(nproc) $TEST_ALL $TEST_PCT $PRECISION
+          # Coverage-based test selection in SHADOW mode on PRs: the selector
+          # prints what it WOULD run, but the full suite still runs (no
+          # --select-enforce). Enforcement is a separate, later change.
+          SELECT=()
+          [ "${{ github.event_name }}" = "pull_request" ] && SELECT=(--only-changes --changed-files "$CHANGED_FILES")
+          /bin/bash mfc.sh test -v --max-attempts 3 -j $(nproc) "${SELECT[@]}" $TEST_ALL $TEST_PCT $PRECISION
         env:
           TEST_ALL: ${{ matrix.mpi == 'mpi' && '--test-all' || '' }}
           TEST_PCT: ${{ matrix.debug == 'reldebug' && '-% 20' || '' }}
           PRECISION: ${{ matrix.precision != '' && format('--{0}', matrix.precision) || '' }}
+          CHANGED_FILES: ${{ needs.file-changes.outputs.changed_files }}
 
       # ── NVHPC build + test (via docker exec into long-lived container) ──
       - name: Build (NVHPC)

tests/coverage_map.json.gz

@@ -464,6 +464,25 @@
             type=str,
             default=None,
         ),
+        Argument(
+            name="build-coverage-map", dest="build_coverage_map", action=ArgAction.STORE_TRUE, default=False, help="Build the gcov coverage map (requires a prior --gcov build). Master-side only."
+        ),
+        Argument(
+            name="only-changes",
+            dest="only_changes",
+            action=ArgAction.STORE_TRUE,
+            default=False,
+            help="Select only tests whose covered files overlap changed files (shadow mode unless --select-enforce).",
+        ),
+        Argument(
+            name="select-enforce",
+            dest="select_enforce",
+            action=ArgAction.STORE_TRUE,
+            default=False,
+            help="With --only-changes, actually skip unselected tests (otherwise shadow: print selection, run all).",
+        ),
+        Argument(name="changed-files", dest="changed_files", type=str, default=None, help="Changed-file list (newline-, space-, or comma-separated; from CI paths-filter). Overrides git detection."),
+        Argument(name="changes-branch", dest="changes_branch", type=str, default="master", help="Branch to diff against for --only-changes."),
     ],
     mutually_exclusive=[
         MutuallyExclusiveGroup(

toolchain/mfc/cli/commands.py

@@ -246,6 +246,11 @@ def get_trace(self) -> str:
     def get_uuid(self) -> str:
         return trace_to_uuid(self.trace)
 
+    def coverage_key(self) -> str:
+        from .coverage import param_hash
+
+        return param_hash(self.params)
+
     def get_dirpath(self):
         return os.path.join(common.MFC_TEST_DIR, self.get_uuid())
 
@@ -371,6 +376,9 @@ class TestCaseBuilder:
     def get_uuid(self) -> str:
         return trace_to_uuid(self.trace)
 
+    def coverage_key(self) -> str:
+        return self.to_case().coverage_key()
+
     def to_case(self) -> TestCase:
         if self.kind == "convergence":
             # Convergence cases drive their own runs — the BASE_CFG mods/path