Implementation Plan: T5-P5-A10-WP1 — CLI Config-Acceptance Probe (Check 34) (#4140)

## Summary

Add `_check_cli_conformance_probes` to
`src/autoskillit/cli/doctor/_doctor_runtime.py` as a new doctor check
(Check 34) that writes a minimal TOML config to a temp directory,
invokes the backend CLI with a config override flag, and reports whether
the CLI accepted the config. Follows the exact gating pattern from
`_check_codex_version`: skip when backend is `None` or
`version_check_command` is empty, skip when CLI is unavailable or times
out.

**Check numbering note:** The issue references "Check 33" but the
current codebase already assigns Check 33 to `_check_codex_graduation`
(line 217–218 of `__init__.py`). This plan assigns the new check as
**Check 34**, appended after Check 33.

## Implementation Plan

Plan file:
`/home/talon/projects/autoskillit-runs/impl-20260628-044315-253111/.autoskillit/temp/make-plan/T5-P5-A10-WP1_cli_conformance_probes_plan_2026-06-28_050100.md`

Closes #4028

🤖 Generated with [Claude Code](https://claude.com/claude-code) via
AutoSkillit
<!-- autoskillit:pipeline-signature
steps=prepare_pr,run_arch_lenses,compose_pr,annotate_pr_diff,review_pr
-->

## Token Usage Summary

| Step | Model | count | uncached | output | cache_read | peak_ctx |
turns | cache_write | time |

|------|-------|-------|----------|--------|------------|----------|-------|-------------|------|
| plan* | opus[1m] | 1 | 61 | 19.5k | 1.8M | 98.9k | 58 | 81.4k | 12m
42s |
| verify* | sonnet | 1 | 76 | 8.1k | 387.0k | 54.1k | 24 | 35.6k | 4m
10s |
| implement* | MiniMax-M3 | 1 | 75.4k | 10.2k | 1.8M | 0 | 76 | 0 | 6m
51s |
| fix* | sonnet | 1 | 126 | 3.9k | 663.6k | 52.6k | 35 | 34.1k | 3m 29s
|
| audit_impl* | sonnet | 1 | 54 | 8.8k | 214.2k | 48.4k | 17 | 40.2k |
3m 17s |
| prepare_pr* | MiniMax-M3 | 1 | 43.4k | 1.8k | 153.9k | 0 | 12 | 0 | 1m
10s |
| compose_pr* | MiniMax-M3 | 1 | 37.1k | 1.5k | 213.4k | 0 | 15 | 0 |
54s |
| **Total** | | | 156.2k | 53.9k | 5.2M | 98.9k | | 191.3k | 32m 35s |

\* *Step used a non-Anthropic provider; caching behavior may differ.*

## Token Efficiency

| Step | LoC Changed | cache_read/LoC | cache_write/LoC | output/LoC |
|------|-------------|----------------|-----------------|------------|
| plan | 0 | — | — | — |
| verify | 0 | — | — | — |
| implement | 173 | 10138.6 | 0.0 | 58.7 |
| fix | 2 | 331794.0 | 17058.5 | 1942.5 |
| audit_impl | 0 | — | — | — |
| prepare_pr | 0 | — | — | — |
| compose_pr | 0 | — | — | — |
| **Total** | **175** | 29773.9 | 1093.1 | 307.8 |

## Model Usage Breakdown

| Model | steps | uncached | output | cache_read | cache_write | time |
|-------|-------|----------|--------|------------|-------------|------|
| opus[1m] | 1 | 61 | 19.5k | 1.8M | 81.4k | 12m 42s |
| sonnet | 3 | 256 | 20.8k | 1.3M | 109.9k | 10m 57s |
| MiniMax-M3 | 3 | 155.9k | 13.5k | 2.1M | 0 | 8m 55s |

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: Trecek

docs/installation.md

@@ -67,8 +67,8 @@ projects need.
 
     autoskillit doctor
 
-Doctor runs 28 checks (23 numbered + 5 lettered sub-checks: `2b`, `2c`, `2d`, `4b`, `7b`); up to 33 with the fleet feature enabled.
-Enumerated by `run_doctor` in `src/autoskillit/cli/_doctor.py`:
+Doctor runs 33 checks (23 numbered + 5 lettered sub-checks: `2b`, `2c`, `2d`, `4b`, `7b` + 5 backend runtime probes, checks 30–34); up to 39 with the fleet feature enabled.
+Enumerated by `run_doctor` in `src/autoskillit/cli/doctor/__init__.py`:
 
 | # | Check | What it verifies |
 |---|-------|------------------|
@@ -101,6 +101,12 @@ Enumerated by `run_doctor` in `src/autoskillit/cli/_doctor.py`:
 | 22 | Feature dependency consistency | Enabled features satisfy their declared dependencies |
 | 23 | Feature registry import consistency | All feature gate modules import without errors |
 | 24–28 | Fleet infrastructure | Sous-chef skill, dispatch guard, stale state, onboarding, clone collisions (fleet feature only) |
+| 29 | Fleet state schema | Fleet state schema version drift (fleet feature only) |
+| 30 | Codex version | Codex CLI version meets minimum requirement |
+| 31 | script(1) binary | PTY binary availability with -qefc support |
+| 32 | MCP timeouts | Codex MCP tool_timeout_sec coherence |
+| 33 | Codex graduation | Multi-criteria graduation readiness (version, probe, matrix, smoke) |
+| 34 | CLI conformance | Backend CLI accepts minimal TOML config probe |
 
 See **[Hooks](safety/hooks.md)** for what each PreToolUse / PostToolUse /
 SessionStart hook actually enforces.

src/autoskillit/cli/doctor/AGENTS.md

@@ -1,6 +1,6 @@
 # doctor/
 
-Diagnostic health checks for the autoskillit installation (33 checks).
+Diagnostic health checks for the autoskillit installation (34 checks).
 
 ## Files
 

src/autoskillit/cli/doctor/__init__.py

@@ -61,6 +61,7 @@
 )
 from ._doctor_runtime import (
     _check_claude_process_state_breakdown,
+    _check_cli_conformance_probes,
     _check_codex_graduation,
     _check_codex_version,
     _check_quota_cache_schema,
@@ -217,6 +218,9 @@ def run_doctor(*, output_json: bool = False) -> None:
     # Check 33: Codex graduation criteria
     results.append(_check_codex_graduation(backend=_backend))
 
+    # Check 34: CLI config-acceptance probe
+    results.append(_check_cli_conformance_probes(backend=_backend))
+
     # Output
     if output_json:
         print(

src/autoskillit/cli/doctor/_doctor_runtime.py

@@ -3,14 +3,17 @@
 from __future__ import annotations
 
 import json
+import shlex
 import subprocess
+import tempfile
 from pathlib import Path
 
 import regex as re
 
 from autoskillit.core import (
     CodingAgentBackend,
     Severity,
+    atomic_write,
     default_log_dir,
     get_logger,
 )
@@ -270,3 +273,47 @@ def _check_codex_graduation(
         summary += f" — EXPERIMENTAL hold: {pending} of 4 criteria pending"
 
     return DoctorResult(Severity.INFO, check_name, summary)
+
+
+def _check_cli_conformance_probes(*, backend: CodingAgentBackend | None = None) -> DoctorResult:
+    check_name = "cli_conformance_probes"
+
+    if backend is None or not backend.capabilities.version_check_command:
+        return DoctorResult(Severity.OK, check_name, "Skipped (no version check command)")
+
+    cli_argv = shlex.split(backend.capabilities.version_check_command)
+    try:
+        subprocess.run(cli_argv, capture_output=True, text=True, timeout=5)
+    except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
+        return DoctorResult(Severity.OK, check_name, "CLI unavailable; skipping config probe")
+
+    cli_binary = cli_argv[0]
+    result = None
+    with tempfile.TemporaryDirectory() as tmpdir:
+        probe_path = Path(tmpdir) / "probe.toml"
+        atomic_write(probe_path, 'model = "test-model"\n')
+        try:
+            result = subprocess.run(
+                [cli_binary, "-c", str(probe_path), "--version"],
+                capture_output=True,
+                text=True,
+                timeout=10,
+            )
+        except subprocess.TimeoutExpired:
+            return DoctorResult(
+                Severity.OK,
+                check_name,
+                f"{cli_binary} config probe timed out; skipping",
+            )
+        except (FileNotFoundError, OSError):
+            return DoctorResult(Severity.OK, check_name, "CLI unavailable; skipping config probe")
+
+    if result is not None and result.returncode == 0:
+        return DoctorResult(Severity.OK, check_name, f"{cli_binary} accepted minimal config probe")
+    if result is None:
+        return DoctorResult(Severity.OK, check_name, "CLI unavailable; skipping config probe")
+    return DoctorResult(
+        Severity.WARNING,
+        check_name,
+        f"{cli_binary} rejected config probe (exit {result.returncode})",
+    )

tests/cli/test_doctor_backend_guards.py

@@ -691,3 +691,117 @@ def test_absent_files_yield_not_yet_run(
         assert "probe=not-yet-run" in result.message
         assert "matrix=not-yet-run" in result.message
         assert "smoke=not-found" in result.message
+
+
+class TestCheckCliConformanceProbes:
+    """Tests for _check_cli_conformance_probes doctor check (Check 34)."""
+
+    def test_skip_for_none_backend(self) -> None:
+        from autoskillit.cli.doctor._doctor_runtime import _check_cli_conformance_probes
+        from autoskillit.core import Severity
+
+        result = _check_cli_conformance_probes(backend=None)
+        assert result.severity == Severity.OK
+        assert result.check == "cli_conformance_probes"
+        assert "skipped" in result.message.lower()
+
+    def test_skip_for_backend_without_version_check_command(self) -> None:
+        from types import SimpleNamespace
+
+        from autoskillit.cli.doctor._doctor_runtime import _check_cli_conformance_probes
+        from autoskillit.core import Severity
+
+        stub = SimpleNamespace(capabilities=SimpleNamespace(version_check_command=""))
+        result = _check_cli_conformance_probes(backend=stub)
+        assert result.severity == Severity.OK
+        assert "skipped" in result.message.lower()
+
+    def test_skip_when_cli_unavailable(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        from autoskillit.cli.doctor._doctor_runtime import _check_cli_conformance_probes
+        from autoskillit.core import Severity
+        from autoskillit.execution.backends.codex import CodexBackend
+
+        def _raise(*a, **kw):
+            raise FileNotFoundError("codex")
+
+        monkeypatch.setattr("autoskillit.cli.doctor._doctor_runtime.subprocess.run", _raise)
+        result = _check_cli_conformance_probes(backend=CodexBackend())
+        assert result.severity == Severity.OK
+        assert "unavailable" in result.message.lower()
+
+    def test_skip_when_cli_times_out(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        import subprocess
+
+        from autoskillit.cli.doctor._doctor_runtime import _check_cli_conformance_probes
+        from autoskillit.core import Severity
+        from autoskillit.execution.backends.codex import CodexBackend
+
+        call_count = 0
+
+        def _stateful_fake(*a, **kw):
+            nonlocal call_count
+            call_count += 1
+            if call_count == 1:
+                return type(
+                    "CompletedProcess", (), {"returncode": 0, "stdout": "", "stderr": ""}
+                )()
+            raise subprocess.TimeoutExpired(cmd="codex", timeout=10)
+
+        monkeypatch.setattr(
+            "autoskillit.cli.doctor._doctor_runtime.subprocess.run", _stateful_fake
+        )
+        result = _check_cli_conformance_probes(backend=CodexBackend())
+        assert result.severity == Severity.OK
+        assert "timed out" in result.message.lower()
+
+    def test_ok_when_config_accepted(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        import subprocess
+
+        from autoskillit.cli.doctor._doctor_runtime import _check_cli_conformance_probes
+        from autoskillit.core import Severity
+        from autoskillit.execution.backends.codex import CodexBackend
+
+        calls = []
+
+        def _fake_run(*a, **kw):
+            calls.append(a)
+            return type(
+                "CompletedProcess",
+                (),
+                {"returncode": 0, "stdout": "codex 0.130.0\n", "stderr": ""},
+            )()
+
+        monkeypatch.setattr(subprocess, "run", _fake_run)
+        result = _check_cli_conformance_probes(backend=CodexBackend())
+        assert result.severity == Severity.OK
+        assert "accepted" in result.message.lower()
+        assert len(calls) == 2
+
+    def test_warning_when_config_rejected(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        import subprocess
+
+        from autoskillit.cli.doctor._doctor_runtime import _check_cli_conformance_probes
+        from autoskillit.core import Severity
+        from autoskillit.execution.backends.codex import CodexBackend
+
+        call_count = 0
+
+        def _fake_run(*a, **kw):
+            nonlocal call_count
+            call_count += 1
+            if call_count == 1:
+                return type(
+                    "CompletedProcess",
+                    (),
+                    {"returncode": 0, "stdout": "codex 0.130.0\n", "stderr": ""},
+                )()
+            return type(
+                "CompletedProcess",
+                (),
+                {"returncode": 1, "stdout": "", "stderr": "unknown flag"},
+            )()
+
+        monkeypatch.setattr(subprocess, "run", _fake_run)
+        result = _check_cli_conformance_probes(backend=CodexBackend())
+        assert result.severity == Severity.WARNING
+        assert "rejected" in result.message.lower() or "exit 1" in result.message

tests/docs/test_doc_counts.py

@@ -237,16 +237,15 @@ def test_quota_thresholds_defaults() -> None:
     assert long_ == pytest.approx(95.0)
 
 
-def test_doctor_check_count_is_40() -> None:
-    # 40 total = 17 numbered base + 7 lettered sub-checks (2b–2e, 4b, 7b, 7c)
-    # + 4 ambient env (18–21) + 2 feature (22–23) + 6 gated franchise (24–29)
-    # + 1 codex version (30) + 1 script binary (31) + 1 MCP timeouts (32)
-    # + 1 codex graduation (33).
-    # The docs claim 17 user-visible checks; the gap is intentional (Check 2/4/7
-    # split into sub-markers here but appear as single entries in docs).
+def test_doctor_check_count_is_41() -> None:
+    # 41 total = 23 numbered (1–17 base + 18–21 ambient env + 22–23 feature)
+    # + 7 lettered sub-checks (2b–2e, 4b, 7b, 7c)
+    # + 6 gated fleet (24–29) + 5 backend runtime (30–34).
+    # Docs list 33 user-visible checks (23 numbered + 5 documented lettered +
+    # 5 backend runtime); 2e and 7c are internal-only sub-checks not shown in docs.
     # Update both tests whenever a new doctor check is added.
     count = _count_doctor_checks()
-    assert count == 40, f"Expected 40 doctor checks; found {count}"
+    assert count == 41, f"Expected 41 doctor checks; found {count}"
 
 
 def test_bundled_recipe_count_is_15() -> None: