Implementation Plan: T5-P5-A11-WP2 — CI-Facing Shell Layer for Probe Canary State Machine (#4142)

Trecek · claude · web-flow · commit e9f153cc7703 · 2026-06-28T15:15:46.000Z
## Summary Create two shell scripts (`scripts/post-probe-failure.sh` and `scripts/create-probe-canary-issue.sh`) and a Python CLI entry point in `_probe_canary.py` that connects them. The shell scripts format CI probe failure data and route it into the existing Python canary state machine — no state-transition logic is reimplemented in shell. The `post-probe-failure.sh` script accepts positional args, validates `.venv/bin/python`, and delegates to `python -m autoskillit._probe_canary post-failure`. The `create-probe-canary-issue.sh` script validates `GITHUB_REPOSITORY` and `GITHUB_TOKEN`, then calls `gh issue create --label autoskillit-canary`. Closes #4030 ## Implementation Plan Plan file: `/home/talon/projects/autoskillit-runs/impl-20260628-071500-126668/.autoskillit/temp/make-plan/t5-p5-a11-wp2-ci-shell-layer_plan_2026-06-28_071500.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) via AutoSkillit  ## Token Usage Summary | Step | Model | count | uncached | output | cache_read | peak_ctx | turns | cache_write | time | |------|-------|-------|----------|--------|------------|----------|-------|-------------|------| | plan* | opus[1m] | 1 | 3.4k | 25.1k | 1.7M | 102.4k | 54 | 84.0k | 16m 25s | | verify* | sonnet | 1 | 840 | 10.7k | 346.5k | 59.6k | 24 | 41.1k | 5m 15s | | implement* | MiniMax-M3 | 1 | 117.2k | 13.2k | 1.6M | 0 | 89 | 0 | 7m 52s | | audit_impl* | sonnet | 1 | 2.1k | 11.1k | 229.1k | 60.6k | 17 | 44.6k | 5m 19s | | prepare_pr* | MiniMax-M3 | 1 | 42.3k | 1.7k | 111.6k | 0 | 11 | 0 | 1m 26s | | compose_pr* | MiniMax-M3 | 1 | 37.4k | 1.6k | 175.8k | 0 | 13 | 0 | 47s | | **Total** | | | 203.4k | 63.3k | 4.2M | 102.4k | | 169.7k | 37m 5s | \* *Step used a non-Anthropic provider; caching behavior may differ.* ## Token Efficiency | Step | LoC Changed | cache_read/LoC | cache_write/LoC | output/LoC | |------|-------------|----------------|-----------------|------------| | plan | 0 | — | — | — | | verify | 0 | — | — | — | | implement | 384 | 4295.7 | 0.0 | 34.3 | | audit_impl | 0 | — | — | — | | prepare_pr | 0 | — | — | — | | compose_pr | 0 | — | — | — | | **Total** | **384** | 11053.8 | 441.9 | 164.9 | ## Model Usage Breakdown | Model | steps | uncached | output | cache_read | cache_write | time | |-------|-------|----------|--------|------------|-------------|------| | opus[1m] | 1 | 3.4k | 25.1k | 1.7M | 84.0k | 16m 25s | | sonnet | 2 | 3.0k | 21.7k | 575.6k | 85.7k | 10m 35s | | MiniMax-M3 | 3 | 197.0k | 16.5k | 1.9M | 0 | 10m 5s | --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
diff --git a/.autoskillit/test-filter-manifest.yaml b/.autoskillit/test-filter-manifest.yaml
@@ -201,6 +201,12 @@ install.sh:
 scripts/verify-test-filter.sh:
   - infra/
 
+scripts/post-probe-failure.sh:
+  - infra/
+
+scripts/create-probe-canary-issue.sh:
+  - infra/
+
 # Python scripts under scripts/ — route to verified test consumers
 scripts/benchmark-testmon.py:
   - infra/
diff --git a/scripts/create-probe-canary-issue.sh b/scripts/create-probe-canary-issue.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+# Create a GitHub issue with the autoskillit-canary label.
+#
+# Usage: scripts/create-probe-canary-issue.sh TITLE BODY
+#
+# Args: $1=TITLE  $2=BODY
+#
+# Environment:
+#   GITHUB_REPOSITORY  — required, owner/repo format
+#   GITHUB_TOKEN       — required, GitHub authentication token
+
+set -euo pipefail
+export LC_ALL=C
+
+TITLE="${1:?Usage: $0 TITLE BODY}"
+BODY="${2:?Usage: $0 TITLE BODY}"
+
+if [[ -z "${GITHUB_REPOSITORY:-}" ]]; then
+    echo "ERROR: GITHUB_REPOSITORY is not set." >&2
+    exit 1
+fi
+
+if [[ -z "${GITHUB_TOKEN:-}" ]]; then
+    echo "ERROR: GITHUB_TOKEN is not set." >&2
+    exit 1
+fi
+
+gh issue create \
+    --repo "${GITHUB_REPOSITORY}" \
+    --title "${TITLE}" \
+    --label "autoskillit-canary" \
+    --body "${BODY}"
diff --git a/scripts/post-probe-failure.sh b/scripts/post-probe-failure.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Route a probe failure record to the Python canary state machine.
+#
+# Usage: scripts/post-probe-failure.sh BACKEND CLI_VERSION FAILURE_TYPE WORKFLOW_RUN_URL
+#
+# Args: $1=BACKEND  $2=CLI_VERSION  $3=FAILURE_TYPE  $4=WORKFLOW_RUN_URL
+#
+# Environment:
+#   CANARY_STATE_FILE  — path to canary state JSON (required)
+#   GITHUB_REPOSITORY  — owner/repo for issue creation (required by Python)
+
+set -euo pipefail
+export LC_ALL=C
+
+BACKEND="${1:?Usage: $0 BACKEND CLI_VERSION FAILURE_TYPE WORKFLOW_RUN_URL}"
+CLI_VERSION="${2:?Usage: $0 BACKEND CLI_VERSION FAILURE_TYPE WORKFLOW_RUN_URL}"
+FAILURE_TYPE="${3:?Usage: $0 BACKEND CLI_VERSION FAILURE_TYPE WORKFLOW_RUN_URL}"
+WORKFLOW_RUN_URL="${4:?Usage: $0 BACKEND CLI_VERSION FAILURE_TYPE WORKFLOW_RUN_URL}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+
+PYTHON="${PROJECT_ROOT}/.venv/bin/python"
+if [[ ! -x "${PYTHON}" ]]; then
+    echo "ERROR: ${PYTHON} not found. Run 'task install-worktree' first." >&2
+    exit 1
+fi
+
+"${PYTHON}" -m autoskillit._probe_canary post-failure \
+    --state-file "${CANARY_STATE_FILE:?CANARY_STATE_FILE is not set}" \
+    --backend "${BACKEND}" \
+    --cli-version "${CLI_VERSION}" \
+    --failure-type "${FAILURE_TYPE}" \
+    --workflow-run-url "${WORKFLOW_RUN_URL}"
diff --git a/src/autoskillit/_probe_canary.py b/src/autoskillit/_probe_canary.py
@@ -165,3 +165,90 @@ def _find_existing(self, title: str) -> int | None:
                 if isinstance(number, int):
                     return number
         return None
+
+
+def _handle_post_failure(
+    *,
+    state_file: str,
+    backend: str,
+    cli_version: str,
+    failure_type: str,
+    workflow_run_url: str,
+) -> int:
+    repo_slug = os.environ.get("GITHUB_REPOSITORY", "")
+    if not repo_slug or "/" not in repo_slug:
+        logger.error("post_failure_missing_github_repository")
+        return 1
+
+    owner, repo = repo_slug.split("/", 1)
+    state_path = Path(state_file)
+    state = CanaryState.load(state_path)
+    try:
+        kind = ErrorKind(failure_type)
+    except ValueError:
+        logger.error("post_failure_invalid_failure_type", failure_type=failure_type)
+        return 1
+    state.record_failure(kind)
+
+    if state.should_report():
+        title = f"[Canary] {backend} probe failure: {kind.value}"
+        body = (
+            f"**Backend:** {backend}\n"
+            f"**CLI Version:** {cli_version}\n"
+            f"**Failure Type:** {kind.value}\n"
+            f"**Workflow Run:** {workflow_run_url}\n"
+            f"**Network Streak:** {state.network_streak}\n"
+            f"**Schema Streak:** {state.schema_streak}\n"
+        )
+        updater = CanaryIssueUpdater(owner=owner, repo=repo)
+        try:
+            updater.ensure_issue(state, title, body)
+        except Exception as exc:
+            logger.error("canary_ensure_issue_failed", error=str(exc))
+
+    state.save(state_path)
+    return 0
+
+
+def _cli_main(argv: list[str] | None = None) -> int:
+    import argparse
+
+    parser = argparse.ArgumentParser(prog="autoskillit._probe_canary")
+    sub = parser.add_subparsers(dest="command")
+
+    post = sub.add_parser(
+        "post-failure", help="Record a probe failure and optionally create an issue"
+    )
+    post.add_argument("--state-file", required=True, help="Path to canary state JSON file")
+    post.add_argument("--backend", required=True, help="Probe backend identifier")
+    post.add_argument("--cli-version", required=True, help="CLI version that ran the probe")
+    post.add_argument(
+        "--failure-type",
+        required=True,
+        choices=[e.value for e in ErrorKind],
+        help="Error kind: network or schema",
+    )
+    post.add_argument("--workflow-run-url", required=True, help="GitHub Actions workflow run URL")
+
+    args = parser.parse_args(argv)
+
+    if args.command is None:
+        parser.print_help()
+        return 1
+
+    if args.command == "post-failure":
+        return _handle_post_failure(
+            state_file=args.state_file,
+            backend=args.backend,
+            cli_version=args.cli_version,
+            failure_type=args.failure_type,
+            workflow_run_url=args.workflow_run_url,
+        )
+
+    return 1
+
+
+if __name__ == "__main__":
+    import sys
+
+    sys.exit(_cli_main())
diff --git a/tests/execution/backends/test_probe_canary.py b/tests/execution/backends/test_probe_canary.py
@@ -12,6 +12,7 @@
     CanaryIssueUpdater,
     CanaryState,
     ErrorKind,
+    _cli_main,
 )
 
 pytestmark = [pytest.mark.layer("execution"), pytest.mark.small]
@@ -202,3 +203,158 @@ def mock_run_gh(args, **kwargs):
         assert num == 42
         assert state.last_issue_number == 42
         assert any(e.get("event") == "canary_issue_edit_failed" for e in cap_logs)
+
+
+class TestCliMain:
+    def test_post_failure_records_and_saves(
+        self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        """post-failure writes updated state to the state file."""
+        monkeypatch.setenv("GITHUB_REPOSITORY", "test-org/test-repo")
+        state_path = tmp_path / "state.json"
+
+        gh_call_count = 0
+
+        def mock_run_gh(args, **kwargs):
+            nonlocal gh_call_count
+            gh_call_count += 1
+            return CompletedProcess(args=args, returncode=1, stdout="", stderr="")
+
+        monkeypatch.setattr("autoskillit._probe_canary.run_gh", mock_run_gh)
+
+        rc = _cli_main(
+            [
+                "post-failure",
+                "--state-file",
+                str(state_path),
+                "--backend",
+                "claude-code",
+                "--cli-version",
+                "1.0.0",
+                "--failure-type",
+                "network",
+                "--workflow-run-url",
+                "https://example.com/run/1",
+            ]
+        )
+        assert rc == 0
+        assert state_path.exists()
+        raw = json.loads(state_path.read_text())
+        assert raw["network_streak"] == 1
+        assert gh_call_count == 0, "run_gh must not be called when streak is below threshold"
+
+    def test_post_failure_threshold_triggers_issue_creation(
+        self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        """When streak reaches N_CONSECUTIVE_FLAKE_GUARD, ensure_issue is called."""
+        monkeypatch.setenv("GITHUB_REPOSITORY", "test-org/test-repo")
+        state_path = tmp_path / "state.json"
+        state_path.write_text(
+            json.dumps(
+                {
+                    "network_streak": N_CONSECUTIVE_FLAKE_GUARD - 1,
+                    "schema_streak": 0,
+                    "last_issue_number": None,
+                }
+            )
+        )
+
+        gh_calls: list[list[str]] = []
+
+        def mock_run_gh(args, **kwargs):
+            gh_calls.append(list(args))
+            if args[0:2] == ["issue", "list"]:
+                return CompletedProcess(args=args, returncode=0, stdout="[]", stderr="")
+            if args[0:2] == ["issue", "create"]:
+                return CompletedProcess(
+                    args=args,
+                    returncode=0,
+                    stdout=json.dumps({"number": 7}),
+                    stderr="",
+                )
+            return CompletedProcess(args=args, returncode=1, stdout="", stderr="")
+
+        monkeypatch.setattr("autoskillit._probe_canary.run_gh", mock_run_gh)
+
+        rc = _cli_main(
+            [
+                "post-failure",
+                "--state-file",
+                str(state_path),
+                "--backend",
+                "claude-code",
+                "--cli-version",
+                "1.0.0",
+                "--failure-type",
+                "network",
+                "--workflow-run-url",
+                "https://example.com/run/2",
+            ]
+        )
+        assert rc == 0
+        assert any(call[0:2] == ["issue", "list"] for call in gh_calls)
+        assert any(call[0:2] == ["issue", "create"] for call in gh_calls)
+        saved = json.loads(state_path.read_text())
+        assert saved["network_streak"] == N_CONSECUTIVE_FLAKE_GUARD
+        assert saved["last_issue_number"] == 7
+
+    def test_post_failure_below_threshold_no_issue(
+        self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        """Below threshold, no gh issue commands are issued."""
+        monkeypatch.setenv("GITHUB_REPOSITORY", "test-org/test-repo")
+        state_path = tmp_path / "state.json"
+
+        def mock_run_gh(args, **kwargs):
+            raise AssertionError(f"Unexpected gh call: {args}")
+
+        monkeypatch.setattr("autoskillit._probe_canary.run_gh", mock_run_gh)
+
+        rc = _cli_main(
+            [
+                "post-failure",
+                "--state-file",
+                str(state_path),
+                "--backend",
+                "claude-code",
+                "--cli-version",
+                "1.0.0",
+                "--failure-type",
+                "network",
+                "--workflow-run-url",
+                "https://example.com/run/3",
+            ]
+        )
+        assert rc == 0
+        assert state_path.exists()
+        raw = json.loads(state_path.read_text())
+        assert raw["network_streak"] == 1
+
+    def test_post_failure_missing_github_repository(
+        self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        """Missing GITHUB_REPOSITORY env var returns 1."""
+        monkeypatch.delenv("GITHUB_REPOSITORY", raising=False)
+        state_path = tmp_path / "state.json"
+
+        rc = _cli_main(
+            [
+                "post-failure",
+                "--state-file",
+                str(state_path),
+                "--backend",
+                "claude-code",
+                "--cli-version",
+                "1.0.0",
+                "--failure-type",
+                "network",
+                "--workflow-run-url",
+                "https://example.com/run/4",
+            ]
+        )
+        assert rc == 1
+
+    def test_cli_no_command_returns_nonzero(self) -> None:
+        """Calling _cli_main with no subcommand returns non-zero."""
+        rc = _cli_main([])
+        assert rc != 0
diff --git a/tests/infra/AGENTS.md b/tests/infra/AGENTS.md
@@ -48,6 +48,7 @@ CI/CD configuration, security, guard coverage, and release sanity tests.
 | `test_pretty_output_hook_infra.py` | Tests: pretty_output hook infrastructure, fail-open, and coverage contracts |
 | `test_pretty_output_integration.py` | End-to-end schema consistency tests for the pretty_output hook |
 | `test_pretty_output_recipe.py` | Tests: pretty_output token/timing, load_recipe, list_recipes, open_kitchen, deduplication |
+| `test_probe_scripts.py` | Tests for CI-facing probe canary shell scripts (post-probe-failure.sh, create-probe-canary-issue.sh) — syntax, executable bit, and env-var validation |
 | `test_pyproject_bounds.py` | Tests for pyproject.toml version lower bounds |
 | `test_recipe_read_guard.py` | Tests for the recipe_read_guard PreToolUse hook — blocks recipe/skill/agent file reads |
 | `test_pyproject_metadata.py` | Verify pyproject.toml contains required public release metadata |
diff --git a/tests/infra/test_probe_scripts.py b/tests/infra/test_probe_scripts.py
