Implementation Plan: Fix Infinite CI Loop — flake_suspected Routing Escape for is_fixable=false

src/autoskillit/recipes/diagrams/implementation-groups.md

@@ -1,4 +1,4 @@
-<!-- autoskillit-recipe-hash: sha256:c8e501dca905a687bfceb05c903a86d7f4bdb9d0057fe39dc510a303d595e1bf -->
+<!-- autoskillit-recipe-hash: sha256:f0b9b6a9aaeedd513a73303b939fda5cfc2c287e9c3dd060bb79df7a99a48c41 -->
 <!-- autoskillit-diagram-format: v7 -->
 ## implementation-groups
 Group-based implementation with per-group plan/implement/test cycles and PR gates.

src/autoskillit/recipes/diagrams/implementation.md

@@ -1,4 +1,4 @@
-<!-- autoskillit-recipe-hash: sha256:6387790f3513145bccf0f7c64f4d1ac8e3b807f61bd3ff217da33b1bdefef946 -->
+<!-- autoskillit-recipe-hash: sha256:e5972fabca3a0bb471f4f99688ceaa78127056b33100574f05f355712fe50ffb -->
 <!-- autoskillit-diagram-format: v7 -->
 # implementation
 

src/autoskillit/recipes/diagrams/remediation.md

@@ -1,4 +1,4 @@
-<!-- autoskillit-recipe-hash: sha256:3a11f66644f46f34ca24a05b186d0d2c7421a6c331a9440d764d0450f9a8ac27 -->
+<!-- autoskillit-recipe-hash: sha256:98efa055bc27ca8057fe1dc01887cbe1e2094b036a71bd34a6469cc52d637c18 -->
 <!-- autoskillit-diagram-format: v7 -->
 ## remediation
 Investigate, rectify, implement, and merge a bug fix with CI and PR gates.

src/autoskillit/recipes/implementation-groups.json

@@ -1527,7 +1527,7 @@
         },
         {
           "when": "${{ result.verdict }} == 'flake_suspected'",
-          "route": "re_push"
+          "route": "check_flake_loop"
         },
         {
           "when": "${{ result.verdict }} == 'ci_only_failure'",
@@ -1542,7 +1542,7 @@
         }
       ],
       "on_failure": "release_issue_failure",
-      "note": "Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci structured output to avoid redundant CI failure re-investigation. Routes via on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses (sibling fix may have landed), flake_suspected retries via re_push (bounded by retries: 2 / on_exhausted: release_issue_failure), ci_only_failure escalates to human. Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.\n"
+      "note": "Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci structured output to avoid redundant CI failure re-investigation. Routes via on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses (sibling fix may have landed), flake_suspected routes to check_flake_loop (bounded by ci_flake_count <= 3, shared across all diagnose_ci callers; escalates to release_issue_failure when exceeded), ci_only_failure escalates to human. Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.\n"
     },
     "pre_resolve_rebase": {
       "tool": "run_python",
@@ -1618,6 +1618,31 @@
       ],
       "note": "Guards the resolve_ci(already_green) → pre_resolve_rebase → diagnose_ci cycle. Caps at 3 rebase-rediagnose iterations. If CI keeps reporting already_green after 3 rebases, a structural issue requires human intervention.\n"
     },
+    "check_flake_loop": {
+      "tool": "run_python",
+      "with": {
+        "callable": "autoskillit.smoke_utils.check_loop_iteration",
+        "current_iteration": "${{ context.ci_flake_count }}",
+        "max_iterations": "3"
+      },
+      "capture": {
+        "ci_flake_count": "${{ result.next_iteration }}"
+      },
+      "on_result": [
+        {
+          "when": "${{ result.max_exceeded }} == true",
+          "route": "release_issue_failure"
+        },
+        {
+          "route": "re_push"
+        }
+      ],
+      "on_failure": "release_issue_failure",
+      "optional_context_refs": [
+        "ci_flake_count"
+      ],
+      "note": "Guards ALL resolve_ci(flake_suspected) → re_push cycles. The ci_flake_count counter is shared across all entry points into diagnose_ci, including CI watch retries, merge queue ejections, dropped-CI-group paths, and conflict resolution escalations. Caps at 3 total flake_suspected verdicts per pipeline run. If CI keeps failing with the same flake_suspected verdict after 3 iterations from any combination of callers, the failure requires human intervention.\n"
+    },
     "re_push": {
       "tool": "push_to_remote",
       "with": {

src/autoskillit/recipes/implementation-groups.yaml

@@ -1373,7 +1373,7 @@ steps:
     - when: "${{ result.verdict }} == 'already_green'"
       route: pre_resolve_rebase
     - when: "${{ result.verdict }} == 'flake_suspected'"
-      route: re_push
+      route: check_flake_loop
     - when: "${{ result.verdict }} == 'ci_only_failure'"
       route: release_issue_failure
     - when: "${{ result.verdict }} == 'no_test_infrastructure'"
@@ -1385,8 +1385,9 @@ steps:
       Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci
       structured output to avoid redundant CI failure re-investigation. Routes via
       on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses
-      (sibling fix may have landed), flake_suspected retries via re_push (bounded by
-      retries: 2 / on_exhausted: release_issue_failure), ci_only_failure escalates to human.
+      (sibling fix may have landed), flake_suspected routes to check_flake_loop (bounded
+      by ci_flake_count <= 3, shared across all diagnose_ci callers; escalates to
+      release_issue_failure when exceeded), ci_only_failure escalates to human.
       Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.
 
   pre_resolve_rebase:
@@ -1449,6 +1450,29 @@ steps:
       Caps at 3 rebase-rediagnose iterations. If CI keeps reporting already_green
       after 3 rebases, a structural issue requires human intervention.
 
+  check_flake_loop:
+    tool: run_python
+    with:
+      callable: "autoskillit.smoke_utils.check_loop_iteration"
+      current_iteration: "${{ context.ci_flake_count }}"
+      max_iterations: "3"
+    capture:
+      ci_flake_count: "${{ result.next_iteration }}"
+    on_result:
+    - when: "${{ result.max_exceeded }} == true"
+      route: release_issue_failure
+    - route: re_push
+    on_failure: release_issue_failure
+    optional_context_refs:
+    - ci_flake_count
+    note: >
+      Guards ALL resolve_ci(flake_suspected) → re_push cycles. The ci_flake_count
+      counter is shared across all entry points into diagnose_ci, including CI watch
+      retries, merge queue ejections, dropped-CI-group paths, and conflict resolution
+      escalations. Caps at 3 total flake_suspected verdicts per pipeline run. If CI
+      keeps failing with the same flake_suspected verdict after 3 iterations from any
+      combination of callers, the failure requires human intervention.
+
   re_push:
     tool: push_to_remote
     with:

src/autoskillit/recipes/implementation.json

@@ -2336,7 +2336,7 @@
         },
         {
           "when": "${{ result.verdict }} == 'flake_suspected'",
-          "route": "re_push"
+          "route": "check_flake_loop"
         },
         {
           "when": "${{ result.verdict }} == 'ci_only_failure'",
@@ -2351,7 +2351,7 @@
         }
       ],
       "on_failure": "release_issue_failure",
-      "note": "Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci structured output to avoid redundant CI failure re-investigation. Routes via on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses (sibling fix may have landed), flake_suspected retries via re_push (bounded by retries: 2 / on_exhausted: release_issue_failure), ci_only_failure escalates to human. Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.\n"
+      "note": "Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci structured output to avoid redundant CI failure re-investigation. Routes via on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses (sibling fix may have landed), flake_suspected routes to check_flake_loop (bounded by ci_flake_count <= 3, shared across all diagnose_ci callers; escalates to release_issue_failure when exceeded), ci_only_failure escalates to human. Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.\n"
     },
     "pre_resolve_rebase": {
       "tool": "run_python",
@@ -2427,6 +2427,31 @@
       ],
       "note": "Guards the resolve_ci(already_green) → pre_resolve_rebase → diagnose_ci cycle. Caps at 3 rebase-rediagnose iterations. If CI keeps reporting already_green after 3 rebases, a structural issue requires human intervention.\n"
     },
+    "check_flake_loop": {
+      "tool": "run_python",
+      "with": {
+        "callable": "autoskillit.smoke_utils.check_loop_iteration",
+        "current_iteration": "${{ context.ci_flake_count }}",
+        "max_iterations": "3"
+      },
+      "capture": {
+        "ci_flake_count": "${{ result.next_iteration }}"
+      },
+      "on_result": [
+        {
+          "when": "${{ result.max_exceeded }} == true",
+          "route": "release_issue_failure"
+        },
+        {
+          "route": "re_push"
+        }
+      ],
+      "on_failure": "release_issue_failure",
+      "optional_context_refs": [
+        "ci_flake_count"
+      ],
+      "note": "Guards ALL resolve_ci(flake_suspected) → re_push cycles. The ci_flake_count counter is shared across all entry points into diagnose_ci, including CI watch retries, merge queue ejections, dropped-CI-group paths, and conflict resolution escalations. Caps at 3 total flake_suspected verdicts per pipeline run. If CI keeps failing with the same flake_suspected verdict after 3 iterations from any combination of callers, the failure requires human intervention.\n"
+    },
     "re_push": {
       "tool": "push_to_remote",
       "with": {

src/autoskillit/recipes/implementation.yaml

@@ -2103,7 +2103,7 @@ steps:
     - when: ${{ result.verdict }} == 'already_green'
       route: pre_resolve_rebase
     - when: ${{ result.verdict }} == 'flake_suspected'
-      route: re_push
+      route: check_flake_loop
     - when: ${{ result.verdict }} == 'ci_only_failure'
       route: release_issue_failure
     - when: ${{ result.verdict }} == 'no_test_infrastructure'
@@ -2114,9 +2114,10 @@ steps:
     note: 'Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci
       structured output to avoid redundant CI failure re-investigation. Routes via
       on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses
-      (sibling fix may have landed), flake_suspected retries via re_push (bounded
-      by retries: 2 / on_exhausted: release_issue_failure), ci_only_failure escalates
-      to human. Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.
+      (sibling fix may have landed), flake_suspected routes to check_flake_loop (bounded
+      by ci_flake_count <= 3, shared across all diagnose_ci callers; escalates to
+      release_issue_failure when exceeded), ci_only_failure escalates to human.
+      Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.
 
       '
   pre_resolve_rebase:
@@ -2178,6 +2179,29 @@ steps:
       cycle. Caps at 3 rebase-rediagnose iterations. If CI keeps reporting already_green
       after 3 rebases, a structural issue requires human intervention.
 
+      '
+  check_flake_loop:
+    tool: run_python
+    with:
+      callable: autoskillit.smoke_utils.check_loop_iteration
+      current_iteration: ${{ context.ci_flake_count }}
+      max_iterations: '3'
+    capture:
+      ci_flake_count: ${{ result.next_iteration }}
+    on_result:
+    - when: ${{ result.max_exceeded }} == true
+      route: release_issue_failure
+    - route: re_push
+    on_failure: release_issue_failure
+    optional_context_refs:
+    - ci_flake_count
+    note: 'Guards ALL resolve_ci(flake_suspected) → re_push cycles. The ci_flake_count
+      counter is shared across all entry points into diagnose_ci, including CI watch
+      retries, merge queue ejections, dropped-CI-group paths, and conflict resolution
+      escalations. Caps at 3 total flake_suspected verdicts per pipeline run. If CI
+      keeps failing with the same flake_suspected verdict after 3 iterations from any
+      combination of callers, the failure requires human intervention.
+
       '
   re_push:
     tool: push_to_remote

src/autoskillit/recipes/remediation.json

@@ -2472,7 +2472,7 @@
         },
         {
           "when": "${{ result.verdict }} == 'flake_suspected'",
-          "route": "re_push"
+          "route": "check_flake_loop"
         },
         {
           "when": "${{ result.verdict }} == 'ci_only_failure'",
@@ -2487,7 +2487,7 @@
         }
       ],
       "on_failure": "release_issue_failure",
-      "note": "Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci structured output to avoid redundant CI failure re-investigation. Routes via on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses (sibling fix may have landed), flake_suspected retries via re_push (bounded by retries: 2 / on_exhausted: release_issue_failure), ci_only_failure escalates to human. Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.\n"
+      "note": "Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci structured output to avoid redundant CI failure re-investigation. Routes via on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses (sibling fix may have landed), flake_suspected routes to check_flake_loop (bounded by ci_flake_count <= 3, shared across all diagnose_ci callers; escalates to release_issue_failure when exceeded), ci_only_failure escalates to human. Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.\n"
     },
     "pre_resolve_rebase": {
       "tool": "run_python",
@@ -2563,6 +2563,31 @@
       ],
       "note": "Guards the resolve_ci(already_green) → pre_resolve_rebase → diagnose_ci cycle. Caps at 3 rebase-rediagnose iterations. If CI keeps reporting already_green after 3 rebases, a structural issue requires human intervention.\n"
     },
+    "check_flake_loop": {
+      "tool": "run_python",
+      "with": {
+        "callable": "autoskillit.smoke_utils.check_loop_iteration",
+        "current_iteration": "${{ context.ci_flake_count }}",
+        "max_iterations": "3"
+      },
+      "capture": {
+        "ci_flake_count": "${{ result.next_iteration }}"
+      },
+      "on_result": [
+        {
+          "when": "${{ result.max_exceeded }} == true",
+          "route": "release_issue_failure"
+        },
+        {
+          "route": "re_push"
+        }
+      ],
+      "on_failure": "release_issue_failure",
+      "optional_context_refs": [
+        "ci_flake_count"
+      ],
+      "note": "Guards ALL resolve_ci(flake_suspected) → re_push cycles. The ci_flake_count counter is shared across all entry points into diagnose_ci, including CI watch retries, merge queue ejections, dropped-CI-group paths, and conflict resolution escalations. Caps at 3 total flake_suspected verdicts per pipeline run. If CI keeps failing with the same flake_suspected verdict after 3 iterations from any combination of callers, the failure requires human intervention.\n"
+    },
     "re_push": {
       "tool": "push_to_remote",
       "with": {

src/autoskillit/recipes/remediation.yaml

@@ -2209,7 +2209,7 @@ steps:
     - when: ${{ result.verdict }} == 'already_green'
       route: pre_resolve_rebase
     - when: ${{ result.verdict }} == 'flake_suspected'
-      route: re_push
+      route: check_flake_loop
     - when: ${{ result.verdict }} == 'ci_only_failure'
       route: release_issue_failure
     - when: ${{ result.verdict }} == 'no_test_infrastructure'
@@ -2219,9 +2219,10 @@ steps:
     note: 'Runs when ci_watch reports a CI failure. Receives ci_failed_jobs from wait_for_ci
       structured output to avoid redundant CI failure re-investigation. Routes via
       on_result: verdict dispatch — real_fix pushes, already_green rebases and re-diagnoses
-      (sibling fix may have landed), flake_suspected retries via re_push (bounded
-      by retries: 2 / on_exhausted: release_issue_failure), ci_only_failure escalates
-      to human. Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.
+      (sibling fix may have landed), flake_suspected routes to check_flake_loop (bounded
+      by ci_flake_count <= 3, shared across all diagnose_ci callers; escalates to
+      release_issue_failure when exceeded), ci_only_failure escalates to human.
+      Retries up to 2 times (3 total attempts) before exhausting to release_issue_failure.
 
       '
   pre_resolve_rebase:
@@ -2282,6 +2283,29 @@ steps:
       cycle. Caps at 3 rebase-rediagnose iterations. If CI keeps reporting already_green
       after 3 rebases, a structural issue requires human intervention.
 
+      '
+  check_flake_loop:
+    tool: run_python
+    with:
+      callable: autoskillit.smoke_utils.check_loop_iteration
+      current_iteration: ${{ context.ci_flake_count }}
+      max_iterations: '3'
+    capture:
+      ci_flake_count: ${{ result.next_iteration }}
+    on_result:
+    - when: ${{ result.max_exceeded }} == true
+      route: release_issue_failure
+    - route: re_push
+    on_failure: release_issue_failure
+    optional_context_refs:
+    - ci_flake_count
+    note: 'Guards ALL resolve_ci(flake_suspected) → re_push cycles. The ci_flake_count
+      counter is shared across all entry points into diagnose_ci, including CI watch
+      retries, merge queue ejections, dropped-CI-group paths, and conflict resolution
+      escalations. Caps at 3 total flake_suspected verdicts per pipeline run. If CI
+      keeps failing with the same flake_suspected verdict after 3 iterations from any
+      combination of callers, the failure requires human intervention.
+
       '
   re_push:
     tool: push_to_remote