fix(workflows): make impl pipeline resilient to transient Claude failures (#5410)

MarkusNeusinger · claude · web-flow · commit 048f38fe6c99 · 2026-04-26T00:00:57.000+02:00
## Summary The implementation pipeline was leaving PRs and issues stuck after a single Claude Code Action hiccup. Three fixes restore self-healing behavior: - **`impl-generate.yml`**: cap raised from **2 → 3** generation attempts, aligning with the existing `impl:{lib}:failed` label description (*"max retries exhausted (3 attempts)"*) and the repair phase's 3-attempt budget. Failure comments now read `Attempt N/3`. - **`impl-repair.yml`**: previously had no failure handler — when the Claude Code Action itself crashed, the workflow ended with `ai-rejected` already removed and re-review never fired, leaving the PR silently stuck. Added a `Handle repair failure` step that restores `ai-rejected` and auto-retries the same attempt **once** via a marker comment, then falls back to manual. - **`impl-review.yml`**: both failure paths (Claude crash → `Handle review failure`, and score=0 from missing `quality_score.txt` → `Validate review output`) immediately surfaced `ai-review-failed`, requiring manual rerun. Both now auto-retry **once** via `repository_dispatch` with a shared marker comment before giving up. The `>=50% after 3 attempts` merge logic in `impl-review.yml` was already correct and is unchanged — these fixes only ensure PRs reach that gate instead of stalling earlier. ## Concrete trigger (not added to the PR but motivated it) Issue #5365 (`area-mountain-panorama`) had **4/9 libraries hard-failed** without ever creating a PR (transient Claude crashes during generate, capped at 2 attempts), **1 PR stuck** with `ai-review-failed` (plotnine #5372), and **1 PR stuck** mid-repair (altair #5370 — repair workflow itself crashed on attempt 1). Manual recovery was triggered earlier in the conversation. ## Test plan - [ ] Trigger a generate that fails twice (e.g., simulate or wait for transient flake) — should auto-retry to attempt 3 instead of stopping at 2 - [ ] Trigger a repair where Claude Code Action crashes — should restore `ai-rejected` and auto-retry the same attempt once via marker comment - [ ] Trigger a review where Claude crashes — should auto-retry via `repository_dispatch` once before adding `ai-review-failed` - [ ] Trigger a review where Claude runs but writes no `quality_score.txt` — same auto-retry behavior - [ ] Verify markers prevent infinite retry loops (each marker only allows one auto-retry) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
diff --git a/.github/workflows/impl-generate.yml b/.github/workflows/impl-generate.yml
@@ -755,16 +755,22 @@ jobs:
         run: |
           echo "::notice::Handling generation failure for $LIBRARY/$SPEC_ID"
 
-          # Count previous failures via hidden marker comments (more reliable than workflow runs)
+          # Count previous failures via hidden marker comments (more reliable than workflow runs).
+          # Paginate so the marker is found even on issues with >30 comments
+          # (which is common because all 9 library impls land on the same issue).
           MARKER="<!-- impl-fail:${SPEC_ID}:${LIBRARY} -->"
-          FAILURE_COUNT=$(gh api "repos/${{ github.repository }}/issues/${ISSUE}/comments" \
-            --jq "[.[] | select(.body | contains(\"$MARKER\"))] | length" 2>/dev/null || echo "0")
+          FAILURE_COUNT=$(gh api --paginate "repos/${{ github.repository }}/issues/${ISSUE}/comments?per_page=100" \
+            --jq "[.[] | select(.body != null and (.body | contains(\"$MARKER\")))] | length" 2>/dev/null || echo "0")
 
           echo "::notice::Previous failures for ${LIBRARY}/${SPEC_ID}: $FAILURE_COUNT"
 
-          # After 1 previous failure (= this is attempt 2) → mark as failed
-          if [ "$FAILURE_COUNT" -ge 1 ]; then
-            echo "::warning::Marking $LIBRARY as failed after 2 generation attempts"
+          # FAILURE_COUNT counts marker comments BEFORE this run.
+          # 0 → this is attempt 1 fail, 1 → attempt 2 fail, 2 → attempt 3 fail.
+          ATTEMPT=$((FAILURE_COUNT + 1))
+
+          # After 2 previous failures (= this is attempt 3) → mark as failed
+          if [ "$FAILURE_COUNT" -ge 2 ]; then
+            echo "::warning::Marking $LIBRARY as failed after 3 generation attempts"
 
             # Create failed label if needed
             gh label create "impl:${LIBRARY}:failed" --color "d73a4a" \
@@ -777,9 +783,9 @@ jobs:
 
             # Post final failure comment with marker
             gh issue comment "$ISSUE" --body "${MARKER}
-          ## :x: ${LIBRARY} Failed (Attempt 2/2)
+          ## :x: ${LIBRARY} Failed (Attempt 3/3)
 
-          The **${LIBRARY}** implementation for \`${SPEC_ID}\` failed after 2 attempts.
+          The **${LIBRARY}** implementation for \`${SPEC_ID}\` failed after 3 attempts.
 
           **Reason:** Claude Code failed to create the implementation file.
 
@@ -792,11 +798,11 @@ jobs:
           :robot: *[impl-generate](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
 
           else
-            # First failure → post comment with marker and auto-retry
+            # Attempt 1 or 2 failed → post comment with marker and auto-retry
             gh issue comment "$ISSUE" --body "${MARKER}
-          ## :warning: ${LIBRARY} Generation Failed (Attempt 1/2)
+          ## :warning: ${LIBRARY} Generation Failed (Attempt ${ATTEMPT}/3)
 
-          First attempt failed. Automatically retrying...
+          Attempt ${ATTEMPT} failed. Automatically retrying...
 
           ---
           :robot: *[impl-generate](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
@@ -810,5 +816,5 @@ jobs:
               -f library="${LIBRARY}" \
               -f issue_number="${ISSUE}"
 
-            echo "::notice::Triggered automatic retry for ${LIBRARY}/${SPEC_ID}"
+            echo "::notice::Triggered automatic retry for ${LIBRARY}/${SPEC_ID} (attempt $((ATTEMPT + 1)))"
           fi
diff --git a/.github/workflows/impl-repair.yml b/.github/workflows/impl-repair.yml
@@ -239,3 +239,57 @@ jobs:
             -f event_type=review-pr \
             -f "client_payload[pr_number]=$PR_NUM"
           echo "::notice::Triggered impl-review.yml via repository_dispatch for PR #$PR_NUM"
+
+      # ========================================================================
+      # Failure handling: when the repair workflow itself crashes (e.g. Claude
+      # Code Action transient failure), restore the ai-rejected label and
+      # auto-retry once. After one auto-retry, fall back to manual.
+      # ========================================================================
+      - name: Handle repair failure
+        if: failure()
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PR_NUM: ${{ inputs.pr_number }}
+          SPEC_ID: ${{ inputs.specification_id }}
+          LIBRARY: ${{ inputs.library }}
+          ATTEMPT: ${{ inputs.attempt }}
+        run: |
+          # Restore ai-rejected label that was removed at start of repair so the
+          # PR state stays consistent (otherwise it looks "stuck approved").
+          gh pr edit "$PR_NUM" --add-label "ai-rejected" 2>/dev/null || true
+
+          MARKER="<!-- repair-retry:${SPEC_ID}:${LIBRARY}:attempt-${ATTEMPT} -->"
+          # Paginate so the marker is found even on PRs with >30 comments.
+          RETRY_COUNT=$(gh api --paginate "repos/${{ github.repository }}/issues/${PR_NUM}/comments?per_page=100" \
+            --jq "[.[] | select(.body != null and (.body | contains(\"$MARKER\")))] | length" 2>/dev/null || echo "0")
+
+          if [ "$RETRY_COUNT" -ge 1 ]; then
+            echo "::error::Repair attempt ${ATTEMPT} crashed twice — giving up"
+            gh pr comment "$PR_NUM" --body "${MARKER}
+          ## :x: Repair Workflow Crashed (Attempt ${ATTEMPT}/3, retry exhausted)
+
+          The repair workflow itself failed twice for this attempt — likely a persistent Claude Code Action issue.
+
+          **Manual restart:**
+          \`\`\`
+          gh workflow run impl-repair.yml -f pr_number=${PR_NUM} -f specification_id=${SPEC_ID} -f library=${LIBRARY} -f attempt=${ATTEMPT}
+          \`\`\`
+
+          ---
+          :robot: *[impl-repair](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
+          else
+            echo "::warning::Repair attempt ${ATTEMPT} crashed — auto-retrying once"
+            gh pr comment "$PR_NUM" --body "${MARKER}
+          ## :wrench: Repair Workflow Crashed (Attempt ${ATTEMPT}/3) — Auto-Retrying
+
+          The repair workflow failed (probably a transient Claude Code Action issue). Automatically re-triggering this attempt...
+
+          ---
+          :robot: *[impl-repair](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
+
+            gh workflow run impl-repair.yml \
+              -f pr_number="$PR_NUM" \
+              -f specification_id="$SPEC_ID" \
+              -f library="$LIBRARY" \
+              -f attempt="$ATTEMPT"
+          fi
diff --git a/.github/workflows/impl-review.yml b/.github/workflows/impl-review.yml
@@ -172,33 +172,53 @@ jobs:
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           PR_NUM: ${{ steps.pr.outputs.pr_number }}
+          SPEC_ID: ${{ steps.pr.outputs.specification_id }}
+          LIBRARY: ${{ steps.pr.outputs.library }}
         run: |
-          echo "::error::AI Review did not produce valid output files"
-          echo "::error::Expected quality_score.txt but got score=0"
-          echo "::error::This indicates Claude Code Action ran but didn't complete the review task"
+          echo "::error::AI Review did not produce valid output files (score=0)"
 
-          # Add ai-review-failed label so it's visible
-          gh pr edit "$PR_NUM" --add-label "ai-review-failed" 2>/dev/null || true
+          MARKER="<!-- review-retry:${SPEC_ID}:${LIBRARY} -->"
+          # Paginate so the marker is found even on PRs with >30 comments.
+          RETRY_COUNT=$(gh api --paginate "repos/${{ github.repository }}/issues/${PR_NUM}/comments?per_page=100" \
+            --jq "[.[] | select(.body != null and (.body | contains(\"$MARKER\")))] | length" 2>/dev/null || echo "0")
 
-          # Post error comment on PR
-          gh pr comment "$PR_NUM" --body "## :x: AI Review Failed
+          if [ "$RETRY_COUNT" -ge 1 ]; then
+            # Already auto-retried once → final fail, require manual rerun
+            gh pr edit "$PR_NUM" --add-label "ai-review-failed" 2>/dev/null || true
+            gh pr comment "$PR_NUM" --body "${MARKER}
+          ## :x: AI Review Failed (auto-retry exhausted)
 
-          The AI review action completed but did not produce valid output files.
+          The AI review action completed but did not produce valid output files. Auto-retry already tried once.
 
           **What happened:**
           - The Claude Code Action ran
           - No \`quality_score.txt\` file was created
-          - No review data was extracted
 
-          **Action required:**
-          Re-run the impl-review workflow manually:
+          **Manual rerun:**
           \`\`\`
           gh workflow run impl-review.yml -f pr_number=$PR_NUM
           \`\`\`
 
           ---
           :robot: *[impl-review](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
+            exit 1
+          fi
 
+          # First failure — post marker and auto-retry via repository_dispatch
+          gh pr comment "$PR_NUM" --body "${MARKER}
+          ## :wrench: AI Review Produced No Score — Auto-Retrying
+
+          The Claude Code Action ran but didn't write \`quality_score.txt\`. Auto-retrying review once...
+
+          ---
+          :robot: *[impl-review](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
+
+          gh api repos/${{ github.repository }}/dispatches \
+            -f event_type=review-pr \
+            -f "client_payload[pr_number]=$PR_NUM"
+          echo "::notice::Auto-re-triggered impl-review.yml for PR #$PR_NUM"
+          # Mark this run as failed so the run status reflects that no verdict
+          # was produced. The auto-retry runs in a separate workflow run.
           exit 1
 
       - name: Add quality score label
@@ -448,19 +468,47 @@ jobs:
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           PR_NUM: ${{ steps.pr.outputs.pr_number }}
+          SPEC_ID: ${{ steps.pr.outputs.specification_id }}
+          LIBRARY: ${{ steps.pr.outputs.library }}
         run: |
-          gh pr edit "$PR_NUM" --add-label "ai-review-failed"
-          gh pr comment "$PR_NUM" --body "## :warning: AI Review Failed
+          MARKER="<!-- review-retry:${SPEC_ID}:${LIBRARY} -->"
+          # Paginate so the marker is found even on PRs with >30 comments.
+          RETRY_COUNT=$(gh api --paginate "repos/${{ github.repository }}/issues/${PR_NUM}/comments?per_page=100" \
+            --jq "[.[] | select(.body != null and (.body | contains(\"$MARKER\")))] | length" 2>/dev/null || echo "0")
 
-          The AI review action failed or timed out.
+          if [ "$RETRY_COUNT" -ge 1 ]; then
+            gh pr edit "$PR_NUM" --add-label "ai-review-failed" 2>/dev/null || true
+            gh pr comment "$PR_NUM" --body "${MARKER}
+          ## :x: AI Review Failed (auto-retry exhausted)
 
-          **Options:**
-          1. Re-run the workflow manually
-          2. Request manual human review
+          The AI review action failed or timed out twice in a row.
+
+          **Manual rerun:**
+          \`\`\`
+          gh workflow run impl-review.yml -f pr_number=$PR_NUM
+          \`\`\`
+
+          ---
+          :robot: *[impl-review](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
+            exit 0
+          fi
+
+          gh pr comment "$PR_NUM" --body "${MARKER}
+          ## :wrench: AI Review Crashed — Auto-Retrying
+
+          The Claude Code Action failed or timed out. Auto-retrying review once...
 
           ---
           :robot: *[impl-review](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
 
+          gh api repos/${{ github.repository }}/dispatches \
+            -f event_type=review-pr \
+            -f "client_payload[pr_number]=$PR_NUM"
+          echo "::notice::Auto-re-triggered impl-review.yml for PR #$PR_NUM"
+          # Mark this run as failed so the run status reflects that no verdict
+          # was produced. The auto-retry runs in a separate workflow run.
+          exit 1
+
       - name: Add verdict label and take action
         if: steps.review.conclusion == 'success' && steps.score.outputs.score != '0'
         env:
