Skip to content

Commit a03e5a3

Browse files
ci(governance-reusable): timeout-minutes + codeload-retry docs (closes #208) (#224)
## Summary - Adds `timeout-minutes: 10` to all 8 jobs in `governance-reusable.yml`. A transient `codeload.github.com` outage that hangs the implicit `actions/checkout` mid-fetch can no longer burn the default 6-hour runner budget per stuck job. - Documents the situation + recommended downstream mitigation in the workflow's header doc block. ## Why not more Codeload's tarball-fetch retry policy is **runner-side**, controlled by GitHub, not surfaced via workflow inputs. The 2026-05-26 incident (cited in #208) saw `actions/checkout` itself fail to download, which happens *before* any per-step `continue-on-error` can fire. The workflow file therefore can't natively make those fetches resilient. The honest mitigation hierarchy: 1. **Workflow-level (this PR)**: bound the cost of a stuck job via `timeout-minutes`. 2. **Operational (today's pattern)**: `gh run rerun --failed --repo <owner>/<repo> <run-id>` — restored 60-80% of failures in the 2026-05-26 sweep. 3. **Opt-in per-repo (future)**: a `workflow_run`-triggered `auto-rerun-on-codeload-fail.yml` that scrubs the failing run's logs for the codeload signature and reruns once. Needs `actions: write`, which this reusable deliberately doesn't request — left as a sidecar pattern for repos that want it. ## Closes Closes #208. ## Test plan - [ ] Workflow YAML parses (governance jobs still discover at PR time) - [ ] No behaviour change for green runs - [ ] 10-minute upper bound observed via GH UI on next post-merge run Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 1a06816 commit a03e5a3

1 file changed

Lines changed: 26 additions & 0 deletions

File tree

.github/workflows/governance-reusable.yml

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,24 @@
3939
# (`advisory` | `blocking`); the default is `advisory` for the
4040
# one-week soak. Flip to `blocking` once the conversion is
4141
# observed-clean.
42+
#
43+
# Codeload-retry resilience (added 2026-05-27, standards#208):
44+
# - Every job carries `timeout-minutes: 10` so a transient
45+
# `codeload.github.com` outage cannot burn the full 6-hour default
46+
# when the runner hangs mid-fetch (observed 2026-05-26: single
47+
# codeload glitch propagated to 44 PRs across the estate).
48+
# - GitHub controls runner-side action-tarball retry; workflow files
49+
# cannot meaningfully override it. The recommended downstream
50+
# mitigation when a transient happens is:
51+
# gh run rerun --failed --repo <owner>/<repo> <run-id>
52+
# which restores 60-80% of failed runs in our experience without
53+
# consuming additional minutes beyond the failing job's residual.
54+
# - Consumer repos that want auto-rerun should add
55+
# `.github/workflows/auto-rerun-on-codeload-fail.yml` — a
56+
# workflow_run-triggered job that detects the codeload-failure
57+
# signature in the failing run's logs and reruns once. Not bundled
58+
# here because it requires `actions: write` permission scope which
59+
# this reusable intentionally avoids.
4260

4361
name: Estate Governance (reusable)
4462

@@ -58,6 +76,7 @@ jobs:
5876
validate-baseline:
5977
name: Validate Hypatia baseline
6078
runs-on: ${{ inputs.runs-on }}
79+
timeout-minutes: 10
6180
permissions:
6281
contents: read
6382
steps:
@@ -123,6 +142,7 @@ jobs:
123142
language-policy:
124143
name: Language / package anti-pattern policy
125144
runs-on: ${{ inputs.runs-on }}
145+
timeout-minutes: 10
126146
permissions:
127147
contents: read
128148
steps:
@@ -366,6 +386,7 @@ jobs:
366386
package-policy:
367387
name: Guix primary / Nix fallback policy
368388
runs-on: ${{ inputs.runs-on }}
389+
timeout-minutes: 10
369390
permissions:
370391
contents: read
371392
steps:
@@ -393,6 +414,7 @@ jobs:
393414
security-policy:
394415
name: Security policy checks
395416
runs-on: ${{ inputs.runs-on }}
417+
timeout-minutes: 10
396418
permissions:
397419
contents: read
398420
steps:
@@ -479,6 +501,7 @@ jobs:
479501
quality:
480502
name: Code quality + docs
481503
runs-on: ${{ inputs.runs-on }}
504+
timeout-minutes: 10
482505
permissions:
483506
contents: read
484507
steps:
@@ -527,6 +550,7 @@ jobs:
527550
wellknown:
528551
name: Well-Known (RFC 9116 + RSR)
529552
runs-on: ${{ inputs.runs-on }}
553+
timeout-minutes: 10
530554
permissions:
531555
contents: read
532556
steps:
@@ -584,6 +608,7 @@ jobs:
584608
workflow-lint:
585609
name: Workflow security linter
586610
runs-on: ${{ inputs.runs-on }}
611+
timeout-minutes: 10
587612
permissions:
588613
contents: read
589614
steps:
@@ -626,6 +651,7 @@ jobs:
626651
trusted-base:
627652
name: Trusted-base reduction policy
628653
runs-on: ${{ inputs.runs-on }}
654+
timeout-minutes: 10
629655
permissions:
630656
contents: read
631657
steps:

0 commit comments

Comments
 (0)