Skip to content

Commit aaee1af

Browse files
fix(ci): staleness gate fails on named defects, not on the calendar (#522)
The Governance staleness gate is currently failing **294 of 350 consumer repos** — and it is failing them for the wrong reason. ## What is actually happening All 294 pin the *same* SHA, `d7c22711` (2026-06-26). Nothing is wrong with any of them individually. `standards` moves ~2.6 commits/day, so a consumer exhausts both window budgets (≤50 commits **or** ≤14 days) about a fortnight after any propagation, and goes red on a timer. `aspasia` is the clean natural experiment — same pin, nothing changed in the repo: | run | pin age | verdict | |---|---|---| | 2026-07-09 | 32 commits / 12d | `::notice`, passing | | 2026-07-21 | 63 commits / 24d | `::error`, hard fail | Holding the fleet green under that rule means re-pinning ~300 repos every fortnight — on the order of **7,800 PRs/year**. A gate that fails everyone on a timer is not a guard. It trains the estate to ignore it, and it buries the findings that matter: the same sweep shows **148 real `Workflow security linter` failures** and **74 anti-pattern failures**, all of which read as noise once the fleet is uniformly red. ## What changes **1. Age outside the window is now a `::notice`, not an error.** The window is still computed and still reported, so `propagate-workflow-pins.sh` and the Hypatia `sha_bump_propagation` rule keep their signal. **2. A named deny-list (`KNOWN_BAD_BEFORE`) replaces age as the hard failure.** Each entry is `<reusable>:<fix-sha>`; a pin that is a strict ancestor of the fix carries that defect and is rejected **at any age**. This is what the window was only ever a proxy for, and it is strictly better in *both* directions — a **recent** pin carrying the defect is now caught, and an old pin carrying none is no longer punished by the calendar. The first entry is `e9c8888769a7` (#441). Before it, `hypatia-scan-reusable` and the `validate-hypatia-baseline` job cached the built Hypatia scanner under a keyless key while the build steps were guarded by `if [ ! -d ]` — so *the first scanner build ever cached was reused forever* and scanner fixes never took effect. A pin older than this reports a **false green**. > ⚠️ **This PR turns zero repos green, by design.** All 294 consumers on `d7c22711` predate the Hypatia fix by one day, so they stay red — now saying *"your Hypatia scanner is frozen, refresh this pin"* instead of *"you are 24 days old"*. Relaxing the window instead would have manufactured a green light across 294 repos running a known-broken scanner. The green comes from re-pinning them, which is a separate change. **3. Integrity no longer rests on the runner's clone.** The `UNKNOWN` / "may be forged" verdict was firing on legitimate pins. `awesome-haskell` pins `governance-reusable@5a93d9d` and was accused on **four consecutive runs over 17 days**, while that commit verifies as a true ancestor of `main` locally in *both* treeless and `--depth 200` clones and via the compare API (`behind=0, ahead=90`). The mechanism was never reproduced off-runner. A supply-chain accusation must not depend on the health of a clone the gate does not control. A hard `FORGED` verdict now requires either a **complete** local clone (not shallow, not partial) or confirmation from `GET /repos/{nwo}/compare/{pin}...{branch}`. Where neither is available the gate emits a `::warning` and passes — *cannot verify* is not *compromised*. Cost is **zero API calls on the happy path**; the server is consulted only when the gate is about to accuse. The deny-list gets the same fallback, so a degraded runner cannot silently skip it: "could not check" is reported as **SKIPPED, never as passed**. ## Verification The hermetic fixture suite goes **12 → 16 cases, 16/16 passing**: ``` PASS: ancient pin (400d, 5 behind) passes with an advisory notice PASS: pin predating a known-bad fix fails regardless of window PASS: in-window pin carrying a known defect still fails PASS: the fixing commit itself is not denied PASS: a deny-list entry for another reusable does not fail this one ``` Out-of-window now asserts exit 0 **and** the advisory notice, via a new `run_case_out` helper — an exit code alone cannot distinguish "passed silently" from "passed with the notice the propagation path depends on". The suite stays hermetic: a complete fixture clone resolves the forged case locally, so no network is required. Also verified against **live standards history** and in **degraded-clone mode**; both give identical verdicts for fresh / denied / old-but-clean / forged. `shellcheck -S warning` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent f9dca6d commit aaee1af

3 files changed

Lines changed: 366 additions & 20 deletions

File tree

docs/decisions/ADR-003-workflow-pin-staleness-window.adoc

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,10 @@ minimal data) instead of `--depth 1`, with a `--depth 200` fallback.
145145
`scripts/tests/check-workflow-staleness-test.sh` (hermetic fixture history)
146146
covers fresh / ahead / in-window-by-commits / in-window-by-days /
147147
out-of-window / forged, plus the retired-enforcer and SARIF structural rules.
148+
Extended by the 2026-07-21 amendment below to 16 cases: out-of-window now
149+
asserts exit 0 *and* the advisory notice, plus deny-list coverage (rejected at
150+
any age, rejected even when in-window, the fixing commit itself not rejected,
151+
and entries scoped per reusable).
148152
`scripts/tests/propagate-workflow-pins-test.sh` covers audit/fix/idempotence
149153
and branch staging. End-to-end against real standards history, a
150154
stapeln-shaped consumer pinned 3 commits behind HEAD passes the relaxed gate
@@ -162,3 +166,60 @@ stapeln-shaped consumer pinned 3 commits behind HEAD passes the relaxed gate
162166
rules out mutable-tag pinning (option C).
163167
* `scripts/propagate-gitignore-67-68.sh` — the audit-first propagation pattern
164168
this helper follows.
169+
170+
== Amendment (2026-07-21): age is advisory; defects are named
171+
172+
The window narrowed the treadmill but did not remove it. `standards` moves
173+
~2.6 commits/day, so a consumer exhausts both budgets roughly a fortnight
174+
after any propagation.
175+
176+
Measured on 2026-07-21: *294 of 350* consumers were red on the single pin
177+
`d7c22711` (2026-06-26) — every one of them green a fortnight earlier, with
178+
nothing changed in any consumer. `aspasia` is the clean natural experiment:
179+
same pin, 2026-07-09 run at 32 commits / 12d = passing notice; 2026-07-21 at
180+
63 commits / 24d = hard error. Keeping the fleet green under that rule means
181+
re-pinning ~300 repos every fortnight — of the order of 7,800 PRs/year.
182+
183+
A gate that fails everyone on a timer is not a guard. It trains the estate to
184+
ignore it and it buries the findings that matter: the same run reports 148
185+
real `Workflow security linter` failures and 74 anti-pattern failures, all of
186+
which read as noise once the fleet is uniformly red.
187+
188+
Two changes, both in `scripts/check-workflow-staleness.sh`:
189+
190+
. *Age outside the window is now a `::notice`, not an error.* The window still
191+
exists and is still reported, so `propagate-workflow-pins.sh` and the Hypatia
192+
`sha_bump_propagation` rule keep their signal.
193+
. *A named deny-list replaces age as the hard failure* (`KNOWN_BAD_BEFORE`).
194+
Each entry is `<reusable>:<fix-sha>`; a pin that is a strict ancestor of the
195+
fix carries that defect and is rejected at any age. This is what the window
196+
was only ever a proxy for, and it is strictly better in both directions — a
197+
*recent* pin carrying the defect is now caught, and an old pin carrying none
198+
is no longer punished by the calendar.
199+
200+
The first entry is `e9c8888769a7` (#441): before it, `hypatia-scan-reusable`
201+
and the `validate-hypatia-baseline` job cached the built Hypatia scanner under
202+
a keyless key while the build steps were guarded by `if [ ! -d ]`, so the first
203+
scanner build ever cached was reused forever and scanner fixes never took
204+
effect. A pin older than this reports a *false green* — which is why it must
205+
be refreshed and cannot be waited out. All 294 consumers on `d7c22711` predate
206+
it by one day, so they remain red, now for a true and actionable reason.
207+
208+
=== Integrity no longer rests on the runner's clone
209+
210+
The `UNKNOWN` verdict ("may be forged") was firing on legitimate pins.
211+
`hyperpolymath/awesome-haskell` pins `governance-reusable@5a93d9d5` and was
212+
accused on four consecutive runs across 17 days, while that commit verifies as
213+
a true ancestor of `main` locally in both treeless and `--depth 200` clones and
214+
via the compare API (`behind=0, ahead=90`). The mechanism was never reproduced
215+
off-runner.
216+
217+
A supply-chain accusation must not depend on the health of a clone the gate
218+
does not control. A hard `FORGED` verdict now requires either a *complete*
219+
local clone (not shallow, not partial) or confirmation from
220+
`GET /repos/{nwo}/compare/{pin}...{branch}`. Where neither is available the
221+
gate emits a `::warning` and passes: *cannot verify* is not *compromised*.
222+
Cost is zero API calls on the happy path — the server is consulted only when
223+
the gate is about to accuse. The deny-list gets the same fallback, so a
224+
degraded runner cannot silently skip it: "could not check" is reported as
225+
SKIPPED, never as passed.

0 commit comments

Comments
 (0)