Skip to content

Commit 700a3c5

Browse files
hyperpolymathclaude
andcommitted
fix(ci): staleness gate fails on named defects, not on the calendar
MEASURED 2026-07-21: 294 of 350 consumers were red on the single pin d7c2271 (2026-06-26) — every one of them green a fortnight earlier, with nothing changed in any consumer. aspasia is the clean natural experiment: same pin, 2026-07-09 run at 32 commits / 12d = passing notice; 2026-07-21 at 63 commits / 24d = hard error. standards moves ~2.6 commits/day, so a consumer exhausts both window budgets ~14 days after any propagation. Holding the fleet green under that rule means re-pinning ~300 repos every fortnight (order 7,800 PRs/year). A gate that fails everyone on a timer is not a guard: it trains the estate to ignore it, and it buries the findings that matter — the same sweep shows 148 real Workflow security linter failures and 74 anti-pattern failures that read as noise once the fleet is uniformly red. 1. Age outside the window is now a ::notice, not an error. The window is still computed and still reported, so propagate-workflow-pins.sh and the Hypatia sha_bump_propagation rule keep their signal. 2. A named deny-list (KNOWN_BAD_BEFORE) replaces age as the hard failure. Each entry is <reusable>:<fix-sha>; a pin that is a strict ancestor of the fix carries that defect and is rejected at any age. This is what the window was only ever a proxy for, and it is strictly better in both directions: a RECENT pin carrying the defect is now caught, and an old pin carrying none is no longer punished by the calendar. First entry is e9c8888 (#441). Before it, hypatia-scan-reusable and the validate-hypatia-baseline job cached the built Hypatia scanner under a keyless key while the build steps were guarded by `if [ ! -d ]`, so the first scanner build ever cached was reused forever and scanner fixes never took effect. A pin older than this reports a FALSE GREEN. All 294 consumers on d7c2271 predate it by one day and stay red — now for a true and actionable reason. 3. Integrity no longer rests on the runner's clone. The UNKNOWN / "may be forged" verdict was firing on legitimate pins: awesome-haskell pins governance-reusable@5a93d9d5 and was accused on four consecutive runs over 17 days, while that commit verifies as a true ancestor of main locally in both treeless and --depth 200 clones and via the compare API (behind=0, ahead=90). The mechanism was never reproduced off-runner. A hard FORGED verdict now requires either a COMPLETE local clone (not shallow, not partial) or confirmation from GET /repos/{nwo}/compare/{pin}...{branch}. Where neither is available the gate warns and passes: "cannot verify" is not "compromised". Zero API calls on the happy path — the server is consulted only when about to accuse. The deny-list gets the same fallback, so a degraded runner cannot silently skip it; "could not check" is reported as SKIPPED, never as passed. Tests: the hermetic fixture suite goes 12 -> 16 cases. Out-of-window now asserts exit 0 AND the advisory notice (a new run_case_out helper — an exit code alone cannot distinguish "passed silently" from "passed with the notice the propagation path depends on"). New coverage: deny-list rejects at any age, rejects even when in-window, does not reject the fixing commit itself, and is scoped per reusable. 16/16 pass, and the suite stays hermetic because a complete fixture clone resolves the forged case without the network. Also verified against live standards history and in degraded-clone mode; both give identical verdicts for fresh / denied / old-but-clean / forged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent f9dca6d commit 700a3c5

3 files changed

Lines changed: 366 additions & 20 deletions

File tree

docs/decisions/ADR-003-workflow-pin-staleness-window.adoc

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,10 @@ minimal data) instead of `--depth 1`, with a `--depth 200` fallback.
145145
`scripts/tests/check-workflow-staleness-test.sh` (hermetic fixture history)
146146
covers fresh / ahead / in-window-by-commits / in-window-by-days /
147147
out-of-window / forged, plus the retired-enforcer and SARIF structural rules.
148+
Extended by the 2026-07-21 amendment below to 16 cases: out-of-window now
149+
asserts exit 0 *and* the advisory notice, plus deny-list coverage (rejected at
150+
any age, rejected even when in-window, the fixing commit itself not rejected,
151+
and entries scoped per reusable).
148152
`scripts/tests/propagate-workflow-pins-test.sh` covers audit/fix/idempotence
149153
and branch staging. End-to-end against real standards history, a
150154
stapeln-shaped consumer pinned 3 commits behind HEAD passes the relaxed gate
@@ -162,3 +166,60 @@ stapeln-shaped consumer pinned 3 commits behind HEAD passes the relaxed gate
162166
rules out mutable-tag pinning (option C).
163167
* `scripts/propagate-gitignore-67-68.sh` — the audit-first propagation pattern
164168
this helper follows.
169+
170+
== Amendment (2026-07-21): age is advisory; defects are named
171+
172+
The window narrowed the treadmill but did not remove it. `standards` moves
173+
~2.6 commits/day, so a consumer exhausts both budgets roughly a fortnight
174+
after any propagation.
175+
176+
Measured on 2026-07-21: *294 of 350* consumers were red on the single pin
177+
`d7c22711` (2026-06-26) — every one of them green a fortnight earlier, with
178+
nothing changed in any consumer. `aspasia` is the clean natural experiment:
179+
same pin, 2026-07-09 run at 32 commits / 12d = passing notice; 2026-07-21 at
180+
63 commits / 24d = hard error. Keeping the fleet green under that rule means
181+
re-pinning ~300 repos every fortnight — of the order of 7,800 PRs/year.
182+
183+
A gate that fails everyone on a timer is not a guard. It trains the estate to
184+
ignore it and it buries the findings that matter: the same run reports 148
185+
real `Workflow security linter` failures and 74 anti-pattern failures, all of
186+
which read as noise once the fleet is uniformly red.
187+
188+
Two changes, both in `scripts/check-workflow-staleness.sh`:
189+
190+
. *Age outside the window is now a `::notice`, not an error.* The window still
191+
exists and is still reported, so `propagate-workflow-pins.sh` and the Hypatia
192+
`sha_bump_propagation` rule keep their signal.
193+
. *A named deny-list replaces age as the hard failure* (`KNOWN_BAD_BEFORE`).
194+
Each entry is `<reusable>:<fix-sha>`; a pin that is a strict ancestor of the
195+
fix carries that defect and is rejected at any age. This is what the window
196+
was only ever a proxy for, and it is strictly better in both directions — a
197+
*recent* pin carrying the defect is now caught, and an old pin carrying none
198+
is no longer punished by the calendar.
199+
200+
The first entry is `e9c8888769a7` (#441): before it, `hypatia-scan-reusable`
201+
and the `validate-hypatia-baseline` job cached the built Hypatia scanner under
202+
a keyless key while the build steps were guarded by `if [ ! -d ]`, so the first
203+
scanner build ever cached was reused forever and scanner fixes never took
204+
effect. A pin older than this reports a *false green* — which is why it must
205+
be refreshed and cannot be waited out. All 294 consumers on `d7c22711` predate
206+
it by one day, so they remain red, now for a true and actionable reason.
207+
208+
=== Integrity no longer rests on the runner's clone
209+
210+
The `UNKNOWN` verdict ("may be forged") was firing on legitimate pins.
211+
`hyperpolymath/awesome-haskell` pins `governance-reusable@5a93d9d5` and was
212+
accused on four consecutive runs across 17 days, while that commit verifies as
213+
a true ancestor of `main` locally in both treeless and `--depth 200` clones and
214+
via the compare API (`behind=0, ahead=90`). The mechanism was never reproduced
215+
off-runner.
216+
217+
A supply-chain accusation must not depend on the health of a clone the gate
218+
does not control. A hard `FORGED` verdict now requires either a *complete*
219+
local clone (not shallow, not partial) or confirmation from
220+
`GET /repos/{nwo}/compare/{pin}...{branch}`. Where neither is available the
221+
gate emits a `::warning` and passes: *cannot verify* is not *compromised*.
222+
Cost is zero API calls on the happy path — the server is consulted only when
223+
the gate is about to accuse. The deny-list gets the same fallback, so a
224+
degraded runner cannot silently skip it: "could not check" is reported as
225+
SKIPPED, never as passed.

0 commit comments

Comments
 (0)