ci: fix the last two main-red roots — Hypatia startup_failure (#451, this repo only) + scorecard-verify false-red (new)#514
Conversation
…false-red scorecard verify Follows #511/#512. Two independent causes, both structural, neither a symptom of the other. ## 1. Hypatia has never run: caller under-grants (standards#451) `hypatia-scan-reusable.yml` declares `security-events: write` (it uploads SARIF). `hypatia-scan.yml` granted `security-events: read`. A called workflow may never request more than its caller grants, so GitHub rejected the run at *startup* — `startup_failure`, zero jobs created, and `gh run view --log-failed` returning "log not found". Granting `read` did not degrade the scan to a no-upload mode. It stopped the scan from existing. The reusable carried a comment asserting that "callers granting only security-events:read skip gracefully" via `continue-on-error: true`. That is a false mental model and worth naming explicitly: no step can be skipped gracefully in a job that was never created. Comment corrected in place so the next reader does not re-derive the wrong conclusion. This is the #451 pattern that startup-fails reusable-workflow callers estate-wide. Fixed here on the canon's own caller first, deliberately, so the fix is demonstrated on one repo before any propagation. ## 2. Seven "fake passes" that were all real (standards#381) `registry-verify.yml` runs `build-scorecards.sh --verify`, which EXECUTES every pass-row's check. Those checks shell out to real tools — but the job installed none, and ubuntu-latest ships neither ripgrep nor xmllint: - k9-svc/M6 called `xmllint` -> exit 127 - 6 release-pre-flight checks invoke v1-audit.sh, which hard-exits 2 without ripgrep, so their greps matched nothing -> exit 1 The verifier reported all seven as "claimed PASS but the pass is not real". That was a false accusation. Measured after installing the tools: 87 grounded pass · 0 broken · 0 unrunnable (exit 0) Every one of the seven claims was true; the checks had simply never run. A false-red is not harmless — it trains maintainers to ignore a gate, the same end state as the fake-green gates #500 removed. Two changes, because installing the tools alone would leave the trap armed for the next missing dependency: - registry-verify.yml installs ripgrep + libxml2-utils. - `run_verify()` refuses to judge pass-rows at all in an incomplete environment, and reports genuinely unrunnable checks as `unrunnable` (still failing — an unverifiable pass must never go green) rather than as false claims. The required-tool list is declared, not inferred. An earlier draft parsed check strings to discover their commands; it misparsed quoted regexes ("a\|b\|c") and `[ $(…) -ge 25 ]`, inventing 35 failures — precisely the sin the guard exists to prevent. Measured instead: xmllint is named by 4 checks, jq by 1, and rg by none but is needed transitively via v1-audit.sh. ## Verification - complete env -> 87 grounded, 0 broken, 0 unrunnable, exit 0 - incomplete env-> fails fast, judges nothing, names the missing tool - k9-svc XML/plist independently confirmed well-formed, so installing xmllint turns that row genuinely green rather than exposing a real fault - all three workflow files parse Deliberately NOT included: `.gitleaks.toml` — covered by #512, which is the better implementation (see PR body). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…381 The previous commit cited standards#381 for the seven tooling-induced "fake passes". That is wrong, and worth correcting in the tree rather than only in review. #381 is REGISTRY.a2ml source_hash drift in `build-registry.sh --check` — the FIRST step of registry-verify.yml, asking for a regen step or pre-commit hook. The fault fixed here is in the THIRD step, `build-scorecards.sh --verify`: a different script, a different mechanism, and nothing in this branch adds a regen step or hook. The scorecard verify false-red was newly diagnosed here and has no pre-existing issue. #381 remains untouched and open. Also confirmed while checking this: `just scorecards-verify` (Justfile:82) is the only other caller of `--verify`, so the new preflight cannot regress a toolless path — `scorecards-check` and `scorecards-check-strict` do not pass `--verify`, and the preflight lives inside `run_verify()`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hyperpolymath
changed the title
Correction + scope caveats (please read before merging)1. The scorecard false-red is NOT #381 — I mis-cited it. The original commit message and PR body tagged standards#381 for the seven tooling-induced "fake passes". Wrong issue. #381 is #381 is untouched and stays open. The scorecard-verify false-red was newly diagnosed here and had no pre-existing issue. Corrected in-tree in bb88cc5 (source comments) and in the title above; I did not rewrite the first commit's message rather than force-push a pushed branch. 2. #451 is fixed for this repo's caller only. The issue is explicitly estate-wide (~300 wrappers). I fixed the canon's own caller and demonstrated the pattern end-to-end — deliberately one repo, so the mechanism is proven before propagation. Do not close #451 on this PR. Estate propagation touches 3. 4. Turning Hypatia on surfaced 54 criticals. The scan ran for the first time (1m56s, SARIF uploaded) and annotated:
That is a consequence of the fix, not a footnote: findings that existed all along were invisible because the workflow never started. It is advisory-only, so it does not block this PR — but it wants triage against #399 (104 findings vs empty baseline) and #178 (Track C human triage), and "does not gate" is itself worth a decision now that the scan actually runs. Verified after the correction: |
|
1 participant
Completes the run to green on
main, after #511 (secret-scanner caller + SPDX) and #512 (gitleaks allowlist). Two remaining causes, independent of each other and of those PRs.1. Hypatia has never run — the caller under-grants (#451)
hypatia-scan-reusable.ymldeclaressecurity-events: write(it uploads SARIF).hypatia-scan.ymlgrantedsecurity-events: **read**.A called workflow may never request more than its caller grants. When it does, GitHub rejects the run at startup — before any job exists:
Granting
readdid not degrade the scan to a no-upload mode. It stopped the scan from existing.The reusable carried this comment:
That is a false mental model, and it is load-bearing — it is the reason the mismatch looked intentional. Nothing skips gracefully in a job that was never created. Corrected in place.
This is exactly the #451 pattern that startup-fails reusable-workflow callers estate-wide. Fixed here on the canon's own caller only — deliberately demonstrated on one repo before any propagation is proposed.
2. Seven "fake passes" that were every one of them real (#381)
registry-verify.ymlrunsbuild-scorecards.sh --verify, which executes every pass-row's check. The job installed no tools, andubuntu-latestships neither ripgrep nor xmllint:k9-svc/M6xmllintdirectlyrelease-pre-flight/*v1-audit.sh, which hard-exits 2 without ripgrep, so the greps match nothingThe verifier reported all seven as "claimed PASS but the pass is not real". Reproduced locally by stripping
rgfromPATH— identical output.Measured with the tools present:
All seven claims were true. The checks had never run. A false-red is not harmless: it trains maintainers to ignore a gate — the same end state as the fake-green gates #500 removed, reached from the opposite direction.
Two changes, since installing the tools alone leaves the trap armed for the next missing dependency:
registry-verify.ymlinstallsripgrep+libxml2-utils.run_verify()refuses to judge pass-rows at all in an incomplete environment, and reports genuinely unrunnable checks asunrunnable— still failing, since an unverifiable pass must never go green — rather than as false claims.The required-tool list is declared, not inferred
An earlier draft parsed check strings to discover their commands. It misparsed quoted regexes (
"a\|b\|c") and[ $(…) -ge 25 ], inventing 35 failures — precisely the sin the guard exists to prevent. Replaced with a measured, documented list:xmllintis named by 4 checks,jqby 1, andrgby none but is required transitively viav1-audit.sh.Verification
87 grounded · 0 broken · 0 unrunnable, exit 0k9-svc/mime/k9.xml+k9.uti.plistindependently confirmed well-formed and the two companion files present — so installing xmllint turns that row genuinely green rather than exposing a real faultNot included, on purpose
.gitleaks.toml— #512 covers it and is the better implementation. I built one independently and then discarded it: my regexes matched against the whole line unanchored, and canary-testing showed that suppressed real tokens.ghp_…PATEXAMPLE_KEY# ...#512's anchored
^…$design has no such hole. Verified independently against gitleaks 8.18.4 (the exact CI build, checksum-matched): all four adversarial fixtures still caught, and0findings on the tree. Adopt #512.🤖 Generated with Claude Code