Make AUDIT_PAT required, fail loudly when missing

nedtwigg · claude · nedtwigg · commit 0eb0fdab8104 · 2026-05-22T13:51:11.000-07:00
The audit's gh queries against repo administration endpoints
(/rulesets, /actions/secrets, /environments/*) require AUDIT_PAT.
Previously the workflow degraded gracefully when AUDIT_PAT was
absent, recording those checks as UNVERIFIABLE. That left half of
the FAIL IFs in SECURITY.md unenforceable, which defeats the
purpose of having the audit at all.

New pre-check step "Verify AUDIT_PAT is provisioned" runs before
Claude. If the secret is empty it writes a FAIL status, a
self-explanatory audit-report.md (so the issue body explains the
problem instead of pointing at logs), and exits non-zero. The
Surface step then files the issue as usual.

Claude's prompt now treats 403 from gh api as a real FAIL ("PAT's
scope has drifted") rather than UNVERIFIABLE — since AUDIT_PAT is
guaranteed present by the time Claude runs, 403 means the token's
permissions are wrong.

SECURITY.md: the AUDIT_PAT FAIL IF now asserts presence in the
env (not just placement); the audit-weakening FAIL IF covers
removing the pre-check; the CI Validation Contract prose calls
the token required rather than optional.

Co-Authored-By: Claude Opus 4.7 (1M context) &lt;noreply@anthropic.com&gt;
diff --git a/.github/workflows/security-audit.yaml b/.github/workflows/security-audit.yaml
@@ -45,16 +45,41 @@ jobs:
       - name: Install workspace dependencies
         run: pnpm install --frozen-lockfile
 
+      - name: Verify AUDIT_PAT is provisioned
+        env:
+          AUDIT_PAT: ${{ secrets.AUDIT_PAT }}
+        run: |
+          if [ -n "$AUDIT_PAT" ]; then
+            echo "AUDIT_PAT present."
+            exit 0
+          fi
+          echo "FAIL" > audit-status.txt
+          cat > audit-report.md <<'EOF'
+          ## Summary
+
+          **Overall status: FAIL.** `AUDIT_PAT` is not present in the
+          `security-audit` environment.
+
+          The audit refuses to run without `AUDIT_PAT`. It needs read-only
+          `Administration`, `Secrets`, and `Environments` access to
+          verify the ruleset, secret, and environment FAIL IFs.
+
+          Provision per `SECURITY.md` > CI Validation Contract: mint a
+          fine-grained PAT scoped to this repository only, then run
+          `gh secret set AUDIT_PAT --env security-audit --repo
+          diffplug/dormouse`.
+          EOF
+          echo "::error::AUDIT_PAT secret is not set in the security-audit environment."
+          exit 1
+
       - name: Audit against SECURITY.md
         uses: anthropics/claude-code-action@4481e6d3c7bbb88db2a928ca3444c536f589c7c1 # v1
         env:
-          # Prefer AUDIT_PAT (fine-grained, read-only Administration +
-          # Secrets + Environments) so checks against /rulesets,
-          # /actions/secrets, and /environments succeed; fall back to
-          # the default workflow token so the audit still runs (with
-          # UNVERIFIABLE results on those endpoints) if AUDIT_PAT is
-          # absent.
-          GH_TOKEN: ${{ secrets.AUDIT_PAT || github.token }}
+          # AUDIT_PAT is fine-grained, read-only Administration +
+          # Secrets + Environments. Required (the previous step
+          # refuses to proceed without it), so no fallback to
+          # github.token.
+          GH_TOKEN: ${{ secrets.AUDIT_PAT }}
         with:
           claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
           # Without an explicit allowlist the action defaults to a
@@ -83,17 +108,17 @@ jobs:
                  references secrets, for security holes the specs don't
                  cover.
 
-            `$GH_TOKEN` is either an `AUDIT_PAT` (fine-grained, read-only
-            Administration + Secrets + Environments) when the repo has
-            that secret configured, or a default `GITHUB_TOKEN` otherwise.
-            If `gh api` returns 403 / "Resource not accessible by
-            integration" on a check (typically `/rulesets`,
-            `/environments/*`, or `/actions/secrets`), the deployment is
-            running without `AUDIT_PAT`; record the check as
-            `UNVERIFIABLE` (not `FAIL`) with a note that minting
-            `AUDIT_PAT` (Administration + Secrets + Environments, all
-            read-only) would cover it. `UNVERIFIABLE` checks do not flip
-            the overall status to FAIL on their own.
+            `$GH_TOKEN` is the `AUDIT_PAT` — a fine-grained,
+            read-only PAT scoped to Administration + Secrets +
+            Environments. A previous workflow step has already verified
+            the token is present, so you can assume it is. If `gh api`
+            returns 403 / "Resource not accessible by integration" on
+            any check, record that check as `FAIL` with a note that
+            `AUDIT_PAT`'s scope has drifted from what SECURITY.md
+            specifies — do not treat it as `UNVERIFIABLE`. Reserve
+            `UNVERIFIABLE` for the rare case where an endpoint is
+            genuinely indeterminable (e.g. a transient network error),
+            and note explicitly why in the report.
 
             Produce a Markdown report with three sections:
               - `## FAIL IF results` — one line per check with PASS/FAIL
diff --git a/SECURITY.md b/SECURITY.md
@@ -47,7 +47,7 @@ This repository runs the [tend](https://github.com/max-sixty/tend) agent harness
 - FAIL IF `dormouse-bot` holds a permission higher than `push` on this repository.
 - FAIL IF `OVSX_PAT` or `VSCE_PAT` appears as a repo-level secret. They must live only in the `vscode-extension-publish` environment.
 - FAIL IF any GitHub environment's deployment-branch-policies admit a ref that is not admin-gated by the `Tag operations` or `Merge access` rulesets. Today this covers `vscode-extension-publish` (`v*` tag, admin-only via `Tag operations`) and `security-audit` (`main` admin-only via `Merge access`, plus `v*` tag).
-- FAIL IF `AUDIT_PAT` exists at the repo level. If the secret is provisioned at all, it must live in the `security-audit` environment so a bot-pushed feature branch cannot reach it.
+- FAIL IF `AUDIT_PAT` is missing from the `security-audit` environment, or is present at the repo level instead. The audit refuses to run without it, and it must be env-scoped so a bot-pushed feature branch cannot reach it.
 - FAIL IF `CHROMATIC_PROJECT_TOKEN` is missing from `secrets.allowed` in `.config/tend.yaml`. The allowlist entry is an explicit acknowledgment that the bot can read this token.
 - FAIL IF `.github/workflows/workflow-audit.yaml` is missing, disabled, or has not produced a successful run in the last 48 hours.
 - FAIL IF any `tend-*.yaml` workflow uses an unpinned action reference (e.g. `@main`, no version). Inside `tend-*.yaml`, both tag pins (`@v6`, `@0.0.25`) and SHA pins are accepted because the file is owned by the upstream generator (`max-sixty/tend`), which currently uses tag pins. All actions in every other workflow — including `workflow-audit.yaml` and `security-audit.yaml` — must follow the SHA-pin rule in "GitHub Actions Policies".
@@ -78,11 +78,11 @@ The audit job declares `environment: security-audit`, whose deployment-branch-po
 
 As a consequence of that env-gating, audit changes are iterated on `main` directly. A `workflow_dispatch` from any other ref is rejected by the environment's deployment-policy before any step runs. To experiment on a branch, widen the env's policy temporarily and revert after.
 
-`AUDIT_PAT` itself is optional. Without it, the audit's `gh` queries against `/rulesets`, `/actions/secrets`, and `/environments/*` return 403 and the affected checks are recorded as `UNVERIFIABLE` rather than `FAIL`. To upgrade them to `PASS`, mint a fine-grained PAT on an admin's account with read-only `Administration` + `Secrets` + `Environments` scoped to `diffplug/dormouse` only, then store it env-scoped:
+`AUDIT_PAT` is **required**. The audit's first step verifies the secret is present and refuses to run otherwise — without it the audit cannot read the administration endpoints needed to verify ruleset bypass actors, repo-level secret listing, and environment policies, so the spec it claims to enforce would be unenforceable in its key sections. Mint a fine-grained PAT on an admin's account with read-only `Administration` + `Secrets` + `Environments` scoped to `diffplug/dormouse` only, then store it env-scoped:
 
 ```bash
 gh secret set AUDIT_PAT --env security-audit --repo diffplug/dormouse --body 'github_pat_…'
 ```
 
 - FAIL IF `.github/workflows/security-audit.yaml` is missing, disabled, or no longer invoked from `release.yml`'s publish path.
-- FAIL IF the audit has been weakened — e.g. the prompt no longer requires the qualitative pass, a `FAIL IF` can be ignored, or the failure-reporting step that opens a `security-audit-failure` issue and exits non-zero has been removed.
+- FAIL IF the audit has been weakened — e.g. the prompt no longer requires the qualitative pass, a `FAIL IF` can be ignored, the failure-reporting step that opens a `security-audit-failure` issue and exits non-zero has been removed, or the `AUDIT_PAT` pre-check is removed or bypassed.
