fix(tooling): derive the docs-audit scope from the filesystem instead of a hand-kept list (#4851) (#4921)

`.claude/workflows/docs-accuracy-audit.js` carried its default audit scope inline,
as a hand-kept `ALL_HANDWRITTEN` array behind a "keep in sync with
`affected-docs.mjs --all`" comment. Nothing checked that promise, and it had
rotted in BOTH directions:

- 16 listed paths no longer existed — 10 of them the whole
  `content/docs/protocol/objectos/**` directory, renamed to `protocol/kernel/`.
  An audit agent pointed at a non-existent file reads nothing and reports
  `fixCount: 0`, indistinguishable in the run summary from a doc that was
  checked and found accurate. That is how #4781 and #4817 sat in
  `protocol/kernel/` for ~2 months under green "full" audits.
- 48 existing docs were absent from the list — including all 9 of
  `protocol/kernel/**` and the whole `content/docs/capabilities/` directory. A
  run logging `FULL audit` was auditing 130 of 178 hand-written docs.

The list stays inline because it must: a workflow script runs in a `node:vm`
context with no require/import/filesystem, so it can neither walk
`content/docs/` nor read a JSON artifact. So it is GENERATED instead —
`scripts/docs-audit/check-audit-scope.mjs --write` derives it from
`affected-docs.mjs --all` (one definition of "hand-written doc"), and
`pnpm check:docs-audit-scope` fails in lint.yml when the block and
`content/docs/` disagree in either direction, naming every entry. Missing
markers or an unparseable block fail too — a gate that cannot find its subject
must go red, not green.

A CI gate only sees the default list, so two more nets cover the rest:
- the workflow preflights its resolved scope (including a caller-supplied
  `args.docs`) and refuses to start, naming every path that does not exist; the
  preflight's own arithmetic is reconciled against the scope, so a verdict that
  cannot account for every path exactly once is a failed preflight, not a pass;
- every audit agent reports `docExists` from the path that actually opens the
  file, and the run throws if any comes back false — a self-check that runs
  somewhere other than the real read path proves nothing about it (#4868).

Same discipline as #4690 / #4777 / #4804 / #4835 / #4868 / #4890.


Claude-Session: https://claude.ai/code/session_018iARDqtrhQgz6fVHDeDkbQ

Co-authored-by: Claude <noreply@anthropic.com>

Author: xuyushun441-sys

.changeset/docs-audit-scope-derived.md

@@ -0,0 +1,47 @@
+---
+---
+
+tooling: derive the docs-accuracy audit's scope from the filesystem and fail loudly when
+it drifts (#4851)
+
+Release-nothing: touches `.claude/workflows/docs-accuracy-audit.js`,
+`scripts/docs-audit/`, the root `check:docs-audit-scope` script and one `lint.yml` step —
+no package code.
+
+`.claude/workflows/docs-accuracy-audit.js` carried its default audit scope inline, as a
+hand-kept `ALL_HANDWRITTEN` array behind a "keep in sync with `affected-docs.mjs --all`"
+comment. Nothing checked that promise, and it had rotted in **both** directions:
+
+- **16 listed paths no longer existed** — 10 of them the whole
+  `content/docs/protocol/objectos/**` directory, renamed to `protocol/kernel/`. A doc
+  path that resolves to nothing produces an audit agent that reads nothing and reports
+  `fixCount: 0`, which in the run summary is indistinguishable from a doc that was
+  checked and found accurate. That is how the accuracy defects in #4781
+  (`runtime-capabilities.mdx` documenting a schema deleted in #3605) and #4817
+  (`http-protocol.mdx` attributing the dispatcher's response shape to
+  `/api/v1/discovery`) survived ~2 months of green "full" audits.
+- **48 existing docs were absent from it** — including all 9 of `protocol/kernel/**` and
+  the entire `content/docs/capabilities/` directory. A run logging
+  `FULL audit (no args.docs given)` was auditing 130 of 178 hand-written docs.
+
+The list stays inline because it must: a workflow script runs in a `node:vm` context
+with no `require`, no `import` and no filesystem, so it can neither walk `content/docs/`
+nor read a JSON artifact. So it is now **generated** rather than hand-kept —
+`node scripts/docs-audit/check-audit-scope.mjs --write` derives it from
+`affected-docs.mjs --all` (one definition of "hand-written doc", not a second walk to
+drift), and `pnpm check:docs-audit-scope` fails in `lint.yml` when the block and
+`content/docs/` disagree in either direction, naming every entry.
+
+Two more nets, because a CI gate can only see the *default* list:
+
+- the workflow **preflights its resolved scope** — including a caller-supplied
+  `args.docs` — and refuses to start, naming every path that does not exist. Its
+  arithmetic is reconciled against the scope, so a preflight that cannot account for
+  every path exactly once is a failed preflight, not a pass;
+- every audit agent now reports `docExists` from the path that actually opens the file,
+  and the run **throws** if any comes back false. The preflight is not the real read
+  path, and a self-check that runs somewhere other than the real path proves nothing
+  about it (#4868).
+
+Same discipline as #4690 / #4777 / #4804 / #4835 / #4868 / #4890: a check whose subject
+has gone missing must go red, never green-by-vacancy.

.claude/workflows/docs-accuracy-audit.js

@@ -94,6 +94,20 @@ jobs:
       - name: Doc/skill authoring guard
         run: pnpm check:doc-authoring
 
+      # #4851: the docs-accuracy-audit workflow carries its default scope inline
+      # (a workflow script runs in a vm with no filesystem, so it cannot enumerate
+      # content/docs/ itself). Hand-kept, that list rotted in BOTH directions —
+      # 16 entries pointing at files that no longer existed after the
+      # protocol/objectos → protocol/kernel rename, and 48 existing docs missing
+      # from it — while every "FULL audit" run reported green, which is how the
+      # accuracy defects in #4781 and #4817 survived ~2 months. This regenerates
+      # the list from the filesystem and fails when the two disagree either way.
+      # It lives in this job deliberately: the change that breaks the list is a
+      # docs rename, so a `packages/**` paths filter would blind the gate to
+      # exactly its own failure mode.
+      - name: Docs-audit scope is derived, not hand-kept
+        run: pnpm check:docs-audit-scope
+
       # ADR-0090 D3 vocabulary ratchet: "role" is reserved-forbidden in docs
       # and skills. Existing occurrences are frozen in the baseline (better-auth
       # boundary, ARIA samples, educational mentions); NEW occurrences fail.

.github/workflows/lint.yml

@@ -34,6 +34,7 @@
     "check:i18n-coverage": "node scripts/check-i18n-coverage.mjs",
     "check:nul-bytes": "node scripts/check-nul-bytes.mjs --self-test && node scripts/check-nul-bytes.mjs",
     "check:doc-authoring": "node scripts/check-doc-authoring.mjs --self-test && node scripts/check-doc-authoring.mjs",
+    "check:docs-audit-scope": "node scripts/docs-audit/affected-docs.mjs --self-test && node scripts/docs-audit/check-audit-scope.mjs --self-test && node scripts/docs-audit/check-audit-scope.mjs",
     "check:role-word": "node scripts/check-role-word.mjs",
     "check:adr-anchors": "node scripts/check-adr-anchors.mjs",
     "check:org-identifier": "node scripts/check-org-identifier.mjs",

package.json

@@ -81,6 +81,44 @@ looks like the obvious next step and is a provable no-op, for two independent re
 A hand-edited CHANGELOG outside a release is also close to nonexistent in practice. Left
 counted, and recorded here so the idea is not rediscovered as a gap.
 
+## 1b. `check-audit-scope.mjs` — the audit workflow's scope, derived not hand-kept
+
+```bash
+node scripts/docs-audit/check-audit-scope.mjs           # verify (also: pnpm check:docs-audit-scope)
+node scripts/docs-audit/check-audit-scope.mjs --write   # regenerate the list from the filesystem
+node scripts/docs-audit/check-audit-scope.mjs --self-test
+```
+
+The `docs-accuracy-audit` workflow (part 3) carries its default scope **inline**, as
+`ALL_HANDWRITTEN`. It has to: a workflow script runs inside a `node:vm` context whose
+only globals are `log`/`phase`/`console`/`budget`/timers plus
+`agent`/`parallel`/`pipeline`/`workflow`/`args`, with code generation disabled — no
+`require`, no `import`, no filesystem. It can neither walk `content/docs/` nor read a
+JSON artifact, so the list cannot be derived *at run time*.
+
+It is therefore derived at *generation* time instead: `--write` rewrites the block from
+`affected-docs.mjs --all` (one definition of "hand-written doc", not two), and the plain
+run is a CI gate in `lint.yml` that fails when the block and `content/docs/` disagree
+**in either direction**.
+
+Both directions matter, and only one had ever been noticed (#4851):
+
+- **listed but missing** — the 10 `content/docs/protocol/objectos/**` paths left behind
+  by the rename to `protocol/kernel/`, plus 6 others. An audit agent pointed at a
+  non-existent file reads nothing and reports `fixCount: 0`, which in the run summary is
+  indistinguishable from a doc that was checked and found accurate. That is how the
+  accuracy defects in #4781 and #4817 sat in `protocol/kernel/` for ~2 months while full
+  audits reported green.
+- **exists but unlisted** — 48 docs, including all of `protocol/kernel/**` and the whole
+  `capabilities/` directory. A run logging `FULL audit (no args.docs given)` was
+  auditing 130 of 178 docs.
+
+The workflow additionally preflights its resolved scope — including a caller-supplied
+`args.docs`, which no CI gate can see — and aborts naming any path that does not exist;
+and each audit agent reports `docExists` from the read path itself, so a preflight that
+was wrong cannot be laundered into a green summary. The gate covers the default list,
+the preflight covers the caller's list, and the read path checks both.
+
 ## 2. CI gate — `.github/workflows/docs-drift-check.yml`
 
 On any PR that touches `packages/**`, runs `affected-docs.mjs` against the base branch
@@ -119,6 +157,8 @@ opens a PR when there are fixes. See the routine prompt for the exact steps.
 
 ---
 
-**Cost note:** a full audit of all 128 hand-written docs is ~2.8M output tokens / ~160
-agents. Always prefer the change-scoped list (`affected-docs.mjs`) over `--all` except for
-the periodic full backstop.
+**Cost note:** a full audit is ~2 agents per doc — measured at ~2.8M output tokens /
+~160 agents when the scope was 128 docs, and the hand-written set is 178 today (run
+`check-audit-scope.mjs` for the current number; don't trust a count written down here).
+Always prefer the change-scoped list (`affected-docs.mjs`) over `--all` except for the
+periodic full backstop.

scripts/docs-audit/README.md

@@ -1,7 +1,7 @@
 #!/usr/bin/env node
 // Map a set of `packages/**` code changes to the hand-written docs that reference
 // the affected packages, so a doc-accuracy audit can be scoped to what actually
-// changed instead of re-auditing all 128 hand-written docs every time.
+// changed instead of re-auditing every hand-written doc (178 of them today) each time.
 //
 // Usage:
 //   node scripts/docs-audit/affected-docs.mjs [sinceRef]   # docs affected by changes since <sinceRef> (default origin/main)