docs(audit): rate docs quality by page and section#154
Conversation
✅ Deploy Preview for tangle-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved PR — ba8771d4
Blanket team auto-approval is enabled for this reviewer service.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: blanket_auto_approve · 2026-06-15T14:42:55Z
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟠 Value Audit — better-approach-exists
| Verdict | better-approach-exists |
| Concerns | 6 (1 strong-concern, 2 medium-concern, 2 low, 1 weak-concern) |
| Heuristic | 1.6s |
| Duplication | 0.0s |
| Interrogation | 150.3s (2 bridge agents) |
| Total | 151.9s |
💰 Value — better-approach-exists
Adds a useful docs-quality scoring script and removes stale OpenClaw/NanoClaw harness references, but ships 56k+ lines of generated audit artifacts and hardcodes a personal dotfiles path to the slop scanner, so the architecture needs redirecting before merge.
- What it does: The change does three things: (1) introduces
scripts/audit-docs-quality.mjs, which walkspages/**/*.mdx, invokes the externaldocs-slop-auditscanner, and computes ten-dimension quality scores (voice, clarity, structure, actionability, completeness, sourceGrounding, boundaryClarity, freshness, examples, audienceFit) for every page and subsection, then writes JSON/CSV/Markdown reports under ` - Goals it achieves: The audit gives the docs team a repeatable, quantified baseline to find the lowest-rated pages/sections and track quality over time. The harness cleanup keeps public docs aligned with the actual published Sandbox SDK/UI surfaces and prevents the OpenCode copy gate from requiring references to non-public harnesses.
- Assessment: The audit script is coherent and the scoring rubric is well reasoned. The OpenClaw/NanoClaw removal is clearly worthwhile. However, the change fights the repo's grain by committing ~1.7 MB of machine-generated data that will be stale immediately, and it relies on a hardcoded absolute path to an external skill that will not exist for other contributors or CI. Those two issues are architectural, not
- Better / existing approach: I searched
scripts/,package.json,.github/workflows/, andCONTENT_AUDIT.md; there is no existing automated docs-quality scanner or equivalent audit artifact pipeline. The better approach is: (1) keep the scoring script but do NOT commit the raw JSON/CSVs—commit only the Markdown summary, or run the audit in CI and upload artifacts viaactions/upload-artifact; (2) vendor `scan-docs-slop.
🎯 Usefulness — sound-with-nits
The doc cleanup and harness-copy gate update are clearly used; the new audit script works but is only manually reachable because it is not wired into package.json/CI and hardcodes a local skill path.
- Integration: The harness-copy gate is already wired into package.json ("check:harness-copy": "node ./scripts/check-harness-copy.mjs") and CI (.github/workflows/checks.yml:111), and this PR keeps that working. The new audit script (scripts/audit-docs-quality.mjs) is not added to package.json scripts or to any workflow, and it defaults to an absolute path outside the repo (/home/drew/code/dotfiles/claude/skills/
- Fit with existing patterns: The script fits the existing scripts/ convention alongside check-harness-copy.mjs and check-tnt-core-sync.mjs. There is no existing docs-quality audit to duplicate. The scoring rubric is arbitrary by design and the markdown output follows the same report style as the existing audits/ directory.
- Real-world viability: The script runs end-to-end on this checkout and reproduces the published numbers. It degrades gracefully when the external scanner is missing (returns empty findings). It is a synchronous, single-process audit tool, so concurrency is not a concern. The main real-world fragility is the hardcoded external scanner path, which will break on machines/CI that do not have that exact skill checkout unless
🔎 Heuristic Signals
🟡 Cruft: console debug added scripts/audit-docs-quality.mjs
+console.log(
🟡 Cruft: todo added scripts/audit-docs-quality.mjs
+const futureWords = /\b(coming soon|planned|roadmap|will support|will allow|will add|todo|tbd)\b/gi;
💰 Value Audit
🔴 Committing 56k+ lines of generated audit artifacts bloats the repo and will stale quickly [better-architecture] ``
Evidence:
audits/2026-06-15-docs-quality.jsonis 52,488 lines / 1.4 MB,audits/2026-06-15-docs-quality-sections.csvis 2,410 lines / 307 KB (seegit diff --stat HEAD~1). These are regenerated outputs, not source docs. A better approach is to commit only the Markdown summary (or nothing at all) and produce the JSON/CSVs as CI artifacts viaactions/upload-artifact, or store snapshots in a separate data location. That keeps the docs repo small and guarantees the artifacts are fresh.
🟠 Audit script defaults to a hardcoded personal dotfiles path for the slop scanner [better-architecture] ``
Evidence:
scripts/audit-docs-quality.mjs:8-10defaultsscannerPathto/home/drew/code/dotfiles/claude/skills/docs-slop-audit/scripts/scan-docs-slop.mjs. It does allow override viaDOCS_SLOP_SCANNER, but the default will fail for every other contributor and in CI, and if the path is missing the script silently returns zero scanner findings (scripts/audit-docs-quality.mjs:187-189), producing misleading scores. Better: vendorscan-docs-slop.mjsintoscripts/or consume it as a real de
🟡 New audit script is not wired into package.json or CI workflows [better-architecture] ``
Evidence:
package.json:14-19only listscheck:harness-copy,check:tnt-core-sync, lint, and format;.github/workflows/checks.ymlhas aharness copyjob but no docs-quality job. If the audit is meant to be repeatable, it should have a script entry and a workflow job so it runs automatically on PRs and prevents drift.
🎯 Usefulness Audit
🟠 Audit script is not wired into package.json or CI and depends on an external skill [integration] ``
scripts/audit-docs-quality.mjs:8-10 hardcodes /home/drew/code/dotfiles/claude/skills/docs-slop-audit/scripts/scan-docs-slop.mjs and the script is not referenced in package.json or .github/workflows. Other scripts in the same directory (check-harness-copy.mjs, check-tnt-core-sync.mjs) are exposed as yarn scripts and run in CI. To make the 'repeatable' claim real for other contributors and CI, either add a package.json script (e.g. 'audit:docs-quality') and a workflow step, or vendor the scanner i
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved PR — 332765c6
Blanket team auto-approval is enabled for this reviewer service.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: blanket_auto_approve · 2026-06-15T14:50:42Z
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Concerns | 6 (1 medium-concern, 2 low, 3 weak-concern) |
| Heuristic | 1.7s |
| Duplication | 0.0s |
| Interrogation | 138.0s (2 bridge agents) |
| Total | 139.7s |
💰 Value — sound-with-nits
Adds a repeatable docs-quality audit that rates every page/section across ten dimensions and publishes JSON/CSV/MD snapshots, plus updates harness docs and the OpenCode copy gate to include OpenClaw/NanoClaw; the docs corrections and audit tooling are worthwhile, but the scanner default path is brit
- What it does: Introduces scripts/audit-docs-quality.mjs (824 lines) that walks pages/**/*.mdx, invokes an external docs-slop scanner, computes heuristic scores for voice, clarity, structure, actionability, completeness, sourceGrounding, boundaryClarity, freshness, examples, and audienceFit, and writes four artifact files (audits/2026-06-15-docs-quality.{json,md,pages.csv,sections.csv}) with 228 page and 2,409 s
- Goals it achieves: Makes docs quality measurable and repeatable so the team can prioritize rewrites using concrete page/section scores and scanner findings; keeps public harness documentation accurate by restoring OpenClaw/NanoClaw coverage; and keeps the OpenCode framing lint aligned with that expanded harness set so copy that names OpenCode without mentioning peers continues to fail CI.
- Assessment: Good on its merits. The harness doc changes are accurate and consistent with the restated copy gate (scripts/check-harness-copy.mjs:7-8 now includes OpenClaw|NanoClaw). The audit script ran cleanly (228 pages, 2409 sections, 22 high / 69 medium / 42 low findings) and produced the claimed artifacts. The scoring rubric is explicit about being heuristic, and the generated report includes a threats-to
- Better / existing approach: I searched scripts/, package.json, .gitignore, and git history for any existing docs-quality audit, slop scanner, or content-rating script and found none. The closest existing tooling is scripts/check-harness-copy.mjs, which the change correctly extends rather than duplicates. So no existing equivalent to reuse. The one design improvement would be to stop hardcoding a machine-specific absolute pat
🎯 Usefulness — sound-with-nits
Useful docs-quality audit and harness-copy alignment, but the audit script needs portable scanner resolution and CI/npm wiring to stay repeatable.
- Integration: The audit script runs manually and produced the claimed outputs (
AUDIT_DATE=2026-06-15 node scripts/audit-docs-quality.mjs→ 228 pages, 2409 sections, 133 findings). The harness-copy gate is integrated:scripts/check-harness-copy.mjs:8now countsOpenClaw|NanoClawas peer harness context, and the changed pages (e.g.pages/infrastructure/harnesses.mdx:14, `pages/blueprints/operator-matrix.m - Fit with existing patterns: Fits the codebase. The page edits extend the existing harness documentation and the established OpenCode-framing copy rule rather than competing with it. The audit script mirrors the other standalone
scripts/*.mjsutilities and writes artifacts underaudits/. No existing equivalent produces page/section quality ratings. - Real-world viability: Holds up for the current single-user, manual run. The script is synchronous and handles the corpus without concurrency issues. The real-world gap is portability/reproducibility: it defaults to an absolute scanner path outside the repo (
scripts/audit-docs-quality.mjs:8-10) and silently returns zero findings when that scanner is missing (scripts/audit-docs-quality.mjs:187-190), so other machines
🔎 Heuristic Signals
🟡 Cruft: console debug added scripts/audit-docs-quality.mjs
+console.log(
🟡 Cruft: todo added scripts/audit-docs-quality.mjs
+const futureWords = /\b(coming soon|planned|roadmap|will support|will allow|will add|todo|tbd)\b/gi;
🎯 Usefulness Audit
🟠 Audit scanner default is a non-portable absolute path and silently degrades [robustness] ``
scripts/audit-docs-quality.mjs:8-10 hard-codes the scanner to
/home/drew/code/dotfiles/claude/skills/docs-slop-audit/scripts/scan-docs-slop.mjs, and 187-190 returns empty findings if that file does not exist. This makes the headline 22-high/69-medium/42-low result non-repeatable on other machines or CI. The env overrideDOCS_SLOP_SCANNERis a workaround, but the default should either vendor the scanner, resolve relative to the repo, or fail loudly with setup instructions.
🟡 Audit script is not wired into package.json or CI [integration] ``
The repo wires
check-harness-copyandcheck-tnt-core-syncinto bothpackage.jsonscripts (lines 14-15) and.github/workflows/checks.yml(lines 34, 111).scripts/audit-docs-quality.mjshas neither. Add anaudit:docsnpm script and a CI step so the audit artifacts do not become a one-off snapshot.
💰 Value Audit
🟡 Scanner path hardcoded to a machine-specific skill outside the repo [against-grain] ``
scripts/audit-docs-quality.mjs:8-10 defaults scannerPath to /home/drew/code/dotfiles/claude/skills/docs-slop-audit/scripts/scan-docs-slop.mjs. Other repo scripts (e.g., scripts/check-tnt-core-sync.mjs:9-28) resolve paths relative to the repo or accept env vars with repo-relative fallbacks. The env var DOCS_SLOP_SCANNER provides an override, but the default is non-portable and will silently skip scanner findings on any other machine. Better: vendor scan-docs-slop.mjs into this repo under scripts/
🟡 Audit script is not registered in package.json [against-grain] ``
package.json:14-15 registers check:harness-copy and check:tnt-core-sync as npm scripts, but the new audit-docs-quality.mjs has no entry (e.g., check:docs-quality or audit:docs). Adding it would make the tool discoverable and consistent with the other repo check scripts.
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
✅ No Blockers —
|
|
Superseded by #155. The replacement PR carries the docs improvements on a clean branch and leaves the generated audit artifacts/script out of the active diff and PR history. |
2 participants
Summary
Headline ratings
Artifacts
Verification