You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"description": "Evaluates project documentation for claim discipline, internal consistency, navigation coherence, and implementation honesty. Designed for READMEs, specs, changelogs, tutorials, and user-facing guides. Complements mechanical checks (validate-docs.py) with semantic and narrative evaluation.",
6
+
"companion_tools": [
7
+
{
8
+
"tool": "tools/validate-docs.py",
9
+
"relationship": "complementary",
10
+
"note": "validate-docs.py handles deterministic mechanical checks: version consistency across files, critic counts vs critic-status.yaml, DEC-019 denominator fractions, and file existence. This rubric handles semantic evaluation: claim accuracy, narrative coherence, navigation structure, and boundary discipline. Run validate-docs.py BEFORE this rubric — its output can serve as evidence for DOC-002, DOC-003, and DOC-005."
11
+
}
12
+
],
13
+
"criteria": [
14
+
{
15
+
"id": "DOC-001",
16
+
"criterion": "Every capability claim matches the actual shipped state — no features described as working that are specified-only or planned",
17
+
"severity": "CRITICAL",
18
+
"evidence_required": "Quote the claim, then show evidence it is not implemented (e.g., 'learning memory' described as active when code shows it is not wired up)",
19
+
"why": "Overclaiming is the #1 credibility killer for open source projects. One false claim undermines all true ones.",
20
+
"category": "claim_discipline",
21
+
"deterministic": false
22
+
},
23
+
{
24
+
"id": "DOC-002",
25
+
"criterion": "Numeric claims are accurate — critic counts, test counts, check counts, coverage percentages match reality",
26
+
"severity": "HIGH",
27
+
"evidence_required": "Quote the numeric claim and show the actual count from source code, test output, or canonical status files (e.g., critic-status.yaml) that contradicts it",
28
+
"why": "Stale numbers (e.g., '10 checks' when 12 exist, '9 critics' when 6 are shipped) signal unmaintained docs.",
29
+
"category": "claim_discipline",
30
+
"deterministic": false
31
+
},
32
+
{
33
+
"id": "DOC-003",
34
+
"criterion": "Version numbers are consistent across all documents — no file says v0.3.0 while another says v0.5.1",
35
+
"severity": "HIGH",
36
+
"evidence_required": "Quote the version references from two or more files that disagree",
37
+
"why": "Inconsistent versions confuse users about what release they are reading about.",
38
+
"category": "consistency",
39
+
"deterministic": false
40
+
},
41
+
{
42
+
"id": "DOC-004",
43
+
"criterion": "Implementation status markers are present and accurate for all major components — shipped features are not marked as planned, and vice versa",
44
+
"severity": "HIGH",
45
+
"evidence_required": "Quote a component described without status markers, or with incorrect markers (e.g., marked 'not yet built' but present in codebase, or marked 'shipped' but not callable)",
46
+
"why": "Users and contributors need to know what works today vs what is planned. Stale status markers are a special case of overclaiming.",
47
+
"category": "claim_discipline",
48
+
"deterministic": false
49
+
},
50
+
{
51
+
"id": "DOC-005",
52
+
"criterion": "All internal cross-references resolve — links to other docs, sections, or files point to targets that exist at the referenced path",
53
+
"severity": "HIGH",
54
+
"evidence_required": "Quote the link or reference and show the target does not exist, has been renamed, or has moved to a different path",
55
+
"why": "Broken links are the most common defect after file restructuring. Every broken link is a dead end for a user trying to learn the system.",
56
+
"category": "consistency",
57
+
"deterministic": true
58
+
},
59
+
{
60
+
"id": "DOC-006",
61
+
"criterion": "No internal or proprietary references leak into public-facing docs — no workspace paths, internal project names, private repo references, or team member names",
62
+
"severity": "CRITICAL",
63
+
"evidence_required": "Quote the text containing the internal reference (e.g., absolute paths, internal tool names, private repo URLs, team names)",
64
+
"why": "Leaking internal structure undermines professional credibility and may expose sensitive organizational information.",
65
+
"category": "boundary",
66
+
"deterministic": false
67
+
},
68
+
{
69
+
"id": "DOC-007",
70
+
"criterion": "Examples and code snippets are runnable — CLI commands use correct syntax, import paths match the actual package structure, output samples reflect current tool behavior",
71
+
"severity": "HIGH",
72
+
"evidence_required": "Quote the example and identify the specific error (wrong flag name, nonexistent module, incorrect syntax, outdated output format)",
73
+
"why": "Broken examples in docs are the fastest way to lose a new user during onboarding.",
74
+
"category": "accuracy",
75
+
"deterministic": false
76
+
},
77
+
{
78
+
"id": "DOC-008",
79
+
"criterion": "Navigation hubs (README, docs index, table of contents) link to all documentation files in their scope — no orphaned docs that are unreachable from any hub",
80
+
"severity": "MEDIUM",
81
+
"evidence_required": "List the documentation files that exist on disk but are not linked from any navigation hub or table of contents",
82
+
"why": "Orphaned docs are invisible docs. If a user can't discover a file through navigation, it effectively doesn't exist.",
83
+
"category": "navigation",
84
+
"deterministic": true
85
+
},
86
+
{
87
+
"id": "DOC-009",
88
+
"criterion": "Onboarding progression is coherent — entry points (README, QUICK_START) lead logically to deeper docs without circular references or missing steps",
89
+
"severity": "MEDIUM",
90
+
"evidence_required": "Trace the onboarding path from README and identify where the reader hits a dead end, a circular loop, or a prerequisite that was never introduced",
91
+
"why": "Documentation is a directed graph. If the happy path has gaps, new users fall off at the first missing link.",
92
+
"category": "navigation",
93
+
"deterministic": false
94
+
},
95
+
{
96
+
"id": "DOC-010",
97
+
"criterion": "Tone and voice are consistent within each document — no unexplained shifts between casual first-person and formal third-person",
98
+
"severity": "LOW",
99
+
"evidence_required": "Quote two passages from the same document that use conflicting voice or register",
100
+
"why": "Inconsistent voice makes a project feel unpolished and confuses the reader about who is speaking.",
101
+
"category": "quality",
102
+
"deterministic": false
103
+
},
104
+
{
105
+
"id": "DOC-011",
106
+
"criterion": "Changelogs and release notes accurately reflect what was actually changed — no omitted breaking changes, no listed changes that weren't included in the release",
107
+
"severity": "HIGH",
108
+
"evidence_required": "Compare the changelog entry against the actual git diff or release contents and identify discrepancies (missing entries, phantom entries, misattributed versions)",
109
+
"why": "Changelogs are the trust contract between maintainers and users. Inaccurate changelogs erode update confidence.",
110
+
"category": "accuracy",
111
+
"deterministic": false
112
+
},
113
+
{
114
+
"id": "DOC-012",
115
+
"criterion": "No stale 'coming soon', 'TODO', 'not yet implemented', or placeholder text remains for features that have since shipped",
116
+
"severity": "MEDIUM",
117
+
"evidence_required": "Quote the placeholder text and show that the referenced feature is now implemented in the codebase",
118
+
"why": "Stale placeholders make shipped features look unfinished and confuse contributors about what work remains.",
0 commit comments