Skip to content

feat: documentation quality rubric + validate-docs companion reference#17

Merged
dacervera merged 1 commit into
mainfrom
feat/documentation-rubric
Mar 12, 2026
Merged

feat: documentation quality rubric + validate-docs companion reference#17
dacervera merged 1 commit into
mainfrom
feat/documentation-rubric

Conversation

@dacervera

Copy link
Copy Markdown
Contributor

What

  • New builtin rubric: documentation.json (12 criteria across 6 categories: claim_discipline, consistency, boundary, accuracy, navigation, quality)
  • Bidirectional cross-reference between validate-docs.py and the documentation rubric via companion_tools metadata
  • Mechanical checks (validate-docs.py) complement semantic evaluation (rubric)

Why

PR #16 validation revealed no documentation rubric existed. The research-synthesis rubric produced >50% NOT_APPLICABLE on markdown docs. This rubric was designed specifically for project documentation: READMEs, specs, changelogs, tutorials, and guides.

Files Changed

  • reference-implementation/quorum/rubrics/builtin/documentation.json (new)
  • tools/validate-docs.py (companion docstring added)

Criteria Summary

Category IDs Severities
claim_discipline DOC-001, 002, 004, 012 1 CRITICAL, 2 HIGH, 1 MEDIUM
consistency DOC-003, 005 2 HIGH
boundary DOC-006 1 CRITICAL
accuracy DOC-007, 011 2 HIGH
navigation DOC-008, 009 2 MEDIUM
quality DOC-010 1 LOW

DOC-005 and DOC-008 are marked deterministic: true for Tester L1 (zero API cost).

Validation

  • validate-docs.py: clean ✅
  • Boundary scan: clean (only pre-existing quorum-runs paths) ✅

…rence

- New builtin rubric: documentation.json (12 criteria, 6 categories)
  Covers claim discipline, consistency, boundary, accuracy, navigation, quality
- Cross-reference between validate-docs.py and documentation rubric
  Mechanical checks (validate-docs) complement semantic evaluation (rubric)
- Rubric includes companion_tools metadata linking the two together
@dacervera

Copy link
Copy Markdown
Contributor Author

CI Triage — Quorum Self-Validation

Verdict: REVISE (10 findings on validate-docs.py)

All findings are pre-existing — this PR only adds 7 lines of docstring text. The file itself was not functionally changed.

2 HIGH — False Positives

Both flag read_file_lines() returning None without caller guards:

  • PC-002 (completeness): read_file_lines returns None but callers don't check
  • PC-004 (correctness): type safety issue with None return

Both call sites (lines 359 and 411) already have if lines is None: continue — the critics flagged the signature without tracing through to the guards. Verified: no unguarded call sites exist.

6 MEDIUM + 2 LOW — Pre-existing Design Debt

PC-005, PC-006, PC-007, PC-008, PC-025 (SRP, god function, duplication, hardcoded deps, mixed I/O), PC-012 (input validation), PC-017 (magic numbers), PC-009 (resource lifecycle). All pre-existing, same findings surfaced on PR #14 CI run. Routed to stale content backlog.

@dacervera
dacervera merged commit 9ca4d6e into main Mar 12, 2026
6 of 7 checks passed
@dacervera
dacervera deleted the feat/documentation-rubric branch March 12, 2026 23:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant