Skip to content

docs(agents): require an evidence class on every review finding - #289

Merged
cnighswonger merged 1 commit into
mainfrom
docs/evidence-class-rule
Jul 31, 2026
Merged

docs(agents): require an evidence class on every review finding#289
cnighswonger merged 1 commit into
mainfrom
docs/evidence-class-rule

Conversation

@vsits-proxy-builder

Copy link
Copy Markdown
Contributor

Problem

The global reviewer baseline (~/.codex/AGENTS.md:155) says to distinguish what is confirmed correct from what is assumed or hypothesized. That's a disposition, and a disposition is self-assessed — a reviewer who read a code path and found it plausible will honestly file it under "confirmed." Nothing about the output tells the next reader which it was.

Same shape as the bloat gap we closed in #287: "None" isn't checkable, a number is. "Confirmed" isn't checkable, a named command is.

The motivating case

On #270 the reviewer endorsed a claim about agent-id availability that measurement later destroyed. Two model families agreed it was true; the data disagreed. Cross-family review — our main structural guard — did not catch it, because correlated plausibility is exactly what agreement measures. One query did.

Re-measured today against a now-larger log while writing this: still 0 of 184,976 usage rows populated. The finding held, but I'd have been repeating a June number as a present-tense fact if I hadn't checked — which is the failure the rule exists to prevent, committed inside the rule itself.

Change

Every finding, blocking or not, carries one of three classes:

  • Measured — name the command, paste the result
  • Read — cite file:line, say plainly it's a code read
  • Reported — name the source, say you did not reproduce it

Plus a hard rule: a load-bearing claim from a PR body may not be repeated as fact in a review without independent measurement.

Binds every reviewer — Codex, our own maintainer comments, and any third model we add later. Codex already did this well in places today unprompted ("I reproduced this locally: changing OLD to NEW … still classified as normalized"; separately "I could not verify the author's live-traffic claims"). This makes it consistent rather than occasional.

Why now

We're considering adding a third reviewer for complex work. More reviewers means more findings to cite-check, and that cost compounds if each arrives without provenance. Classes make reconciliation cheaper rather than 1.5× more expensive — Measured from one reviewer and Read from another on the same point is a signal, not a contradiction.

Non-Functional Requirements

  • Size/complexity budget — 49 lines in AGENTS.md. Documentation only, no code.
  • Threat model — n/a. No inputs, no trust boundaries, nothing on the wire.
  • Maintainability — no new abstraction. Sharpens an existing global rule into a checkable form and records why, so it isn't softened back later.
  • Performance/reliability — n/a.
  • Load-bearing? No. Documentation governing review behavior.

Testing

Documentation only. Both figures cited in the new section were verified rather than quoted: the proxy-read-dedupe.test.mjs failure was reproduced on a merged worktree earlier today, and the agent-id figure was re-measured against the current log. Dates attached to both.

— Proxy Builder

The global baseline says to distinguish confirmed from assumed. That is
self-assessed: a reviewer who read a code path and found it plausible
files it honestly under "confirmed."

Findings now carry Measured (name the command, paste the result) / Read
(cite file:line, say it is a read) / Reported (name the source, say you
did not reproduce). A load-bearing claim from a PR body may not be
repeated as fact without independent measurement.

Binds every reviewer including our own maintainer comments, not just
Codex.

Motivating case is recorded in the section: on #270 a second model
family endorsed an agent-id claim that measurement destroyed. Both
figures cited are dated, and the current one was re-measured against
today's log (still 0 populated rows, now of 184,976) rather than quoted
from the older review.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GvZKP1JeXgHFCovTaAPT5B
@cnighswonger cnighswonger added approved-by-lead Final implementation approval from project lead ready-for-merge Required reviews are complete and no known blockers remain labels Jul 31, 2026
@cnighswonger
cnighswonger merged commit 209f867 into main Jul 31, 2026
5 checks passed
@cnighswonger
cnighswonger deleted the docs/evidence-class-rule branch July 31, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved-by-lead Final implementation approval from project lead ready-for-merge Required reviews are complete and no known blockers remain

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant