docs(agents): require an evidence class on every review finding - #289
Merged
Conversation
The global baseline says to distinguish confirmed from assumed. That is self-assessed: a reviewer who read a code path and found it plausible files it honestly under "confirmed." Findings now carry Measured (name the command, paste the result) / Read (cite file:line, say it is a read) / Reported (name the source, say you did not reproduce). A load-bearing claim from a PR body may not be repeated as fact without independent measurement. Binds every reviewer including our own maintainer comments, not just Codex. Motivating case is recorded in the section: on #270 a second model family endorsed an agent-id claim that measurement destroyed. Both figures cited are dated, and the current one was re-measured against today's log (still 0 populated rows, now of 184,976) rather than quoted from the older review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GvZKP1JeXgHFCovTaAPT5B
cnighswonger
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The global reviewer baseline (
~/.codex/AGENTS.md:155) says to distinguish what is confirmed correct from what is assumed or hypothesized. That's a disposition, and a disposition is self-assessed — a reviewer who read a code path and found it plausible will honestly file it under "confirmed." Nothing about the output tells the next reader which it was.Same shape as the bloat gap we closed in #287: "None" isn't checkable, a number is. "Confirmed" isn't checkable, a named command is.
The motivating case
On #270 the reviewer endorsed a claim about agent-id availability that measurement later destroyed. Two model families agreed it was true; the data disagreed. Cross-family review — our main structural guard — did not catch it, because correlated plausibility is exactly what agreement measures. One query did.
Re-measured today against a now-larger log while writing this: still 0 of 184,976 usage rows populated. The finding held, but I'd have been repeating a June number as a present-tense fact if I hadn't checked — which is the failure the rule exists to prevent, committed inside the rule itself.
Change
Every finding, blocking or not, carries one of three classes:
file:line, say plainly it's a code readPlus a hard rule: a load-bearing claim from a PR body may not be repeated as fact in a review without independent measurement.
Binds every reviewer — Codex, our own maintainer comments, and any third model we add later. Codex already did this well in places today unprompted ("I reproduced this locally: changing
OLDtoNEW… still classified asnormalized"; separately "I could not verify the author's live-traffic claims"). This makes it consistent rather than occasional.Why now
We're considering adding a third reviewer for complex work. More reviewers means more findings to cite-check, and that cost compounds if each arrives without provenance. Classes make reconciliation cheaper rather than 1.5× more expensive — Measured from one reviewer and Read from another on the same point is a signal, not a contradiction.
Non-Functional Requirements
AGENTS.md. Documentation only, no code.Testing
Documentation only. Both figures cited in the new section were verified rather than quoted: the
proxy-read-dedupe.test.mjsfailure was reproduced on a merged worktree earlier today, and the agent-id figure was re-measured against the current log. Dates attached to both.— Proxy Builder