[codex] Enforce substantive headless reviews

[CompleteDotTech]

Summary

• Enforces substantive hosted review output instead of accepting generic completion text.
• Parses review final text into structured findings, tests, no-findings rationale, supporting-context rationale, and limitations.
• Strengthens review-mode prompts with required sections, bounded exploration, file-backed reasoning, and review-only behavior.
• Filters noisy .git/directory trace evidence and caps hosted review mode to exactly 40 turns.

Context

• Related issue/user request: Hosted Review Mode — security & memory hardening track OpenCoven/coven-code#119.
• Stacked after CompleteTech-LLC-AI-Research/coven-code PR [codex] Trace supporting review files #8.
• Scope: crates/cli headless review prompt/result parsing/enforcement and hosted review query defaults.

Changes

• Adds conservative markdown-section parsing for Findings, No-findings justification, Tests/commands considered, Supporting context used, and Confidence/limitations.
• Marks empty/generic review-mode output as partial with limitations instead of reporting misleading success.
• Requires structured findings, file-backed no-findings rationale, or explicit limitations before review evidence can be complete.
• Requires supporting-context rationale when supporting files were traced.
• Keeps hosted review mode review-only unless the comment explicitly asks for code changes.
• Bounds hosted review exploration and gives review mode a fixed 40-turn budget.
• Filters .git internals and directories from supporting-file evidence.

Validation

• cargo fmt --all from src-rust/ - passed.
• cargo check --workspace from src-rust/ - passed.
• cargo clippy --workspace --all-targets -- -D warnings from src-rust/ - passed.
• cargo test --workspace from src-rust/ - passed.
• Focused tests: cargo test --package claurst headless - passed, 28 matching tests after the cap regression was added.
• Focused tests: cargo test --package claurst hosted_review_headless_gets_extra_turn_budget - passed.
• Focused tests: cargo test --package claurst hosted_review_headless_caps_existing_turn_budget - passed.
• Transient unrelated TUI check: cargo test -p claurst-tui osc8::tests::detects_simple_http_url -- --nocapture - passed on rerun after one full-suite attempt failed there.

Dogfood

• Initial enforcement run proved generic output is downgraded to partial: [codex] Enforce substantive headless reviews #9 (comment)
• Empty final-output run after review-only prompt still downgraded to partial: [codex] Enforce substantive headless reviews #9 (comment)
• Trace filtering and larger budget removed .git noise but still caught generic output as partial: [codex] Enforce substantive headless reviews #9 (comment)
• Bounded review prompt produced structured review with a real finding: [codex] Enforce substantive headless reviews #9 (comment)
• Final cap fix produced success, review.evidence_status: complete, no findings, and a file-backed no-findings justification: [codex] Enforce substantive headless reviews #9 (comment)

PR Readiness

• Diff is limited to intended headless/hosted-review files.
• Generated files were not edited by hand.
• No secrets, credentials, local paths, or unrelated logs are included.
• Remaining risk: parser remains intentionally conservative; non-sectioned review prose is marked partial by design.

[CompleteDotTech]

@cody please review this PR.

Use the required hosted-review sections exactly. Inspect the changed file and relevant supporting context. If there are no findings, provide a specific file-backed no-findings justification and explain the supporting context used. Do not post a generic completion message.

[covencat]

Cody dogfood result

Status: partial

Cody made 1 commit across 0 files.

Cody

Made partial progress on the requested change.

Files changed

No files changed.

Evidence

• PR: [codex] Enforce substantive headless reviews #9
• Base: codex/headless-review-tool-read-tracing @ 017d44e78b43e699427247252942f333de763da8
• Head: codex/headless-review-quality-enforcement @ 831b72e24af6493e41f0c9bc0e4d7c627cbbe961
• Checked-out workspace HEAD: 831b72e24af6493e41f0c9bc0e4d7c627cbbe961
• Changed files supplied to agent: 1
• Review context SHA-256: 02d0cfbe109029aa2f7bc02b3fe6c11eef8bf871922b95d413a33b2f7b175640
• Files: src-rust/crates/cli/src/headless.rs

Structured review

• Mode: pull_request
• Evidence status: partial
• Reviewed files: 1
• Reviewed file list: src-rust/crates/cli/src/headless.rs
• Supporting files inspected: 39
• Supporting file list: .git/FETCH_HEAD, .git/HEAD, .git/packed-refs, .github, .github/PULL_REQUEST_TEMPLATE.md, .github/release.yml, .github/workflows/auto-release.yml, .github/workflows/npm-publish.yml, .github/workflows/pages.yml, .github/workflows/patch-release.yml, .github/workflows/release.yml, .github/workflows/rust-ci.yml, .github/workflows/tui-tests.yml, AGENTS.md, ATTRIBUTION.md, CONTRIBUTING.md, COVEN.md, LICENSE.md, PROVENANCE.md, README.md
• Supporting file list truncated after 20 entries.
• Findings: 0
• Tests reported by runtime: 0
• Limitations: 3
  • Review narrative was generic and did not explain the review outcome.
  • Review output did not explain supporting context with traced file references.
  • Review output did not include structured findings, a file-backed no-findings justification, or an explicit limitation explaining why substantive review was not possible.

Files changed: 0
Commits: 1

Task 59c3cdb0-78ee-11f1-98d0-c77293aeb998. Publication is enabled on the hosted test adapter only.

[CompleteDotTech]

@cody please review this updated PR again.

This run is specifically checking whether hosted review mode now produces a substantive structured review instead of a generic completion. Do not modify files or commit. Use the required sections, inspect supporting context, and if there are no findings include a specific file-backed no-findings justification plus the supporting context used.

[covencat]

Cody dogfood result

Status: partial

Cody made 1 commit across 0 files.

Cody

Made partial progress on the requested change.

Files changed

No files changed.

Evidence

• PR: [codex] Enforce substantive headless reviews #9
• Base: codex/headless-review-tool-read-tracing @ 017d44e78b43e699427247252942f333de763da8
• Head: codex/headless-review-quality-enforcement @ e723dfd1ee0d90e0a9f81bf7ea52caed16b194bd
• Checked-out workspace HEAD: e723dfd1ee0d90e0a9f81bf7ea52caed16b194bd
• Changed files supplied to agent: 1
• Review context SHA-256: 94ec998ed7841962f4f44662e304ea3a6cb479c300ae65d0b1085ecac29e1cd2
• Files: src-rust/crates/cli/src/headless.rs

Structured review

• Mode: pull_request
• Evidence status: partial
• Reviewed files: 1
• Reviewed file list: src-rust/crates/cli/src/headless.rs
• Supporting files inspected: 155
• Supporting file list: .git, .git/FETCH_HEAD, .git/HEAD, .git/config, .git/hooks/applypatch-msg.sample, .git/hooks/commit-msg.sample, .git/hooks/fsmonitor-watchman.sample, .git/hooks/post-update.sample, .git/hooks/pre-applypatch.sample, .git/hooks/pre-commit.sample, .git/hooks/pre-merge-commit.sample, .git/hooks/pre-push.sample, .git/hooks/pre-rebase.sample, .git/hooks/pre-receive.sample, .git/hooks/prepare-commit-msg.sample, .git/hooks/push-to-checkout.sample, .git/hooks/sendemail-validate.sample, .git/hooks/update.sample, .git/objects/pack/pack-002cce33b3632c490a8956df113b0a9ae9c6ea00.idx, .git/objects/pack/pack-002cce33b3632c490a8956df113b0a9ae9c6ea00.pack
• Supporting file list truncated after 20 entries.
• Findings: 0
• Tests reported by runtime: 0
• Limitations: 3
  • Review narrative was generic and did not explain the review outcome.
  • Review output did not explain supporting context with traced file references.
  • Review output did not include structured findings, a file-backed no-findings justification, or an explicit limitation explaining why substantive review was not possible.

Files changed: 0
Commits: 1

Task a563a090-78f0-11f1-9c6f-b43162ffd001. Publication is enabled on the hosted test adapter only.

[CompleteDotTech]

@cody please review this updated PR again.

This run checks the increased hosted review turn budget and filtered supporting-file trace. Do not modify files or commit. Use the required sections, inspect supporting context, and if there are no findings include a specific file-backed no-findings justification plus the supporting context used.

[covencat]

Cody dogfood result

Status: partial

Cody made 1 commit across 0 files.

Cody

Made partial progress on the requested change.

Files changed

No files changed.

Evidence

• PR: [codex] Enforce substantive headless reviews #9
• Base: codex/headless-review-tool-read-tracing @ 017d44e78b43e699427247252942f333de763da8
• Head: codex/headless-review-quality-enforcement @ e3fa52758eb48977347f1adf0a6ad8ed045f172b
• Checked-out workspace HEAD: e3fa52758eb48977347f1adf0a6ad8ed045f172b
• Changed files supplied to agent: 2
• Review context SHA-256: e10c41c9da7a2e4786ba6e66198fbe813632d4ad1df34a22b85ca276125b097d
• Files: src-rust/crates/cli/src/headless.rs, src-rust/crates/cli/src/main.rs

Structured review

• Mode: pull_request
• Evidence status: partial
• Reviewed files: 2
• Reviewed file list: src-rust/crates/cli/src/headless.rs, src-rust/crates/cli/src/main.rs
• Supporting files inspected: 129
• Supporting file list: .github/PULL_REQUEST_TEMPLATE.md, .github/release.yml, .github/workflows/auto-release.yml, .github/workflows/npm-publish.yml, .github/workflows/pages.yml, .github/workflows/patch-release.yml, .github/workflows/release.yml, .github/workflows/rust-ci.yml, .github/workflows/tui-tests.yml, docs/advanced.md, docs/configuration.md, docs/headless-contract.md, docs/shared/hosted-review-phase-1-pr.md, docs/shared/hosted-review-phase-2-pr.md, docs/shared/hosted-review-phase-3-pr.md, docs/shared/hosted-review-phase-4-pr.md, docs/shared/hosted-review-phase-5-pr.md, src-rust/crates/cli/tests/headless_contract/result.example.json, src-rust/crates/cli/tests/headless_contract/result.schema.json, src-rust/crates/cli/tests/headless_contract/session-brief.example.json
• Supporting file list truncated after 20 entries.
• Findings: 0
• Tests reported by runtime: 0
• Limitations: 3
  • Review narrative was generic and did not explain the review outcome.
  • Review output did not explain supporting context with traced file references.
  • Review output did not include structured findings, a file-backed no-findings justification, or an explicit limitation explaining why substantive review was not possible.

Files changed: 0
Commits: 1

Task cdd0d4b0-78f2-11f1-9e71-099f7b7e21c5. Publication is enabled on the hosted test adapter only.

[CompleteDotTech]

@cody please review this updated PR again.

This run checks the bounded review prompt, 40-turn review budget, and filtered supporting-file trace. Do not modify files or commit. Inspect the changed files and only targeted supporting context, then stop tool use and produce the required review sections. If there are no findings, include a specific file-backed no-findings justification and the supporting context used.

[covencat]

Cody dogfood result

Status: success

Files inspected

Files inspected

• src-rust/crates/cli/src/main.rs
• src-rust/crates/cli/src/headless.rs
• docs/headless-contract.md

Supporting context used

• src-rust/crates/query/src/lib.rs - confirmed the query loop enforces the already-resolved config.max_turns, so an oversized value passed from the CLI/headless setup directly expands the hosted review turn budget.
• src-rust/crates/core/src/lib.rs - checked related max_turns configuration definitions/default handling to understand where agent/CLI limits originate.

Findings

• [medium] src-rust/crates/cli/src/main.rs:1576 Hosted review budget is a floor, not a bound - The hosted review setup uses query_config.max_turns = query_config.max_turns.max(40), which guarantees at least 40 turns but does not bound the review to 40 turns. If the CLI default, an agent definition, or other configuration supplies a higher value, the downstream query loop will honor that higher max_turns, so hosted reviews can exceed the intended 40-turn review budget. Recommendation: set the hosted review budget to exactly 40, or clamp with min(40) after any higher-precedence overrides if the contract is “at most 40 turns”; add a regression test where the incoming max_turns is greater than 40.

No-findings justification

Not applicable; one finding is listed above.

Tests/commands considered

• cargo test -p claurst-cli headless - not run: review requested inspection only and no file modifications.
• cargo test -p claurst-cli hosted_review_context_sets_review_budget - not run: targeted test inferred from inspected Rust test names, but commands were not executed to keep the review bounded.

Confidence/limitations

Medium-high confidence. I inspected the changed hosted-review/headless files and targeted supporting max_turns enforcement context. Limitation: the patch text supplied through the hosted context was truncated in tool output, so the review is based on the checked-out PR workspace contents and targeted source inspection rather than a full visible patch transcript.

Evidence

• PR: [codex] Enforce substantive headless reviews #9
• Base: codex/headless-review-tool-read-tracing @ 017d44e78b43e699427247252942f333de763da8
• Head: codex/headless-review-quality-enforcement @ a7744a147a262a552b3cc482bb6237fdf1e99698
• Checked-out workspace HEAD: a7744a147a262a552b3cc482bb6237fdf1e99698
• Changed files supplied to agent: 2
• Review context SHA-256: 03390c4f06792102e10629aa7fad6e41ed890759933374e4ab80d675de013632
• Files: src-rust/crates/cli/src/headless.rs, src-rust/crates/cli/src/main.rs

Structured review

• Mode: pull_request
• Evidence status: complete
• Reviewed files: 2
• Reviewed file list: src-rust/crates/cli/src/headless.rs, src-rust/crates/cli/src/main.rs
• Supporting files inspected: 181
• Supporting file list: .github/workflows/rust-ci.yml, AGENTS.md, PROVENANCE.md, docs/SECURITY_PERF_AUDIT.md, docs/advanced.md, docs/auth.md, docs/commands.md, docs/configuration.md, docs/headless-contract.md, docs/installation.md, docs/package-lock.json, docs/shared/githubStars.js, docs/shared/hosted-review-phase-1-pr.md, docs/shared/hosted-review-phase-2-pr.md, docs/shared/hosted-review-phase-3-pr.md, docs/shared/hosted-review-phase-4-pr.md, docs/shared/hosted-review-phase-5-pr.md, docs/src/code-highlight.js, docs/src/content/advanced.js, docs/src/content/agents.js
• Supporting file list truncated after 20 entries.
• Findings: 1
  1. medium src-rust/crates/cli/src/main.rs:1576 - Hosted review budget is a floor, not a bound - The hosted review setup uses `query_config
• Tests reported by runtime: 2
  • cargo test -p claurst-cli headless: not_run - review requested inspection only and no file modifications.
  • cargo test -p claurst-cli hosted_review_context_sets_review_budget: not_run - targeted test inferred from inspected Rust test names, but commands were not executed to keep the review bounded.
• Limitations: 1
  • Medium-high confidence. I inspected the changed hosted-review/headless files and targeted supporting max_turns enforcement context. Limitation: the patch text supplied through the hosted context was truncated in tool output, so the review is based on the checked-out PR workspace contents and targeted source inspection rather than a full visible patch transcript.

Files changed: 0
Commits: 1

Task edd6f630-78f3-11f1-9afa-bba5a35d8d31. Publication is enabled on the hosted test adapter only.

[CompleteDotTech]

@cody please review this updated PR again.

This run checks the final cap fix for the prior finding: hosted review mode should use a bounded 40-turn budget, not a 40-turn floor. Do not modify files or commit. Inspect the changed files and targeted supporting context, then stop tool use and produce the required review sections. If there are no findings, include a specific file-backed no-findings justification and the supporting context used.

[covencat]

Cody dogfood result

Status: success

Files inspected

Files inspected

• src-rust/crates/cli/src/main.rs

Supporting context used

• src-rust/crates/cli/src/headless.rs - Confirmed SessionBrief::review_mode() returns review mode only for address-review-comment tasks or review_context.kind == "pull_request", which is the condition used by the hosted review defaulting logic.
• src-rust/crates/query/src/lib.rs - Confirmed the query loop enforces max_turns via effective_max_turns and stops when turn > effective_max_turns, so the configured value is the actual execution bound.

Findings

None.

No-findings justification

The updated hosted review defaulting logic in src-rust/crates/cli/src/main.rs:1568-1579 sets query_config.max_turns directly to HOSTED_REVIEW_MAX_TURNS (40) whenever the GitHub session is in review mode. This avoids the prior “40-turn floor” behavior because higher existing values are overwritten rather than preserved or raised with max(...).

The changed tests back this up:

• src-rust/crates/cli/src/main.rs:5293-5312 verifies an existing max_turns: 99 is capped to 40.
• src-rust/crates/cli/src/main.rs:5315-5333 verifies non-review headless sessions keep their existing max_turns.
• src-rust/crates/cli/src/main.rs:5271-5290 verifies hosted review sessions receive the intended 40-turn review budget.

The supporting control flow also matches the intent: src-rust/crates/cli/src/headless.rs:180-193 identifies PR review mode from review_context.kind == "pull_request", and src-rust/crates/query/src/lib.rs:877-920 enforces the resulting max_turns as the query loop limit.

Tests/commands considered

• cargo test -p claurst-cli hosted_review_headless_caps_existing_turn_budget - not run: review request explicitly asked not to modify files; I performed a bounded code inspection only.
• cargo test -p claurst-cli hosted_review_headless_gets_extra_turn_budget - not run: targeted test was inspected in source but not executed.
• cargo test -p claurst-cli non_review_headless_keeps_turn_budget - not run: targeted test was inspected in source but not executed.

Confidence/limitations

High confidence for the reviewed behavior. I inspected the changed file and the targeted supporting code that determines review mode and enforces max_turns. Limitation: I did not run tests or inspect unrelated repository areas, keeping the review bounded to the hosted-review turn-budget change and its direct supporting context.

Evidence

• PR: [codex] Enforce substantive headless reviews #9
• Base: codex/headless-review-tool-read-tracing @ 017d44e78b43e699427247252942f333de763da8
• Head: codex/headless-review-quality-enforcement @ 7f5a3a12f1bae3f9613ef967b9e7831fa99dda16
• Checked-out workspace HEAD: 7f5a3a12f1bae3f9613ef967b9e7831fa99dda16
• Changed files supplied to agent: 2
• Review context SHA-256: e8304240f2943bdbaf9ecd57a9bd80c647bd139f90cdd6c8dca35ee7b1abad6a
• Files: src-rust/crates/cli/src/headless.rs, src-rust/crates/cli/src/main.rs

Structured review

• Mode: pull_request
• Evidence status: complete
• Reviewed files: 2
• Reviewed file list: src-rust/crates/cli/src/headless.rs, src-rust/crates/cli/src/main.rs
• Supporting files inspected: 168
• Supporting file list: .github/workflows/rust-ci.yml, 1014: / Apply todo nudge on turns > 2., 116: / Fallback model name. Used when the primary model returns overloaded, 1480: / inject_reasoning_for_tool_turns can find it later., 1947: / and returns immediately with an empty Vec., 2024: / maybe_trigger() checks gates + acquires lock. If it returns, 2346:/ incomplete (status != "completed"). Returns empty string otherwise., 561:/ Returns an empty Vec immediately; results (if any) are lost., 605:/ in user turns)., 639:/ under budget. Returns the (possibly modified) message list and the, 838:/ returns when the model issues an end_turn or an error/limit is hit., 842:/ appended as a plain user message between turns. Callers that do not need, 877: / If an agent defines a max_turns override, respect it (agent wins over config)., 881: .and_then(|a| a.max_turns), 882: .unwrap_or(config.max_turns);, 916: .unwrap_or_else(|| Message, 930: / Mirrors the TS messageQueueManager drain between turns., 986: / and the conversation has more than 2 turns, append a reminder., 9:/ 6. Manages stop conditions (end_turn, max_turns, cancellation), docs/AUDIT-2026-06.md
• Supporting file list truncated after 20 entries.
• Findings: 0
• No-findings reason: The updated hosted review defaulting logic in src-rust/crates/cli/src/main.rs:1568-1579 sets query_config.max_turns directly to HOSTED_REVIEW_MAX_TURNS (40) whenever the GitHub session is in review mode. This avoids the prior “40-turn floor” behavior because higher existing values are overwritten rather than preserved or raised with max(...). The changed tests back this up: src-rust/crates/cli/src/main.rs:5293-5312 verifies an existing max_turns: 99 is capped to 40. src-rust/crates/cli/src/main.rs:5315-5333 verifies non-review headless sessions keep their existing max_turns. src-rust/crates/cli/src/main.rs:5271-5290 verifies hosted review sessions receive the intended 40-turn review budget. The supporting control flow also matches the intent: src-rust/crates/cli/src/headless.rs:180-193 identifies PR review mode from review_context.kind == "pull_request", and src-rust/crates/query/src/lib.rs:877-920 enforces the resulting max_turns as the query loop limit.
• Tests reported by runtime: 3
  • cargo test -p claurst-cli hosted_review_headless_caps_existing_turn_budget: not_run - review request explicitly asked not to modify files; I performed a bounded code inspection only.
  • cargo test -p claurst-cli hosted_review_headless_gets_extra_turn_budget: not_run - targeted test was inspected in source but not executed.
  • cargo test -p claurst-cli non_review_headless_keeps_turn_budget: not_run - targeted test was inspected in source but not executed.
• Limitations: 1
  • High confidence for the reviewed behavior. I inspected the changed file and the targeted supporting code that determines review mode and enforces max_turns. Limitation: I did not run tests or inspect unrelated repository areas, keeping the review bounded to the hosted-review turn-budget change and its direct supporting context.

Files changed: 0
Commits: 1

Task 483576a0-78f5-11f1-81a1-b3bb947e56a4. Publication is enabled on the hosted test adapter only.