Skip to content

Commit b4ffd47

Browse files
datalogics-kamdatalogics-kam
committed
agents: Add PR Review Auditor with policy-first review process
- Introduced a new skill, `pr-review-auditor`, for rigorous PR reviews. - Added `SKILL.md` with detailed guidelines, rules, and review workflow. - Configured agent interface with `openai.yaml` to standardize usage. - Provided a policy checklist (`review-checklist.md`) for compliance enforcement. - Implemented helper scripts: - `collect_pr_context.sh` for deriving PR context and policy inventory. - `review_range.sh` for in-depth commit and diff analysis. - Updated `.pre-commit-config.yaml` to exclude `.agents/` directory. Assisted-by: Codex
1 parent 20bce70 commit b4ffd47

6 files changed

Lines changed: 292 additions & 0 deletions

File tree

‎.agents/skills/pr-review-auditor/SKILL.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
---
2+
name: pr-review-auditor
3+
description: Review the currently checked-out PR by discovering its upstream base branch with GitHub CLI, diffing base..HEAD, and performing a correctness-first code review that enforces repository policy with special priority on *GUIDELINES*.md rules and then AGENTS.md. Use when asked to do a full PR review, /review-style audit, commit-range quality check, or policy-compliance review of branch changes.
4+
---
5+
6+
# PR Review Auditor
7+
8+
## Overview
9+
10+
Review branch changes from PR base to `HEAD` with findings-first output.
11+
Prioritize correctness risks, then CI regressions, then test gaps, and enforce
12+
policy precedence: `*GUIDELINES*.md` before `AGENTS.md`.
13+
14+
## Workflow
15+
16+
1. Collect context from the active branch. Run `scripts/collect_pr_context.sh`
17+
from repo root to detect:
18+
19+
- checked-out branch,
20+
- PR number and base branch (via `gh pr view`),
21+
- merge-base commit,
22+
- review range (`origin/<base>..HEAD`),
23+
- changed files,
24+
- available policy files (`*GUIDELINES*.md`, `AGENTS.md`).
25+
26+
2. Inspect commits and patches in range. Run `scripts/review_range.sh` with the
27+
range from step 1. This prints:
28+
29+
- commits with full messages (`git log --format=fuller`),
30+
- per-commit changed files,
31+
- unified patches for detailed behavior review.
32+
33+
3. Run a three-pass policy protocol (Markdown-only). Pass 1: Rule extraction
34+
35+
- Read all `*GUIDELINES*.md` files first, then `AGENTS.md`.
36+
- Extract rules section-by-section by heading. For each heading, list actionable
37+
rules before moving to the next heading.
38+
- Extract every actionable rule into a numbered list with source citation.
39+
- Keep wording close to source text and avoid merging unrelated rules.
40+
- Minimum extraction floors:
41+
- `TESTING_GUIDELINES.md`: at least 20 rules unless a heading-by-heading audit
42+
proves fewer.
43+
- `AGENTS.md`: at least 15 rules unless a heading-by-heading audit proves
44+
fewer.
45+
- If below floor, include an explicit justification section that names each
46+
heading and why no additional actionable rule exists. Pass 2: Evidence
47+
mapping
48+
- For each extracted rule, assign `pass`, `fail`, or `n/a`.
49+
- Provide concrete evidence (`file:line`, command output, or explicit absence).
50+
- If rules conflict, cite both and follow `*GUIDELINES*.md`. Pass 3: Adversarial
51+
gap check
52+
- Re-read diffs and policy files only to find missed rules/findings.
53+
- Add new findings only when they include new source citations and evidence.
54+
- Require at least 5 "candidate missed rules" per policy file during this pass,
55+
then mark each as `new rule` or `duplicate` with rationale.
56+
57+
4. Perform review in severity order.
58+
59+
- Correctness/behavioral regressions.
60+
- CI and workflow regressions.
61+
- Test gaps or weak assertions.
62+
- Secondary maintainability/style concerns.
63+
64+
5. Produce findings-first output. List issues with file/line references and
65+
policy citation when applicable. Separate policy findings into two explicit
66+
sections:
67+
68+
- `GUIDELINES findings`
69+
- `AGENTS findings`
70+
71+
6. Provide a remediation plan. Include:
72+
73+
- ordered fix plan,
74+
- tests to run and why,
75+
- tests not run and why,
76+
- assumptions and open questions.
77+
78+
## Output Requirements
79+
80+
- Findings first, ordered by severity.
81+
- Include commit-message quality notes when relevant.
82+
- Cite exact paths and lines for each finding.
83+
- Include `files reviewed`, `tests run`, and `tests not run`.
84+
- If no findings exist, state that explicitly and list residual risks.
85+
- Include a `Rule Coverage Matrix` section in Markdown.
86+
- Do not mark the review complete until every extracted rule has a status and
87+
evidence.
88+
- Include a `Rule Extraction Summary` with:
89+
- total rules per source file,
90+
- per-heading rule counts,
91+
- whether extraction floors were met,
92+
- explicit justification if any floor was not met.
93+
94+
## Rule Coverage Matrix Template
95+
96+
Use this exact table shape:
97+
98+
| Rule ID | Source | Rule Summary | Status (pass/fail/n-a) | Evidence |
99+
| ------- | --------------------- | ------------ | ---------------------- | ------------------ |
100+
| G-001 | TESTING_GUIDELINES.md | ... | pass | tests/test_x.py:42 |
101+
| A-001 | AGENTS.md | ... | fail | src/module.py:87 |
102+
103+
## Trigger Examples
104+
105+
- "Figure out this PR's base branch and do a /review from base to HEAD."
106+
- "Audit all commits in this branch and check AGENTS + GUIDELINES compliance."
107+
- "Review patches in this PR and give me a fix plan with test gaps."
108+
109+
## Resources
110+
111+
- `scripts/collect_pr_context.sh`: derive PR base/range and policy file
112+
inventory.
113+
- `scripts/review_range.sh`: print full commit messages and patches for a range.
114+
- `references/review-checklist.md`: compact review checklist and severity
115+
rubric.

‎.agents/skills/pr-review-auditor/agents/openai.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
interface:
2+
display_name: "PR Review Auditor"
3+
short_description: "Review base..HEAD with GUIDELINES-first policy checks."
4+
default_prompt: "Use $pr-review-auditor to review the current PR from upstream base to HEAD with findings-first output and strict GUIDELINES/AGENTS compliance checks."

‎.agents/skills/pr-review-auditor/references/review-checklist.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# PR Review Checklist
2+
3+
## Policy Precedence
4+
5+
1. `*GUIDELINES*.md` rules are highest-priority compliance checks.
6+
2. `AGENTS.md` rules are required secondary checks.
7+
3. If rules conflict, cite both files and follow `*GUIDELINES*.md`.
8+
9+
## Review Scope
10+
11+
- Determine PR base branch and audit `origin/<base>..HEAD`.
12+
- Review full commit messages, changed files, and relevant patches.
13+
- Focus on behavior, CI impact, tests, then style.
14+
15+
## Three-Pass Completeness Protocol
16+
17+
1. Rule extraction pass:
18+
19+
- Enumerate every actionable rule from `*GUIDELINES*.md`, then `AGENTS.md`.
20+
- Extract heading-by-heading and record per-heading counts.
21+
- Assign stable rule IDs (for example `G-001`, `A-001`).
22+
- Meet extraction floors unless explicitly justified:
23+
- `TESTING_GUIDELINES.md`: >= 20 rules.
24+
- `AGENTS.md`: >= 15 rules.
25+
- If below floor, justify heading-by-heading why no additional actionable
26+
rules exist.
27+
28+
2. Evidence mapping pass:
29+
30+
- Map every rule to `pass`, `fail`, or `n/a`.
31+
- Attach evidence for every status (path/line or concrete command output).
32+
33+
3. Gap pass:
34+
35+
- Re-check changed files and policy text for missing rules/findings.
36+
- Add only findings backed by new source citations and evidence.
37+
- Generate at least 5 missed-rule candidates per policy file, then classify each
38+
as `new rule` or `duplicate` with rationale.
39+
40+
## Severity Order
41+
42+
1. Correctness risk: behavior bug, data loss, runtime failure, API mismatch.
43+
2. CI/workflow regression: broken matrix, wrong triggers, missing job parity.
44+
3. Test gap: missing or weak coverage for changed behavior.
45+
4. Maintainability/style: readability or consistency issues with low break risk.
46+
47+
## Mandatory Output Sections
48+
49+
- Findings (ordered by severity, each with path/line evidence).
50+
- Rule Coverage Matrix (every extracted rule must be represented).
51+
- Rule Extraction Summary (totals per file, per-heading counts, floors met/not
52+
met).
53+
- GUIDELINES findings.
54+
- AGENTS findings.
55+
- Open questions and assumptions.
56+
- Remediation plan with validation steps.
57+
- Files reviewed.
58+
- Tests run.
59+
- Tests not run and why.
60+
61+
## Completion Gate
62+
63+
- The review is incomplete if any extracted rule is missing from the matrix.
64+
- The review is incomplete if any matrix row lacks status or evidence.
65+
- The review is incomplete if extraction floors are missed without explicit
66+
heading-by-heading justification.

‎.agents/skills/pr-review-auditor/scripts/collect_pr_context.sh‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
branch="$(git rev-parse --abbrev-ref HEAD)"
5+
base_remote="origin"
6+
if git remote get-url upstream > /dev/null 2>&1; then
7+
base_remote="upstream"
8+
fi
9+
10+
context_source="gh-pr-view"
11+
pr_number="N/A"
12+
head_ref="${branch}"
13+
base_ref=""
14+
pr_json=""
15+
gh_error=""
16+
17+
if command -v gh > /dev/null 2>&1 && command -v jq > /dev/null 2>&1; then
18+
gh_error_file="$(mktemp)"
19+
if ! pr_json="$(gh pr view --json number,baseRefName,headRefName 2> "${gh_error_file}")"; then
20+
pr_json=""
21+
fi
22+
if [[ -z "${pr_json}" ]]; then
23+
gh_error="$(< "${gh_error_file}")"
24+
fi
25+
rm -f "${gh_error_file}"
26+
fi
27+
28+
if [[ -n "${pr_json}" && "${pr_json}" != "null" ]]; then
29+
pr_number="$(jq -r '.number' <<< "${pr_json}")"
30+
base_ref="$(jq -r '.baseRefName' <<< "${pr_json}")"
31+
head_ref="$(jq -r '.headRefName' <<< "${pr_json}")"
32+
else
33+
context_source="remote-head-fallback"
34+
remote_head_ref="$(git symbolic-ref --quiet --short "refs/remotes/${base_remote}/HEAD" || true)"
35+
if [[ -z "${remote_head_ref}" ]]; then
36+
echo "error: unable to derive base branch from refs/remotes/${base_remote}/HEAD" >&2
37+
echo "hint: run 'git remote set-head ${base_remote} --auto' and retry" >&2
38+
if [[ -n "${gh_error}" ]]; then
39+
echo "gh pr view: ${gh_error}" >&2
40+
fi
41+
exit 1
42+
fi
43+
base_ref="${remote_head_ref#${base_remote}/}"
44+
if [[ -n "${gh_error}" ]]; then
45+
echo "warning: gh pr view failed; using ${base_remote}/HEAD fallback" >&2
46+
echo "gh pr view: ${gh_error}" >&2
47+
fi
48+
fi
49+
50+
fetch_error_file="$(mktemp)"
51+
if ! git fetch --quiet "${base_remote}" "${base_ref}" 2> "${fetch_error_file}"; then
52+
if git rev-parse --verify --quiet "${base_remote}/${base_ref}" > /dev/null; then
53+
echo "warning: git fetch failed; using cached ${base_remote}/${base_ref}" >&2
54+
cat "${fetch_error_file}" >&2
55+
else
56+
echo "error: unable to fetch ${base_remote}/${base_ref} and no local cached ref exists" >&2
57+
cat "${fetch_error_file}" >&2
58+
rm -f "${fetch_error_file}"
59+
exit 1
60+
fi
61+
fi
62+
rm -f "${fetch_error_file}"
63+
64+
merge_base="$(git merge-base "${base_remote}/${base_ref}" HEAD)"
65+
review_range="${base_remote}/${base_ref}..HEAD"
66+
67+
echo "CONTEXT_SOURCE=${context_source}"
68+
echo "PR_NUMBER=${pr_number}"
69+
echo "CURRENT_BRANCH=${branch}"
70+
echo "PR_HEAD_BRANCH=${head_ref}"
71+
echo "PR_BASE_BRANCH=${base_ref}"
72+
echo "PR_BASE_REMOTE=${base_remote}"
73+
echo "MERGE_BASE=${merge_base}"
74+
echo "REVIEW_RANGE=${review_range}"
75+
echo
76+
77+
echo "CHANGED_FILES:"
78+
git diff --name-only "${review_range}"
79+
echo
80+
81+
echo "POLICY_FILES:"
82+
find . -type f \( -name '*GUIDELINES*.md' -o -name 'AGENTS.md' \) | sort

‎.agents/skills/pr-review-auditor/scripts/review_range.sh‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
range="${1:-}"
5+
if [[ -z "${range}" ]]; then
6+
echo "usage: $(basename "$0") <git-range>" >&2
7+
echo "example: $(basename "$0") origin/main..HEAD" >&2
8+
exit 1
9+
fi
10+
11+
echo "=== RANGE ==="
12+
echo "${range}"
13+
echo
14+
15+
echo "=== COMMITS (full messages) ==="
16+
git log --format=fuller "${range}"
17+
echo
18+
19+
echo "=== COMMIT FILE SUMMARIES ==="
20+
git log --name-status --format='commit %H%nAuthor: %an <%ae>%nDate: %ad%n%n%s%n%b' "${range}"
21+
echo
22+
23+
echo "=== PATCHES ==="
24+
git log --patch --stat "${range}"

‎.pre-commit-config.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
# Verbose regular expressions '(?x)': https://docs.python.org/3.9/library/re.html#re.X
33
exclude: |
44
(?x)^(
5+
.agents/|
56
.idea/|
67
.venv/
78
)

0 commit comments

Comments
 (0)