Skip to content

Commit dc34455

Browse files
committed
docs: add code review agent design scaffold
Add design docs and a Skill scaffold for the code review Agent proposal. Updates #92 RELEASE NOTES: Added design documentation and example scaffold for a Skill-based code review Agent.
1 parent e113610 commit dc34455

7 files changed

Lines changed: 1272 additions & 0 deletions

File tree

Lines changed: 332 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,332 @@
1+
# Code Review Agent Design
2+
3+
This page describes a proposed automatic code-review Agent that combines Agent Skills, sandboxed execution, structured findings, filters, telemetry, and SQL storage. It is a design and scaffold document: it defines the expected architecture and contracts before a production-ready implementation is added.
4+
5+
## Overview
6+
7+
The code-review Agent receives a unified diff, a PR patch, or local repository changes and produces a structured review report. A future runnable implementation should support inputs such as:
8+
9+
- `--diff-file`: read a saved unified diff or patch.
10+
- `--repo-path`: read local working-tree changes from a repository.
11+
- Test fixtures: run deterministic samples in dry-run or fake-model mode.
12+
13+
Expected outputs are:
14+
15+
- `review_report.json`: machine-readable findings, filter decisions, sandbox runs, and metrics.
16+
- `review_report.md`: human-readable summary.
17+
- SQL records for review tasks, input summaries, sandbox runs, findings, filter decisions, monitoring summaries, and final reports.
18+
19+
## Non-goals for the scaffold
20+
21+
The first contribution should stay small and reviewable. It should not claim to implement the full review bot.
22+
23+
- No GitHub or PR comment posting.
24+
- No automatic file modification or fix application.
25+
- No host execution of model-generated commands.
26+
- No production scheduler, webhook server, or credentials management.
27+
- No complete sandbox, database, or model loop implementation yet.
28+
29+
## Architecture
30+
31+
A complete implementation should follow this flow:
32+
33+
```text
34+
diff / repo changes
35+
-> diff parser
36+
-> code-review Skill
37+
-> Filter governance
38+
-> sandboxed checks
39+
-> structured finding validation
40+
-> dedupe and noise filtering
41+
-> SQLite storage
42+
-> JSON/Markdown report
43+
```
44+
45+
The design separates three concerns:
46+
47+
1. **Review policy** lives in the `code-review` Skill and reference docs.
48+
2. **Execution safety** lives in sandbox and Filter layers.
49+
3. **Auditability** lives in structured output, SQL storage, and monitoring summaries.
50+
51+
## Existing building blocks
52+
53+
The implementation should reuse existing tRPC-Agent patterns instead of inventing a parallel framework.
54+
55+
- Skills: [`trpc_agent_sdk/skills/__init__.py`](../../../trpc_agent_sdk/skills/__init__.py)
56+
- `SkillToolSet`
57+
- `create_default_skill_repository`
58+
- `SkillLoadTool`
59+
- `SkillRunTool`
60+
- Skill examples:
61+
- [`examples/skills`](../../../examples/skills)
62+
- [`examples/skills_with_container`](../../../examples/skills_with_container)
63+
- [`examples/skills_with_cube`](../../../examples/skills_with_cube)
64+
- Code execution:
65+
- [`BaseCodeExecutor`](../../../trpc_agent_sdk/code_executors/_base_code_executor.py)
66+
- [`ContainerCodeExecutor`](../../../trpc_agent_sdk/code_executors/container/_container_code_executor.py)
67+
- Filter governance:
68+
- [`FilterABC`](../../../trpc_agent_sdk/abc/_filter.py)
69+
- [`FilterResult`](../../../trpc_agent_sdk/abc/_filter.py)
70+
- SQL storage:
71+
- [`SqlStorage`](../../../trpc_agent_sdk/storage/_sql.py)
72+
- [`SqlKey`](../../../trpc_agent_sdk/storage/_sql.py)
73+
- [`SqlCondition`](../../../trpc_agent_sdk/storage/_sql.py)
74+
75+
## Skill design
76+
77+
The code-review Skill is the reusable review policy package. A first scaffold can live under the example directory:
78+
79+
```text
80+
examples/code_review_agent/skills/code-review/
81+
SKILL.md
82+
references/
83+
finding_schema.md
84+
security_boundary.md
85+
scripts/
86+
# future static checks / diff helpers
87+
```
88+
89+
`SKILL.md` should define the review workflow and require structured output. Future rule documents should cover at least four categories from the issue:
90+
91+
- Security risks.
92+
- Async errors.
93+
- Resource leaks.
94+
- Missing tests.
95+
- Sensitive information leaks.
96+
- Database transaction or connection lifecycle issues.
97+
98+
The Skill should not instruct the model to modify files. It should ask the model to produce candidate findings, while downstream validation, deduplication, and governance decide which findings are kept.
99+
100+
## Structured finding schema
101+
102+
Every high-confidence finding should be structured. Minimum fields:
103+
104+
| Field | Description |
105+
| --- | --- |
106+
| `severity` | `info`, `low`, `medium`, `high`, or `critical`. |
107+
| `category` | Example: `security`, `async`, `resource_leak`, `test_coverage`, `secrets`, `database_lifecycle`. |
108+
| `file` | Repository-relative file path. |
109+
| `line` | New-file line number from the diff. |
110+
| `title` | One-line summary. |
111+
| `evidence` | Concrete diff or code evidence. |
112+
| `recommendation` | Actionable fix guidance. |
113+
| `confidence` | `low`, `medium`, or `high`. |
114+
| `source` | Example: `skill`, `sandbox`, `filter`, or `fake_model`. |
115+
116+
Useful future fields:
117+
118+
- `fingerprint`: stable dedupe key.
119+
- `line_start` / `line_end`: line span.
120+
- `needs_human_review`: whether the result must be manually checked before promotion.
121+
- `raw_source`: optional trace for debugging.
122+
123+
Low-confidence or weakly evidenced findings should be stored as warnings or `needs_human_review`; they should not be mixed into high-confidence findings.
124+
125+
## Diff parsing contract
126+
127+
The diff parser should support unified diffs from saved files, PR patches, or local `git diff` output. It should produce a compact, model-friendly representation while preserving enough structure for line anchoring.
128+
129+
Recommended internal shape:
130+
131+
```text
132+
DiffFile
133+
old_path
134+
new_path
135+
status
136+
hunks[]
137+
138+
DiffHunk
139+
old_start
140+
old_count
141+
new_start
142+
new_count
143+
changed_lines[]
144+
145+
ChangedLine
146+
old_line_number
147+
new_line_number
148+
kind: added | removed | context
149+
text
150+
```
151+
152+
Rules:
153+
154+
- Findings should anchor to new-file changed lines whenever possible.
155+
- Deleted-only lines should not be used as final review comment anchors.
156+
- Renames should preserve both old and new paths.
157+
- Binary diffs should be summarized as unsupported instead of sent as raw binary content.
158+
159+
## Sandbox policy
160+
161+
The production path should be container-first or Cube/E2B-first. Local execution is useful for trusted development but should not be the default for untrusted diffs or model-generated commands.
162+
163+
Sandbox requirements:
164+
165+
- Mount repository inputs read-only.
166+
- Write outputs only under a controlled workspace/output directory.
167+
- Enforce command timeout.
168+
- Enforce output-size limits.
169+
- Pass only allowlisted environment variables.
170+
- Do not pass secrets into the sandbox.
171+
- Redact sensitive values from stdout, stderr, reports, and SQL records.
172+
- Record failures without crashing the entire review task.
173+
174+
The container Skill example in `examples/skills_with_container` is the closest starting point for this path.
175+
176+
## Filter governance
177+
178+
Filters should act as policy gates before and after sandbox/model work.
179+
180+
Pre-execution checks should deny or mark `needs_human_review` for:
181+
182+
- High-risk scripts or commands.
183+
- Forbidden repository paths.
184+
- Non-allowlisted network access.
185+
- Over-budget execution.
186+
- Requests that require secrets unavailable in dry-run mode.
187+
188+
Post-processing checks should:
189+
190+
- Validate finding schema.
191+
- Drop findings not anchored to changed lines.
192+
- Deduplicate repeated findings.
193+
- Downgrade low-confidence or speculative findings.
194+
- Redact secrets.
195+
196+
Every filter decision should be written to the report and database, including the decision, reason, and filter name.
197+
198+
## SQLite schema proposal
199+
200+
A minimal SQLite-backed implementation can use the existing SQL storage patterns. Suggested tables:
201+
202+
### `review_tasks`
203+
204+
- `id`
205+
- `repo_path`
206+
- `base_ref`
207+
- `head_ref`
208+
- `mode`: `dry_run` or `apply`
209+
- `status`
210+
- `created_at`
211+
- `completed_at`
212+
- `metadata_json`
213+
214+
### `review_input_summaries`
215+
216+
- `id`
217+
- `task_id`
218+
- `diff_sha256`
219+
- `file_count`
220+
- `hunk_count`
221+
- `changed_line_count`
222+
- `summary_json`
223+
224+
### `sandbox_runs`
225+
226+
- `id`
227+
- `task_id`
228+
- `runtime`: `container`, `cube`, or `local_dev`
229+
- `command`
230+
- `exit_code`
231+
- `duration_ms`
232+
- `stdout_excerpt`
233+
- `stderr_excerpt`
234+
- `status`
235+
236+
### `review_findings`
237+
238+
- `id`
239+
- `task_id`
240+
- `fingerprint`
241+
- `severity`
242+
- `category`
243+
- `file`
244+
- `line_start`
245+
- `line_end`
246+
- `title`
247+
- `evidence`
248+
- `recommendation`
249+
- `confidence`
250+
- `source`
251+
252+
### `filter_decisions`
253+
254+
- `id`
255+
- `task_id`
256+
- `finding_fingerprint`
257+
- `filter_name`
258+
- `decision`: `allow`, `deny`, `drop`, `merge`, `downgrade`, or `needs_human_review`
259+
- `reason`
260+
- `created_at`
261+
262+
### `review_reports`
263+
264+
- `id`
265+
- `task_id`
266+
- `json_path`
267+
- `markdown_path`
268+
- `summary`
269+
- `created_at`
270+
271+
### `monitoring_summaries`
272+
273+
- `id`
274+
- `task_id`
275+
- `total_duration_ms`
276+
- `sandbox_duration_ms`
277+
- `tool_call_count`
278+
- `filter_interception_count`
279+
- `finding_count`
280+
- `severity_distribution_json`
281+
- `exception_distribution_json`
282+
283+
## Dedupe and noise control
284+
285+
A stable finding fingerprint should include:
286+
287+
- normalized file path;
288+
- line span or nearest changed-line anchor;
289+
- category;
290+
- normalized title/evidence;
291+
- optional diff hunk hash.
292+
293+
The review pipeline should not surface duplicates for the same file, line, and category. Low-confidence issues should be kept in warnings or human-review sections instead of promoted to high-confidence findings.
294+
295+
## Monitoring and audit
296+
297+
Reports should include:
298+
299+
- total review duration;
300+
- sandbox execution duration;
301+
- tool call count;
302+
- filter interception count;
303+
- finding count;
304+
- severity distribution;
305+
- exception type distribution;
306+
- sandbox failures and timeouts;
307+
- filter deny / human-review decisions.
308+
309+
Audit events should record important lifecycle steps such as diff collection, skill loading, sandbox command requested, command allowed/denied, structured parsing success/failure, database write success/failure, and report rendering.
310+
311+
## Dry-run and fake-model mode
312+
313+
Dry-run should be the default. It should:
314+
315+
- parse the diff;
316+
- load the Skill policy;
317+
- exercise sandbox policy decisions when configured;
318+
- validate and filter findings;
319+
- write only local artifacts/database records;
320+
- render reports;
321+
- never post external comments;
322+
- never modify repository files.
323+
324+
Fake-model mode should make the pipeline testable without real model API keys. It can return deterministic findings from fixtures to verify parsing, storage, filtering, and report rendering.
325+
326+
## Future implementation phases
327+
328+
1. **Docs and scaffold**: design docs, example README, and `code-review` Skill scaffold.
329+
2. **Parser and schema**: unified diff parser, Pydantic finding models, fake-model path.
330+
3. **Sandbox and storage**: container-first sandbox runner and SQLite persistence.
331+
4. **Filters and metrics**: dedupe, redaction, human-review routing, monitoring summaries, and 8 fixture tests.
332+
5. **Optional integrations**: PR comments, CI entrypoint, dashboards, or Cube/E2B remote sandbox configuration.

0 commit comments

Comments
 (0)