|
| 1 | +# Code Review Agent Design |
| 2 | + |
| 3 | +This page describes a proposed automatic code-review Agent that combines Agent Skills, sandboxed execution, structured findings, filters, telemetry, and SQL storage. It is a design and scaffold document: it defines the expected architecture and contracts before a production-ready implementation is added. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The code-review Agent receives a unified diff, a PR patch, or local repository changes and produces a structured review report. A future runnable implementation should support inputs such as: |
| 8 | + |
| 9 | +- `--diff-file`: read a saved unified diff or patch. |
| 10 | +- `--repo-path`: read local working-tree changes from a repository. |
| 11 | +- Test fixtures: run deterministic samples in dry-run or fake-model mode. |
| 12 | + |
| 13 | +Expected outputs are: |
| 14 | + |
| 15 | +- `review_report.json`: machine-readable findings, filter decisions, sandbox runs, and metrics. |
| 16 | +- `review_report.md`: human-readable summary. |
| 17 | +- SQL records for review tasks, input summaries, sandbox runs, findings, filter decisions, monitoring summaries, and final reports. |
| 18 | + |
| 19 | +## Non-goals for the scaffold |
| 20 | + |
| 21 | +The first contribution should stay small and reviewable. It should not claim to implement the full review bot. |
| 22 | + |
| 23 | +- No GitHub or PR comment posting. |
| 24 | +- No automatic file modification or fix application. |
| 25 | +- No host execution of model-generated commands. |
| 26 | +- No production scheduler, webhook server, or credentials management. |
| 27 | +- No complete sandbox, database, or model loop implementation yet. |
| 28 | + |
| 29 | +## Architecture |
| 30 | + |
| 31 | +A complete implementation should follow this flow: |
| 32 | + |
| 33 | +```text |
| 34 | +diff / repo changes |
| 35 | + -> diff parser |
| 36 | + -> code-review Skill |
| 37 | + -> Filter governance |
| 38 | + -> sandboxed checks |
| 39 | + -> structured finding validation |
| 40 | + -> dedupe and noise filtering |
| 41 | + -> SQLite storage |
| 42 | + -> JSON/Markdown report |
| 43 | +``` |
| 44 | + |
| 45 | +The design separates three concerns: |
| 46 | + |
| 47 | +1. **Review policy** lives in the `code-review` Skill and reference docs. |
| 48 | +2. **Execution safety** lives in sandbox and Filter layers. |
| 49 | +3. **Auditability** lives in structured output, SQL storage, and monitoring summaries. |
| 50 | + |
| 51 | +## Existing building blocks |
| 52 | + |
| 53 | +The implementation should reuse existing tRPC-Agent patterns instead of inventing a parallel framework. |
| 54 | + |
| 55 | +- Skills: [`trpc_agent_sdk/skills/__init__.py`](../../../trpc_agent_sdk/skills/__init__.py) |
| 56 | + - `SkillToolSet` |
| 57 | + - `create_default_skill_repository` |
| 58 | + - `SkillLoadTool` |
| 59 | + - `SkillRunTool` |
| 60 | +- Skill examples: |
| 61 | + - [`examples/skills`](../../../examples/skills) |
| 62 | + - [`examples/skills_with_container`](../../../examples/skills_with_container) |
| 63 | + - [`examples/skills_with_cube`](../../../examples/skills_with_cube) |
| 64 | +- Code execution: |
| 65 | + - [`BaseCodeExecutor`](../../../trpc_agent_sdk/code_executors/_base_code_executor.py) |
| 66 | + - [`ContainerCodeExecutor`](../../../trpc_agent_sdk/code_executors/container/_container_code_executor.py) |
| 67 | +- Filter governance: |
| 68 | + - [`FilterABC`](../../../trpc_agent_sdk/abc/_filter.py) |
| 69 | + - [`FilterResult`](../../../trpc_agent_sdk/abc/_filter.py) |
| 70 | +- SQL storage: |
| 71 | + - [`SqlStorage`](../../../trpc_agent_sdk/storage/_sql.py) |
| 72 | + - [`SqlKey`](../../../trpc_agent_sdk/storage/_sql.py) |
| 73 | + - [`SqlCondition`](../../../trpc_agent_sdk/storage/_sql.py) |
| 74 | + |
| 75 | +## Skill design |
| 76 | + |
| 77 | +The code-review Skill is the reusable review policy package. A first scaffold can live under the example directory: |
| 78 | + |
| 79 | +```text |
| 80 | +examples/code_review_agent/skills/code-review/ |
| 81 | + SKILL.md |
| 82 | + references/ |
| 83 | + finding_schema.md |
| 84 | + security_boundary.md |
| 85 | + scripts/ |
| 86 | + # future static checks / diff helpers |
| 87 | +``` |
| 88 | + |
| 89 | +`SKILL.md` should define the review workflow and require structured output. Future rule documents should cover at least four categories from the issue: |
| 90 | + |
| 91 | +- Security risks. |
| 92 | +- Async errors. |
| 93 | +- Resource leaks. |
| 94 | +- Missing tests. |
| 95 | +- Sensitive information leaks. |
| 96 | +- Database transaction or connection lifecycle issues. |
| 97 | + |
| 98 | +The Skill should not instruct the model to modify files. It should ask the model to produce candidate findings, while downstream validation, deduplication, and governance decide which findings are kept. |
| 99 | + |
| 100 | +## Structured finding schema |
| 101 | + |
| 102 | +Every high-confidence finding should be structured. Minimum fields: |
| 103 | + |
| 104 | +| Field | Description | |
| 105 | +| --- | --- | |
| 106 | +| `severity` | `info`, `low`, `medium`, `high`, or `critical`. | |
| 107 | +| `category` | Example: `security`, `async`, `resource_leak`, `test_coverage`, `secrets`, `database_lifecycle`. | |
| 108 | +| `file` | Repository-relative file path. | |
| 109 | +| `line` | New-file line number from the diff. | |
| 110 | +| `title` | One-line summary. | |
| 111 | +| `evidence` | Concrete diff or code evidence. | |
| 112 | +| `recommendation` | Actionable fix guidance. | |
| 113 | +| `confidence` | `low`, `medium`, or `high`. | |
| 114 | +| `source` | Example: `skill`, `sandbox`, `filter`, or `fake_model`. | |
| 115 | + |
| 116 | +Useful future fields: |
| 117 | + |
| 118 | +- `fingerprint`: stable dedupe key. |
| 119 | +- `line_start` / `line_end`: line span. |
| 120 | +- `needs_human_review`: whether the result must be manually checked before promotion. |
| 121 | +- `raw_source`: optional trace for debugging. |
| 122 | + |
| 123 | +Low-confidence or weakly evidenced findings should be stored as warnings or `needs_human_review`; they should not be mixed into high-confidence findings. |
| 124 | + |
| 125 | +## Diff parsing contract |
| 126 | + |
| 127 | +The diff parser should support unified diffs from saved files, PR patches, or local `git diff` output. It should produce a compact, model-friendly representation while preserving enough structure for line anchoring. |
| 128 | + |
| 129 | +Recommended internal shape: |
| 130 | + |
| 131 | +```text |
| 132 | +DiffFile |
| 133 | + old_path |
| 134 | + new_path |
| 135 | + status |
| 136 | + hunks[] |
| 137 | +
|
| 138 | +DiffHunk |
| 139 | + old_start |
| 140 | + old_count |
| 141 | + new_start |
| 142 | + new_count |
| 143 | + changed_lines[] |
| 144 | +
|
| 145 | +ChangedLine |
| 146 | + old_line_number |
| 147 | + new_line_number |
| 148 | + kind: added | removed | context |
| 149 | + text |
| 150 | +``` |
| 151 | + |
| 152 | +Rules: |
| 153 | + |
| 154 | +- Findings should anchor to new-file changed lines whenever possible. |
| 155 | +- Deleted-only lines should not be used as final review comment anchors. |
| 156 | +- Renames should preserve both old and new paths. |
| 157 | +- Binary diffs should be summarized as unsupported instead of sent as raw binary content. |
| 158 | + |
| 159 | +## Sandbox policy |
| 160 | + |
| 161 | +The production path should be container-first or Cube/E2B-first. Local execution is useful for trusted development but should not be the default for untrusted diffs or model-generated commands. |
| 162 | + |
| 163 | +Sandbox requirements: |
| 164 | + |
| 165 | +- Mount repository inputs read-only. |
| 166 | +- Write outputs only under a controlled workspace/output directory. |
| 167 | +- Enforce command timeout. |
| 168 | +- Enforce output-size limits. |
| 169 | +- Pass only allowlisted environment variables. |
| 170 | +- Do not pass secrets into the sandbox. |
| 171 | +- Redact sensitive values from stdout, stderr, reports, and SQL records. |
| 172 | +- Record failures without crashing the entire review task. |
| 173 | + |
| 174 | +The container Skill example in `examples/skills_with_container` is the closest starting point for this path. |
| 175 | + |
| 176 | +## Filter governance |
| 177 | + |
| 178 | +Filters should act as policy gates before and after sandbox/model work. |
| 179 | + |
| 180 | +Pre-execution checks should deny or mark `needs_human_review` for: |
| 181 | + |
| 182 | +- High-risk scripts or commands. |
| 183 | +- Forbidden repository paths. |
| 184 | +- Non-allowlisted network access. |
| 185 | +- Over-budget execution. |
| 186 | +- Requests that require secrets unavailable in dry-run mode. |
| 187 | + |
| 188 | +Post-processing checks should: |
| 189 | + |
| 190 | +- Validate finding schema. |
| 191 | +- Drop findings not anchored to changed lines. |
| 192 | +- Deduplicate repeated findings. |
| 193 | +- Downgrade low-confidence or speculative findings. |
| 194 | +- Redact secrets. |
| 195 | + |
| 196 | +Every filter decision should be written to the report and database, including the decision, reason, and filter name. |
| 197 | + |
| 198 | +## SQLite schema proposal |
| 199 | + |
| 200 | +A minimal SQLite-backed implementation can use the existing SQL storage patterns. Suggested tables: |
| 201 | + |
| 202 | +### `review_tasks` |
| 203 | + |
| 204 | +- `id` |
| 205 | +- `repo_path` |
| 206 | +- `base_ref` |
| 207 | +- `head_ref` |
| 208 | +- `mode`: `dry_run` or `apply` |
| 209 | +- `status` |
| 210 | +- `created_at` |
| 211 | +- `completed_at` |
| 212 | +- `metadata_json` |
| 213 | + |
| 214 | +### `review_input_summaries` |
| 215 | + |
| 216 | +- `id` |
| 217 | +- `task_id` |
| 218 | +- `diff_sha256` |
| 219 | +- `file_count` |
| 220 | +- `hunk_count` |
| 221 | +- `changed_line_count` |
| 222 | +- `summary_json` |
| 223 | + |
| 224 | +### `sandbox_runs` |
| 225 | + |
| 226 | +- `id` |
| 227 | +- `task_id` |
| 228 | +- `runtime`: `container`, `cube`, or `local_dev` |
| 229 | +- `command` |
| 230 | +- `exit_code` |
| 231 | +- `duration_ms` |
| 232 | +- `stdout_excerpt` |
| 233 | +- `stderr_excerpt` |
| 234 | +- `status` |
| 235 | + |
| 236 | +### `review_findings` |
| 237 | + |
| 238 | +- `id` |
| 239 | +- `task_id` |
| 240 | +- `fingerprint` |
| 241 | +- `severity` |
| 242 | +- `category` |
| 243 | +- `file` |
| 244 | +- `line_start` |
| 245 | +- `line_end` |
| 246 | +- `title` |
| 247 | +- `evidence` |
| 248 | +- `recommendation` |
| 249 | +- `confidence` |
| 250 | +- `source` |
| 251 | + |
| 252 | +### `filter_decisions` |
| 253 | + |
| 254 | +- `id` |
| 255 | +- `task_id` |
| 256 | +- `finding_fingerprint` |
| 257 | +- `filter_name` |
| 258 | +- `decision`: `allow`, `deny`, `drop`, `merge`, `downgrade`, or `needs_human_review` |
| 259 | +- `reason` |
| 260 | +- `created_at` |
| 261 | + |
| 262 | +### `review_reports` |
| 263 | + |
| 264 | +- `id` |
| 265 | +- `task_id` |
| 266 | +- `json_path` |
| 267 | +- `markdown_path` |
| 268 | +- `summary` |
| 269 | +- `created_at` |
| 270 | + |
| 271 | +### `monitoring_summaries` |
| 272 | + |
| 273 | +- `id` |
| 274 | +- `task_id` |
| 275 | +- `total_duration_ms` |
| 276 | +- `sandbox_duration_ms` |
| 277 | +- `tool_call_count` |
| 278 | +- `filter_interception_count` |
| 279 | +- `finding_count` |
| 280 | +- `severity_distribution_json` |
| 281 | +- `exception_distribution_json` |
| 282 | + |
| 283 | +## Dedupe and noise control |
| 284 | + |
| 285 | +A stable finding fingerprint should include: |
| 286 | + |
| 287 | +- normalized file path; |
| 288 | +- line span or nearest changed-line anchor; |
| 289 | +- category; |
| 290 | +- normalized title/evidence; |
| 291 | +- optional diff hunk hash. |
| 292 | + |
| 293 | +The review pipeline should not surface duplicates for the same file, line, and category. Low-confidence issues should be kept in warnings or human-review sections instead of promoted to high-confidence findings. |
| 294 | + |
| 295 | +## Monitoring and audit |
| 296 | + |
| 297 | +Reports should include: |
| 298 | + |
| 299 | +- total review duration; |
| 300 | +- sandbox execution duration; |
| 301 | +- tool call count; |
| 302 | +- filter interception count; |
| 303 | +- finding count; |
| 304 | +- severity distribution; |
| 305 | +- exception type distribution; |
| 306 | +- sandbox failures and timeouts; |
| 307 | +- filter deny / human-review decisions. |
| 308 | + |
| 309 | +Audit events should record important lifecycle steps such as diff collection, skill loading, sandbox command requested, command allowed/denied, structured parsing success/failure, database write success/failure, and report rendering. |
| 310 | + |
| 311 | +## Dry-run and fake-model mode |
| 312 | + |
| 313 | +Dry-run should be the default. It should: |
| 314 | + |
| 315 | +- parse the diff; |
| 316 | +- load the Skill policy; |
| 317 | +- exercise sandbox policy decisions when configured; |
| 318 | +- validate and filter findings; |
| 319 | +- write only local artifacts/database records; |
| 320 | +- render reports; |
| 321 | +- never post external comments; |
| 322 | +- never modify repository files. |
| 323 | + |
| 324 | +Fake-model mode should make the pipeline testable without real model API keys. It can return deterministic findings from fixtures to verify parsing, storage, filtering, and report rendering. |
| 325 | + |
| 326 | +## Future implementation phases |
| 327 | + |
| 328 | +1. **Docs and scaffold**: design docs, example README, and `code-review` Skill scaffold. |
| 329 | +2. **Parser and schema**: unified diff parser, Pydantic finding models, fake-model path. |
| 330 | +3. **Sandbox and storage**: container-first sandbox runner and SQLite persistence. |
| 331 | +4. **Filters and metrics**: dedupe, redaction, human-review routing, monitoring summaries, and 8 fixture tests. |
| 332 | +5. **Optional integrations**: PR comments, CI entrypoint, dashboards, or Cube/E2B remote sandbox configuration. |
0 commit comments