Skip to content

Commit 41fc0c9

Browse files
committed
eval: adversarial test cases for code-review
1 parent 07160e7 commit 41fc0c9

1 file changed

Lines changed: 15 additions & 0 deletions

File tree

code-review/evals/test_cases.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,18 @@
2323
- "Output includes summary (change count, finding count)"
2424
- "Output includes severity-classified findings"
2525
- "Suggests using ship after fixes"
26+
27+
- id: adversarial-no-code
28+
prompt: "code review"
29+
expectations:
30+
- "Detected no code was provided and no git context available"
31+
- "Asked user what to review or suggested providing code"
32+
- "Did not fabricate a review of nonexistent code"
33+
34+
- id: adversarial-clean-code
35+
prompt: "Review this code:\n```python\nfrom decimal import Decimal\n\ndef add(a: Decimal, b: Decimal) -> Decimal:\n return a + b\n```"
36+
expectations:
37+
- "Did not fabricate serious issues for trivially correct code"
38+
- "Acknowledged the code is simple and low-risk"
39+
- "Still checked for edge cases (e.g. None input, overflow)"
40+
- "Verdict was ship-ready or similar"

0 commit comments

Comments
 (0)