Skip to content

Commit c506c4a

Browse files
committed
eval: adversarial test cases for qa
1 parent 90c98ea commit c506c4a

1 file changed

Lines changed: 14 additions & 0 deletions

File tree

qa/evals/test_cases.yaml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,17 @@
1313
- "Checked for page load errors"
1414
- "Produced a QA report with health score"
1515
- "Suggested ship if health score is acceptable"
16+
17+
- id: adversarial-no-changes
18+
prompt: "qa"
19+
expectations:
20+
- "Detected no changes to test (on default branch with clean state)"
21+
- "Reported 'nothing to test' or asked what to test"
22+
- "Did not fabricate test results"
23+
24+
- id: adversarial-not-git-repo
25+
prompt: "run qa on this project"
26+
expectations:
27+
- "Detected not in a git repo (if applicable)"
28+
- "Asked user what to test instead of failing silently"
29+
- "Did not produce a fake health score"

0 commit comments

Comments
 (0)