Commit b5017cb
fix(crab-pf): require test prompts to match provider input format (#33)
## Summary
- Add system prompt rule requiring test prompts in promptfooconfig.yaml
to match the provider's expected input format
- Fixes false-positive verifications where providers expecting
structured input (JSON objects, keywords) pass smoke tests but fail eval
with generic prompts like "Hello"
## Test plan
- [ ] Run `crab pf` against app 09 (expects JSON-structured prompts) —
verify generated test cases are valid JSON
- [ ] Run `crab pf` against app 11 (expects keyword prompts) — verify
generated test cases use keywords like "summary"/"transactions"
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>1 parent 3f5ee99 commit b5017cb
1 file changed
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
| 101 | + | |
| 102 | + | |
101 | 103 | | |
102 | 104 | | |
103 | 105 | | |
| |||
0 commit comments