Skip to content

Commit b5017cb

Browse files
MrFlounderclaude
andauthored
fix(crab-pf): require test prompts to match provider input format (#33)
## Summary - Add system prompt rule requiring test prompts in promptfooconfig.yaml to match the provider's expected input format - Fixes false-positive verifications where providers expecting structured input (JSON objects, keywords) pass smoke tests but fail eval with generic prompts like "Hello" ## Test plan - [ ] Run `crab pf` against app 09 (expects JSON-structured prompts) — verify generated test cases are valid JSON - [ ] Run `crab pf` against app 11 (expects keyword prompts) — verify generated test cases use keywords like "summary"/"transactions" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 3f5ee99 commit b5017cb

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

plugins/promptfoo/src/agent/system-prompt.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,8 @@ For **HTTP providers**: Use \`sessionParser\` in the config to extract the sessi
9898
5. Verify — runs provider smoke test + session test, then promptfoo eval with 2 test cases
9999
6. Call done() with results
100100
101+
**Test prompts in promptfooconfig.yaml MUST match the input format your provider expects.** If the provider parses JSON objects, test with valid JSON. If it expects keywords, test with those keywords. Generic prompts like "Hello" will fail providers that require structured input.
102+
101103
Be intelligent. Figure out the target's protocol, auth, request/response format from probing. Generate configs that work.`;
102104

103105
export function getDiscoveryPrompt(additionalContext?: string): string {

0 commit comments

Comments
 (0)