Skip to content

Commit 9eb612d

Browse files
MrFlounderclaude
MrFlounder
and
claude
committed
docs: add promptfoo plugin documentation
- Add crab pf section to README - Add detailed docs/promptfoo-plugin.md Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 951b191 commit 9eb612d

2 files changed

Lines changed: 142 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,25 @@ crab config scan # Auto-detect .env files and ports
134134
crab config # Show current configuration
135135
```
136136

137+
### Promptfoo Target Discovery (`crab pf`)
138+
139+
AI-powered agent that analyzes any target and generates working promptfoo configurations.
140+
141+
```bash
142+
crab pf install # Install the plugin
143+
crab pf --file target.txt # Analyze from file
144+
crab pf "curl -X POST http://..." # Analyze curl command
145+
crab pf --file api.json --output ./config # Specify output dir
146+
crab pf --file spec.yaml --verbose # Show detailed output
147+
crab pf uninstall # Remove the plugin
148+
```
149+
150+
**Supported formats:** curl commands, OpenAPI specs, Postman collections, Burp exports, plain text descriptions
151+
152+
**Requirements:** Node.js, `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`
153+
154+
The agent probes the target, figures out the protocol (HTTP, WebSocket, polling, etc.), generates the config, and verifies it works.
155+
137156
### Other Commands
138157

139158
```bash

‎docs/promptfoo-plugin.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Promptfoo Target Discovery Plugin
2+
3+
An AI-powered agent that analyzes any target specification and generates working [promptfoo](https://promptfoo.dev) configurations for red-teaming.
4+
5+
## Installation
6+
7+
```bash
8+
crab pf install
9+
```
10+
11+
**Requirements:**
12+
- Node.js 18+
13+
- API key: `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`
14+
15+
## Usage
16+
17+
```bash
18+
# From a file (any format)
19+
crab pf --file target.txt
20+
21+
# From a curl command
22+
crab pf "curl -X POST http://localhost:8080/chat -d '{\"message\":\"hi\"}'"
23+
24+
# Specify output directory
25+
crab pf --file api-spec.json --output ./my-config
26+
27+
# Use Anthropic instead of OpenAI
28+
crab pf --file target.txt --provider anthropic:claude-sonnet-4-20250514
29+
30+
# Verbose output (see agent reasoning)
31+
crab pf --file target.txt --verbose
32+
```
33+
34+
## Supported Input Formats
35+
36+
| Format | Example |
37+
|--------|---------|
38+
| **Curl** | `curl -X POST http://api.example.com/chat -H "Authorization: Bearer $TOKEN" -d '{"message":"hi"}'` |
39+
| **OpenAPI/Swagger** | `openapi.json`, `swagger.yaml` |
40+
| **Postman** | Exported collection JSON |
41+
| **Burp Suite** | Exported XML |
42+
| **Plain text** | Any description of the API |
43+
44+
## What It Does
45+
46+
1. **Parses** the input to understand the target
47+
2. **Probes** the target to verify connectivity and discover request/response format
48+
3. **Generates** a promptfoo YAML config (and custom provider.js if needed)
49+
4. **Verifies** the config works with a mini red-team test
50+
51+
## Output
52+
53+
The agent creates:
54+
55+
```
56+
output-dir/
57+
├── promptfooconfig.yaml # Main config file
58+
├── provider.js # Custom provider (if needed for WebSocket, polling, etc.)
59+
└── package.json # Dependencies (if provider.js uses external packages)
60+
```
61+
62+
## Target Types
63+
64+
| Type | Provider |
65+
|------|----------|
66+
| Simple HTTP | Built-in `http` provider |
67+
| HTTP with auth | Built-in `http` provider with env vars |
68+
| Session-based | Built-in `http` provider with `sessionParser` |
69+
| WebSocket | Custom `provider.js` |
70+
| Async/Polling | Custom `provider.js` |
71+
| GraphQL | Built-in `http` provider |
72+
73+
## Example
74+
75+
```bash
76+
# Start your target
77+
cd my-api && npm start
78+
79+
# Generate config
80+
export OPENAI_API_KEY=sk-...
81+
crab pf --file my-api-docs.txt --output ./redteam-config --verbose
82+
83+
# Run red-team
84+
cd ./redteam-config
85+
promptfoo eval
86+
```
87+
88+
## Options
89+
90+
| Flag | Description |
91+
|------|-------------|
92+
| `--file`, `-f` | Input file path |
93+
| `--output`, `-o` | Output directory (default: current dir) |
94+
| `--provider` | LLM provider (default: `openai:gpt-4o`) |
95+
| `--verbose`, `-v` | Show detailed agent output |
96+
| `--max-turns` | Max agent iterations (default: 30) |
97+
98+
## Environment Variables
99+
100+
| Variable | Description |
101+
|----------|-------------|
102+
| `OPENAI_API_KEY` | OpenAI API key |
103+
| `ANTHROPIC_API_KEY` | Anthropic API key |
104+
| `DISCOVERY_PROVIDER` | Default provider (e.g., `anthropic:claude-sonnet-4-20250514`) |
105+
106+
## Uninstall
107+
108+
```bash
109+
crab pf uninstall
110+
```
111+
112+
## How It Works
113+
114+
The plugin uses an LLM agent loop with tools:
115+
116+
1. **probe** - Send HTTP requests to test connectivity
117+
2. **probe_ws** - Test WebSocket endpoints
118+
3. **write_config** - Generate promptfoo YAML
119+
4. **write_provider** - Generate custom JS/Python providers
120+
5. **verify** - Run `promptfoo eval` to test
121+
6. **done** - Signal completion
122+
123+
The agent decides which tools to use based on the target. For simple HTTP APIs, it uses the built-in provider. For complex targets (WebSocket, polling), it generates custom provider code.

0 commit comments

Comments
 (0)