|
| 1 | +# MCP Test Harness |
| 2 | + |
| 3 | +A universal testing tool for Model Context Protocol (MCP) servers. Test any MCP server implementation for protocol compliance, performance, and reliability. |
| 4 | + |
| 5 | +[](https://crates.io/crates/mcp-test-harness) |
| 6 | +[](https://docs.rs/mcp-test-harness) |
| 7 | +[](LICENSE) |
| 8 | + |
| 9 | +## Features |
| 10 | + |
| 11 | +- 🔧 **Universal MCP Server Testing** - Works with any MCP server implementation |
| 12 | +- 🚀 **Multiple Transport Support** - stdio, HTTP, WebSocket |
| 13 | +- 📊 **Performance Monitoring** - Built-in benchmarking and regression detection |
| 14 | +- 🛡️ **Protocol Compliance** - Comprehensive MCP protocol validation |
| 15 | +- 🐳 **Docker Ready** - Containerized testing for CI/CD pipelines |
| 16 | +- 📋 **Flexible Configuration** - YAML/JSON configuration with templates |
| 17 | +- �� **Auto-Discovery** - Automatic MCP server detection |
| 18 | +- 📈 **Rich Reporting** - Table, JSON, YAML output formats |
| 19 | + |
| 20 | +## Quick Start |
| 21 | + |
| 22 | +### Installation |
| 23 | + |
| 24 | +```bash |
| 25 | +# Install from crates.io |
| 26 | +cargo install mcp-test-harness |
| 27 | + |
| 28 | +# Or build from source |
| 29 | +git clone https://github.com/rustic-ai/codeprism |
| 30 | +cd prism/mcp-test-harness-standalone |
| 31 | +cargo install --path . |
| 32 | +``` |
| 33 | + |
| 34 | +### Basic Usage |
| 35 | + |
| 36 | +```bash |
| 37 | +# Test a stdio MCP server |
| 38 | +mcp-test-harness test \ |
| 39 | + --config configs/examples/basic-mcp-server.yaml \ |
| 40 | + --server-cmd "node my-mcp-server.js" |
| 41 | + |
| 42 | +# Test an HTTP MCP server |
| 43 | +mcp-test-harness test \ |
| 44 | + --config my-http-server-config.yaml |
| 45 | + |
| 46 | +# Generate a configuration template |
| 47 | +mcp-test-harness template --server-type filesystem --output my-config.yaml |
| 48 | + |
| 49 | +# Discover running MCP servers |
| 50 | +mcp-test-harness discover --port-range "3000-3010" |
| 51 | + |
| 52 | +# Run performance benchmarks |
| 53 | +mcp-test-harness benchmark \ |
| 54 | + --config my-config.yaml \ |
| 55 | + --iterations 1000 \ |
| 56 | + --duration 60 |
| 57 | +``` |
| 58 | + |
| 59 | +### Docker Usage |
| 60 | + |
| 61 | +```bash |
| 62 | +# Pull the image |
| 63 | +docker pull mcp-test-harness:latest |
| 64 | + |
| 65 | +# Run tests with Docker |
| 66 | +docker run --rm -v $(pwd)/configs:/configs \ |
| 67 | + mcp-test-harness test --config /configs/my-server.yaml |
| 68 | + |
| 69 | +# Use in CI/CD |
| 70 | +docker run --rm -v $(pwd):/workspace -w /workspace \ |
| 71 | + mcp-test-harness test --config test-config.yaml --output json |
| 72 | +``` |
| 73 | + |
| 74 | +## Configuration |
| 75 | + |
| 76 | +### Basic Configuration Structure |
| 77 | + |
| 78 | +```yaml |
| 79 | +global: |
| 80 | + max_global_concurrency: 2 |
| 81 | + timeout_seconds: 30 |
| 82 | + fail_fast: false |
| 83 | + |
| 84 | +server: |
| 85 | + transport: "stdio" # or "http", "websocket" |
| 86 | + command: "node server.js" |
| 87 | + args: ["--port", "3000"] |
| 88 | + working_dir: "/path/to/server" |
| 89 | + env: |
| 90 | + NODE_ENV: "test" |
| 91 | + |
| 92 | +performance: |
| 93 | + enable_monitoring: true |
| 94 | + baseline_storage_path: "baselines/" |
| 95 | + regression_detection: |
| 96 | + warning_threshold_percent: 25.0 |
| 97 | + error_threshold_percent: 50.0 |
| 98 | + |
| 99 | +test_suites: |
| 100 | + - name: "MCP Protocol Compliance" |
| 101 | + test_cases: |
| 102 | + - id: "initialize" |
| 103 | + tool_name: "initialize" |
| 104 | + enabled: true |
| 105 | + input_params: |
| 106 | + protocolVersion: "2024-11-05" |
| 107 | + expected: |
| 108 | + patterns: |
| 109 | + - key: "protocolVersion" |
| 110 | + validation: { type: "exists" } |
| 111 | + required: true |
| 112 | +``` |
| 113 | +
|
| 114 | +### Server Types |
| 115 | +
|
| 116 | +#### stdio Server |
| 117 | +```yaml |
| 118 | +server: |
| 119 | + transport: "stdio" |
| 120 | + command: "node" |
| 121 | + args: ["server.js"] |
| 122 | +``` |
| 123 | +
|
| 124 | +#### HTTP Server |
| 125 | +```yaml |
| 126 | +server: |
| 127 | + transport: "http" |
| 128 | + url: "http://localhost:3000" |
| 129 | + connection_timeout: 10 |
| 130 | +``` |
| 131 | +
|
| 132 | +#### WebSocket Server |
| 133 | +```yaml |
| 134 | +server: |
| 135 | + transport: "websocket" |
| 136 | + url: "ws://localhost:3000" |
| 137 | +``` |
| 138 | +
|
| 139 | +## Validation Patterns |
| 140 | +
|
| 141 | +The test harness supports comprehensive response validation: |
| 142 | +
|
| 143 | +```yaml |
| 144 | +expected: |
| 145 | + patterns: |
| 146 | + # Check if field exists |
| 147 | + - key: "status" |
| 148 | + validation: { type: "exists" } |
| 149 | + required: true |
| 150 | + |
| 151 | + # Check exact value |
| 152 | + - key: "status" |
| 153 | + validation: { type: "equals", value: "ok" } |
| 154 | + required: true |
| 155 | + |
| 156 | + # Check numeric range |
| 157 | + - key: "count" |
| 158 | + validation: { type: "range", min: 1.0, max: 100.0 } |
| 159 | + required: true |
| 160 | + |
| 161 | + # Check array structure |
| 162 | + - key: "items" |
| 163 | + validation: { type: "array" } |
| 164 | + required: true |
| 165 | + |
| 166 | + # Check array length |
| 167 | + - key: "items" |
| 168 | + validation: { type: "array_min_length", min_length: 1 } |
| 169 | + required: true |
| 170 | + |
| 171 | + # Check boolean value |
| 172 | + - key: "enabled" |
| 173 | + validation: { type: "boolean", value: true } |
| 174 | + required: true |
| 175 | +``` |
| 176 | +
|
| 177 | +## Performance Testing |
| 178 | +
|
| 179 | +### Basic Performance Requirements |
| 180 | +
|
| 181 | +```yaml |
| 182 | +performance_requirements: |
| 183 | + max_execution_time_ms: 5000 |
| 184 | + max_memory_usage_mb: 64 |
| 185 | +``` |
| 186 | +
|
| 187 | +### Benchmarking |
| 188 | +
|
| 189 | +```bash |
| 190 | +# Run benchmarks |
| 191 | +mcp-test-harness benchmark \ |
| 192 | + --config server-config.yaml \ |
| 193 | + --iterations 1000 \ |
| 194 | + --duration 60 \ |
| 195 | + --output json > benchmark-results.json |
| 196 | +``` |
| 197 | + |
| 198 | +## Templates |
| 199 | + |
| 200 | +Generate configuration templates for common server types: |
| 201 | + |
| 202 | +```bash |
| 203 | +# Filesystem server template |
| 204 | +mcp-test-harness template --server-type filesystem --output fs-config.yaml |
| 205 | + |
| 206 | +# Database server template |
| 207 | +mcp-test-harness template --server-type database --output db-config.yaml |
| 208 | + |
| 209 | +# API wrapper template |
| 210 | +mcp-test-harness template --server-type api --output api-config.yaml |
| 211 | + |
| 212 | +# List available templates |
| 213 | +mcp-test-harness list |
| 214 | +``` |
| 215 | + |
| 216 | +## CI/CD Integration |
| 217 | + |
| 218 | +### GitHub Actions |
| 219 | + |
| 220 | +```yaml |
| 221 | +name: MCP Server Tests |
| 222 | +on: [push, pull_request] |
| 223 | + |
| 224 | +jobs: |
| 225 | + test: |
| 226 | + runs-on: ubuntu-latest |
| 227 | + steps: |
| 228 | + - uses: actions/checkout@v4 |
| 229 | + - name: Test MCP Server |
| 230 | + run: | |
| 231 | + docker run --rm -v $(pwd):/workspace -w /workspace \ |
| 232 | + mcp-test-harness test \ |
| 233 | + --config .github/mcp-test-config.yaml \ |
| 234 | + --output json > test-results.json |
| 235 | + - name: Upload Results |
| 236 | + uses: actions/upload-artifact@v3 |
| 237 | + with: |
| 238 | + name: test-results |
| 239 | + path: test-results.json |
| 240 | +``` |
| 241 | +
|
| 242 | +### GitLab CI |
| 243 | +
|
| 244 | +```yaml |
| 245 | +mcp-tests: |
| 246 | + image: mcp-test-harness:latest |
| 247 | + script: |
| 248 | + - mcp-test-harness test --config ci-config.yaml --output junit > results.xml |
| 249 | + artifacts: |
| 250 | + reports: |
| 251 | + junit: results.xml |
| 252 | +``` |
| 253 | +
|
| 254 | +## Contributing |
| 255 | +
|
| 256 | +1. Fork the repository |
| 257 | +2. Create your feature branch (`git checkout -b feature/amazing-feature`) |
| 258 | +3. Commit your changes (`git commit -m 'Add amazing feature'`) |
| 259 | +4. Push to the branch (`git push origin feature/amazing-feature`) |
| 260 | +5. Open a Pull Request |
| 261 | + |
| 262 | +## License |
| 263 | + |
| 264 | +This project is licensed under either of |
| 265 | + |
| 266 | +* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0) |
| 267 | +* MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT) |
| 268 | + |
| 269 | +at your option. |
| 270 | + |
| 271 | +## Support |
| 272 | + |
| 273 | +- 📖 [Documentation](https://docs.rs/mcp-test-harness) |
| 274 | +- 🐛 [Issue Tracker](https://github.com/rustic-ai/codeprism/issues) |
| 275 | +- 💬 [Discussions](https://github.com/rustic-ai/codeprism/discussions) |
| 276 | +- �� [Email Support](mailto:team@codeprism.ai) |
0 commit comments