Skip to content

Commit e51ccc3

Browse files
committed
feat(mcp): implement standalone MCP test harness binary
- Create self-contained binary independent of CodePrism codebase - Support multiple transport types: stdio, HTTP, WebSocket - Universal MCP server configuration system - Comprehensive validation engine with pattern matching - Performance benchmarking and monitoring capabilities - Cross-platform build scripts and Docker support - Template system for common MCP server types - Command-line interface with validation, testing, and discovery Key Features: - Zero CodePrism dependencies, pure MCP testing - Works with any MCP server implementation - Built-in configuration templates (filesystem, database, API, custom) - Rich output formats (table, JSON, YAML) - Automatic server discovery and profiling - Comprehensive protocol compliance validation - One-command installation via cargo install closes #102
1 parent e039c2c commit e51ccc3

10 files changed

Lines changed: 3362 additions & 0 deletions

File tree

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
[package]
2+
name = "mcp-test-harness"
3+
version = "0.1.0"
4+
edition = "2021"
5+
description = "Standalone test harness for Model Context Protocol (MCP) servers"
6+
license = "MIT OR Apache-2.0"
7+
repository = "https://github.com/rustic-ai/codeprism"
8+
keywords = ["mcp", "testing", "protocol", "ai", "llm"]
9+
categories = ["development-tools::testing", "command-line-utilities"]
10+
authors = ["CodePrism Team <team@codeprism.ai>"]
11+
readme = "README.md"
12+
13+
[[bin]]
14+
name = "mcp-test-harness"
15+
path = "src/main.rs"
16+
17+
[dependencies]
18+
# Core dependencies - minimal and focused
19+
tokio = { version = "1.0", features = ["full"] }
20+
serde = { version = "1.0", features = ["derive"] }
21+
serde_json = "1.0"
22+
serde_yaml = "0.9"
23+
clap = { version = "4.0", features = ["derive"] }
24+
anyhow = "1.0"
25+
thiserror = "1.0"
26+
tracing = "0.1"
27+
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
28+
chrono = { version = "0.4", features = ["serde"] }
29+
30+
# HTTP and JSON-RPC for MCP communication
31+
reqwest = { version = "0.12", features = ["json"] }
32+
jsonrpc-core = "18.0"
33+
34+
# Process management for server lifecycle
35+
process_control = "4.0"
36+
which = "6.0"
37+
38+
# Configuration and validation
39+
jsonschema = "0.18"
40+
uuid = { version = "1.0", features = ["v4"] }
41+
42+
# Performance and monitoring
43+
instant = "0.1"
44+
45+
# Optional features for advanced functionality
46+
regex = "1.0"
47+
walkdir = "2.0"
48+
49+
[dev-dependencies]
50+
tempfile = "3.0"
51+
assert_cmd = "2.0"
52+
predicates = "3.0"
53+
54+
[features]
55+
default = ["performance-monitoring"]
56+
performance-monitoring = []
57+
docker-support = []
58+
extended-validation = []
59+
60+
[workspace]
61+
# This is a standalone package, not part of the parent workspace
Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
# MCP Test Harness
2+
3+
A universal testing tool for Model Context Protocol (MCP) servers. Test any MCP server implementation for protocol compliance, performance, and reliability.
4+
5+
[![Crate](https://img.shields.io/crates/v/mcp-test-harness.svg)](https://crates.io/crates/mcp-test-harness)
6+
[![Documentation](https://docs.rs/mcp-test-harness/badge.svg)](https://docs.rs/mcp-test-harness)
7+
[![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-blue.svg)](LICENSE)
8+
9+
## Features
10+
11+
- 🔧 **Universal MCP Server Testing** - Works with any MCP server implementation
12+
- 🚀 **Multiple Transport Support** - stdio, HTTP, WebSocket
13+
- 📊 **Performance Monitoring** - Built-in benchmarking and regression detection
14+
- 🛡️ **Protocol Compliance** - Comprehensive MCP protocol validation
15+
- 🐳 **Docker Ready** - Containerized testing for CI/CD pipelines
16+
- 📋 **Flexible Configuration** - YAML/JSON configuration with templates
17+
- �� **Auto-Discovery** - Automatic MCP server detection
18+
- 📈 **Rich Reporting** - Table, JSON, YAML output formats
19+
20+
## Quick Start
21+
22+
### Installation
23+
24+
```bash
25+
# Install from crates.io
26+
cargo install mcp-test-harness
27+
28+
# Or build from source
29+
git clone https://github.com/rustic-ai/codeprism
30+
cd prism/mcp-test-harness-standalone
31+
cargo install --path .
32+
```
33+
34+
### Basic Usage
35+
36+
```bash
37+
# Test a stdio MCP server
38+
mcp-test-harness test \
39+
--config configs/examples/basic-mcp-server.yaml \
40+
--server-cmd "node my-mcp-server.js"
41+
42+
# Test an HTTP MCP server
43+
mcp-test-harness test \
44+
--config my-http-server-config.yaml
45+
46+
# Generate a configuration template
47+
mcp-test-harness template --server-type filesystem --output my-config.yaml
48+
49+
# Discover running MCP servers
50+
mcp-test-harness discover --port-range "3000-3010"
51+
52+
# Run performance benchmarks
53+
mcp-test-harness benchmark \
54+
--config my-config.yaml \
55+
--iterations 1000 \
56+
--duration 60
57+
```
58+
59+
### Docker Usage
60+
61+
```bash
62+
# Pull the image
63+
docker pull mcp-test-harness:latest
64+
65+
# Run tests with Docker
66+
docker run --rm -v $(pwd)/configs:/configs \
67+
mcp-test-harness test --config /configs/my-server.yaml
68+
69+
# Use in CI/CD
70+
docker run --rm -v $(pwd):/workspace -w /workspace \
71+
mcp-test-harness test --config test-config.yaml --output json
72+
```
73+
74+
## Configuration
75+
76+
### Basic Configuration Structure
77+
78+
```yaml
79+
global:
80+
max_global_concurrency: 2
81+
timeout_seconds: 30
82+
fail_fast: false
83+
84+
server:
85+
transport: "stdio" # or "http", "websocket"
86+
command: "node server.js"
87+
args: ["--port", "3000"]
88+
working_dir: "/path/to/server"
89+
env:
90+
NODE_ENV: "test"
91+
92+
performance:
93+
enable_monitoring: true
94+
baseline_storage_path: "baselines/"
95+
regression_detection:
96+
warning_threshold_percent: 25.0
97+
error_threshold_percent: 50.0
98+
99+
test_suites:
100+
- name: "MCP Protocol Compliance"
101+
test_cases:
102+
- id: "initialize"
103+
tool_name: "initialize"
104+
enabled: true
105+
input_params:
106+
protocolVersion: "2024-11-05"
107+
expected:
108+
patterns:
109+
- key: "protocolVersion"
110+
validation: { type: "exists" }
111+
required: true
112+
```
113+
114+
### Server Types
115+
116+
#### stdio Server
117+
```yaml
118+
server:
119+
transport: "stdio"
120+
command: "node"
121+
args: ["server.js"]
122+
```
123+
124+
#### HTTP Server
125+
```yaml
126+
server:
127+
transport: "http"
128+
url: "http://localhost:3000"
129+
connection_timeout: 10
130+
```
131+
132+
#### WebSocket Server
133+
```yaml
134+
server:
135+
transport: "websocket"
136+
url: "ws://localhost:3000"
137+
```
138+
139+
## Validation Patterns
140+
141+
The test harness supports comprehensive response validation:
142+
143+
```yaml
144+
expected:
145+
patterns:
146+
# Check if field exists
147+
- key: "status"
148+
validation: { type: "exists" }
149+
required: true
150+
151+
# Check exact value
152+
- key: "status"
153+
validation: { type: "equals", value: "ok" }
154+
required: true
155+
156+
# Check numeric range
157+
- key: "count"
158+
validation: { type: "range", min: 1.0, max: 100.0 }
159+
required: true
160+
161+
# Check array structure
162+
- key: "items"
163+
validation: { type: "array" }
164+
required: true
165+
166+
# Check array length
167+
- key: "items"
168+
validation: { type: "array_min_length", min_length: 1 }
169+
required: true
170+
171+
# Check boolean value
172+
- key: "enabled"
173+
validation: { type: "boolean", value: true }
174+
required: true
175+
```
176+
177+
## Performance Testing
178+
179+
### Basic Performance Requirements
180+
181+
```yaml
182+
performance_requirements:
183+
max_execution_time_ms: 5000
184+
max_memory_usage_mb: 64
185+
```
186+
187+
### Benchmarking
188+
189+
```bash
190+
# Run benchmarks
191+
mcp-test-harness benchmark \
192+
--config server-config.yaml \
193+
--iterations 1000 \
194+
--duration 60 \
195+
--output json > benchmark-results.json
196+
```
197+
198+
## Templates
199+
200+
Generate configuration templates for common server types:
201+
202+
```bash
203+
# Filesystem server template
204+
mcp-test-harness template --server-type filesystem --output fs-config.yaml
205+
206+
# Database server template
207+
mcp-test-harness template --server-type database --output db-config.yaml
208+
209+
# API wrapper template
210+
mcp-test-harness template --server-type api --output api-config.yaml
211+
212+
# List available templates
213+
mcp-test-harness list
214+
```
215+
216+
## CI/CD Integration
217+
218+
### GitHub Actions
219+
220+
```yaml
221+
name: MCP Server Tests
222+
on: [push, pull_request]
223+
224+
jobs:
225+
test:
226+
runs-on: ubuntu-latest
227+
steps:
228+
- uses: actions/checkout@v4
229+
- name: Test MCP Server
230+
run: |
231+
docker run --rm -v $(pwd):/workspace -w /workspace \
232+
mcp-test-harness test \
233+
--config .github/mcp-test-config.yaml \
234+
--output json > test-results.json
235+
- name: Upload Results
236+
uses: actions/upload-artifact@v3
237+
with:
238+
name: test-results
239+
path: test-results.json
240+
```
241+
242+
### GitLab CI
243+
244+
```yaml
245+
mcp-tests:
246+
image: mcp-test-harness:latest
247+
script:
248+
- mcp-test-harness test --config ci-config.yaml --output junit > results.xml
249+
artifacts:
250+
reports:
251+
junit: results.xml
252+
```
253+
254+
## Contributing
255+
256+
1. Fork the repository
257+
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
258+
3. Commit your changes (`git commit -m 'Add amazing feature'`)
259+
4. Push to the branch (`git push origin feature/amazing-feature`)
260+
5. Open a Pull Request
261+
262+
## License
263+
264+
This project is licensed under either of
265+
266+
* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
267+
* MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
268+
269+
at your option.
270+
271+
## Support
272+
273+
- 📖 [Documentation](https://docs.rs/mcp-test-harness)
274+
- 🐛 [Issue Tracker](https://github.com/rustic-ai/codeprism/issues)
275+
- 💬 [Discussions](https://github.com/rustic-ai/codeprism/discussions)
276+
- �� [Email Support](mailto:team@codeprism.ai)

0 commit comments

Comments
 (0)