Skip to content

Commit 9110ccc

Browse files
pcarletonpcarleton
authored
Merge branch 'main' into fweinberger/fix-npm-install
2 parents d9ea985 + 768f155 commit 9110ccc

23 files changed

Lines changed: 2248 additions & 623 deletions

‎.github/workflows/ci.yml‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ on:
33
branches:
44
- main
55
pull_request:
6+
release:
7+
types: [published]
68

79
permissions:
810
contents: read
@@ -19,9 +21,34 @@ jobs:
1921
- uses: actions/checkout@v4
2022
- uses: actions/setup-node@v4
2123
with:
22-
node-version: 22
24+
node-version: 24
2325
cache: npm
2426

2527
- run: npm ci
28+
- run: npm run check
29+
- run: npm run build
2630
- run: npm test
27-
- run: npm run lint
31+
32+
publish:
33+
runs-on: ubuntu-latest
34+
if: github.event_name == 'release'
35+
environment: release
36+
needs: [test]
37+
38+
permissions:
39+
contents: read
40+
id-token: write
41+
42+
steps:
43+
- uses: actions/checkout@v4
44+
- uses: actions/setup-node@v4
45+
with:
46+
node-version: 24
47+
cache: npm
48+
registry-url: 'https://registry.npmjs.org'
49+
50+
- run: npm ci
51+
52+
- run: npm publish --provenance --access public
53+
env:
54+
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

‎.github/workflows/pr-publish.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
name: Publish PR to pkg.pr.new
2+
permissions:
3+
contents: read
4+
on:
5+
pull_request:
6+
push:
7+
branches:
8+
- '**'
9+
tags:
10+
- '!**'
11+
12+
jobs:
13+
pr-publish:
14+
runs-on: ubuntu-latest
15+
16+
steps:
17+
- uses: actions/checkout@v4
18+
- uses: actions/setup-node@v4
19+
with:
20+
node-version: 24
21+
cache: npm
22+
23+
- run: npm ci
24+
- name: Build
25+
run: npm run build
26+
- name: Publish
27+
run: npx pkg-pr-new publish

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
node_modules
22
results/
3-
lefthook-local.yml
3+
lefthook-local.yml
4+
dist

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2024 Anthropic, PBC
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 68 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# MCP Conformance Test Framework
22

3-
A framework for testing MCP (Model Context Protocol) client implementations against the specification.
3+
A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
44

55
> [!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
66
@@ -9,56 +9,110 @@ A framework for testing MCP (Model Context Protocol) client implementations agai
99
### Testing Clients
1010

1111
```bash
12-
npm install
13-
npm run start -- --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
12+
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
1413
```
1514

1615
### Testing Servers
1716

1817
```bash
19-
npm run test:server -- --server-url http://localhost:3000/mcp --all
18+
# Run all server scenarios (default)
19+
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
20+
21+
# Run a single scenario
22+
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
23+
```
24+
25+
### List Available Scenarios
26+
27+
```bash
28+
npx @modelcontextprotocol/conformance list
2029
```
2130

2231
## Overview
2332

24-
The conformance test framework validates MCP client implementations by:
33+
The conformance test framework validates MCP implementations by:
34+
35+
**For Clients:**
2536

2637
1. Starting a test server for the specified scenario
2738
2. Running the client implementation with the test server URL
2839
3. Capturing MCP protocol interactions
2940
4. Running conformance checks against the specification
3041
5. Generating detailed test results
3142

43+
**For Servers:**
44+
45+
1. Connecting to the running server as an MCP client
46+
2. Sending test requests and capturing responses
47+
3. Running conformance checks against server behavior
48+
4. Generating detailed test results
49+
3250
## Usage
3351

52+
### Client Testing
53+
3454
```bash
35-
npm run start -- --command "<client-command>" --scenario <scenario-name>
55+
npx @modelcontextprotocol/conformance client --command "<client-command>" --scenario <scenario-name> [options]
3656
```
3757

58+
**Options:**
59+
3860
- `--command` - The command to run your MCP client (can include flags)
3961
- `--scenario` - The test scenario to run (e.g., "initialize")
62+
- `--timeout` - Timeout in milliseconds (default: 30000)
63+
- `--verbose` - Show verbose output
4064

4165
The framework appends the server URL as the final argument to your command.
4266

67+
### Server Testing
68+
69+
```bash
70+
npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
71+
```
72+
73+
**Options:**
74+
75+
- `--url` - URL of the server to test
76+
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize". Runs all available scenarios by default
77+
4378
## Test Results
4479

45-
Results are saved to `results/<scenario>-<timestamp>/`:
80+
**Client Testing** - Results are saved to `results/<scenario>-<timestamp>/`:
4681

4782
- `checks.json` - Array of conformance check results with pass/fail status
4883
- `stdout.txt` - Client stdout output
4984
- `stderr.txt` - Client stderr output
5085

86+
**Server Testing** - Results are saved to `results/server-<scenario>-<timestamp>/`:
87+
88+
- `checks.json` - Array of conformance check results with pass/fail status
89+
5190
## Example Clients
5291

5392
- `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks)
5493
- `examples/clients/typescript/test-broken.ts` - Invalid client missing required fields (fails checks)
5594

5695
## Available Scenarios
5796

97+
### Client Scenarios
98+
5899
- **initialize** - Tests MCP client initialization handshake
59100
- Validates protocol version
60101
- Validates clientInfo (name and version)
61102
- Validates server response handling
103+
- **tools-call** - Tests tool invocation
104+
- **auth/basic-dcr** - Tests OAuth Dynamic Client Registration flow
105+
- **auth/basic-metadata-var1** - Tests OAuth with authorization metadata
106+
107+
### Server Scenarios
108+
109+
Run `npx @modelcontextprotocol/conformance list --server` to see all available server scenarios, including:
110+
111+
- **server-initialize** - Tests server initialization and capabilities
112+
- **tools-list** - Tests tool listing endpoint
113+
- **tools-call-\*** - Various tool invocation scenarios
114+
- **resources-\*** - Resource management scenarios
115+
- **prompts-\*** - Prompt management scenarios
62116

63117
## Architecture
64118

@@ -67,8 +121,13 @@ See `src/runner/DESIGN.md` for detailed architecture documentation.
67121
### Key Components
68122

69123
- **Runner** (`src/runner/`) - Orchestrates test execution and result generation
70-
- **Scenarios** (`src/scenarios/`) - Test scenarios with custom server implementations
71-
- **Checks** (`src/checks.ts`) - Conformance validation functions
124+
- `client.ts` - Client testing implementation
125+
- `server.ts` - Server testing implementation
126+
- `utils.ts` - Shared utilities
127+
- `index.ts` - Public API exports
128+
- **CLI** (`src/index.ts`) - Command-line interface using Commander.js
129+
- **Scenarios** (`src/scenarios/`) - Test scenarios with expected behaviors
130+
- **Checks** (`src/checks/`) - Conformance validation functions
72131
- **Types** (`src/types.ts`) - Shared type definitions
73132

74133
## Adding New Scenarios

‎SERVER_REQUIREMENTS.md‎

Lines changed: 68 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,72 @@ If no progress token provided, just execute with delays.
397397

398398
**Implementation Note**: If the client doesn't support elicitation (no `elicitation` capability), return an error.
399399

400+
#### `test_elicitation_sep1034_defaults`
401+
402+
**Arguments**: None
403+
404+
**Behavior**: Request user input from the client using `elicitation/create` with default values for all primitive types (SEP-1034)
405+
406+
**Elicitation Request**:
407+
408+
```json
409+
{
410+
"method": "elicitation/create",
411+
"params": {
412+
"message": "Please review and update the form fields with defaults",
413+
"requestedSchema": {
414+
"type": "object",
415+
"properties": {
416+
"name": {
417+
"type": "string",
418+
"description": "User name",
419+
"default": "John Doe"
420+
},
421+
"age": {
422+
"type": "integer",
423+
"description": "User age",
424+
"default": 30
425+
},
426+
"score": {
427+
"type": "number",
428+
"description": "User score",
429+
"default": 95.5
430+
},
431+
"status": {
432+
"type": "string",
433+
"description": "User status",
434+
"enum": ["active", "inactive", "pending"],
435+
"default": "active"
436+
},
437+
"verified": {
438+
"type": "boolean",
439+
"description": "Verification status",
440+
"default": true
441+
}
442+
},
443+
"required": []
444+
}
445+
}
446+
}
447+
```
448+
449+
**Returns**: Text content with the elicitation result
450+
451+
```json
452+
{
453+
"content": [
454+
{
455+
"type": "text",
456+
"text": "Elicitation completed: action=<accept/decline/cancel>, content={...}"
457+
}
458+
]
459+
}
460+
```
461+
462+
**Implementation Note**: This tool tests SEP-1034 support for default values across all primitive types (string, integer, number, enum, boolean). If the client doesn't support elicitation (no `elicitation` capability), return an error.
463+
464+
**Reference**: [SEP-1034](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1034)
465+
400466
---
401467

402468
## 3. Resources Requirements
@@ -956,8 +1022,8 @@ MCP Conformance Test Server running on http://localhost:3000
9561022

9571023
```bash
9581024
# Single test
959-
npm run test:server -- --server-url http://localhost:3000/mcp --scenario server-initialize
1025+
npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
9601026

9611027
# All tests
962-
npm run test:server -- --server-url http://localhost:3000/mcp --all
1028+
npm run start -- server --url http://localhost:3000/mcp
9631029
```

‎examples/servers/typescript/README.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -147,8 +147,7 @@ This server implements the MCP Server Conformance Requirements specified in `../
147147
To run conformance tests against this server:
148148

149149
```bash
150-
cd ../../../
151-
npm run test:server -- --server-url http://localhost:3000/mcp --all
150+
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
152151
```
153152

154153
## Implementation Notes

0 commit comments

Comments
 (0)