Skip to content

Commit 2d52de5

Browse files
pcarletonpcarleton
authored
docs: add SDK integration guide and fix spec references (#121)
* docs: add SDK integration guide and fix spec references - Add SDK_INTEGRATION.md with guide for SDK maintainers on integrating conformance tests - Link to guide from README.md - Update spec references to use 2025-11-25 instead of draft URLs * fix: prettier formatting and correct action version to v0.1.10
1 parent 245e44e commit 2d52de5

3 files changed

Lines changed: 218 additions & 8 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ A framework for testing MCP (Model Context Protocol) client and server implement
55
> [!WARNING]
66
> This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
77
8+
**For SDK maintainers:** See [SDK Integration Guide](./SDK_INTEGRATION.md) for a streamlined guide on integrating conformance tests into your SDK repository.
9+
810
## Quick Start
911

1012
### Testing Clients

‎SDK_INTEGRATION.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# Using MCP Conformance Tests in SDK Repositories
2+
3+
This guide explains how to integrate the MCP conformance test suite into your language SDK repository. The conformance framework tests your MCP implementation against the protocol specification to ensure compatibility.
4+
5+
## Quick Start
6+
7+
Install and run conformance tests:
8+
9+
```bash
10+
# Client testing (framework starts a test server, runs your client against it)
11+
npx @modelcontextprotocol/conformance client --command "your-client-command" --scenario initialize
12+
13+
# Server testing (your server must already be running)
14+
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
15+
```
16+
17+
## Two Testing Modes
18+
19+
### Client Testing
20+
21+
The framework **starts a test server** and spawns your client against it. Your client receives the server URL as its final command-line argument.
22+
23+
```bash
24+
# Run a single scenario
25+
npx @modelcontextprotocol/conformance client \
26+
--command "python tests/conformance/client.py" \
27+
--scenario initialize
28+
29+
# Run a suite of tests
30+
npx @modelcontextprotocol/conformance client \
31+
--command "python tests/conformance/client.py" \
32+
--suite auth
33+
```
34+
35+
**Available client suites:** `all`, `core`, `extensions`, `auth`, `metadata`, `sep-835`
36+
37+
Your client should:
38+
39+
1. Accept the server URL as its last argument
40+
2. Read `MCP_CONFORMANCE_SCENARIO` env var to determine which scenario is being tested
41+
3. Read `MCP_CONFORMANCE_CONTEXT` env var for scenario-specific data (e.g., OAuth credentials)
42+
43+
### Server Testing
44+
45+
Your server must be **running before** invoking the conformance tool. The framework connects to it as an MCP client.
46+
47+
```bash
48+
# Start your server first
49+
your-server --port 3001 &
50+
51+
# Then run conformance tests
52+
npx @modelcontextprotocol/conformance server \
53+
--url http://localhost:3001/mcp \
54+
--suite active
55+
```
56+
57+
**Available server suites:** `active` (default), `all`, `pending`
58+
59+
**Note:** Server testing requires you to manage server lifecycle (start, health-check, cleanup) yourself.
60+
61+
---
62+
63+
## Expected Failures (Baseline) File
64+
65+
The expected-failures feature lets your CI pass while you work on fixing known issues. It catches regressions by failing when:
66+
67+
- A previously passing test starts failing (regression)
68+
- A previously failing test starts passing (stale baseline - remove the entry)
69+
70+
### File Format
71+
72+
Create a YAML file (e.g., `conformance-baseline.yml`):
73+
74+
```yaml
75+
server:
76+
- tools-call-with-progress
77+
- resources-subscribe
78+
client:
79+
- auth/client-credentials-jwt
80+
```
81+
82+
### Usage
83+
84+
```bash
85+
npx @modelcontextprotocol/conformance server \
86+
--url http://localhost:3000/mcp \
87+
--expected-failures ./conformance-baseline.yml
88+
```
89+
90+
### Exit Code Behavior
91+
92+
| Scenario Result | In Baseline? | Exit Code | Meaning |
93+
| --------------- | ------------ | --------- | ----------------------------- |
94+
| Fails | Yes | 0 | Expected failure |
95+
| Fails | No | 1 | Unexpected regression |
96+
| Passes | Yes | 1 | Stale baseline - remove entry |
97+
| Passes | No | 0 | Normal pass |
98+
99+
---
100+
101+
## GitHub Action
102+
103+
The conformance repo provides a reusable GitHub Action that handles Node.js setup and conformance execution.
104+
105+
### Client Testing Example
106+
107+
```yaml
108+
name: Conformance Tests
109+
on: [push, pull_request]
110+
111+
jobs:
112+
conformance:
113+
runs-on: ubuntu-latest
114+
steps:
115+
- uses: actions/checkout@v4
116+
117+
- name: Set up your SDK
118+
run: |
119+
# Your SDK setup (pip install, npm install, etc.)
120+
pip install -e .
121+
122+
- uses: modelcontextprotocol/conformance@v0.1.10
123+
with:
124+
mode: client
125+
command: 'python tests/conformance/client.py'
126+
suite: auth
127+
expected-failures: ./conformance-baseline.yml
128+
```
129+
130+
### Server Testing Example
131+
132+
```yaml
133+
name: Conformance Tests
134+
on: [push, pull_request]
135+
136+
jobs:
137+
conformance:
138+
runs-on: ubuntu-latest
139+
steps:
140+
- uses: actions/checkout@v4
141+
142+
- name: Set up and start server
143+
run: |
144+
pip install -e .
145+
python -m myserver --port 3001 &
146+
# Wait for server to be ready
147+
timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
148+
149+
- uses: modelcontextprotocol/conformance@v0.1.10
150+
with:
151+
mode: server
152+
url: http://localhost:3001/mcp
153+
suite: active
154+
expected-failures: ./conformance-baseline.yml
155+
```
156+
157+
### Action Inputs
158+
159+
| Input | Required | Description |
160+
| ------------------- | ----------- | ----------------------------------------------- |
161+
| `mode` | Yes | `server` or `client` |
162+
| `url` | Server mode | URL of the server to test |
163+
| `command` | Client mode | Command to run the client |
164+
| `expected-failures` | No | Path to YAML baseline file |
165+
| `suite` | No | Test suite to run |
166+
| `scenario` | No | Run a single scenario by name |
167+
| `timeout` | No | Timeout in ms for client tests (default: 30000) |
168+
| `verbose` | No | Show verbose output (default: false) |
169+
| `node-version` | No | Node.js version (default: 20) |
170+
171+
---
172+
173+
## Writing Conformance Clients/Servers
174+
175+
### Example Client Pattern
176+
177+
See [`src/conformance/everything-client.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-client.ts) in the TypeScript SDK for a reference implementation. The recommended pattern is a single client that routes behavior based on the scenario:
178+
179+
```python
180+
import os
181+
import sys
182+
import json
183+
184+
def main():
185+
server_url = sys.argv[-1] # URL passed as last argument
186+
scenario = os.environ.get("MCP_CONFORMANCE_SCENARIO", "")
187+
context = json.loads(os.environ.get("MCP_CONFORMANCE_CONTEXT", "{}"))
188+
189+
if scenario.startswith("auth/"):
190+
run_auth_scenario(server_url, scenario, context)
191+
else:
192+
run_default_scenario(server_url)
193+
194+
if __name__ == "__main__":
195+
main()
196+
```
197+
198+
### Example Server Pattern
199+
200+
See [`src/conformance/everything-server.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-server.ts) in the TypeScript SDK for a reference implementation that handles all server scenarios.
201+
202+
---
203+
204+
## Additional Resources
205+
206+
- [Conformance README](./README.md)
207+
- [Design documentation](./src/runner/DESIGN.md)
208+
- [TypeScript SDK conformance examples](https://github.com/modelcontextprotocol/typescript-sdk/tree/main/src/conformance)

‎src/scenarios/client/auth/spec-references.ts‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,15 +19,15 @@ export const SpecReferences: { [key: string]: SpecReference } = {
1919
},
2020
MCP_PRM_DISCOVERY: {
2121
id: 'MCP-2025-06-18-PRM-discovery',
22-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#protected-resource-metadata-discovery-requirements'
22+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#protected-resource-metadata-discovery-requirements'
2323
},
2424
MCP_AUTH_DISCOVERY: {
2525
id: 'MCP-Authorization-metadata-discovery',
26-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#authorization-server-metadata-discovery'
26+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#authorization-server-metadata-discovery'
2727
},
2828
MCP_DCR: {
2929
id: 'MCP-Dynamic-client-registration',
30-
url: 'https://modelcontextprotocol.io/specification/draft/basic/client#dynamic-client-registration'
30+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/client#dynamic-client-registration'
3131
},
3232
OAUTH_2_1_AUTHORIZATION_ENDPOINT: {
3333
id: 'OAUTH-2.1-authorization-endpoint',
@@ -39,23 +39,23 @@ export const SpecReferences: { [key: string]: SpecReference } = {
3939
},
4040
MCP_ACCESS_TOKEN_USAGE: {
4141
id: 'MCP-Access-token-usage',
42-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#access-token-usage'
42+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#access-token-usage'
4343
},
4444
MCP_SCOPE_SELECTION_STRATEGY: {
4545
id: 'MCP-Scope-selection-strategy',
46-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-selection-strategy'
46+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-selection-strategy'
4747
},
4848
MCP_SCOPE_CHALLENGE_HANDLING: {
4949
id: 'MCP-Scope-challenge-handling',
50-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-challenge-handling'
50+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-challenge-handling'
5151
},
5252
MCP_AUTH_ERROR_HANDLING: {
5353
id: 'MCP-Auth-error-handling',
54-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#error-handling'
54+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#error-handling'
5555
},
5656
MCP_CLIENT_ID_METADATA_DOCUMENTS: {
5757
id: 'MCP-Client-ID-Metadata-Documents',
58-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#client-id-metadata-documents'
58+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#client-id-metadata-documents'
5959
},
6060
IETF_CIMD: {
6161
id: 'IETF-OAuth-Client-ID-Metadata-Document',

0 commit comments

Comments
 (0)