Skip to content

Commit 760e277

Browse files
jancurnclaude
andauthored
Add MCP conformance testing script and on-demand workflow (#196)
Introduce `npm run test:conformance`, which drives the `@modelcontextprotocol/conformance` framework against mcpc to validate adherence to the MCP specification. Scenarios are mapped to mcpc CLI commands by a small Node adapter at `test/conformance/client.mjs` (currently covering `initialize`; more scenarios can be added over time). Also add a separate `Conformance` GitHub Actions workflow that runs the suite on demand (`workflow_dispatch`) and uploads the framework's `results/` directory as a build artifact. Co-authored-by: Claude <noreply@anthropic.com>
1 parent 1a020cb commit 760e277

5 files changed

Lines changed: 157 additions & 0 deletions

File tree

.github/workflows/conformance.yml

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: Conformance
2+
3+
# Runs the @modelcontextprotocol/conformance suite against mcpc to check
4+
# adherence to the MCP specification. Triggered on-demand only; it is not
5+
# part of the default CI pipeline because the conformance framework is still
6+
# evolving and test coverage via the mcpc adapter is a work in progress.
7+
8+
on:
9+
workflow_dispatch:
10+
inputs:
11+
scenario:
12+
description: "Scenario name to run (e.g. 'initialize', 'tools-call'). Leave blank to use the default configured in package.json."
13+
required: false
14+
type: string
15+
verbose:
16+
description: "Enable verbose output from the conformance framework"
17+
required: false
18+
type: boolean
19+
default: false
20+
21+
jobs:
22+
conformance:
23+
name: MCP conformance
24+
runs-on: ubuntu-latest
25+
26+
steps:
27+
- uses: actions/checkout@v6
28+
29+
- name: Set up Node.js 24
30+
uses: actions/setup-node@v6
31+
with:
32+
node-version: 24
33+
cache: npm
34+
35+
- name: Install dependencies
36+
run: npm ci
37+
38+
- name: Build
39+
run: npm run build
40+
41+
- name: Run conformance tests
42+
run: |
43+
CMD=(npx -y @modelcontextprotocol/conformance client
44+
--command "node test/conformance/client.mjs"
45+
--scenario "${SCENARIO:-initialize}")
46+
if [ "${VERBOSE}" = "true" ]; then
47+
CMD+=(--verbose)
48+
fi
49+
"${CMD[@]}"
50+
env:
51+
SCENARIO: ${{ inputs.scenario }}
52+
VERBOSE: ${{ inputs.verbose }}
53+
54+
- name: Upload conformance results
55+
if: always()
56+
uses: actions/upload-artifact@v4
57+
with:
58+
name: conformance-results
59+
path: results/
60+
if-no-files-found: ignore

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ test/coverage/unit
1616
test/coverage/merged
1717
test/runs/
1818

19+
# Conformance test output (written to CWD by @modelcontextprotocol/conformance)
20+
results/
21+
1922
# Logs
2023
*.log
2124
logs/

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Added
11+
12+
- New `npm run test:conformance` script (and on-demand `Conformance` GitHub Actions workflow) that runs the `@modelcontextprotocol/conformance` framework against mcpc to verify adherence to the MCP specification. Starts with the `initialize` scenario; additional scenarios can be added to `test/conformance/client.mjs` as coverage grows.
13+
1014
### Changed
1115

1216
- OAuth callback ports for the hosted CIMD changed from the contiguous range 13316–13325 to 6 non-contiguous ports (13316, 13163, 31316, 31613, 16133, 16313) so one unrelated process is less likely to claim all of them. `localhost` variants dropped from the CIMD's `redirect_uris` in favor of `127.0.0.1` only (per RFC 8252 §8.3, which recommends the IP literal to avoid DNS resolution ambiguity).

package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@
4141
"test:coverage:merge": "test/coverage/coverage-merge.sh",
4242
"test:e2e": "./test/e2e/run.sh --keep",
4343
"test:e2e:bun": "./test/e2e/run.sh --no-build --runtime bun",
44+
"test:conformance": "npm run build && npx -y @modelcontextprotocol/conformance client --command \"node test/conformance/client.mjs\" --scenario initialize",
4445
"lint": "eslint src/**/*.ts && prettier --check \"src/**/*.ts\" \"test/**/*.ts\"",
4546
"lint:fix": "eslint src/**/*.ts --fix && prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
4647
"format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",

test/conformance/client.mjs

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/usr/bin/env node
2+
3+
// Adapter that drives the mcpc CLI from the
4+
// `@modelcontextprotocol/conformance` framework.
5+
//
6+
// The framework invokes this script with the test server URL appended as the
7+
// last positional argument and sets `MCP_CONFORMANCE_SCENARIO` (plus an
8+
// optional `MCP_CONFORMANCE_CONTEXT` JSON blob) in the environment. Per
9+
// scenario, we translate the expected behaviour into one or more mcpc
10+
// sub-commands against a freshly-created session, then tear the session down
11+
// again.
12+
//
13+
// Only a small set of scenarios is wired up today; unsupported ones exit
14+
// non-zero so the framework records them as failures (track them in
15+
// `test/conformance/expected-failures.yml` to keep CI green until coverage
16+
// grows).
17+
18+
import { spawn } from 'node:child_process';
19+
import { mkdtemp, rm } from 'node:fs/promises';
20+
import { tmpdir } from 'node:os';
21+
import { fileURLToPath } from 'node:url';
22+
import { dirname, resolve } from 'node:path';
23+
24+
const scenario = process.env.MCP_CONFORMANCE_SCENARIO;
25+
const serverUrl = process.argv[process.argv.length - 1];
26+
27+
if (!scenario) {
28+
console.error('MCP_CONFORMANCE_SCENARIO environment variable is not set');
29+
process.exit(1);
30+
}
31+
if (!serverUrl || !/^https?:\/\//i.test(serverUrl)) {
32+
console.error(`Missing or invalid server URL (got: ${serverUrl ?? ''})`);
33+
process.exit(1);
34+
}
35+
36+
const here = dirname(fileURLToPath(import.meta.url));
37+
const mcpcBin = resolve(here, '..', '..', 'bin', 'mcpc');
38+
const homeDir = await mkdtemp(`${tmpdir()}/mcpc-conformance-`);
39+
const sessionName = `conformance-${process.pid}`;
40+
const env = { ...process.env, MCPC_HOME_DIR: homeDir, MCPC_JSON: '1' };
41+
42+
function runMcpc(args) {
43+
return new Promise((res, rej) => {
44+
const child = spawn(mcpcBin, args, { env, stdio: 'inherit' });
45+
child.once('error', rej);
46+
child.once('exit', (code) => {
47+
if (code === 0) res();
48+
else rej(new Error(`mcpc ${args.join(' ')} exited with code ${code}`));
49+
});
50+
});
51+
}
52+
53+
async function cleanup() {
54+
try {
55+
await runMcpc([`@${sessionName}`, 'close']);
56+
} catch {
57+
// Best effort — the session may never have been created.
58+
}
59+
await rm(homeDir, { recursive: true, force: true }).catch(() => {});
60+
}
61+
62+
async function main() {
63+
switch (scenario) {
64+
case 'initialize':
65+
// Connecting triggers the full MCP initialize handshake via the
66+
// bridge process. That is all the conformance server needs to
67+
// observe for this scenario.
68+
await runMcpc(['connect', serverUrl, `@${sessionName}`]);
69+
return;
70+
71+
case 'tools-call':
72+
await runMcpc(['connect', serverUrl, `@${sessionName}`]);
73+
await runMcpc([`@${sessionName}`, 'tools-list']);
74+
return;
75+
76+
default:
77+
console.error(`Scenario not implemented by mcpc conformance adapter: ${scenario}`);
78+
process.exit(1);
79+
}
80+
}
81+
82+
try {
83+
await main();
84+
} catch (err) {
85+
console.error(err instanceof Error ? err.message : String(err));
86+
process.exitCode = 1;
87+
} finally {
88+
await cleanup();
89+
}

0 commit comments

Comments
 (0)