Skip to content

Commit 0cffc60

Browse files
koickoic
committed
Support running MCP client conformance tests against the Ruby SDK
## Motivation and Context PR #248 introduced server conformance testing using `@modelcontextprotocol/conformance`. This extends coverage to the client side so that `MCP::Client` behavior can also be validated against the MCP specification. ### Scope Client conformance is driven by the conformance runner, which spawns test servers for each scenario and invokes `conformance/client.rb` against them. The client script performs the MCP initialize handshake and executes scenario-specific operations (for example listing tools or calling a tool). Because `MCP::Client` does not yet support the full initialize handshake natively (pending #210), the client uses a lightweight `ConformanceTransport` that handles both JSON and SSE responses and performs the handshake manually. ### Usage Run all conformance tests (server + client): ```console bundle exec rake conformance ``` Run a single client scenario: ```console bundle exec rake conformance SCENARIO=initialize ``` List available scenarios: ```console bundle exec rake conformance_list ```
1 parent 0d700d7 commit 0cffc60

File tree

6 files changed

+202
-17
lines changed

6 files changed

+202
-17
lines changed

‎Rakefile‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,22 +20,25 @@ desc "Run MCP conformance tests (PORT, SCENARIO, SPEC_VERSION, VERBOSE)"
2020
task :conformance do |t|
2121
next unless npx_available?(t.name)
2222

23-
require_relative "conformance/runner"
23+
require_relative "conformance/server_runner"
24+
require_relative "conformance/client_runner"
2425

2526
options = {}
2627
options[:port] = Integer(ENV["PORT"]) if ENV["PORT"]
2728
options[:scenario] = ENV["SCENARIO"] if ENV["SCENARIO"]
2829
options[:spec_version] = ENV["SPEC_VERSION"] if ENV["SPEC_VERSION"]
2930
options[:verbose] = true if ENV["VERBOSE"]
3031

31-
Conformance::Runner.new(**options).run
32+
Conformance::ServerRunner.new(**options).run
33+
Conformance::ClientRunner.new(**options.except(:port)).run
3234
end
3335

3436
desc "List available conformance scenarios"
3537
task :conformance_list do |t|
3638
next unless npx_available?(t.name)
3739

3840
system("npx", "--yes", "@modelcontextprotocol/conformance", "list", "--server")
41+
system("npx", "--yes", "@modelcontextprotocol/conformance", "list", "--client")
3942
end
4043

4144
desc "Start the conformance server (PORT)"

‎conformance/README.md‎

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -15,22 +15,22 @@ Validates the Ruby SDK's conformance to the MCP specification using [`@modelcont
1515
bundle exec rake conformance
1616
```
1717

18-
Starts the conformance server, runs all active scenarios against it, prints a pass/fail
19-
summary for each scenario, and exits with a non-zero status code if any unexpected failures
20-
are detected. Scenarios listed in `expected_failures.yml` are allowed to fail without
21-
affecting the exit code.
18+
Runs both server and client conformance tests in sequence. Server conformance starts the
19+
conformance server and tests it. Client conformance spawns test servers for each scenario
20+
and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml`
21+
are allowed to fail without affecting the exit code.
2222

2323
### Environment variables
2424

25-
| Variable | Description | Default |
26-
|----------------|--------------------------------------|---------|
27-
| `PORT` | Server port | `9292` |
28-
| `SCENARIO` | Run a single scenario by name | (all) |
29-
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
30-
| `VERBOSE` | Show raw JSON output when set | (off) |
25+
| Variable | Description | Default |
26+
|----------------|---------------------------------------|---------|
27+
| `PORT` | Server port (server conformance only) | `9292` |
28+
| `SCENARIO` | Run a single scenario by name | (all) |
29+
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
30+
| `VERBOSE` | Show raw JSON output when set | (off) |
3131

3232
```bash
33-
# Run a single scenario
33+
# Run a single server scenario
3434
bundle exec rake conformance SCENARIO=ping
3535

3636
# Use a different port with verbose output
@@ -91,8 +91,10 @@ submissions.
9191
```
9292
conformance/
9393
server.rb # Conformance server (Rack + Puma, default port 9292)
94-
runner.rb # Starts the server, runs npx conformance, exits with result code
95-
expected_failures.yml # Baseline of known-failing scenarios
94+
server_runner.rb # Starts the server, runs npx conformance server, exits with result code
95+
client.rb # Conformance client (invoked by npx conformance client)
96+
client_runner.rb # Runs npx conformance client, exits with result code
97+
expected_failures.yml # Baseline of known-failing scenarios (server and client)
9698
README.md # This file
9799
```
98100

‎conformance/client.rb‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# frozen_string_literal: true
2+
3+
# Conformance test client for the MCP Ruby SDK.
4+
# Invoked by the conformance runner:
5+
# MCP_CONFORMANCE_SCENARIO=<scenario> bundle exec ruby conformance/client.rb <server-url>
6+
#
7+
# The server URL is passed as the last positional argument.
8+
# The scenario name is read from the MCP_CONFORMANCE_SCENARIO environment variable,
9+
# which is set automatically by the conformance test runner.
10+
11+
require "net/http"
12+
require "json"
13+
require "securerandom"
14+
require "uri"
15+
require_relative "../lib/mcp"
16+
17+
# A transport that handles both JSON and SSE (text/event-stream) responses.
18+
# The standard `MCP::Client::HTTP` transport only accepts application/json,
19+
# but the MCP `StreamableHTTPServerTransport` may return text/event-stream.
20+
class ConformanceTransport
21+
def initialize(url:)
22+
@uri = URI(url)
23+
end
24+
25+
def send_request(request:)
26+
http = Net::HTTP.new(@uri.host, @uri.port)
27+
req = Net::HTTP::Post.new(@uri.path.empty? ? "/" : @uri.path)
28+
req["Content-Type"] = "application/json"
29+
req["Accept"] = "application/json, text/event-stream"
30+
req.body = JSON.generate(request)
31+
32+
response = http.request(req)
33+
34+
case response.content_type
35+
when "application/json"
36+
JSON.parse(response.body)
37+
when "text/event-stream"
38+
parse_sse_response(response.body)
39+
else
40+
raise "Unexpected content type: #{response.content_type}"
41+
end
42+
end
43+
44+
private
45+
46+
def parse_sse_response(body)
47+
body.each_line do |line|
48+
next unless line.start_with?("data: ")
49+
50+
data = line.delete_prefix("data: ").strip
51+
next if data.empty?
52+
53+
return JSON.parse(data)
54+
end
55+
nil
56+
end
57+
end
58+
59+
scenario = ENV["MCP_CONFORMANCE_SCENARIO"]
60+
server_url = ARGV.last
61+
62+
unless scenario && server_url
63+
abort("Usage: MCP_CONFORMANCE_SCENARIO=<scenario> ruby conformance/client.rb <server-url>")
64+
end
65+
66+
#
67+
# TODO: Once https://github.com/modelcontextprotocol/ruby-sdk/pull/210 is merged,
68+
# replace `ConformanceTransport` and the manual initialize handshake below with:
69+
#
70+
# ```
71+
# transport = MCP::Client::HTTP.new(url: server_url)
72+
# client = MCP::Client.new(transport: transport)
73+
# client.connect(client_info: { ... }, protocol_version: "2025-11-25")
74+
# ```
75+
#
76+
# After that `ConformanceTransport` will be removed.
77+
#
78+
transport = ConformanceTransport.new(url: server_url)
79+
80+
# MCP initialize handshake (the MCP::Client API does not expose this yet).
81+
transport.send_request(request: {
82+
jsonrpc: "2.0",
83+
id: SecureRandom.uuid,
84+
method: "initialize",
85+
params: {
86+
clientInfo: { name: "ruby-sdk-conformance-client", version: MCP::VERSION },
87+
protocolVersion: "2025-11-25",
88+
capabilities: {},
89+
},
90+
})
91+
92+
client = MCP::Client.new(transport: transport)
93+
94+
case scenario
95+
when "initialize"
96+
client.tools
97+
when "tools_call"
98+
tools = client.tools
99+
add_numbers = tools.find { |t| t.name == "add_numbers" }
100+
abort("Tool add_numbers not found") unless add_numbers
101+
client.call_tool(tool: add_numbers, arguments: { a: 1, b: 2 })
102+
else
103+
abort("Unknown or unsupported scenario: #{scenario}")
104+
end

‎conformance/client_runner.rb‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# frozen_string_literal: true
2+
3+
# Runs `npx @modelcontextprotocol/conformance client` against the conformance client script.
4+
require "English"
5+
6+
module Conformance
7+
class ClientRunner
8+
def initialize(scenario: nil, spec_version: nil, verbose: false)
9+
@scenario = scenario
10+
@spec_version = spec_version
11+
@verbose = verbose
12+
end
13+
14+
def run
15+
command = build_command
16+
puts "Command: #{command.join(" ")}\n\n"
17+
18+
system(*command)
19+
conformance_exit_code = $CHILD_STATUS.exitstatus
20+
exit(conformance_exit_code || 1) unless conformance_exit_code == 0
21+
end
22+
23+
private
24+
25+
def build_command
26+
expected_failures_yml = File.expand_path("expected_failures.yml", __dir__)
27+
client_script = File.expand_path("client.rb", __dir__)
28+
29+
npx_command = [
30+
"npx",
31+
"--yes",
32+
"@modelcontextprotocol/conformance",
33+
"client",
34+
"--command",
35+
"bundle exec ruby #{client_script}",
36+
]
37+
npx_command += if @scenario
38+
["--scenario", @scenario]
39+
else
40+
["--suite", "all"]
41+
end
42+
npx_command += ["--spec-version", @spec_version] if @spec_version
43+
npx_command += ["--verbose"] if @verbose
44+
npx_command += ["--expected-failures", expected_failures_yml]
45+
npx_command
46+
end
47+
end
48+
end

‎conformance/expected_failures.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,31 @@ server:
33
- tools-call-elicitation
44
- elicitation-sep1034-defaults
55
- elicitation-sep1330-enums
6+
7+
client:
8+
# TODO: SSE reconnection not implemented in Ruby client.
9+
- sse-retry
10+
# TODO: Elicitation not implemented in Ruby client.
11+
- elicitation-sep1034-client-defaults
12+
# TODO: OAuth/auth not implemented in Ruby client.
13+
- auth/metadata-default
14+
- auth/metadata-var1
15+
- auth/metadata-var2
16+
- auth/metadata-var3
17+
- auth/basic-cimd
18+
- auth/scope-from-www-authenticate
19+
- auth/scope-from-scopes-supported
20+
- auth/scope-omitted-when-undefined
21+
- auth/scope-step-up
22+
- auth/scope-retry-limit
23+
- auth/token-endpoint-auth-basic
24+
- auth/token-endpoint-auth-post
25+
- auth/token-endpoint-auth-none
26+
- auth/pre-registration
27+
- auth/2025-03-26-oauth-metadata-backcompat
28+
- auth/2025-03-26-oauth-endpoint-fallback
29+
- auth/client-credentials-jwt
30+
- auth/client-credentials-basic
31+
- auth/cross-app-access-complete-flow
32+
- auth/offline-access-scope
33+
- auth/offline-access-not-supported

conformance/runner.rb renamed to conformance/server_runner.rb

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
require_relative "server"
77

88
module Conformance
9-
class Runner
9+
class ServerRunner
1010
# Timeout for waiting for the Puma server to start.
1111
SERVER_START_TIMEOUT = 20
1212
SERVER_POLL_INTERVAL = 0.5
@@ -83,7 +83,7 @@ def run_conformance(command, server_pid:)
8383
terminate_server(server_pid)
8484
end
8585

86-
exit(conformance_exit_code || 1)
86+
exit(conformance_exit_code || 1) unless conformance_exit_code == 0
8787
end
8888

8989
def terminate_server(pid)

0 commit comments

Comments
 (0)