Skip to content

Commit e88b341

Browse files
devcrocodkpavlov
devcrocod
and
kpavlov
authored
feat!: add mcp conformance test infra (#585); add SSE reconnection with retry support (#596) (#585)
Adds a comprehensive conformance test suite for the Kotlin MCP SDK, covering core protocol operations, tool calls, elicitation, resources, prompts, and 20 OAuth/auth scenarios - Conformance server and client implementations - OAuth/auth test scenarios: JWT, authorization code flow, client credentials, PKCE, scope handling, cross-app access, client registration - CI workflow - Baseline file for tracking expected failures - Shell script fixes: - #592 - #593 - #596 ## Remaining known failures (tracked issues, will be fixed directly in `main`) - [x] `tools-call-with-logging`, `tools-call-with-progress`, `tools-call-sampling`, `tools-call-elicitation`, `elicitation-sep1034-defaults`- see #599, - [x] `elicitation-sep1330-enums` - #587 #600 - [x] `initialize` - #588 - [x] `tools_call`, `auth/scope-step-up`, `auth/scope-retry-limit` - #589 - [ ] `elicitation-sep1034-client-defaults` - #414 - [x] `sse-retry` - #590 - [ ] `resources-templates-read` - #591 ## Breaking Changes from #596 - `StreamableHttpClientTransport` and `mcpStreamableHttp`/`mcpStreamableHttpTransport`: old constructors accepting `Duration` timeout are now `@Deprecated` — use the new overloads with `ReconnectionOptions` instead - `StreamableHttpClientTransport.close()` no longer calls `terminateSession()` automatically ## Types of changes - [x] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Checklist - [x] I have read the [MCP Documentation](https://modelcontextprotocol.io) - [x] My code follows the repository's style guidelines - [x] New and existing tests pass locally - [x] I have added appropriate error handling - [x] I have added or updated documentation as needed --------- Co-authored-by: Konstantin Pavlov <1517853+kpavlov@users.noreply.github.com>
1 parent 338f948 commit e88b341

42 files changed

Lines changed: 3186 additions & 1122 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/conformance.yml‎

Lines changed: 137 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -7,67 +7,166 @@ on:
77
push:
88
branches: [ main ]
99

10+
env:
11+
JAVA_VERSION: '21'
12+
JAVA_DISTRIBUTION: temurin
13+
NODE_VERSION: '22'
14+
15+
1016
concurrency:
1117
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
1218
# Cancel only when the run is NOT on `main` branch
1319
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
1420

1521
jobs:
16-
run-conformance:
17-
runs-on: ${{ matrix.os }}
18-
name: Run Conformance Tests on ${{ matrix.os }}
22+
server:
23+
runs-on: ubuntu-latest
24+
name: Conformance Server Tests
25+
timeout-minutes: 20
26+
27+
steps:
28+
- uses: actions/checkout@v6
29+
30+
- name: Set up JDK
31+
uses: actions/setup-java@v5
32+
with:
33+
java-version: ${{ env.JAVA_VERSION }}
34+
distribution: ${{ env.JAVA_DISTRIBUTION }}
35+
36+
- name: Setup Gradle
37+
uses: gradle/actions/setup-gradle@v5
38+
with:
39+
cache-read-only: ${{ github.ref != 'refs/heads/main' }}
40+
41+
- name: Build
42+
run: ./gradlew :conformance-test:installDist
43+
44+
- name: Start server
45+
run: |
46+
MCP_PORT=3001 conformance-test/build/install/conformance-test/bin/conformance-test &
47+
for i in $(seq 1 30); do
48+
if curl -s -o /dev/null http://localhost:3001/mcp; then
49+
echo "Server is ready"
50+
break
51+
fi
52+
sleep 1
53+
done
54+
55+
- name: Run conformance tests
56+
uses: modelcontextprotocol/conformance@v0.1.15
57+
with:
58+
mode: server
59+
url: http://localhost:3001/mcp
60+
suite: active
61+
node-version: ${{ env.NODE_VERSION }}
62+
expected-failures: ./conformance-test/conformance-baseline.yml
63+
64+
client:
65+
runs-on: ubuntu-latest
66+
name: "Conformance Client Tests: ${{ matrix.scenario }}"
1967
timeout-minutes: 20
20-
env:
21-
JAVA_OPTS: "-Xmx8g -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Dkotlin.daemon.jvm.options=-Xmx6g"
2268

2369
strategy:
2470
fail-fast: false
2571
matrix:
26-
include:
27-
- os: ubuntu-latest
28-
max-workers: 3
29-
- os: windows-latest
30-
max-workers: 3
31-
- os: macos-latest
32-
max-workers: 2
72+
scenario:
73+
- initialize
74+
- tools_call
75+
- elicitation-sep1034-client-defaults
76+
- sse-retry
77+
78+
steps:
79+
- uses: actions/checkout@v6
80+
81+
- name: Set up JDK
82+
uses: actions/setup-java@v5
83+
with:
84+
java-version: ${{ env.JAVA_VERSION }}
85+
distribution: ${{ env.JAVA_DISTRIBUTION }}
86+
87+
- name: Setup Gradle
88+
uses: gradle/actions/setup-gradle@v5
89+
with:
90+
cache-read-only: ${{ github.ref != 'refs/heads/main' }}
91+
92+
- name: Build
93+
run: ./gradlew :conformance-test:installDist
94+
95+
- name: Run conformance tests
96+
uses: modelcontextprotocol/conformance@v0.1.15
97+
with:
98+
mode: client
99+
command: conformance-test/build/install/conformance-test/bin/conformance-client
100+
scenario: ${{ matrix.scenario }}
101+
node-version: ${{ env.NODE_VERSION }}
102+
expected-failures: ./conformance-test/conformance-baseline.yml
103+
104+
auth:
105+
runs-on: ubuntu-latest
106+
name: Conformance Auth Tests
107+
timeout-minutes: 20
33108

34109
steps:
35110
- uses: actions/checkout@v6
36111

37-
- name: Set up JDK 21
112+
- name: Set up JDK
38113
uses: actions/setup-java@v5
39114
with:
40-
java-version: '21'
41-
distribution: 'temurin'
115+
java-version: ${{ env.JAVA_VERSION }}
116+
distribution: ${{ env.JAVA_DISTRIBUTION }}
42117

43-
- name: Setup Node.js
44-
uses: actions/setup-node@v6
118+
- name: Setup Gradle
119+
uses: gradle/actions/setup-gradle@v5
45120
with:
46-
node-version: '22' # increase only after https://github.com/nodejs/node/issues/56645 will be fixed
121+
cache-read-only: ${{ github.ref != 'refs/heads/main' }}
47122

48-
- name: Setup Conformance Tests
49-
working-directory: conformance-test
50-
run: |-
51-
npm install -g @modelcontextprotocol/conformance@0.1.8
123+
- name: Build
124+
run: ./gradlew :conformance-test:installDist
125+
126+
- name: Run conformance tests
127+
uses: modelcontextprotocol/conformance@v0.1.15
128+
with:
129+
mode: client
130+
command: conformance-test/build/install/conformance-test/bin/conformance-client
131+
suite: auth
132+
node-version: ${{ env.NODE_VERSION }}
133+
expected-failures: ./conformance-test/conformance-baseline.yml
134+
135+
auth-scenarios:
136+
runs-on: ubuntu-latest
137+
name: "Conformance Auth Scenario: ${{ matrix.scenario }}"
138+
timeout-minutes: 20
139+
140+
strategy:
141+
fail-fast: false
142+
matrix:
143+
scenario:
144+
- auth/client-credentials-jwt
145+
- auth/client-credentials-basic
146+
- auth/cross-app-access-complete-flow
147+
148+
steps:
149+
- uses: actions/checkout@v6
150+
151+
- name: Set up JDK
152+
uses: actions/setup-java@v5
153+
with:
154+
java-version: ${{ env.JAVA_VERSION }}
155+
distribution: ${{ env.JAVA_DISTRIBUTION }}
52156

53157
- name: Setup Gradle
54158
uses: gradle/actions/setup-gradle@v5
55159
with:
56-
add-job-summary: 'always'
57160
cache-read-only: ${{ github.ref != 'refs/heads/main' }}
58-
gradle-home-cache-includes: |
59-
caches
60-
notifications
61-
sdks
62-
../.konan/**
63-
64-
- name: Run Conformance Tests
65-
run: |-
66-
./gradlew :conformance-test:test --no-daemon --max-workers ${{ matrix.max-workers }}
67-
68-
- name: Upload Conformance Results
69-
if: always()
70-
uses: actions/upload-artifact@v7
161+
162+
- name: Build
163+
run: ./gradlew :conformance-test:installDist
164+
165+
- name: Run conformance tests
166+
uses: modelcontextprotocol/conformance@v0.1.15
71167
with:
72-
name: conformance-results-${{ matrix.os }}
73-
path: conformance-test/results/
168+
mode: client
169+
command: conformance-test/build/install/conformance-test/bin/conformance-client
170+
scenario: ${{ matrix.scenario }}
171+
node-version: ${{ env.NODE_VERSION }}
172+
expected-failures: ./conformance-test/conformance-baseline.yml

‎build.gradle.kts‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,15 @@ dependencies {
2323
subprojects {
2424
apply(plugin = "org.jlleitschuh.gradle.ktlint")
2525
apply(plugin = "org.jetbrains.kotlinx.kover")
26-
apply(plugin = "dev.detekt")
2726

28-
detekt {
29-
config = files("$rootDir/config/detekt/detekt.yml")
30-
buildUponDefaultConfig = true
31-
failOnSeverity.set(FailOnSeverity.Error)
27+
if (name != "conformance-test" && name != "docs") {
28+
apply(plugin = "dev.detekt")
29+
30+
detekt {
31+
config = files("$rootDir/config/detekt/detekt.yml")
32+
buildUponDefaultConfig = true
33+
failOnSeverity.set(FailOnSeverity.Error)
34+
}
3235
}
3336
}
3437

‎conformance-test/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/results/

‎conformance-test/README.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# MCP Conformance Tests
2+
3+
Conformance tests for the Kotlin MCP SDK. Uses the external
4+
[`@modelcontextprotocol/conformance`](https://www.npmjs.com/package/@modelcontextprotocol/conformance)
5+
runner (pinned to **0.1.15**) to validate compliance with the MCP specification.
6+
7+
## Prerequisites
8+
9+
- **JDK 17+**
10+
- **Node.js 18+** and `npx` (for the conformance runner)
11+
- **curl** (used to poll server readiness)
12+
13+
## Quick Start
14+
15+
Run **all** suites (server, client core, client auth) from the project root:
16+
17+
```bash
18+
./conformance-test/run-conformance.sh all
19+
```
20+
21+
## Commands
22+
23+
```
24+
./conformance-test/run-conformance.sh <command> [extra-args...]
25+
```
26+
27+
| Command | What it does |
28+
|---------------|--------------------------------------------------------------------------------------|
29+
| `list` | [List scenarios available in MCP Conformance Test Framework][list-scenarios-command] |
30+
| `server` | Starts the Ktor conformance server, runs the server test suite against it |
31+
| `client` | Runs the client test suite (`initialize`, `tools_call`, `elicitation`, `sse-retry`) |
32+
| `client-auth` | Runs the client auth test suite (20 OAuth scenarios) |
33+
| `all` | Runs all three suites sequentially |
34+
35+
Any `[extra-args]` are forwarded to the conformance runner (e.g. `--verbose`).
36+
37+
## What the Script Does
38+
39+
1. **Builds** the module via `./gradlew :conformance-test:installDist`
40+
2. For `server` — starts the conformance server on `localhost:3001`, polls until ready
41+
3. Invokes `npx @modelcontextprotocol/conformance@0.1.15` with the appropriate arguments
42+
4. Saves results to `conformance-test/results/<command>/`
43+
5. Cleans up the server process on exit
44+
6. Exits non-zero if any suite fails
45+
46+
## Environment Variables
47+
48+
| Variable | Default | Description |
49+
|------------|---------|---------------------------------|
50+
| `MCP_PORT` | `3001` | Port for the conformance server |
51+
52+
## Project Structure
53+
54+
```
55+
conformance-test/
56+
├── run-conformance.sh # Single entry point script
57+
├── conformance-baseline.yml # Expected failures for known SDK limitations
58+
└── src/main/kotlin/.../conformance/
59+
├── ConformanceServer.kt # Ktor server entry point (StreamableHTTP, DNS rebinding, EventStore)
60+
├── ConformanceClient.kt # Scenario-based client entry point (MCP_CONFORMANCE_SCENARIO routing)
61+
├── ConformanceTools.kt # 18 tool registrations
62+
├── ConformanceResources.kt # 5 resource registrations (static, binary, template, watched, dynamic)
63+
├── ConformancePrompts.kt # 5 prompt registrations (simple, args, image, embedded, dynamic)
64+
├── ConformanceCompletions.kt # completion/complete handler
65+
├── InMemoryEventStore.kt # EventStore impl for SSE resumability (SEP-1699)
66+
└── auth/ # OAuth client for 20 auth scenarios
67+
├── registration.kt # Scenario handler registration
68+
├── utils.kt # Shared utilities: JSON instance, constants, extractOrigin()
69+
├── discovery.kt # Protected Resource Metadata + AS Metadata discovery
70+
├── pkce.kt # PKCE code verifier/challenge generation + AS capability check
71+
├── tokenExchange.kt # Token endpoint interaction (exchange code, error handling)
72+
├── authCodeFlow.kt # Main Authorization Code flow handler (runAuthClient + interceptor)
73+
├── scopeHandling.kt # Scope selection strategy + step-up 403 handling
74+
├── clientRegistration.kt # Client registration logic (pre-reg, CIMD, dynamic)
75+
├── JWTScenario.kt # Client Credentials JWT scenario
76+
├── basicScenario.kt # Client Credentials Basic scenario
77+
└── crossAppAccessScenario.kt # Cross-App Access (SEP-990) scenario
78+
```
79+
80+
## Test Suites
81+
82+
### Server Suite
83+
84+
Tests the conformance server against all server scenarios:
85+
86+
| Category | Scenarios |
87+
|-------------|-------------------------------------------------------------------------------------------------------------------------------------|
88+
| Lifecycle | initialize, ping |
89+
| Tools | text, image, audio, embedded, multiple, progress, logging, error, sampling, elicitation, dynamic, reconnection, JSON Schema 2020-12 |
90+
| Resources | list, read-text, read-binary, templates, subscribe, dynamic |
91+
| Prompts | simple, with-args, with-image, with-embedded-resource, dynamic |
92+
| Completions | complete |
93+
| Security | DNS rebinding protection |
94+
95+
### Client Core Suite
96+
97+
| Scenario | Description |
98+
|---------------------------------------|-----------------------------------------------|
99+
| `initialize` | Connect, list tools, close |
100+
| `tools_call` | Connect, call `add_numbers(a=5, b=3)`, close |
101+
| `elicitation-sep1034-client-defaults` | Elicitation with `applyDefaults` capability |
102+
| `sse-retry` | Call `test_reconnection`, verify reconnection |
103+
104+
### Client Auth Suite
105+
106+
17 OAuth Authorization Code scenarios + 2 Client Credentials scenarios (`jwt`, `basic`) + 1 Cross-App Access scenario = 20 total.
107+
108+
> [!NOTE]
109+
> Auth scenarios are implemented using Ktor's `HttpClient` plugins (`HttpSend` interceptor,
110+
> `ktor-client-auth`) as a standalone OAuth client. They do not use the SDK's built-in auth support.
111+
112+
## Known SDK Limitations
113+
114+
8 scenarios are expected to fail due to current SDK limitations (tracked in [
115+
`conformance-baseline.yml`](conformance-baseline.yml).
116+
117+
| Scenario | Suite | Root Cause |
118+
|---------------------------------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
119+
| `tools-call-with-logging` | server | Notifications from tool handlers have no `relatedRequestId`; transport routes them to the standalone SSE stream instead of the request-specific stream |
120+
| `tools-call-with-progress` | server | *(same as above)* |
121+
| `tools-call-sampling` | server | *(same as above)* |
122+
| `tools-call-elicitation` | server | *(same as above)* |
123+
| `elicitation-sep1034-defaults` | server | *(same as above)* |
124+
| `elicitation-sep1330-enums` | server | *(same as above)* |
125+
| `resources-templates-read` | server | SDK does not implement `addResourceTemplate()` with URI pattern matching; resources are looked up by exact URI |
126+
| `elicitation-sep1034-client-defaults` | client | SDK does not fill in `default` values from the elicitation request schema before sending the response |
127+
128+
These failures reveal SDK gaps and are intentionally not fixed in this module.
129+
130+
[list-scenarios-command]: https://github.com/modelcontextprotocol/conformance/tree/main?tab=readme-ov-file#list-available-scenarios

0 commit comments

Comments
 (0)