chore: additional server conformance tests for SEP-2575#297
Merged
pcarleton merged 9 commits intoMay 21, 2026
Conversation
commit: |
Yuan325
force-pushed
the
additional-sep-2575
branch
from
6fd7dbe to
ed183fb
Compare
Yuan325
force-pushed
the
additional-sep-2575
branch
from
ed183fb to
72d8739
Compare
…tions/listen
The scenario sent params.subscriptions: [{type: 'tools/list-changed'}], a
shape that does not exist in the SEP. SubscriptionsListenRequestParams
requires params.notifications: {toolsListChanged?, promptsListChanged?, ...}
(see schema.ts SubscriptionFilter and the canonical
SubscriptionsListenRequest example). A compliant server would either reject
the old request as invalid params or treat it as subscribing to nothing,
failing the ack/list-changed checks.
Also makes the everything-server's acknowledgment echo the agreed
notifications filter, as the SubscriptionsAcknowledgedNotification schema
requires.
…ames The list-changed checks and the everything-server used notifications/tools/list-changed (hyphen); the spec's method is notifications/tools/list_changed (underscore) per ToolListChangedNotification / PromptListChangedNotification in schema.ts. With the hyphen form the notification-filter check cannot catch a real leak, and the list-changed checks fail a server that emits the correct name. Also drops the f.params.toolsListChanged === true fallback match: that field is the subscriptions/listen request filter, not a notification parameter, so it never appears on a compliant server's notification.
…s out listenToStream aborts the fetch after timeoutMs. If the abort fired while blocked in reader.read(), the rejection propagated to the outer catch and the helper returned [] — discarding every frame already received. A compliant server holds a subscriptions/listen stream open indefinitely, so the timeout is the *normal* way these reads end; previously every subscription check would report a false 'no frames received' FAILURE against such a server. The current everything-server only avoided this by res.end()ing immediately after the ack.
The notification-filter and list-changed checks fired the mutation trigger *before* opening the subscriptions/listen stream. A compliant server only delivers list-changed notifications to streams that are open at the time of the change, so the SHOULD checks would fail any server that does not replay past changes to new subscribers, and the filter check could never observe a real leak. listenToStream now takes an onFirstFrame hook; the three trigger-based checks open the stream, wait for the acknowledgment, then fire the mutation on a separate connection and keep reading. The everything-server now keeps subscriptions/listen streams open, registers them with their filters, and fans the list_changed notification out to open streams when a trigger tool runs - instead of unconditionally emitting a list-changed frame on every stream open (which itself violated the notification-filter requirement in spirit: it announced a change when nothing had changed).
…ication server-tags-subscription-id only inspected the acknowledgment frame, so a server that tags the ack but omits io.modelcontextprotocol/subscriptionId from subsequent notifications passed. The check now triggers a tool-list change once the stream is acknowledged and asserts every notification frame on the stream carries the id in params._meta. Also drops the f.body.method / f.params.method fallbacks when resolving a frame's method - those shapes don't exist in JSON-RPC and only masked malformed frames.
…-changed checks
The spec text behind server-sends-{tools,prompts}-list-changed-on-subscription
is a SHOULD ('servers that declared the listChanged capability SHOULD send a
notification...'). Severity follows the spec keyword: MUST -> FAILURE,
SHOULD -> WARNING. runCheck now accepts a warning flag and the two
list-changed checks use it on their failure paths.
…osed http-server-no-independent-requests-on-stream and server-no-log-without-loglevel reported SUCCESS when the server rejected the tools/call outright (e.g. test_streaming_elicitation / test_logging_tool do not exist): a single error frame contains no independent request and no log notification, so the check passed without exercising anything. The sibling list-changed checks already SKIP when their trigger tool is missing; these now do the same.
…om traceability Marks the section-C rows of modelcontextprotocol#296 as excluded rather than untested: no-prior-context, no-connection-reuse-required, disconnect-is-cancel and stops-on-cancel describe internal server state that a black-box harness cannot observe on the wire. With these excluded and the part-B checks landed, every testable server-side SEP-2575 requirement is covered; the remaining untested rows are all client-side and blocked on a SEP-2575-aware reference client.
pcarleton
approved these changes
May 21, 2026
Member
|
@Yuan325 I added a few fixes on top. I might check toolbox's impl since a few tests diverged from the spec. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added the 7 additional checks from part B of #296. This adds additional conformance tests for modelcontextprotocol/modelcontextprotocol#2575 to validate the stateless-MCP behavior.
Motivation and Context
Ensure that servers adhere to the SEP.
How Has This Been Tested?
Locally and against MCP Toolbox's PR for SEP-2575.
Breaking Changes
n/a
Types of changes
Checklist