Skip to content

ci(conformance): integrate MCP server conformance test pipeline#3106

Open
anubhav756 wants to merge 14 commits into
mainfrom
anubhav-conformance
Open

ci(conformance): integrate MCP server conformance test pipeline#3106
anubhav756 wants to merge 14 commits into
mainfrom
anubhav-conformance

Conversation

@anubhav756
Copy link
Copy Markdown
Contributor

@anubhav756 anubhav756 commented Apr 20, 2026

Description

This PR integrates the MCP conformance test runner to establish automated auditing framework.

Conformance Dashboard

#3152

Changes

  • Deploys a GHA workflow triggering every morning at 6:00 AM.
    • It executes the conformance runner, and automatically maintains exactly one single open issue to track gaps without inbox flooding.
  • Baseline YAML config masks out expected failures (like legacy SSE and LLM sampling) to protect our daily PR presubmits from failing.
  • Patches a Host header by configuring --allowed-hosts localhost on server startup.

Note

We have opened an upstream PR to fix the Repository Health table, in the generated GitHub issue, in the official conformance tool: modelcontextprotocol/conformance#239
image

@anubhav756 anubhav756 self-assigned this Apr 20, 2026
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from e500d8e to 3ca25fe Compare April 20, 2026 22:09
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a conformance-baseline.yml file to the root directory. The reviewer suggests moving this file to a more appropriate location, such as the tests/ or ci/ directory, to maintain a clean project structure and align with existing organizational patterns.

Comment thread tests/conformance/conformance-baseline.yml
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 8 times, most recently from ce64dad to b24e7b5 Compare April 20, 2026 23:11
@anubhav756 anubhav756 marked this pull request as ready for review April 20, 2026 23:14
@anubhav756 anubhav756 requested a review from a team as a code owner April 20, 2026 23:14
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 6 times, most recently from 29cd648 to c33af02 Compare April 23, 2026 13:06
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 7 times, most recently from 298a8b3 to 4de1fd5 Compare May 5, 2026 12:32
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from 4de1fd5 to 695b98a Compare May 6, 2026 08:02
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch 3 times, most recently from 695b98a to 9f161fb Compare May 11, 2026 08:30
@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from 9f161fb to f5665e8 Compare May 15, 2026 08:53
@twishabansal
Copy link
Copy Markdown
Contributor

Is it intentional to run this workflow once a day but also run it on PRs?

@anubhav756
Copy link
Copy Markdown
Contributor Author

Is it intentional to run this workflow once a day but also run it on PRs?

Same as googleapis/mcp-toolbox-sdk-python#637 (comment)

@anubhav756 anubhav756 force-pushed the anubhav-conformance branch from e9f1605 to f650009 Compare May 15, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants