Skip to content

Commit b5eb914

Browse files
committed
[CI] Add weekly conformance run against latest upstream release
The PR pipeline only catches conformance changes when someone opens a PR. Upstream `@modelcontextprotocol/conformance` releases between PRs go untested. This adds a weekly cron + `workflow_dispatch` trigger pinned to `@latest`, and opens a tracking issue on scheduled failures.
1 parent 06917f6 commit b5eb914

1 file changed

Lines changed: 90 additions & 0 deletions

File tree

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
name: conformance-weekly
2+
3+
# Runs the MCP conformance suite weekly against the latest
4+
# @modelcontextprotocol/conformance release. The on:pull_request pipeline
5+
# pins to whatever version is available at PR time; this schedule catches
6+
# upstream releases that add scenarios between PRs.
7+
on:
8+
schedule:
9+
- cron: '0 6 * * 1' # Mondays 06:00 UTC
10+
workflow_dispatch:
11+
12+
permissions:
13+
contents: read
14+
issues: write
15+
16+
jobs:
17+
server:
18+
name: conformance / server (latest)
19+
runs-on: ubuntu-latest
20+
steps:
21+
- uses: actions/checkout@v6
22+
- uses: actions/setup-node@v6
23+
with:
24+
node-version: '22'
25+
- run: composer install --prefer-dist --no-progress --no-interaction
26+
- name: Start conformance server
27+
run: |
28+
mkdir -p tests/Conformance/sessions tests/Conformance/logs
29+
chmod -R 777 tests/Conformance/sessions tests/Conformance/logs
30+
docker compose -f tests/Conformance/Fixtures/docker-compose.yml up -d
31+
sleep 5
32+
- name: Run conformance tests
33+
working-directory: ./tests/Conformance
34+
run: npx --yes @modelcontextprotocol/conformance@latest server --url http://localhost:8000/ --expected-failures conformance-baseline.yml
35+
- name: Show logs on failure
36+
if: failure()
37+
run: |
38+
docker compose -f tests/Conformance/Fixtures/docker-compose.yml logs
39+
cat tests/Conformance/logs/conformance.log 2>/dev/null || true
40+
find tests/Conformance/results -name checks.json -exec sh -c 'echo "--- $1 ---"; cat "$1"' _ {} \; 2>/dev/null || true
41+
42+
client:
43+
name: conformance / client (latest)
44+
runs-on: ubuntu-latest
45+
steps:
46+
- uses: actions/checkout@v6
47+
- uses: shivammathur/setup-php@v2
48+
with:
49+
php-version: '8.4'
50+
coverage: none
51+
- uses: actions/setup-node@v6
52+
with:
53+
node-version: '22'
54+
- run: composer install --prefer-dist --no-progress --no-interaction
55+
- run: mkdir -p tests/Conformance/logs
56+
- name: Run conformance tests
57+
working-directory: ./tests/Conformance
58+
run: npx --yes @modelcontextprotocol/conformance@latest client --command "php ${{ github.workspace }}/tests/Conformance/client.php" --suite all --expected-failures conformance-baseline.yml
59+
- name: Show logs on failure
60+
if: failure()
61+
run: |
62+
cat tests/Conformance/logs/client-conformance.log 2>/dev/null || true
63+
find tests/Conformance/results -name checks.json -exec sh -c 'echo "--- $1 ---"; cat "$1"' _ {} \; 2>/dev/null || true
64+
65+
notify:
66+
name: Open issue on failure
67+
runs-on: ubuntu-latest
68+
needs: [server, client]
69+
if: failure() && github.event_name == 'schedule'
70+
env:
71+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
72+
GH_REPO: ${{ github.repository }}
73+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
74+
steps:
75+
- name: File or comment tracking issue
76+
run: |
77+
existing=$(gh issue list --label conformance-weekly --state open --json number --jq '.[0].number // empty')
78+
if [ -n "$existing" ]; then
79+
gh issue comment "$existing" --body "New failure on $(date -u +%FT%TZ): $RUN_URL"
80+
else
81+
gh issue create \
82+
--title '[conformance] Weekly conformance run failed' \
83+
--label conformance-weekly \
84+
--body "Weekly conformance against \`@modelcontextprotocol/conformance@latest\` failed.
85+
86+
- Run: $RUN_URL
87+
- Triggered: $(date -u +%FT%TZ)
88+
89+
Upstream likely published a release whose scenarios the SDK does not satisfy. Either fix the SDK, update the conformance fixtures, or add the new failure to \`tests/Conformance/conformance-baseline.yml\`."
90+
fi

0 commit comments

Comments
 (0)