Skip to content

Commit f448e87

Browse files
jeffhandleyCopilot
jeffhandley
and
Copilot
committed
Add conformance-tier-audit skill and update skill output paths
Add a new conformance-tier-audit skill that orchestrates MCP SDK tier audits by starting the conformance server, pre-building the client, cloning the conformance repo, and delegating to its mcp-sdk-tier-audit skill. Update both conformance-tier-audit and issue-triage skills to write output files to artifacts/skill-output/ (already gitignored via artifacts/). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent dfa9342 commit f448e87

2 files changed

Lines changed: 174 additions & 1 deletion

File tree

‎.github/skills/conformance-tier-audit/SKILL.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
---
2+
name: conformance-tier-audit
3+
description: >-
4+
Run an MCP SDK conformance tier audit for the C# MCP SDK. Starts the conformance server,
5+
pre-builds the conformance client, clones the conformance repo, and delegates
6+
to its mcp-sdk-tier-audit skill for all evaluation and reporting.
7+
argument-hint: '[--port <port>] [--framework <tfm>] [--branch <branch>]'
8+
compatibility: >-
9+
Requires: Node.js >= 20, .NET SDK (net9.0+),
10+
and internet access to clone the github.com/modelcontextprotocol/conformance repo.
11+
---
12+
13+
# Conformance Tier Audit — C# MCP SDK
14+
15+
This skill orchestrates a tier audit by preparing the C# SDK's conformance server and client, then delegating to the `mcp-sdk-tier-audit` skill from the `modelcontextprotocol/conformance` repo for all tier evaluation, scoring, and report generation.
16+
17+
## Step 0: Pre-flight Checks
18+
19+
### 0a. Parse arguments
20+
21+
Extract optional overrides from the user's input (all have defaults):
22+
23+
- **port** (default: `3001`): Port for the conformance server
24+
- **framework** (default: `net9.0`): Target framework for `dotnet run`. Available: `net8.0`, `net9.0`, `net10.0`
25+
- **branch** (default: current branch): Git branch for GitHub API checks. Derive from: `git rev-parse --abbrev-ref HEAD`
26+
27+
## Step 1: Start the Conformance Server
28+
29+
Start the C# SDK's conformance server as a detached background process from the SDK root (the cwd):
30+
31+
```
32+
dotnet run --project tests/ModelContextProtocol.ConformanceServer --framework <framework> -p:NuGetAudit=false -- --urls http://localhost:<port>
33+
```
34+
35+
Use `mode: async, detach: true` so the server persists.
36+
37+
Wait a few seconds, then verify it's reachable:
38+
39+
**PowerShell** (Windows):
40+
```powershell
41+
curl -sf http://localhost:<port> -o NUL -w '%{http_code}'
42+
```
43+
44+
**Bash** (Linux/macOS):
45+
```bash
46+
curl -sf http://localhost:<port> -o /dev/null -w '%{http_code}'
47+
```
48+
49+
A `400` response is expected and means the server is running (it rejects plain GET requests).
50+
51+
If the server fails to start, check stderr for build errors. Common issues:
52+
- **NU1903 (NuGet vulnerability)**: The `-p:NuGetAudit=false` flag should suppress this.
53+
- **Multiple TFMs**: The `--framework` flag is required because the project multi-targets.
54+
55+
## Step 2: Pre-build the Conformance Client
56+
57+
**CRITICAL**: Pre-build the conformance client before the audit runs tests. The conformance runner executes 26 scenarios in parallel — without pre-building, each `dotnet run` invocation triggers a full compilation, causing massive CPU contention and 30-second timeouts.
58+
59+
**PowerShell** (Windows):
60+
```powershell
61+
dotnet build tests\ModelContextProtocol.ConformanceClient --framework <framework> -p:NuGetAudit=false --nologo -v q
62+
```
63+
64+
**Bash** (Linux/macOS):
65+
```bash
66+
dotnet build tests/ModelContextProtocol.ConformanceClient --framework <framework> -p:NuGetAudit=false --nologo -v q
67+
```
68+
69+
## Step 3: Clone the Conformance Repo
70+
71+
Clone the conformance repo to a temporary directory and build it:
72+
73+
**PowerShell** (Windows):
74+
```powershell
75+
$conformanceDir = Join-Path $env:TEMP "mcp-conformance-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
76+
git clone --depth 1 https://github.com/modelcontextprotocol/conformance.git $conformanceDir
77+
cd $conformanceDir
78+
npm install --silent
79+
npm run build
80+
```
81+
82+
**Bash** (Linux/macOS):
83+
```bash
84+
conformanceDir=$(mktemp -d)
85+
git clone --depth 1 https://github.com/modelcontextprotocol/conformance.git "$conformanceDir"
86+
cd "$conformanceDir"
87+
npm install --silent
88+
npm run build
89+
```
90+
91+
Store the conformance directory path for cleanup later.
92+
93+
## Step 4: Delegate to the Conformance Repo's Audit Skill
94+
95+
Read the `mcp-sdk-tier-audit` skill from the cloned conformance repo:
96+
97+
```
98+
$conformanceDir/.claude/skills/mcp-sdk-tier-audit/SKILL.md
99+
```
100+
101+
Follow that skill's instructions end-to-end, providing these inputs:
102+
103+
| Input | Value |
104+
|-------|-------|
105+
| `--repo` | `modelcontextprotocol/csharp-sdk` |
106+
| `--branch` | `<branch>` |
107+
| `--conformance-server-url` | `http://localhost:<port>` |
108+
| `--client-cmd` | See platform-specific commands below |
109+
110+
Where `<sdk-path>` is the absolute path to the SDK checkout (the original cwd, not the conformance temp dir).
111+
112+
**PowerShell** (Windows) — use backslashes and `%MCP_CONFORMANCE_SCENARIO%`:
113+
```
114+
dotnet run --project <sdk-path>\tests\ModelContextProtocol.ConformanceClient --framework <framework> -p:NuGetAudit=false --no-build -- %MCP_CONFORMANCE_SCENARIO%
115+
```
116+
117+
**Bash** (Linux/macOS) — use forward slashes and `$MCP_CONFORMANCE_SCENARIO`:
118+
```
119+
dotnet run --project <sdk-path>/tests/ModelContextProtocol.ConformanceClient --framework <framework> -p:NuGetAudit=false --no-build -- $MCP_CONFORMANCE_SCENARIO
120+
```
121+
122+
### Windows Quoting Note
123+
124+
The Windows `--client-cmd` uses `%MCP_CONFORMANCE_SCENARIO%` — the conformance runner sets this as an environment variable and spawns the client with `shell: true`, so the Windows shell expands it. If the tier-check CLI reports 0/N client scenarios with 0 checks passed AND 0 checks failed, the command is being parsed incorrectly due to the CLI wrapping it in single quotes (which don't work on Windows cmd.exe). In that case, run the client suite directly:
125+
126+
**PowerShell** (Windows):
127+
```powershell
128+
cd $conformanceDir
129+
node dist/index.js client `
130+
--command "dotnet run --project <sdk-path>\tests\ModelContextProtocol.ConformanceClient --framework <framework> -p:NuGetAudit=false --no-build -- %MCP_CONFORMANCE_SCENARIO%" `
131+
--suite all `
132+
-o <output-dir>
133+
```
134+
135+
**Bash** (Linux/macOS):
136+
```bash
137+
cd "$conformanceDir"
138+
node dist/index.js client \
139+
--command "dotnet run --project <sdk-path>/tests/ModelContextProtocol.ConformanceClient --framework <framework> -p:NuGetAudit=false --no-build -- $MCP_CONFORMANCE_SCENARIO" \
140+
--suite all \
141+
-o <output-dir>
142+
```
143+
144+
### Output Location Override
145+
146+
The conformance skill may specify its own output location. Override it: write all output files to `artifacts/skill-output/` at the SDK repo root. Create the directory if it doesn't exist. The `artifacts/` directory is already gitignored.
147+
148+
## Step 5: Cleanup
149+
150+
Stop the conformance server process. Remove the temporary conformance repo directory:
151+
152+
**PowerShell** (Windows):
153+
```powershell
154+
Remove-Item -Recurse -Force $conformanceDir
155+
```
156+
157+
**Bash** (Linux/macOS):
158+
```bash
159+
rm -rf "$conformanceDir"
160+
```
161+
162+
## Usage Examples
163+
164+
```
165+
# Default settings (port 3001, net9.0, current branch)
166+
/conformance-tier-audit
167+
168+
# Custom port and framework
169+
/conformance-tier-audit --port 3003 --framework net10.0
170+
171+
# Specific branch for GitHub API checks
172+
/conformance-tier-audit --branch main
173+
```

‎.github/skills/issue-triage/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,7 @@ This step adds significant value but also significant API calls. If the user ask
140140
Produce the triage report following the template in [references/report-format.md](references/report-format.md). The report must follow the BLUF structure with urgency-descending ordering.
141141

142142
**Output destination:**
143-
- **Default (local file):** Save as `{YYYY-MM-DD}-mcp-issue-triage.md` in the current working directory. If a file with that name already exists, suffix with `-2`, `-3`, etc.
143+
- **Default (local file):** Save as `artifacts/skill-output/{YYYY-MM-DD}-mcp-issue-triage.md` at the SDK repo root. Create the `artifacts/skill-output/` directory if it doesn't exist (the `artifacts/` directory is already gitignored). If a file with that name already exists, suffix with `-2`, `-3`, etc.
144144
- **Gist (if requested):** If the user asked to save as a gist, create a **secret** gist using `gh gist create` with a `--desc` describing the report. No confirmation is needed — create the gist, then notify the user with a clickable link to it.
145145

146146
The user may request a gist with phrases like "save as a gist", "create a gist", "gist it", "post to gist", etc.

0 commit comments

Comments
 (0)