Add partial-run support and fix implicit skip handling in tier audit#236
Open
jeffhandley wants to merge 3 commits intomodelcontextprotocol:mainfrom
Open
Add partial-run support and fix implicit skip handling in tier audit#236jeffhandley wants to merge 3 commits intomodelcontextprotocol:mainfrom
jeffhandley wants to merge 3 commits intomodelcontextprotocol:mainfrom
Conversation
Previously, omitting --conformance-server-url or --client-cmd would produce a definitive tier classification despite incomplete data. Extract resolveTierCheckPlan() to detect missing inputs and treat them as implicit skips. partial_run is derived from actual check statuses (any skipped check -> partial) rather than only from the explicit --skip-conformance flag. - Add resolveTierCheckPlan() with unit test for implicit skip - Add checks/skipped.ts with factory functions for skipped payloads - Handle skipped status in tier-logic, output formatters (empty circle symbol), and conformance matrix display - Add partial_run field to TierScorecard type - Split CLI checks docs into Conformance Tests / Repository Health - Update SKILL.md partial-run guidance and report template Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Expand the tier-check CLI with granular skip flags so users can run targeted subsets of the audit: --skip-server-conformance skip only server conformance --skip-client-conformance skip only client conformance --skip-conformance skip both (server + client) --skip-repo-health skip all GitHub-backed repo-health checks The skill layer adds --scope presets (server, client, conformance, repo-health, full) that expand to the appropriate skip flags, plus --skip-docs-eval for the AI documentation evaluation. --skip-repo-health also suppresses the AI policy evaluation in the skill layer, keeping repo-health behavior aligned end-to-end. When any check is skipped the run is partial: tier classification shows N/A and skipped rows display as empty circles. A nothing-to-run guard exits early when every check category is excluded. - Expand resolveTierCheckPlan() with skip flag parameters - Add CLI options and nothing-to-run guard - Token now optional when --skip-repo-health excludes all API calls - Add unit tests for nothing-to-run and explicit scope exclusions - Add scope presets and skip flag docs to SKILL.md and READMEs - Add partial-run examples to root README.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two improvements to the
tier-checkCLI, split into separate commits for reviewability.Motivation and Context
When the tier audit skill is invoked with only server or only client conformance results (or neither), the omitted suite's checks were reported as failures rather than being recognized as intentionally absent. This produced misleading tier classifications and noisy reports. Additionally, there was no way to scope an audit to just conformance tests or just repository health checks, meaning every run paid the full cost even when only part of the audit was needed.
How Has This Been Tested?
resolveTierCheckPlan()covering implicit skip detection, nothing-to-run validation, and explicit scope exclusions (3 new tests)eslint+prettier) passes on each commit independentlyBreaking Changes
None. All new CLI flags are optional and the default behavior (full audit) is unchanged.
Types of changes
Checklist
Additional context
Commit 1 — Fix misleading tier results when conformance inputs are omitted
resolveTierCheckPlan()to detect when conformance inputs are absent and mark those suites as skipped rather than failedpartial_runfield toTierScorecard(derived from actual check statuses)○, conformance tables and totals gracefully handle absent suites, and a partial-run banner is shown when applicableauthorization-server-metadata.tsCommit 2 — Add partial-run support with
--skip-*flags and--scopepresets--skip-server-conformance,--skip-client-conformance,--skip-conformance,--skip-repo-health--scopepresets in SKILL.md (server, client, conformance, repo-health, full) that the AI skill layer expands to--skip-*flagsSample output
"/mcp-sdk-tier-audit for modelcontextprotocol/csharp-sdk with a scope of repo-health only"
MCP SDK Tier Audit: modelcontextprotocol/csharp-sdk
Date: 2026-04-17
Branch: main
Scope: partial run — repo-health only (conformance and docs eval skipped via
--scope repo-health)Auditor: mcp-sdk-tier-audit skill (automated + subagent evaluation)
Tier Assessment: N/A (partial run)
This is a partial assessment covering repository health checks only. Server/client conformance tests and documentation coverage evaluation were excluded by scope. No definitive tier classification is assigned.
Requirements Summary
Tier Determination
No tier assigned — partial run (repo-health only). A full audit is required for tier classification.
All repo-health checks that ran are passing at Tier 1 level. Conformance testing (1a, 1b) and documentation evaluation (5) remain unevaluated and are required before a tier can be assigned.
Server Conformance
○ Not run — excluded by scope (
--scope repo-health)Client Conformance
○ Not run — excluded by scope (
--scope repo-health)Issue Triage Details
All 12 required labels are present:
bug,enhancement,question,needs confirmation,needs repro,ready for work,good first issue,help wanted,P0,P1,P2,P3. Issue types are not used (label-based triage).The triage rate of 97.4% exceeds both the Tier 1 threshold (≥ 90%) and the Tier 2 threshold (≥ 80%). Only 2 issues exceeded the SLA window. Median and p95 triage times (58.4 h and 61.7 h respectively) are well within the 2-business-day Tier 1 standard.
Critical Bug Resolution
No open P0 issues. No P0s were closed during the evaluation window. The
all_p0s_resolved_within_7dandall_p0s_resolved_within_14dflags both reporttrue, satisfying both Tier 1 (7-day) and Tier 2 (14-day) standards.Stable Release & Spec Tracking
Meets Tier 1 requirements for both stable release availability and spec tracking responsiveness.
Policy Evaluation
Dependency Update Policy: PASS
Dependabot is configured with weekly NuGet and GitHub Actions updates, grouped dependencies, ignore rules for product dependencies, and PR limits. A
DEPENDENCY_POLICY.mdordocs/dependency-policy.mdfile is not present, but the active Dependabot configuration (.github/dependabot.yml) demonstrates a published and enforced update policy.Roadmap: PASS (Tier 1 and Tier 2)
docs/roadmap.mdis present. It references org-level spec revision project boards, documents current focus areas (next spec revision, Tasks experimental support, end-to-end scenarios), and links to GitHub milestones for planned versions.Versioning Policy: PASS (Tier 1)
docs/versioning.mdcomprehensively documents SemVer 2.0.0 adherence, defines what constitutes breaking changes (incompatible API changes, spec schema changes), explains how breaking changes are communicated (release notes,breaking-changelabels,[Obsolete]attributes with MCP-prefixed diagnostics), and details the support policy for MAJOR/MINOR/PATCH versions.Missing Policy Files
The following policy files were not found but are not blocking for tier compliance:
CHANGELOG.md— not present (release notes may be published via GitHub Releases instead)BREAKING_CHANGES.md— not present (breaking changes are documented indocs/versioning.mdand communicated via labels)Documentation Coverage
○ Not run — excluded by scope (
--scope repo-health)Raw CLI Data
CLI JSON output (click to expand)
{ "repo": "modelcontextprotocol/csharp-sdk", "branch": "main", "timestamp": "2026-04-17T06:56:51.484Z", "version": "1.2.0", "partial_run": true, "checks": { "conformance": { "status": "skipped" }, "client_conformance": { "status": "skipped" }, "labels": { "status": "pass", "present": 12, "required": 12, "missing": [], "found": ["bug","enhancement","question","needs confirmation","needs repro","ready for work","good first issue","help wanted","P0","P1","P2","P3"], "uses_issue_types": false }, "triage": { "status": "pass", "compliance_rate": 0.9736842105263158, "total_issues": 114, "triaged_within_sla": 111, "exceeding_sla": 2, "median_hours": 58.4, "p95_hours": 61.7 }, "p0_resolution": { "status": "pass", "open_p0s": 0, "open_p0_details": [], "closed_within_7d": 0, "closed_within_14d": 0, "closed_total": 0, "all_p0s_resolved_within_7d": true, "all_p0s_resolved_within_14d": true }, "stable_release": { "status": "pass", "version": "1.2.0", "is_stable": true, "is_prerelease": false }, "policy_signals": { "status": "partial", "files": { "CHANGELOG.md": false, "SECURITY.md": true, "CONTRIBUTING.md": true, "DEPENDENCY_POLICY.md": false, "docs/dependency-policy.md": false, ".github/dependabot.yml": true, ".github/renovate.json": false, "renovate.json": false, "ROADMAP.md": false, "docs/roadmap.md": true, "VERSIONING.md": false, "docs/versioning.md": true, "BREAKING_CHANGES.md": false } }, "spec_tracking": { "status": "pass", "latest_spec_release": "2025-11-25T21:17:42Z", "latest_sdk_release": "2025-11-25T23:44:39Z", "sdk_release_within_30d": true, "days_gap": 0 } } }Remediation Guide: modelcontextprotocol/csharp-sdk
Date: 2026-04-17
Scope: partial run — repo-health only (conformance and docs eval skipped via
--scope repo-health)Current Tier: N/A (partial run)
Findings from scoped run
All repo-health checks passed. No remediation items were identified in the evaluated scope.
Not evaluated (excluded by scope)
The following areas were not assessed in this partial run and require a full audit (
--scope full) to evaluate:A full audit is required to enumerate all tier gaps and assign a definitive tier classification.
Recommended Next Steps
--scope full) to obtain server/client conformance pass rates, documentation coverage scores, and a definitive tier classification."/mcp-sdk-tier-audit for modelcontextprotocol/csharp-sdk. Use the main branch. Skip repo health and docs."
MCP SDK Tier Audit: modelcontextprotocol/csharp-sdk
Date: 2026-04-17
Branch: main
Scope: partial run — conformance only (repo health and docs eval skipped via
--skip-repo-health --skip-docs-eval)Auditor: mcp-sdk-tier-audit skill (automated + subagent evaluation)
Tier Assessment: N/A (partial run)
This is a partial assessment covering server and client conformance tests only. Both server and client achieve a 100% pass rate on all date-versioned scenarios. Three extension-only client auth failures were observed (not scored for tier). No tier is assigned — repo health and documentation evaluation were not performed.
Requirements Summary
Tier Determination
No tier assigned — partial run (conformance only). A full audit is required for tier classification.
Conformance Matrix
Informational (not scored for tier):
No baseline file found.
Server Conformance Details
Pass rate: 100% (30/30)
All 30 server scenarios pass. They span spec versions 2025-06-18 and 2025-11-25.
sse-multiple-streams— PASSelicitation-sep1330-enums— PASSelicitation-sep1034-defaults— PASSdns-rebinding-protection— PASSAll 30 scenarios: PASS
Client Conformance Details
Date-versioned pass rate: 100% (23/23)
Full suite (incl. informational): 26/26 scored + 3 extension failures
Core Scenarios (4/4)
Auth Scenarios — Date-versioned (19/19)
2025-03-26 (2/2):
2025-06-18 (3/3):
2025-11-25 (14/14):
Auth Scenarios — Draft (3/3, informational)
Draft scenarios are informational and not scored for tier classification.
Auth Scenarios — Extension (0/3, informational)
Extension scenarios are informational and not scored for tier classification. All three failures show 8 of 10 checks passing.
Issue Triage Details
○ Not run — excluded by scope (
--skip-repo-health)Documentation Coverage
○ Not run — excluded by scope (
--skip-docs-eval)Policy Evaluation
○ Not run — excluded by scope (
--skip-repo-health)Remediation Guide: modelcontextprotocol/csharp-sdk
Date: 2026-04-17
Scope: partial run — conformance only (repo health and docs eval skipped via
--skip-repo-health --skip-docs-eval)Current Tier: N/A (partial run)
Findings from scoped run
All date-versioned conformance scenarios (server and client) pass at 100%. No remediation is required for tier-scored conformance tests.
Extension Scenario Failures (informational)
The 3 failing extension scenarios each pass 8/10 checks:
These are extension scenarios and do not block tier advancement. They represent client_credentials grant type support which is not yet part of the dated MCP spec.
Not evaluated (excluded by scope)
The following areas were not assessed in this partial run and require a full audit to evaluate:
A full audit (
--scope full) is required to enumerate all tier gaps and assign a definitive tier classification.Recommended Next Steps
baseline.ymlto document expected extension failures