Skip to content

Commit 2e5451c

Browse files
jeffhandleyCopilot
andcommitted
Remove own tier logic; fix action summary and artifact output
Two issues fixed: 1. Tier logic: The prompt was telling the agent to 'Apply tier logic' itself (Step 2.4). Now the prompt explicitly says: 'Do not apply your own tier logic or scoring — use only the conformance skill's thresholds, rules, and templates.' All tier determination is delegated to the conformance repo's mcp-sdk-tier-audit skill. 2. Action summary: The report was not appearing on the workflow summary page because the agent wasn't writing to $GITHUB_STEP_SUMMARY reliably. Now Step 3 is restructured with explicit requirements: - MUST write /tmp/audit-report.md (artifact) - MUST cat it to $GITHUB_STEP_SUMMARY (action summary) - Both required BEFORE Step 4 (publish) - 'Action Summary' mode now explicitly calls noop - Issue body must be identical to the action summary content Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 3ca689b commit 2e5451c

File tree

2 files changed

+89
-85
lines changed

2 files changed

+89
-85
lines changed

.github/workflows/sdk-tier-audit.lock.yml

Lines changed: 18 additions & 23 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/sdk-tier-audit.md

Lines changed: 71 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@ permissions:
66
issues: read
77
pull-requests: read
88

9+
network:
10+
allowed:
11+
- defaults
12+
- node
13+
- dotnet
14+
- github
15+
916
safe-outputs:
1017
create-issue:
1118
title-prefix: "[C# SDK Tier Audit] "
@@ -21,43 +28,51 @@ tools:
2128

2229
if: github.repository_owner == 'modelcontextprotocol' || github.event_name == 'workflow_dispatch'
2330

24-
env:
25-
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
26-
2731
concurrency:
2832
group: tier-audit-${{ github.event.inputs.scope || 'Conformance + Repo Health' }}
2933
cancel-in-progress: true
3034

35+
timeout-minutes: 120
36+
3137
runtimes:
3238
node:
33-
version: "22"
39+
version: "24"
3440
dotnet:
3541
version: "10.0"
3642

37-
network:
38-
allowed:
39-
- defaults
40-
- node
41-
- dotnet
42-
- github
43-
44-
timeout-minutes: 120
43+
env:
44+
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
4545

4646
steps:
47-
- name: Write tier-check scorecard to file
47+
- name: Write tier-check outputs to files
4848
env:
4949
TIER_CHECK_JSON: ${{ needs.tier-check.outputs.tier_check_json }}
50-
run: echo "$TIER_CHECK_JSON" > /tmp/tier-check-scorecard.json
50+
AUDIT_SCOPE: ${{ needs.tier-check.outputs.scope }}
51+
AUDIT_OUTPUT: ${{ needs.tier-check.outputs.output }}
52+
AUDIT_SDK_REPO: ${{ needs.tier-check.outputs.csharp_sdk_repo }}
53+
AUDIT_SDK_BRANCH: ${{ needs.tier-check.outputs.csharp_sdk_branch }}
54+
AUDIT_CONF_REPO: ${{ needs.tier-check.outputs.conformance_repo }}
55+
AUDIT_CONF_BRANCH: ${{ needs.tier-check.outputs.conformance_branch }}
56+
run: |
57+
echo "$TIER_CHECK_JSON" > /tmp/tier-check-scorecard.json
58+
mkdir -p /tmp/audit-params
59+
echo "$AUDIT_SCOPE" > /tmp/audit-params/scope
60+
echo "$AUDIT_OUTPUT" > /tmp/audit-params/output
61+
echo "$AUDIT_SDK_REPO" > /tmp/audit-params/csharp-sdk-repo
62+
echo "$AUDIT_SDK_BRANCH" > /tmp/audit-params/csharp-sdk-branch
63+
echo "$AUDIT_CONF_REPO" > /tmp/audit-params/conformance-repo
64+
echo "$AUDIT_CONF_BRANCH" > /tmp/audit-params/conformance-branch
5165
5266
post-steps:
53-
- name: Upload audit report
67+
- name: Write audit report to action summary
5468
if: always()
55-
uses: actions/upload-artifact@v4
56-
with:
57-
name: audit-report
58-
path: /tmp/audit-report.md
59-
retention-days: 90
60-
if-no-files-found: ignore
69+
run: |
70+
if [ -f /tmp/audit-report.md ]; then
71+
cat /tmp/audit-report.md >> "$GITHUB_STEP_SUMMARY"
72+
else
73+
echo "## Tier Audit: No Report" >> "$GITHUB_STEP_SUMMARY"
74+
echo "The agent did not produce /tmp/audit-report.md." >> "$GITHUB_STEP_SUMMARY"
75+
fi
6176
6277
on:
6378
schedule: weekly on thursday around 6:30am utc-5
@@ -266,12 +281,18 @@ Perform a tier audit of the C# MCP SDK. The deterministic tier-check scorecard h
266281

267282
## Inputs
268283

269-
- **Scope**: ${{ needs.tier-check.outputs.scope }}
270-
- **Output mode**: ${{ needs.tier-check.outputs.output }}
271-
- **C# SDK**: ${{ needs.tier-check.outputs.csharp_sdk_repo }} (branch: ${{ needs.tier-check.outputs.csharp_sdk_branch }})
272-
- **Conformance**: ${{ needs.tier-check.outputs.conformance_repo }} (branch: ${{ needs.tier-check.outputs.conformance_branch }})
284+
All parameters are written to `/tmp/audit-params/` by a pre-agent step. Read them:
273285

274-
The scorecard is at `/tmp/tier-check-scorecard.json` (written by a pre-agent step from the `tier-check` job output).
286+
```bash
287+
SCOPE=$(cat /tmp/audit-params/scope)
288+
OUTPUT_MODE=$(cat /tmp/audit-params/output)
289+
SDK_REPO=$(cat /tmp/audit-params/csharp-sdk-repo)
290+
SDK_BRANCH=$(cat /tmp/audit-params/csharp-sdk-branch)
291+
CONF_REPO=$(cat /tmp/audit-params/conformance-repo)
292+
CONF_BRANCH=$(cat /tmp/audit-params/conformance-branch)
293+
```
294+
295+
The tier-check scorecard is at `/tmp/tier-check-scorecard.json`.
275296

276297
## Tier-Check Scorecard (pre-computed)
277298

@@ -283,33 +304,33 @@ cat /tmp/tier-check-scorecard.json
283304

284305
## Step 1: Setup
285306

286-
Clone both repositories for the AI-assisted evaluations. Use shallow clones.
307+
Read the parameters from `/tmp/audit-params/` and clone both repositories for the AI-assisted evaluations. Use shallow clones.
287308

288309
```bash
289-
git clone --depth 1 -b <csharp_sdk_branch> https://github.com/<csharp_sdk_repo>.git /tmp/csharp-sdk
290-
git clone --depth 1 -b <conformance_branch> https://github.com/<conformance_repo>.git /tmp/conformance
310+
SDK_REPO=$(cat /tmp/audit-params/csharp-sdk-repo)
311+
SDK_BRANCH=$(cat /tmp/audit-params/csharp-sdk-branch)
312+
CONF_REPO=$(cat /tmp/audit-params/conformance-repo)
313+
CONF_BRANCH=$(cat /tmp/audit-params/conformance-branch)
314+
315+
git clone --depth 1 -b "$SDK_BRANCH" "https://github.com/${SDK_REPO}.git" /tmp/csharp-sdk
316+
git clone --depth 1 -b "$CONF_BRANCH" "https://github.com/${CONF_REPO}.git" /tmp/conformance
291317
```
292318

293319
You do NOT need to build either repo or start any servers — the conformance tests have already run.
294320

295321
## Step 2: AI-Assisted Evaluation
296322

297-
Read the **"Any Other AI Coding Agent"** section from `/tmp/conformance/.claude/skills/mcp-sdk-tier-audit/README.md`. Follow steps **2 through 5** only (skip step 1 — the CLI has already run):
298-
299-
2. **Evaluate documentation coverage** using the prompt in `references/docs-coverage-prompt.md`
300-
3. **Evaluate policies** using the prompt in `references/policy-evaluation-prompt.md` — pass the `policy_signals` section from the tier-check JSON above
301-
4. **Apply tier logic** using the thresholds in `references/tier-requirements.md` — combine the scorecard above with your evaluation results
302-
5. **Generate report** using the template in `references/report-template.md`
323+
Read the **"Any Other AI Coding Agent"** section from `/tmp/conformance/.claude/skills/mcp-sdk-tier-audit/README.md`. Follow those instructions exactly — steps 2 through 5 (skip step 1, the CLI has already run). The conformance skill's instructions are the single source of truth for tier logic, documentation evaluation criteria, policy evaluation criteria, and report templates. **Do not apply your own tier logic or scoring — use only the conformance skill's thresholds, rules, and templates.**
303324

304-
The SDK checkout at `/tmp/csharp-sdk` is the local path for documentation and policy evaluations.
325+
The tier-check scorecard JSON is at `/tmp/tier-check-scorecard.json`. The SDK checkout at `/tmp/csharp-sdk` is the local path for documentation and policy evaluations.
305326

306-
Write the assessment and remediation reports to `/tmp/conformance/results/`:
327+
The conformance skill will produce assessment and remediation reports. Write them to `/tmp/conformance/results/`:
307328
- `results/<YYYY-MM-DD>-csharp-sdk-assessment.md`
308329
- `results/<YYYY-MM-DD>-csharp-sdk-remediation.md`
309330

310-
## Step 3: Compose the Audit Report
331+
## Step 3: Compose and Save the Audit Report
311332

312-
After the evaluation completes, compose the full audit report as a single markdown file at `/tmp/audit-report.md`. This file is used for both issue creation and the action summary artifact.
333+
After the evaluation completes, compose a single markdown file at `/tmp/audit-report.md`. This same content is used for both output modes: issue body and action summary.
313334

314335
### Report structure
315336

@@ -328,53 +349,41 @@ The report must contain these sections in order:
328349

329350
Use horizontal rules (`---`) to separate sections.
330351

331-
### Write the report
352+
### Save the report file
332353

333-
Write the composed report to `/tmp/audit-report.md`. This file will be uploaded as a workflow artifact by a post-execution step.
354+
Write the composed report to `/tmp/audit-report.md`. A post-execution step will write it to the GitHub Action Summary (visible on the workflow run's summary page).
334355

335-
### Write the action summary
336-
337-
Always write the full audit report to the GitHub Step Summary so it appears on the workflow run's summary page:
338-
339-
```bash
340-
cat /tmp/audit-report.md >> "$GITHUB_STEP_SUMMARY"
341-
```
356+
Verify the file exists before proceeding to Step 4.
342357

343358
## Step 4: Publish Results
344359

360+
Read the output mode: `cat /tmp/audit-params/output`
361+
345362
### If output mode is "Create Issue"
346363

347364
Create a GitHub issue using the `create-issue` safe output.
348365

349-
**Issue title** — The dynamic part (after the `[C# SDK Tier Audit] ` prefix):
366+
**Issue title**Read the scope from `cat /tmp/audit-params/scope`. The dynamic part (after the `[C# SDK Tier Audit] ` prefix):
350367

351368
- **"Conformance + Repo Health" scope**: `<YYYY-MM-DD> - Tier <N>`
352369
- **"Repo Health" scope**: `<YYYY-MM-DD> - Tier <N> (Repo Health)`
353370

354371
Where `<YYYY-MM-DD>` is today's date and `<N>` is the computed tier number (1, 2, or 3).
355372

356-
**Issue body** — Use the contents of `/tmp/audit-report.md` as the issue body.
373+
**Issue body** — Use the exact contents of `/tmp/audit-report.md` as the issue body. The issue body must be identical to what was written to the action summary.
357374

358375
### If output mode is "Action Summary"
359376

360-
Do NOT create an issue. The report is already written to `$GITHUB_STEP_SUMMARY` and will be uploaded as an artifact. No further action is needed.
377+
Do NOT create an issue. Call `noop` with a message like "Audit complete — results in Action Summary." The report will be displayed on the workflow run's summary page by a post-execution step.
361378

362379
## Failure Handling
363380

364381
If the evaluation fails at any step, or if the audit does not produce assessment/remediation results:
365382

366383
1. **Do NOT create an issue.** Do not use the `create-issue` safe output.
367-
2. **Write a GitHub Step Summary** explaining what happened:
368-
369-
```bash
370-
echo "## Tier Audit: No Results" >> "$GITHUB_STEP_SUMMARY"
371-
echo "" >> "$GITHUB_STEP_SUMMARY"
372-
echo "The tier audit did not produce results. Reason: <describe the failure>" >> "$GITHUB_STEP_SUMMARY"
373-
echo "" >> "$GITHUB_STEP_SUMMARY"
374-
echo "No issue was filed for this run." >> "$GITHUB_STEP_SUMMARY"
375-
```
376-
377-
3. The `noop` safe output will apply automatically when no `create-issue` output is produced.
384+
2. **Write a failure report** to `/tmp/audit-report.md` explaining what happened. The post-execution step will display it on the action summary page.
385+
3. Call `noop` with a message describing the failure.
386+
4. The post-execution step will handle displaying the failure on the action summary page.
378387

379388
---
380389

0 commit comments

Comments
 (0)