You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Remove own tier logic; fix action summary and artifact output
Two issues fixed:
1. Tier logic: The prompt was telling the agent to 'Apply tier logic'
itself (Step 2.4). Now the prompt explicitly says: 'Do not apply
your own tier logic or scoring — use only the conformance skill's
thresholds, rules, and templates.' All tier determination is
delegated to the conformance repo's mcp-sdk-tier-audit skill.
2. Action summary: The report was not appearing on the workflow
summary page because the agent wasn't writing to
$GITHUB_STEP_SUMMARY reliably. Now Step 3 is restructured with
explicit requirements:
- MUST write /tmp/audit-report.md (artifact)
- MUST cat it to $GITHUB_STEP_SUMMARY (action summary)
- Both required BEFORE Step 4 (publish)
- 'Action Summary' mode now explicitly calls noop
- Issue body must be identical to the action summary content
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
You do NOT need to build either repo or start any servers — the conformance tests have already run.
294
320
295
321
## Step 2: AI-Assisted Evaluation
296
322
297
-
Read the **"Any Other AI Coding Agent"** section from `/tmp/conformance/.claude/skills/mcp-sdk-tier-audit/README.md`. Follow steps **2 through 5** only (skip step 1 — the CLI has already run):
298
-
299
-
2.**Evaluate documentation coverage** using the prompt in `references/docs-coverage-prompt.md`
300
-
3.**Evaluate policies** using the prompt in `references/policy-evaluation-prompt.md` — pass the `policy_signals` section from the tier-check JSON above
301
-
4.**Apply tier logic** using the thresholds in `references/tier-requirements.md` — combine the scorecard above with your evaluation results
302
-
5.**Generate report** using the template in `references/report-template.md`
323
+
Read the **"Any Other AI Coding Agent"** section from `/tmp/conformance/.claude/skills/mcp-sdk-tier-audit/README.md`. Follow those instructions exactly — steps 2 through 5 (skip step 1, the CLI has already run). The conformance skill's instructions are the single source of truth for tier logic, documentation evaluation criteria, policy evaluation criteria, and report templates. **Do not apply your own tier logic or scoring — use only the conformance skill's thresholds, rules, and templates.**
303
324
304
-
The SDK checkout at `/tmp/csharp-sdk` is the local path for documentation and policy evaluations.
325
+
The tier-check scorecard JSON is at `/tmp/tier-check-scorecard.json`. The SDK checkout at `/tmp/csharp-sdk` is the local path for documentation and policy evaluations.
305
326
306
-
Write the assessment and remediation reports to `/tmp/conformance/results/`:
327
+
The conformance skill will produce assessment and remediation reports. Write them to `/tmp/conformance/results/`:
307
328
-`results/<YYYY-MM-DD>-csharp-sdk-assessment.md`
308
329
-`results/<YYYY-MM-DD>-csharp-sdk-remediation.md`
309
330
310
-
## Step 3: Compose the Audit Report
331
+
## Step 3: Compose and Save the Audit Report
311
332
312
-
After the evaluation completes, compose the full audit report as a single markdown file at `/tmp/audit-report.md`. This file is used for both issue creation and the action summary artifact.
333
+
After the evaluation completes, compose a single markdown file at `/tmp/audit-report.md`. This same content is used for both output modes: issue body and action summary.
313
334
314
335
### Report structure
315
336
@@ -328,53 +349,41 @@ The report must contain these sections in order:
328
349
329
350
Use horizontal rules (`---`) to separate sections.
330
351
331
-
### Write the report
352
+
### Save the report file
332
353
333
-
Write the composed report to `/tmp/audit-report.md`. This file will be uploaded as a workflow artifact by a post-execution step.
354
+
Write the composed report to `/tmp/audit-report.md`. A post-execution step will write it to the GitHub Action Summary (visible on the workflow run's summary page).
334
355
335
-
### Write the action summary
336
-
337
-
Always write the full audit report to the GitHub Step Summary so it appears on the workflow run's summary page:
338
-
339
-
```bash
340
-
cat /tmp/audit-report.md >>"$GITHUB_STEP_SUMMARY"
341
-
```
356
+
Verify the file exists before proceeding to Step 4.
342
357
343
358
## Step 4: Publish Results
344
359
360
+
Read the output mode: `cat /tmp/audit-params/output`
361
+
345
362
### If output mode is "Create Issue"
346
363
347
364
Create a GitHub issue using the `create-issue` safe output.
348
365
349
-
**Issue title** — The dynamic part (after the `[C# SDK Tier Audit] ` prefix):
366
+
**Issue title** — Read the scope from `cat /tmp/audit-params/scope`. The dynamic part (after the `[C# SDK Tier Audit] ` prefix):
Where `<YYYY-MM-DD>` is today's date and `<N>` is the computed tier number (1, 2, or 3).
355
372
356
-
**Issue body** — Use the contents of `/tmp/audit-report.md` as the issue body.
373
+
**Issue body** — Use the exact contents of `/tmp/audit-report.md` as the issue body. The issue body must be identical to what was written to the action summary.
357
374
358
375
### If output mode is "Action Summary"
359
376
360
-
Do NOT create an issue. The report is already written to `$GITHUB_STEP_SUMMARY` and will be uploaded as an artifact. No further action is needed.
377
+
Do NOT create an issue. Call `noop` with a message like "Audit complete — results in Action Summary." The report will be displayed on the workflow run's summary page by a post-execution step.
361
378
362
379
## Failure Handling
363
380
364
381
If the evaluation fails at any step, or if the audit does not produce assessment/remediation results:
365
382
366
383
1.**Do NOT create an issue.** Do not use the `create-issue` safe output.
367
-
2.**Write a GitHub Step Summary** explaining what happened:
368
-
369
-
```bash
370
-
echo"## Tier Audit: No Results">>"$GITHUB_STEP_SUMMARY"
371
-
echo"">>"$GITHUB_STEP_SUMMARY"
372
-
echo"The tier audit did not produce results. Reason: <describe the failure>">>"$GITHUB_STEP_SUMMARY"
373
-
echo"">>"$GITHUB_STEP_SUMMARY"
374
-
echo"No issue was filed for this run.">>"$GITHUB_STEP_SUMMARY"
375
-
```
376
-
377
-
3. The `noop` safe output will apply automatically when no `create-issue` output is produced.
384
+
2.**Write a failure report** to `/tmp/audit-report.md` explaining what happened. The post-execution step will display it on the action summary page.
385
+
3. Call `noop` with a message describing the failure.
386
+
4. The post-execution step will handle displaying the failure on the action summary page.
0 commit comments