Add static eval for repo skills/agents via skill-validator#13537
Merged
JanKrivanek merged 4 commits intoApr 15, 2026
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds automated static validation for repository Copilot skills and agents using skill-validator, and updates existing repo references/content to satisfy validator expectations.
Changes:
- Added a PR/push-triggered workflow to run
skill-validatorand upload results as an artifact. - Added a
workflow_runfollow-up workflow to post (or update) a PR comment with summarized validator results. - Added/updated skills and agent content (including a shared workflow doc reference fix) to align with the validator.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/skill-validation.yml | Runs skill-validator on skills/agents, summarizes results, uploads artifact for downstream comment workflow. |
| .github/workflows/skill-validation-comment.yml | Downloads validator artifact and posts/updates a PR comment summarizing findings. |
| .github/workflows/shared/review-shared.md | Updates agent filename reference to .agent.md. |
| .github/skills/optimizing-msbuild-performance/SKILL.md | New performance-focused skill content. |
| .github/skills/maintaining-binary-log-compatibility/SKILL.md | New binlog compatibility skill content. |
| .github/skills/integrating-sdk-and-msbuild/SKILL.md | New SDK↔MSBuild integration skill content. |
| .github/skills/authoring-errors-and-warnings/SKILL.md | New diagnostics authoring skill content. |
| .github/skills/assessing-breaking-changes/SKILL.md | New breaking-change assessment skill content. |
| .github/skills/changewaves/SKILL.md | Fixes skill link target to the new assessing-breaking-changes skill. |
| .github/agents/expert-reviewer.agent.md | Adds/renames the expert reviewer agent definition and guidance. |
Copilot's findings
- Files reviewed: 4/10 changed files
- Comments generated: 3
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Review Summary
Renames (items 1-6): Clean. All old directory names have been grepped across the repo — no stale references remain in .md, .yml, or other files. The changewaves/SKILL.md cross-reference fix (item 7) and review-shared.md agent path update (item 8) are correct.
Security model: The workflow_run split between skill-validation.yml (read-only, runs PR code) and skill-validation-comment.yml (write permissions, never checks out PR code) correctly follows the safe fork-PR pattern. ✅
Findings
| Severity | Finding |
|---|---|
Hardcoded counts in skill-validation.yml:131-134 — skill-count (13), agent-count (1), total (14) are static and will silently go stale. Should be computed dynamically. |
|
| i️ | Unpinned nightly binary — skill-validator-nightly tag is mutable; a bad upstream release could break CI. Consider pinning or adding checksum verification. |
| i️ | No graceful handling if artifact upload was skipped — the comment workflow will fail hard if the validation workflow was cancelled before uploading. |
| i️ | Success summary is sparse — on a clean run the step summary just says "All checks passed" with no scope detail. |
No breaking changes, no security issues, no stale references found. The hardcoded counts are the main item worth addressing before merge.
Note
🔒 Integrity filter blocked 1 item
The following item were blocked because they don't meet the GitHub integrity level.
- #13537
pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
To allow these resources, lower min-integrity in your GitHub frontmatter:
tools:
github:
min-integrity: approved # merged | approved | unapproved | noneGenerated by Expert Code Review (on open) for issue #13537 · ● 2.5M
… handling, success summary
ViktorHofer
approved these changes
Apr 14, 2026
This was referenced Apr 16, 2026
JanKrivanek
added a commit
that referenced
this pull request
Apr 21, 2026
#13537 followup ### Context The PR above introduced skills/agents files checking within PRs touching skills/agents But it checked all the files - possibly repeating same and unactionable findings ### Changes Made We now filter only for the touched skills and agent files
PureWeen
pushed a commit
to dotnet/maui
that referenced
this pull request
Apr 21, 2026
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Description of Change Adopting skills-validator This is inspired and sourced from similar integrations: * github/awesome-copilot#1195 * dotnet/runtime#126814 * dotnet/msbuild#13537 One important difference is that in this case we run as well in the LLM judging mode cc: @PureWeen - for awareness and for help with iterating on this :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Let's use the skill-validator to check for basic problems of our skills and agents
This is inspired and sourced from similar integrations:
Changes Made
Testing