Add static eval via skill-validator#1195
Merged
aaronpowell merged 2 commits intoMar 30, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds GitHub Actions workflows to run the skill-validator tool for static (non-LLM) evaluation of skills/agents: one workflow posts a nightly report to Discussions (with an Issue fallback), and another posts a non-blocking comment on PRs affecting skills/agents.
Changes:
- Added a nightly scheduled workflow that runs
skill-validatoracross all skills/agents and publishes a formatted report (Discussion preferred, Issue fallback). - Added a PR-gate workflow that runs
skill-validatoron changed skills/agents and posts/updates a single PR comment with the results.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| .github/workflows/skill-quality-report.yml | Nightly scheduled scan that generates and publishes a “Skill Quality Report” discussion (or issue fallback). |
| .github/workflows/skill-check.yml | PR workflow that validates changed skills/agents and posts the results as a PR comment (non-blocking). |
aaronpowell
approved these changes
Mar 30, 2026
Merged
This was referenced Apr 14, 2026
JanKrivanek
added a commit
to dotnet/msbuild
that referenced
this pull request
Apr 15, 2026
### Context Let's use the skill-validator to check for basic problems of our skills and agents This is inspired and sourced from similar integrations: * github/awesome-copilot#1195 * dotnet/runtime#126814 ### Changes Made - Added pipeline using the skill-validator - Fixed the current findings (renames of folders and agent file) ### Testing - skill-validator is the test harness for the skills
PureWeen
pushed a commit
to dotnet/maui
that referenced
this pull request
Apr 21, 2026
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Description of Change Adopting skills-validator This is inspired and sourced from similar integrations: * github/awesome-copilot#1195 * dotnet/runtime#126814 * dotnet/msbuild#13537 One important difference is that in this case we run as well in the LLM judging mode cc: @PureWeen - for awareness and for help with iterating on this :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(replacement of #1186)
Motivation
Add static non-LLM evaluation of skills by skill-validator from dotnet/skills
As a:
Prereq
go to https://github.com/github/awesome-copilot/discussions/categories
Click New category
Set:
Click Create