Sync .github/skills directory with azure-sdk-tools for PR 16380#49863
Merged
helen229 merged 5 commits intoJul 17, 2026
Merged
Conversation
Wires environment.skills: [<skill dirs>] into the mock workflow eval YAML specs so Vally loads the relevant skill per-eval instead of relying on --skill-dir at the pipeline level.
- Add missing environment.skills block to sdk-local-update-metadata-001 (all other positive stimuli in this file already had it) - Replace ambiguous/fictional 'contoso' package paths with the real fixture package (sdk/compute/azure-mgmt-compute, azure-resourcemanager-compute) so the model doesn't need to ask for clarification on a made-up path - Add explicit version/release-type/date to the version-update stimuli since the tool genuinely requires that info Verified locally: suite score went from ~42% to consistently 83-92% across repeated runs (threshold 80%).
The trigger.eval.yaml suite reproducibly failed at ~70%% because vague/ informational prompts (SDK release checklist, walk me through the release process, how to publish a new SDK version, release a new version of azure-sdk) weren't covered by the skill's narrow trigger phrase list, so the model didn't reliably invoke the skill. Added those phrasings to the description, Triggers section, and Examples, and clarified that the skill should also handle how-to/ checklist style questions, not just direct action requests. Verified locally: trigger.eval.yaml went from 70%% to 100%% across two repeated runs (threshold 80%%).
Per Copilot PR review feedback: anti-trigger and negative stimuli in these eval specs didn't set environment.skills, so the skill was never loaded/available for those prompts. That made the disallowed skill-invocation / tool-calls grader checks trivially true (the skill literally couldn't be invoked), providing no real signal. Added environment.skills: [..] to all anti-trigger/negative stimuli in: - azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml - azsdk-common-pipeline-fixer/evals/trigger.eval.yaml - azsdk-common-pipeline-analysis/evals/trigger.eval.yaml - azsdk-common-prepare-release-plan/evals/trigger.eval.yaml - azsdk-common-sdk-release/evals/trigger.eval.yaml - azsdk-common-generate-sdk-locally/evals/eval.yaml (sdk-local-negative-001) Verified locally: all 6 suites still pass with the skill loaded and available - the model correctly declines to invoke it on out-of-scope prompts. No new failures introduced (5 trigger suites at 100%, generate-sdk-locally-eval at 91.7%, same pre-existing unrelated failure as before on sdk-local-analyzer-errors-001).
azure-sdk-automation
Bot
added
EngSys
|
Azure Pipelines: 36 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot could not run the full agentic suite for this review because it was automatically requested on a bot-authored pull request. Request a review from Copilot under Reviewers to retry with the full agentic suite. Improved support for bot-authored pull requests is coming soon.
Updates skill definitions and evaluation stimuli to improve routing for Azure SDK release/plan workflows and introduce a new pipeline-based SDK generation skill.
Changes:
- Add
environment.skills: - ..to multiple eval stimuli so the executor loads the relevant skill context. - Expand trigger wording in existing
azsdk-common-sdk-releaseandazsdk-common-prepare-release-planskills to cover more natural language queries. - Add new
azsdk-common-generate-sdk-pipelineskill documentation describing the pipeline generation workflow and required MCP tools.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/azsdk-common-sdk-release/evals/trigger.eval.yaml | Ensures release skill is available during trigger/anti-trigger eval runs. |
| .github/skills/azsdk-common-sdk-release/SKILL.md | Broadens trigger phrasing to include release-process/how-to requests. |
| .github/skills/azsdk-common-prepare-release-plan/evals/trigger.eval.yaml | Ensures release-plan skill is available during trigger/anti-trigger eval runs. |
| .github/skills/azsdk-common-prepare-release-plan/SKILL.md | Updates description/trigger phrasing for common “prepare/plan” intents. |
| .github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml | Ensures pipeline-fixer skill is available during trigger/anti-trigger eval runs. |
| .github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml | Ensures pipeline-analysis skill is available during trigger/anti-trigger eval runs. |
| .github/skills/azsdk-common-generate-sdk-pipeline/SKILL.md | Adds documentation for pipeline SDK generation via azsdk_run_generate_sdk. |
| .github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml | Loads skills in local-generation evals and expands prompts with concrete details. |
| .github/skills/azsdk-common-generate-sdk-locally/SKILL.md | Clarifies boundary between local vs pipeline generation and tool usage. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml | Ensures APIView skill is available during trigger/anti-trigger eval runs. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml | Loads real skill context for tool-call eval execution. |
helen229
approved these changes
Jul 17, 2026
helen229
deleted the
sync-.github-skills-fix/vally-skill-eval-environment-skills-16380
branch
July 17, 2026 19:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync .github/skills directory with azure-sdk-tools for PR Azure/azure-sdk-tools#16380 See eng/common workflow