Add skill-eval-authoring, tool-eval-authoring, and workflow-eval-authoring skills

[helen229]

Summary

Adds three new meta-skills under .github/skills/ that teach the agent (and contributors) how to write robust Vally evals:

• skill-eval-authoring — routing/capability evals for .github/skills/*/evals/. Packages the four-layer pattern (routing -> tool-use -> output shape -> judgment), a grader catalog, and an anti-patterns reference, adapted from the stale draft in Add skill-eval-authoring skill and harden release-plan eval exemplar #15836 for the current pinned @microsoft/vally-cli (0.7.0).
• tool-eval-authoring — hermetic single-tool evals under evals/tools/.
• workflow-eval-authoring — multi-tool / multi-turn / mock / live evals under evals/workflows/.

Each skill has a SKILL.md, references/authoring-guide.md, and its own evals/trigger.eval.yaml. All three are registered in .github/skills/.vally.yaml and the skills README.md.

Bug caught during local validation

The first draft of all three trigger.eval.yaml files mounted each skill only against itself (skills: [..]), so boundary anti-triggers (e.g., "add trigger tests for my Agent Skill" tested against tool-eval-authoring) trivially failed: with no better-fitting sibling skill available, the model invoked the only skill it could see. Fixed by mounting all three sibling skills together for every stimulus (matching release-planner-workflows.eval.yaml); documented as anti-pattern A7 in the new anti-patterns.md.

Validation

vally lint <skill> --strict   -> 3/3 checks passed, all three skills
vally eval -e skill-eval-authoring/evals/trigger.eval.yaml -e tool-eval-authoring/evals/trigger.eval.yaml -e workflow-eval-authoring/evals/trigger.eval.yaml --skill-dir .
-> all three: passed 100% (21/21 stimuli total)

Fixes #15874

[github-actions]

📊 GEPA Skill Quality Scores

Skill	Quality	Triggers	Tests
azsdk-common-pipeline-analysis	⚠️ 0.57	N/A	---
azsdk-common-pipeline-fixer	⚠️ 0.57	N/A	---
sensei	⚠️ 0.57	N/A	---
azsdk-common-prepare-release-plan	✅ 0.86	N/A	---
azsdk-common-apiview-feedback-resolution	✅ 1.00	N/A	---
azsdk-common-generate-sdk-locally	✅ 1.00	N/A	---
azsdk-common-generate-sdk-pipeline	✅ 1.00	N/A	---
azsdk-common-sdk-release	✅ 1.00	N/A	---
azure-typespec-author	✅ 1.00	N/A	---
markdown-token-optimizer	✅ 1.00	N/A	---
sdk-ai-bot-eval-dataset	✅ 1.00	N/A	---
sdk-ai-bot-run-evaluation	✅ 1.00	N/A	---
skill-authoring	✅ 1.00	N/A	---
skill-eval-authoring	✅ 1.00	N/A	---
tool-eval-authoring	✅ 1.00	N/A	---
workflow-eval-authoring	✅ 1.00	N/A	---

13/16 skills at quality ≥ 0.80

How to improve

# Score a specific skill
python .github/skills/sensei/scripts/gepa/auto_evaluator.py score --skill <name> --skills-dir .github/skills --tests-dir tests

# Optimize a skill with GEPA
python .github/skills/sensei/scripts/gepa/auto_evaluator.py optimize --skill <name> --skills-dir .github/skills --tests-dir tests

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
62 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR introduces three new “meta-skills” under .github/skills/ to standardize how contributors author robust Vally evals (skill routing/capability evals, single-tool prompt-to-tool evals, and multi-tool workflow evals). In support of that, it also reshapes the repo’s top-level eval suite layout under evals/ (flattening to evals/tools/ and evals/workflows/), and updates pipelines/scripts/tests/docs to match the new paths.

Changes:

• Add three new eval-authoring skills (skill-eval-authoring, tool-eval-authoring, workflow-eval-authoring) with references + trigger suites, and register them in .github/skills/.vally.yaml and .github/skills/README.md.
• Restructure eval suite paths under evals/ (e.g., evals/tools/*, evals/workflows/{mock,live}/*) and update related scripts, pipeline globs, docs, and handler comments.
• Add/port tool prompt-to-tool eval coverage YAMLs and supporting fixtures under evals/fixtures/.

Reviewed changes

Copilot reviewed 36 out of 51 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Pipeline/AnalyzeAndArtifactsHandlers.cs	Updates doc comment references to the relocated eval fixtures/workflow paths.
evals/workflows/mock/typespec-generation-step02.eval.yaml	Updates relative paths for skills and git worktree fixture after eval layout change.
evals/workflows/mock/rename-client-property.eval.yaml	Updates relative paths for skills and git worktree fixture after eval layout change.
evals/workflows/mock/release-planner-workflows.eval.yaml	Updates references and relative paths for skills/git fixtures after layout change.
evals/workflows/mock/multi-turn-release-workflows.eval.yaml	Updates relative skill/git paths after layout change.
evals/workflows/mock/fix-pipeline.eval.yaml	Updates relative skill/fixture paths after layout change.
evals/workflows/mock/check-public-repo-then-validate.eval.yaml	Updates relative skill/git paths after layout change.
evals/workflows/mock/analyze-failed-pipeline.eval.yaml	Updates relative skill path after layout change.
evals/workflows/live/release-planner.eval.yaml	Updates relative skill/git paths after layout change.
evals/tools/prompt-to-tool-verify.eval.yaml	Adds prompt-to-tool coverage for azsdk_verify_setup.
evals/tools/prompt-to-tool-typespec.eval.yaml	Adds prompt-to-tool coverage for TypeSpec-related tools and some full-context stimuli.
evals/tools/prompt-to-tool-releaseplan.eval.yaml	Adds prompt-to-tool coverage for release plan related tools, with documented known-passability constraints.
evals/tools/prompt-to-tool-pipeline.eval.yaml	Adds prompt-to-tool coverage for pipeline tools plus routing-related stimuli.
evals/tools/prompt-to-tool-package.eval.yaml	Adds prompt-to-tool coverage for package build/test/release tools.
evals/tools/prompt-to-tool-github.eval.yaml	Adds prompt-to-tool coverage for GitHub-related tools with notes on fragile cases.
evals/tools/prompt-to-tool-engsys.eval.yaml	Adds prompt-to-tool coverage for engsys tools with notes on intentionally-omitted tools.
evals/tools/prompt-to-tool-config.eval.yaml	Adds prompt-to-tool coverage for service label tools.
evals/tools/prompt-to-tool-apiview.eval.yaml	Adds prompt-to-tool coverage for APIView tools.
evals/tools/add-arm-resource.eval.yaml	Updates fixture-relative paths after eval layout change.
evals/README.md	Renames and rewrites eval suite documentation to match new layout and usage commands.
evals/fixtures/README.md	Updates fixture path guidance to match new eval directory layout.
evals/fixtures/Microsoft.Widget/Widget/tspconfig.yaml	Adds TypeSpec fixture config for the ARM resource authoring scenario.
evals/fixtures/Microsoft.Widget/Widget/main.tsp	Adds TypeSpec fixture entrypoint for Microsoft.Widget ARM scenario.
evals/fixtures/Microsoft.Widget/Widget/employee.tsp	Adds TypeSpec fixture models/ops for Microsoft.Widget ARM scenario.
evals/fixtures/analyze-pipeline/QueueClientOptionsTests/code/QueueClientOptionsTests.cs	Adds fixture test file used by pipeline analyze/fix scenarios.
evals/fixtures/analyze-pipeline/QueueClientOptionsTests/code/QueueClientOptions.cs	Adds fixture source file used by pipeline analyze/fix scenarios.
evals/.vally.yaml	Updates suite roots and glob patterns to the new tools/ + workflows/ structure.
evals/.gitignore	Ignores eval output directories (vally-results/, results/).
eng/common/scripts/eval/test/init-eval-git-fixtures.test.ts	Updates tests for new eval directory structure and invariant checks.
eng/common/scripts/eval/test/collect-stimuli.test.ts	Updates matrix discovery tests to the new eval path patterns.
eng/common/scripts/eval/lib/glob.ts	Updates documentation comment examples to the new eval globs.
eng/common/scripts/eval/init-eval-git-fixtures.ts	Updates default discovery patterns to tools/* and workflows/mock/*.
eng/common/scripts/eval/collect-stimuli.ts	Updates default discovery patterns to tools/* and workflows/mock/*.
eng/common/scripts/allow-relative-links.txt	Updates relative-link allowlist from azsdk-evals/** to evals/**.
eng/common/pipelines/workflow-eval.yml	Updates pipeline path filters and vallyRoot to evals/.
eng/common/pipelines/templates/stages/archetype-eval.yml	Updates default eval globs to match new layout.
eng/common/pipelines/live-eval.yml	Updates vallyRoot and live-tier eval glob for new layout.
.github/skills/workflow-eval-authoring/SKILL.md	New skill: guidance for authoring multi-tool/multi-turn/mock/live workflow evals.
.github/skills/workflow-eval-authoring/references/authoring-guide.md	New workflow-eval authoring reference guide with local run steps.
.github/skills/workflow-eval-authoring/evals/trigger.eval.yaml	New trigger/boundary suite for workflow-eval-authoring.
.github/skills/tool-eval-authoring/SKILL.md	New skill: guidance for hermetic single-tool eval authoring under evals/tools/.
.github/skills/tool-eval-authoring/references/authoring-guide.md	New tool-eval authoring reference guide with patterns/checklist.
.github/skills/tool-eval-authoring/evals/trigger.eval.yaml	New trigger/boundary suite for tool-eval-authoring.
.github/skills/skill-eval-authoring/SKILL.md	New skill: guidance for per-skill routing/capability eval authoring under .github/skills/*/evals/.
.github/skills/skill-eval-authoring/references/grader-catalog.md	New reference: catalog of grader types and selection guidance.
.github/skills/skill-eval-authoring/references/four-layer-pattern.md	New reference: four-layer evaluation pattern (routing → tool-use → output → judgment).
.github/skills/skill-eval-authoring/references/authoring-guide.md	New reference: skill-eval authoring guide and local run steps.
.github/skills/skill-eval-authoring/references/anti-patterns.md	New reference: anti-patterns and fixes for eval robustness.
.github/skills/skill-eval-authoring/evals/trigger.eval.yaml	New trigger/boundary suite for skill-eval-authoring.
.github/skills/README.md	Registers the three new meta-skills and updates skill anatomy docs.
.github/skills/.vally.yaml	Registers the three new skills’ eval paths for lint/eval discovery.

[azure-sdk-automation]

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

[azure-sdk-automation]

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

[azure-sdk-automation]

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

[samvaity]

"4 levels up = repo root" but the paths below are now ../../../ (3 levels). Same issue in 4 other workflow eval files (check-public-repo-then-validate, multi-turn-release-workflows, release-planner-workflows, typespec-generation-step02).

[samvaity]

The reference docs (four-layer-pattern.md, grader-catalog.md, anti-patterns.md) live only under skill-eval-authoring/references/. The other two skills' authoring guides cover some of the same concepts (weights summing to 1.0, grader types) but don't link here. Worth cross-linking or moving to a shared location?

[samvaity]

The nightly suite glob **/*.eval.yaml would also match anything dropped into fixtures/ (or any new subdirectory). Previously the evals/ prefix scoped it. Low risk but could be tightened to ["tools/**/*.eval.yaml", "workflows/**/*.eval.yaml"] to match the ci suite pattern.