Fix #16226: route all-language SDK generation to the pipeline tool#16295
Open
helen229 wants to merge 6 commits into
Open
Fix #16226: route all-language SDK generation to the pipeline tool#16295helen229 wants to merge 6 commits into
helen229 wants to merge 6 commits into
Conversation
- Enrich azsdk_run_generate_sdk description with trigger phrases and usage (pipeline / no-local-clone generation, one language per call). - Disambiguate azsdk_release_sdk description (publishes an already-generated package; does NOT generate SDK code). - Add azsdk-common-generate-sdk-pipeline skill routing all-language / pipeline / release-plan generation to azsdk_run_generate_sdk; update the local-gen skill redirect to point at it. - Bump @microsoft/vally-cli 0.5.0 -> 0.7.0 (multi-turn eval support). - Add mock workflow evals for local-vs-pipeline routing (single-turn scenarios in release-planner-workflows + multi-turn conversations in multi-turn-release-workflows). - CHANGELOG entry.
github-actions
Bot
added
the
azsdk-cli
📊 GEPA Skill Quality Scores
11/12 skills at quality ≥ 0.80 How to improve# Score a specific skill
python .github/skills/sensei/scripts/gepa/auto_evaluator.py score --skill <name> --skills-dir .github/skills --tests-dir tests
# Optimize a skill with GEPA
python .github/skills/sensei/scripts/gepa/auto_evaluator.py optimize --skill <name> --skills-dir .github/skills --tests-dir tests |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes routing for prompts like “run SDK generation for all languages for release <id>” so the agent uses the pipeline SDK generation tool (azsdk_run_generate_sdk) rather than release/publish tooling, and adds/updates eval coverage to prevent regressions.
Changes:
- Clarifies MCP tool descriptions to disambiguate pipeline SDK generation (
azsdk_run_generate_sdk) from package release (azsdk_release_sdk). - Adds a dedicated pipeline-generation skill (
azsdk-common-generate-sdk-pipeline) and updates local-generation skill guidance to redirect all-language/pipeline requests to it. - Bumps
@microsoft/vally-clito 0.7.0 and adds new mock workflow eval scenarios, including a new multi-turn suite.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/workflow-scenarios/mock/release-planner-workflows.eval.yaml | Adds mock routing scenarios covering pipeline-vs-local generation and disallowed tool/skill selections. |
| tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/workflow-scenarios/mock/multi-turn-release-workflows.eval.yaml | New multi-turn routing capability suite to exercise session-resume behavior and mid-conversation routing changes. |
| tools/azsdk-cli/Azure.Sdk.Tools.Cli/Tools/ReleasePlan/SpecWorkFlowTool.cs | Expands azsdk_run_generate_sdk tool description to steer prompts toward pipeline generation and away from release/link tools. |
| tools/azsdk-cli/Azure.Sdk.Tools.Cli/Tools/Package/SdkReleaseTool.cs | Clarifies azsdk_release_sdk as publish-only and points generation prompts to azsdk_run_generate_sdk. |
| tools/azsdk-cli/Azure.Sdk.Tools.Cli/CHANGELOG.md | Documents the routing fix and the new pipeline skill. |
| eng/skill-eval/package.json | Pins @microsoft/vally-cli to 0.7.0. |
| eng/skill-eval/package-lock.json | Lockfile update for the Vally CLI bump and transitive dependencies. |
| .github/skills/azsdk-common-generate-sdk-pipeline/SKILL.md | New skill defining the pipeline-generation workflow and routing rules. |
| .github/skills/azsdk-common-generate-sdk-locally/SKILL.md | Updates local-generation rules to redirect all-language/pipeline requests to the new pipeline skill. |
Files not reviewed (1)
- eng/skill-eval/package-lock.json: Generated file
- SKILL.md: azsdk_run_generate_sdk requires explicit TypeSpec project path, SDK release type (beta/stable) and language (validated); instruct the agent to fetch these from azsdk_get_release_plan first, then pass per language. - Fix prettier CI failure on the new SKILL.md (formatting). - Vally 0.7 added a scoring.weights must-sum-to-1.0 validator; normalize weights across all affected eval files (1/N per grader, threshold unchanged so semantics are preserved). - Use config: (repo convention) instead of defaults: in the new multi-turn eval for consistency.
raych1
reviewed
Jul 7, 2026
raych1
reviewed
Jul 7, 2026
…lures The vague/multi-step scenarios (link-spec-pr-then-regenerate, vague-then-clarify, vague-then-local) asserted skill/tool routing pinned to a specific conversation turn. But the agent legitimately front-loads work onto the first actionable turn (or spreads a required tool across steps), so the required skills/tools landed in a different turn slice than the grader inspected -> false failures despite correct routing. Assert these as whole-conversation properties instead. Re-grading the recorded trajectories confirms all three now pass 2/2 graders.
…l description 1. locally/SKILL.md: redirect all-language/pipeline requests to the azsdk-common-generate-sdk-pipeline skill without hard-coding azsdk_run_generate_sdk (that tool is one of several the pipeline skill owns). 2. SdkReleaseTool.cs: trim the duplicated 'generate an SDK from a TypeSpec project' wording while keeping the 'all languages / release plan' disambiguation cue that steers routing away from the release tool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #16226.
Problem
The azsdk agent ran the wrong tools for "run SDK generation for all languages for release " — it reached for
azsdk_release_sdk/ PR-link tools instead of the pipeline generation tool.Changes
SpecWorkFlowTool.cs,SdkReleaseTool.cs)azsdk_run_generate_sdk: enriched with trigger phrases, that it accepts a release plan / work item ID, and that it generates one language per call (call once per language).azsdk_release_sdk: clarified it publishes an already-generated package and does not generate SDK code.azsdk-common-generate-sdk-pipelineroutes all-language / pipeline / release-plan generation toazsdk_run_generate_sdk(get_release_plan->run_generate_sdk->get_sdk_pull_request_link). The local-generation skill''s redirect now points at it.eng/skill-eval/package.json+ lock) for multi-turn eval support.release-planner-workflows.eval.yaml,multi-turn-release-workflows.eval.yaml(vague opening -> clarify to local vs pipeline; local -> switch to pipeline mid-session).