Sync .github/skills directory with azure-sdk-tools for PR 16380 (#48117)

* Add environment.skills to workflow-scenario eval specs

Wires environment.skills: [<skill dirs>] into the mock workflow eval YAML specs so Vally loads the relevant skill per-eval instead of relying on --skill-dir at the pipeline level.

* Fix flaky generate-sdk-locally eval stimuli

- Add missing environment.skills block to sdk-local-update-metadata-001
  (all other positive stimuli in this file already had it)
- Replace ambiguous/fictional 'contoso' package paths with the real
  fixture package (sdk/compute/azure-mgmt-compute, azure-resourcemanager-compute)
  so the model doesn't need to ask for clarification on a made-up path
- Add explicit version/release-type/date to the version-update stimuli
  since the tool genuinely requires that info

Verified locally: suite score went from ~42% to consistently 83-92%
across repeated runs (threshold 80%).

* Broaden sdk-release skill trigger phrases

The trigger.eval.yaml suite reproducibly failed at ~70%% because vague/
informational prompts (SDK release checklist, walk me through the
release process, how to publish a new SDK version, release a new
version of azure-sdk) weren't covered by the skill's narrow trigger
phrase list, so the model didn't reliably invoke the skill.

Added those phrasings to the description, Triggers section, and
Examples, and clarified that the skill should also handle how-to/
checklist style questions, not just direct action requests.

Verified locally: trigger.eval.yaml went from 70%% to 100%% across two
repeated runs (threshold 80%%).

* Add environment.skills to anti-trigger/negative eval stimuli

Per Copilot PR review feedback: anti-trigger and negative stimuli in
these eval specs didn't set environment.skills, so the skill was never
loaded/available for those prompts. That made the disallowed
skill-invocation / tool-calls grader checks trivially true (the skill
literally couldn't be invoked), providing no real signal.

Added environment.skills: [..] to all anti-trigger/negative stimuli in:
- azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml
- azsdk-common-pipeline-fixer/evals/trigger.eval.yaml
- azsdk-common-pipeline-analysis/evals/trigger.eval.yaml
- azsdk-common-prepare-release-plan/evals/trigger.eval.yaml
- azsdk-common-sdk-release/evals/trigger.eval.yaml
- azsdk-common-generate-sdk-locally/evals/eval.yaml (sdk-local-negative-001)

Verified locally: all 6 suites still pass with the skill loaded and
available - the model correctly declines to invoke it on out-of-scope
prompts. No new failures introduced (5 trigger suites at 100%,
generate-sdk-locally-eval at 91.7%, same pre-existing unrelated failure
as before on sdk-local-analyzer-errors-001).

* Mirror .github/skills after patch conflict

---------

Co-authored-by: helen229 <gaoh@microsoft.com>
Co-authored-by: azure-sdk <azuresdk@microsoft.com>

Author: 3 people

.github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml

@@ -20,6 +20,10 @@ scoring:
 stimuli:
   - name: apiview-basic-001
     prompt: "My Python SDK PR has APIView comments that I need to address. Can you help me retrieve and resolve them?"
+    environment:
+      # Load the real skill so the executor can route to it and call its tools.
+      skills:
+        - ..
     graders:
       - type: tool-calls
         config:

.github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml

@@ -20,79 +20,118 @@ scoring:
 stimuli:
   - name: trigger-resolve-apiview-feedback-on-my-pr
     prompt: "resolve APIView feedback on my PR"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           required: ["azsdk-common-apiview-feedback-resolution"]
   - name: trigger-fix-apiview-comments
     prompt: "fix APIView comments"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           required: ["azsdk-common-apiview-feedback-resolution"]
   - name: trigger-address-api-review-feedback
     prompt: "address API review feedback"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           required: ["azsdk-common-apiview-feedback-resolution"]
   - name: trigger-apiview-shows-issues-with-my-sdk
     prompt: "APIView shows issues with my SDK"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           required: ["azsdk-common-apiview-feedback-resolution"]
   - name: trigger-help-me-with-apiview-suggestions
     prompt: "help me with APIView suggestions"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           required: ["azsdk-common-apiview-feedback-resolution"]
   - name: trigger-fix-comments-apiview-url-1
     prompt: "Help me fix these comments: https://spa.apiview.dev/review/0qw98r9qwur9q823r?activeApiRevisionId=afiq948tquajf8jq9f"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           required: ["azsdk-common-apiview-feedback-resolution"]
   - name: trigger-address-feedback-apiview-url-2
     prompt: "Address this feedback: https://spa.apiview.dev/review/akldfja9qjta498tjaejg0?activeApiRevisionId=odiufa9sefighj98439"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           required: ["azsdk-common-apiview-feedback-resolution"]
   - name: trigger-resolve-comments-staging-url
     prompt: "Resolve these comments: https://apiviewstagingtest.com/review/akldfja9qjta498tjaejg0?activeApiRevisionId=odiufa9sefighj98439"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           required: ["azsdk-common-apiview-feedback-resolution"]
 
   - name: anti-trigger-deploy-my-azure-function
     prompt: "deploy my Azure Function"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           disallowed: ["azsdk-common-apiview-feedback-resolution"]
   - name: anti-trigger-fix-my-pipeline-build-error
     prompt: "fix my pipeline build error"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           disallowed: ["azsdk-common-apiview-feedback-resolution"]
   - name: anti-trigger-create-a-new-typespec-definition
     prompt: "create a new TypeSpec definition"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           disallowed: ["azsdk-common-apiview-feedback-resolution"]
   - name: anti-trigger-how-do-i-release-an-sdk-package
     prompt: "how do I release an SDK package"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:
           disallowed: ["azsdk-common-apiview-feedback-resolution"]
   - name: anti-trigger-set-up-azure-storage-account
     prompt: "set up Azure Storage account"
+    environment:
+      skills:
+        - ..
     graders:
       - type: skill-invocation
         config:

.github/skills/azsdk-common-generate-sdk-locally/SKILL.md

@@ -20,6 +20,8 @@ DO NOT USE FOR: publishing to package registries, CI pipeline configuration, API
 
 ## Rules
 
+- This skill generates a **single language** SDK **locally**. If the user asks to "generate SDK for **all languages**", to generate **without a local clone**, or wants SDK **pull requests created** for them (release plan / pipeline flow), do **not** generate locally — call `azure-sdk-mcp:azsdk_run_generate_sdk` instead. It runs the SDK generation pipeline for each language and produces the SDK pull requests.
+- Never use `azure-sdk-mcp:azsdk_get_sdk_pull_request_link` or `azure-sdk-mcp:azsdk_get_pull_request` to _generate_ an SDK; those only retrieve links for SDKs that were already generated.
 - Requires the `azure-sdk-mcp` server for the MCP workflow; without MCP, use `npm exec --prefix eng/common/tsp-client -- tsp-client` CLI.
 - Verify the target language repo and the correct TypeSpec configuration file before generation.
 - After generation or customization, run the check and test steps before updating metadata or finalizing changes.
@@ -83,6 +85,7 @@ Prerequisites: azure-sdk-mcp server must be running. Without MCP, use `npx tsp-c
 
 ## Troubleshooting
 
+- For "generate SDK for all languages", pipeline-based generation, or when no local SDK clone exists, call `azure-sdk-mcp:azsdk_run_generate_sdk` instead of the local generate flow, and never substitute `azure-sdk-mcp:azsdk_get_sdk_pull_request_link` / `azure-sdk-mcp:azsdk_get_pull_request` for generation.
 - Run `azure-sdk-mcp:azsdk_verify_setup` to confirm MCP and tools.
 - If build fails with type conflicts, breaking changes, analyzer errors, or customization drift, use `azure-sdk-mcp:azsdk_customized_code_update` to apply customizations.
 - The customization tool uses a two-phase approach: TypeSpec decorators first (Phase A), then code repairs if needed (Phase B).

.github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml

@@ -20,6 +20,9 @@ scoring:
 stimuli:
   - name: sdk-local-basic-001
     prompt: "Generate the Python SDK locally from my TypeSpec project at specification/compute/resource-manager/Microsoft.Compute/ComputeRP/tspconfig.yaml. My azure-sdk-for-python repo is at ~/repos/azure-sdk-for-python."
+    environment:
+      skills:
+        - ..
     graders:
       - type: tool-calls
         config:
@@ -29,6 +32,9 @@ stimuli:
 
   - name: sdk-local-negative-001
     prompt: "I want to use the Azure DevOps pipeline to generate SDKs automatically and have PRs created for me. I don't want to set up local tools."
+    environment:
+      skills:
+        - ..
     graders:
       - type: tool-calls
         config:
@@ -37,7 +43,10 @@ stimuli:
             - name: azure-sdk-mcp-azsdk_package_generate_code
 
   - name: sdk-local-full-001
-    prompt: "Generate the Java SDK locally, build it, run tests, and prepare it for a PR. Use the SDK generation tools to do the full workflow."
+    prompt: "Generate the Java SDK locally for the compute service, build it, run tests, and prepare it for a PR. My azure-sdk-for-java repo is at ~/repos/azure-sdk-for-java and the package is sdk/compute/azure-resourcemanager-compute. Bump the version to 1.2.0-beta.1 (beta, release date 2026-07-16). Use the SDK generation tools to do the full workflow."
+    environment:
+      skills:
+        - ..
     graders:
       - type: tool-calls
         config:
@@ -53,6 +62,9 @@ stimuli:
 
   - name: sdk-local-edge-001
     prompt: "I generated the .NET SDK locally but the build is failing with compilation errors. The error says a type name conflicts with a reserved keyword. Use the customization tools to fix it."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:
@@ -63,7 +75,10 @@ stimuli:
             - name: azure-sdk-mcp-azsdk_customized_code_update
 
   - name: sdk-local-breaking-changes-001
-    prompt: "After regenerating the Java SDK, the build is failing because the service team renamed 'displayName' to 'name' in the TypeSpec. Our customization class still references getField('displayName'). Use the SDK customization tools to fix this breaking change."
+    prompt: "After regenerating the Java SDK package at sdk/compute/azure-resourcemanager-compute (~/repos/azure-sdk-for-java), the build is failing because the service team renamed 'displayName' to 'name' in the TypeSpec. Our customization class still references getField('displayName'). Use the SDK customization tools to fix this breaking change."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:
@@ -74,7 +89,10 @@ stimuli:
             - name: azure-sdk-mcp-azsdk_customized_code_update
 
   - name: sdk-local-customization-001
-    prompt: "The Java SDK build is failing after regeneration. The error says 'variable operationId is already defined in class AnalyzeOperationDetails'. The service team added operationId to the TypeSpec, but our customization already injects that field. Use the SDK customization tools to resolve this duplicate."
+    prompt: "The Java SDK package at sdk/compute/azure-resourcemanager-compute (~/repos/azure-sdk-for-java) is failing to build after regeneration. The error says 'variable operationId is already defined in class AnalyzeOperationDetails'. The service team added operationId to the TypeSpec, but our customization already injects that field. Use the SDK customization tools to resolve this duplicate."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:
@@ -85,7 +103,10 @@ stimuli:
             - name: azure-sdk-mcp-azsdk_customized_code_update
 
   - name: sdk-local-hide-operation-001
-    prompt: "Hide the get_create_project_status polling operation from the Python SDK public API. It's an internal operation that shouldn't be exposed to users. Use the SDK customization tools."
+    prompt: "Hide the get_create_project_status polling operation from the public API of the Python SDK package at sdk/compute/azure-mgmt-compute (~/repos/azure-sdk-for-python). It's an internal operation that shouldn't be exposed to users. Use the SDK customization tools."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:
@@ -96,7 +117,10 @@ stimuli:
             - name: azure-sdk-mcp-azsdk_customized_code_update
 
   - name: sdk-local-rename-client-001
-    prompt: "Rename the AIProjectConnectionEntraIDCredential model to use 'Id' instead of 'ID' in the .NET SDK. This was flagged in API review. Use the SDK customization tools to apply this rename."
+    prompt: "Rename the AIProjectConnectionEntraIDCredential model to use 'Id' instead of 'ID' in the .NET SDK package at sdk/compute/Azure.ResourceManager.Compute (~/repos/azure-sdk-for-net). This was flagged in API review. Use the SDK customization tools to apply this rename."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:
@@ -108,6 +132,9 @@ stimuli:
 
   - name: sdk-local-update-changelog-001
     prompt: "Update the changelog for the Python SDK package at sdk/compute/azure-mgmt-compute."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:
@@ -118,7 +145,10 @@ stimuli:
             - name: azure-sdk-mcp-azsdk_package_update_changelog_content
 
   - name: sdk-local-update-metadata-001
-    prompt: "Update the package metadata for the .NET SDK at sdk/contoso/Azure.ResourceManager.Contoso."
+    prompt: "Update the package metadata for the .NET SDK package at sdk/compute/Azure.ResourceManager.Compute (~/repos/azure-sdk-for-net)."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:
@@ -129,7 +159,10 @@ stimuli:
             - name: azure-sdk-mcp-azsdk_package_update_metadata
 
   - name: sdk-local-update-version-001
-    prompt: "Update the package version for the Java SDK at sdk/contoso/azure-resourcemanager-contoso."
+    prompt: "Update the package version for the Java SDK package at sdk/compute/azure-resourcemanager-compute (~/repos/azure-sdk-for-java) to 1.2.0-beta.1, release type beta, release date 2026-07-16."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:
@@ -141,6 +174,9 @@ stimuli:
 
   - name: sdk-local-analyzer-errors-001
     prompt: "The .NET SDK build is failing with analyzer errors: AZC0030 says the model name ends with 'Parameters' and AZC0012 says type name 'Tasks' is too generic. Use the SDK customization tools to fix these analyzer errors."
+    environment:
+      skills:
+        - ..
     graders:
       - type: output-not-contains
         config:

.github/skills/azsdk-common-generate-sdk-pipeline/SKILL.md

@@ -0,0 +1,55 @@
+---
+name: azsdk-common-generate-sdk-pipeline
+license: MIT
+metadata:
+  version: "1.0.0"
+  distribution: shared
+description: 'Run the Azure SDK generation pipeline for a release plan and create the generated SDK pull requests, for one language or for all languages. **UTILITY SKILL**. USE FOR: "run SDK generation for all languages", "generate SDK for release plan <id>", "generate SDK for release <id>", "pipeline SDK generation", "generate SDK without a local clone", "create SDK pull requests". DO NOT USE FOR: generating a single SDK locally from a local clone (use azsdk-common-generate-sdk-locally), releasing/publishing an already-generated package (use azsdk-common-sdk-release), API design review. INVOKES: azure-sdk-mcp:azsdk_get_release_plan, azure-sdk-mcp:azsdk_run_generate_sdk, azure-sdk-mcp:azsdk_get_sdk_pull_request_link.'
+compatibility: "azure-sdk-mcp server, existing release plan work item. Supports .NET, Java, JavaScript, Python, Go"
+---
+
+# Generate SDK via Pipeline
+
+This skill runs the Azure SDK generation pipeline for a release plan and produces the generated SDK pull request(s). It is the correct workflow when the user asks to generate SDKs **for all languages**, to generate **for a release plan / release ID**, or to generate **without a local clone** — none of which the local-generation workflow handles.
+
+## Triggers
+
+USE FOR: run SDK generation for all languages, generate SDK for a release plan, generate SDK for a release ID, pipeline SDK generation, generate SDK without a local clone, create SDK pull requests
+WHEN: "run SDK generation, run SDK generation for all languages for release plan <id>", "generate SDK for release plan <id>", "generate SDK for release <id>", "pipeline SDK generation", "generate SDK without a local clone"
+DO NOT USE FOR: generating a single SDK locally from a local clone (use `azsdk-common-generate-sdk-locally`); releasing or publishing an already-generated package (use `azsdk-common-sdk-release`); API design review
+
+## Rules
+
+- **Always call `azure-sdk-mcp:azsdk_run_generate_sdk`** to generate. Do **not** use `azure-sdk-mcp:azsdk_release_sdk` (that publishes an already-generated package, it does not generate) or `azure-sdk-mcp:azsdk_get_sdk_pull_request_link` / `azure-sdk-mcp:azsdk_get_pull_request` (those only retrieve links) to generate an SDK.
+- `azure-sdk-mcp:azsdk_run_generate_sdk` generates **one language per call**. To generate for **all languages**, call it **once per language** the release plan targets.
+- A **release plan work item ID (or release plan ID)** is required. `azsdk_run_generate_sdk` **also** requires the **TypeSpec project path** and the **SDK release type** (`beta` or `stable`) as explicit inputs, and it validates them — it does **not** read them from the release plan. First call `azure-sdk-mcp:azsdk_get_release_plan` to obtain the TypeSpec project path, SDK release type, and target languages, then pass those into each generation call along with the work item ID.
+- Requires the `azure-sdk-mcp` server; there is no CLI fallback for the pipeline generation workflow.
+- Private Preview release plans cannot generate SDKs via the pipeline — only the API spec PR needs to merge. If needed for validation, direct the user to generate locally via `azsdk-common-generate-sdk-locally`.
+
+## MCP Tools
+
+| Tool                                            | Purpose                                                        |
+| ----------------------------------------------- | -------------------------------------------------------------- |
+| `azure-sdk-mcp:azsdk_get_release_plan`          | Fetch the release plan to determine target languages / details |
+| `azure-sdk-mcp:azsdk_run_generate_sdk`          | Run the generation pipeline (once per language)                |
+| `azure-sdk-mcp:azsdk_get_sdk_pull_request_link` | Retrieve the generated SDK pull request link after generation  |
+
+## Steps
+
+1. **Collect release plan** — Get the release plan work item ID (or release plan ID) from the user, then call `azure-sdk-mcp:azsdk_get_release_plan` to fetch it.
+2. **Determine languages** — If the user asked for "all languages", determine the languages the release plan targets (e.g. via `azure-sdk-mcp:azsdk_get_release_plan`). Otherwise use the single language requested.
+3. **Generate per language** — For each target language, run `azure-sdk-mcp:azsdk_run_generate_sdk` passing the **TypeSpec project path**, the **SDK release type** (`beta` or `stable`), and the **language** taken from the release plan, plus the release plan **work item ID**. These inputs are required and validated — an invalid TypeSpec project path, or a release type other than `beta`/`stable`, fails the run.
+4. **Report pull requests** — After each run completes, retrieve and show the generated SDK pull request link with `azure-sdk-mcp:azsdk_get_sdk_pull_request_link`.
+
+## Examples
+
+- "Run SDK generation for all languages for release 12345"
+- "Generate the SDK for release plan 12345"
+- "Generate the Python and .NET SDKs for release 12345 using the pipeline"
+
+## Troubleshooting
+
+- If the agent tries `azsdk_release_sdk`, get-artifact, or get-service-details tools instead of generating, redirect it to `azure-sdk-mcp:azsdk_run_generate_sdk` — that is the only tool that runs the generation pipeline.
+- If the correct tool is not yet available in the session, activate/load the `azure-sdk-mcp` TypeSpec SDK toolset first, then call `azure-sdk-mcp:azsdk_run_generate_sdk`.
+- If generation fails with a missing SDK details error, ensure the release plan has SDK details populated for the language before re-running.
+- Requires the `azure-sdk-mcp` server. No CLI fallback — prompt the user to configure MCP if unavailable.