Skip to content

Commit 812b801

Browse files
azure-sdk-automation[bot]ReilleyMilne
azure-sdk-automation[bot]
and
ReilleyMilne
authored
Sync .github/skills directory with azure-sdk-tools for PR 16300 (#48046)
* Update pipeline troubleshooting skill and vally fixtures * Split pipeline troubleshooting into analysis + fixer skills, added Vally eval suite. * Fixed project name resolution for untrusted DevOps pipelines. Removed unnecessary evaluation files and fixtures for pipeline analysis and fixing. Updated and added skills files for pipeline analysis and fixing. * Fixes to PipelineIdentifierHelper Updated skills/tools descriptions. Deleted redundant eval in prompt-to-tool-pipeline.eval.yaml * Changed skills formatting and updated Vally README to have absolute paths. * Updated to align with review comments. * Updated failure patterns to use "az account show" * Prettier formatting. --------- Co-authored-by: ReilleyMilne <reilleymilne@gmail.com> Co-authored-by: ReilleyMilne <91291100+ReilleyMilne@users.noreply.github.com>
1 parent 7365223 commit 812b801

10 files changed

Lines changed: 367 additions & 271 deletions

File tree

‎.github/skills/azsdk-common-pipeline-analysis/SKILL.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
---
2+
name: azsdk-common-pipeline-analysis
3+
license: MIT
4+
metadata:
5+
version: "1.0.0"
6+
distribution: shared
7+
description: 'Analyze Azure SDK CI/CD pipeline failures into a structured diagnosis, and define the required output format. Load this skill before calling azsdk_analyze_pipeline, which returns raw failure data that this skill interprets and formats. USE FOR: "pipeline failed", "build failure", "CI check failing", "tests failing in CI", "analyze pipeline", "debug SDK pipeline". DO NOT USE FOR: local build issues without pipeline context, API design review, SDK publishing, applying code fixes (instead use azsdk-common-pipeline-fixer). INVOKES: azure-sdk-mcp:azsdk_analyze_pipeline, azure-sdk-mcp:azsdk_get_pipeline_llm_artifacts, azure-sdk-mcp:azsdk_get_pr_checks, azure-sdk-mcp:azsdk_get_pipeline_status.'
8+
compatibility: "azure-sdk-mcp server, Azure DevOps pipeline build ID or GitHub PR link"
9+
---
10+
11+
# Pipeline Analysis
12+
13+
This skill analyzes Azure SDK CI/CD pipeline failures and provides a structured diagnosis including root cause, affected files, and concrete instructions for how to fix each issue. It does NOT apply fixes - it tells you exactly what's wrong and how to resolve it.
14+
15+
## Rules
16+
17+
- Load this skill **before** running `azsdk_analyze_pipeline` — the tool returns raw failure data that this skill interprets and formats per [output format](references/output-format.md).
18+
- Requires the `azure-sdk-mcp` server; without it, inspect logs in the Azure DevOps UI.
19+
- Analysis-only: never edit files or apply fixes — use `azsdk-common-pipeline-fixer` for changes.
20+
- Run `azsdk_analyze_pipeline` first, then categorize each failure and cite specific files/lines.
21+
- For infrastructure failures (network timeouts, agent crashes, throttling), recommend retry, not code changes.
22+
23+
## MCP Tools
24+
25+
| Tool | Purpose |
26+
| ------------------------------------------------ | ---------------------------------------------- |
27+
| `azure-sdk-mcp:azsdk_analyze_pipeline` | Analyze pipeline failure (logs + test results) |
28+
| `azure-sdk-mcp:azsdk_get_pipeline_llm_artifacts` | Download test result artifacts from pipeline |
29+
| `azure-sdk-mcp:azsdk_get_pr_checks` | Get pipeline/check results linked to a PR |
30+
| `azure-sdk-mcp:azsdk_get_pipeline_status` | Get pipeline run status |
31+
32+
## Steps
33+
34+
1. **Identify** - Get the build ID, pipeline URL, or PR link (use the PR link when triggered from a PR comment).
35+
2. **Analyze** - Run `azsdk_analyze_pipeline`. It returns `failed_tasks` (log errors from failed steps) and `failed_test_titles` (failed tests grouped by file).
36+
3. **Categorize** each failure: test, build/compilation, validation/lint, or infrastructure.
37+
4. **Diagnose** - Give each failure's root cause and affected file(s)/line(s), and note if several share one root cause. See [failure patterns](references/failure-patterns.md).
38+
5. **Report** - Use the [output format](references/output-format.md): root cause, affected files, per-failure fix + verify command, and fixable vs infrastructure. Recommend `azsdk-common-pipeline-fixer` to apply fixes.
39+
40+
## Examples
41+
42+
- "My pipeline build 6447834 failed, what went wrong?"
43+
- PR comment: `@copilot Analyze the failed pipeline on this PR`
44+
- "Why is my CI red? Build ID is 6455939"
45+
46+
## Troubleshooting
47+
48+
- If `azsdk_analyze_pipeline` returns no data, verify the build ID is correct and the run has completed.
49+
- If `failed_test_titles` is empty, rely on `failed_tasks` log analysis (test-artifact upload may not be configured).
50+
- For a private/internal pipeline, the user may need to `az login`. See [failure patterns](references/failure-patterns.md).

‎.github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
name: azsdk-common-pipeline-analysis-trigger-eval
2+
description: Trigger and anti-trigger tests for the azsdk-common-pipeline-analysis skill
3+
type: capability
4+
5+
environment: azsdk-mcp-mock
6+
7+
tags:
8+
area: azsdk-common-pipeline-analysis
9+
type: ci-gate
10+
11+
config:
12+
runs: 1
13+
timeout: "90s"
14+
model: claude-opus-4.6
15+
executor: copilot-sdk
16+
17+
scoring:
18+
threshold: 0.8
19+
20+
stimuli:
21+
# Should trigger
22+
- name: trigger-my-ci-pipeline-is-failing
23+
prompt: "my Azure SDK CI pipeline build 6447834 is failing, can you help troubleshoot what went wrong?"
24+
graders:
25+
- type: skill-invocation
26+
config:
27+
required: ["azsdk-common-pipeline-analysis"]
28+
- name: trigger-debug-pipeline-build-error
29+
prompt: "debug the build error in my SDK generation pipeline, build 6447834"
30+
graders:
31+
- type: skill-invocation
32+
config:
33+
required: ["azsdk-common-pipeline-analysis"]
34+
- name: trigger-pipeline-analysis-help
35+
prompt: "I'm stuck on a failing Azure SDK pipeline — can you help me troubleshoot it?"
36+
graders:
37+
- type: skill-invocation
38+
config:
39+
required: ["azsdk-common-pipeline-analysis"]
40+
- name: trigger-analyze-failing-sdk-pipeline
41+
prompt: "analyze my failing Azure SDK generation pipeline https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6447834 and tell me what went wrong"
42+
graders:
43+
- type: skill-invocation
44+
config:
45+
required: ["azsdk-common-pipeline-analysis"]
46+
- name: trigger-ci-build-broke-after-my-change
47+
prompt: "the SDK CI build broke after my change, help me troubleshoot the pipeline"
48+
graders:
49+
- type: skill-invocation
50+
config:
51+
required: ["azsdk-common-pipeline-analysis"]
52+
53+
# Should NOT trigger
54+
# "apply the fix / auto-fix" prompts belong to azsdk-common-pipeline-fixer, not
55+
# this analysis-only skill (see each skill's DO NOT USE FOR boundary).
56+
- name: anti-trigger-auto-fix-and-commit
57+
prompt: "auto-fix the failing tests in my pipeline and commit the changes to my PR"
58+
graders:
59+
- type: skill-invocation
60+
config:
61+
disallowed: ["azsdk-common-pipeline-analysis"]
62+
- name: anti-trigger-write-a-typespec-definition
63+
prompt: "write a TypeSpec definition"
64+
graders:
65+
- type: skill-invocation
66+
config:
67+
disallowed: ["azsdk-common-pipeline-analysis"]
68+
- name: anti-trigger-create-a-new-skill
69+
prompt: "create a new skill"
70+
graders:
71+
- type: skill-invocation
72+
config:
73+
disallowed: ["azsdk-common-pipeline-analysis"]
74+
- name: anti-trigger-optimize-markdown-tokens
75+
prompt: "optimize markdown tokens"
76+
graders:
77+
- type: skill-invocation
78+
config:
79+
disallowed: ["azsdk-common-pipeline-analysis"]
80+
- name: anti-trigger-how-do-i-release-an-sdk-package
81+
prompt: "how do I release an SDK package"
82+
graders:
83+
- type: skill-invocation
84+
config:
85+
disallowed: ["azsdk-common-pipeline-analysis"]
86+
- name: anti-trigger-resolve-apiview-feedback
87+
prompt: "resolve APIView feedback"
88+
graders:
89+
- type: skill-invocation
90+
config:
91+
disallowed: ["azsdk-common-pipeline-analysis"]

‎.github/skills/azsdk-common-pipeline-analysis/references/failure-patterns.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Pipeline Failure Patterns and Resolution
2+
3+
## Test Failures
4+
5+
| Pattern | Likely Cause | Fix |
6+
| ---------------------------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
7+
| AssertionError (expected vs actual) | Logic bug or unintended behavior change | Fix the code logic or update expected value if change was intentional |
8+
| Recording file path does not exist | Missing test recording for a new/modified test | Best effort: use the `az` CLI to check auth (try `az account show`); if authenticated, run tests in record mode; else skip and report re-recording is required |
9+
| Playback failure / ResourceNotFoundError | Recorded responses don't match current API calls | Best effort: use the `az` CLI to check auth (try `az account show`); if authenticated, re-record the session; else skip and report re-recording is required |
10+
| HttpResponseError (5xx) in live tests | Service-side issue during live test run | Infrastructure - retry the pipeline |
11+
| Timeout / test hung | Async issue, deadlock, or slow external dependency | Add timeout, fix async logic, or mock the slow dependency |
12+
| FileNotFoundError / missing fixture | Test data not committed or path changed | Add the fixture file or fix the path reference |
13+
14+
## Build/Compilation Failures
15+
16+
| Pattern | Likely Cause | Fix |
17+
| ------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
18+
| Type mismatch (mypy, pyright, tsc) | Code passes wrong type to a function/constructor | Fix the type at the cited file:line |
19+
| Missing import / unresolved reference | New dependency not imported, or missing from the project | Add the import; if the package isn't a declared dependency, add it to the project manifest (e.g. pyproject.toml, .csproj, package.json) |
20+
| Incompatible type in generics | Generic type parameter doesn't satisfy constraint | Fix the type parameter or constraint |
21+
| Syntax error | Malformed code | Fix the syntax at the cited line |
22+
| Missing types/models (TypeSpec) | TypeSpec compilation issue | Fix TypeSpec definition, regenerate |
23+
24+
## Validation/Lint Failures
25+
26+
| Pattern | Likely Cause | Fix |
27+
| ------------------------------ | ------------------------------------ | -------------------------------------------------- |
28+
| Changelog validation error | Missing or malformed changelog entry | Update CHANGELOG.md content |
29+
| Breaking change detected | API surface changed incompatibly | Revert the breaking change or add suppression |
30+
| Lint/format violation | Code doesn't match style rules | Run the formatter (black, prettier, dotnet format) |
31+
| API compatibility check failed | Public API signature changed | Update API review or revert |
32+
33+
## Infrastructure Failures (Recommend Retry)
34+
35+
| Pattern | Likely Cause | Action |
36+
| ----------------------------------- | --------------------------------------- | -------------- |
37+
| Name or service not known (DNS) | Transient network issue | Retry pipeline |
38+
| HTTP 429 / rate limited | Throttling from package registry or API | Wait and retry |
39+
| Agent disconnected / process killed | CI agent resource exhaustion | Retry pipeline |
40+
| Timeout downloading packages | Network congestion or registry slowness | Retry pipeline |
41+
| Certificate / SSL error | Transient PKI issue | Retry pipeline |
42+
43+
## Cascading Failure Detection
44+
45+
When many tests fail simultaneously, look for:
46+
47+
- A single build/setup step that failed (all downstream tests fail with import errors)
48+
- A shared test fixture or setup method that throws
49+
- A missing environment variable or config file
50+
- A dependency that failed to install
51+
52+
**Rule of thumb:** If >10 tests fail with similar error patterns, identify the common root cause rather than fixing each individually.
53+
54+
## TypeSpec/SDK Generation Failures
55+
56+
| Pattern | Likely Cause | Fix |
57+
| -------------------------------- | ----------------------------------- | ------------------------------------- |
58+
| Missing client.tsp changes | TypeSpec customization needed | Use `azsdk_customized_code_update` |
59+
| tspconfig.yaml emitter error | Incorrect emitter configuration | Fix tspconfig.yaml |
60+
| Breaking change in generated API | TypeSpec change affects SDK surface | Add `@clientName` decorator or revert |

‎.github/skills/azsdk-common-pipeline-analysis/references/output-format.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Pipeline Analysis Output Format
2+
3+
Write the diagnosis as markdown so it reads cleanly in a GitHub PR comment or an
4+
Azure DevOps pipeline summary. Order it: root cause first, then each failure,
5+
then a short fix summary.
6+
7+
Include these parts:
8+
9+
1. **Header** — build ID or PR link, package path, and overall status.
10+
2. **Root cause** — one or two plain sentences naming the underlying problem. If
11+
failures are unrelated, list each root cause separately.
12+
3. **Failures** — for every failure: its category (test / build / validation /
13+
infrastructure), the exact `file:line` from the analysis output, the error
14+
text, and the specific fix.
15+
4. **Verify** — the local command that reproduces each fixable failure
16+
(e.g. `python -m mypy --isolate <path>`, `dotnet test --filter <test>`).
17+
5. **Summary** — counts of fixable-in-code, infrastructure (retry), and
18+
needs-human-input failures. Recommend `azsdk-common-pipeline-fixer` only when
19+
something is code-fixable.
20+
21+
## Rules
22+
23+
- Always cite the real `file:line` from the analysis output; never paraphrase it.
24+
- Group cascading failures under their shared root cause instead of listing each.
25+
- Mark infrastructure failures (network, throttling, agent crash) as retry, not code.
26+
- See [failure patterns](failure-patterns.md) for category-to-fix mappings.

‎.github/skills/azsdk-common-pipeline-fixer/SKILL.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
name: azsdk-common-pipeline-fixer
3+
license: MIT
4+
metadata:
5+
version: "1.0.0"
6+
distribution: shared
7+
description: 'Automatically fix Azure SDK CI/CD pipeline failures by applying code changes and verifying locally. USE FOR: "fix pipeline failure", "fix CI", "fix failing tests", "auto-fix and commit the fix", "fix build error", "fix mypy/pylint/type-check/lint errors", "auto-fix pipeline", "resolve pipeline failure". DO NOT USE FOR: pipeline analysis (instead use azsdk-common-pipeline-analysis), API design review, SDK publishing. INVOKES: azure-sdk-mcp:azsdk_package_build_code, azure-sdk-mcp:azsdk_package_run_check, azure-sdk-mcp:azsdk_package_run_tests, azure-sdk-mcp:azsdk_verify_setup.'
8+
compatibility: "azure-sdk-mcp server, local azure-sdk-for-{language} clone, language build tools"
9+
---
10+
11+
# Pipeline Fixer
12+
13+
This skill automatically fixes Azure SDK CI/CD pipeline failures. It analyzes the failure, identifies the root cause, applies code changes, and verifies the fix locally using build, check, and test tools before committing.
14+
15+
## Rules
16+
17+
- Requires the `azure-sdk-mcp` server and a local clone of the SDK language repo.
18+
- Work on the PR branch — the failing code is there, not main.
19+
- Analyze first, then apply the minimal code change for the root cause.
20+
- Verify ONLY via the azsdk MCP package tools (build → check → tests); never raw shell build/test commands.
21+
- Never fix infrastructure failures (timeouts, crashes, throttling) — recommend retry. Iterate up to 3 times, then report.
22+
23+
## MCP Tools
24+
25+
| Tool | Purpose |
26+
| ---------------------------------------- | ------------------------------------- |
27+
| `azure-sdk-mcp:azsdk_verify_setup` | Verify local environment is ready |
28+
| `azure-sdk-mcp:azsdk_package_build_code` | Build the package locally |
29+
| `azure-sdk-mcp:azsdk_package_run_check` | Run validation/lint/typecheck locally |
30+
| `azure-sdk-mcp:azsdk_package_run_tests` | Run tests locally |
31+
32+
**Prerequisites:** azure-sdk-mcp server required. Local clone of the affected SDK language repo. Language build tools installed.
33+
34+
## Steps
35+
36+
1. **Find analysis** - Reuse an existing analysis from the PR comments; if none, view the `azsdk-common-pipeline-analysis` skill.
37+
2. **Classify** - Fixable (test, type, lint, import, assertion errors) vs retry (infrastructure) vs escalate (breaking API, credentials).
38+
3. **Locate** - Identify the affected package, its path, and the files/lines to change.
39+
4. **Fix** - Read the failing code at the cited lines and apply the minimal fix.
40+
5. **Verify** - Run `azsdk_package_build_code` → `azsdk_package_run_check` → `azsdk_package_run_tests` on the package; all must pass. If the environment isn't ready, run `azsdk_verify_setup`. Never substitute raw shell build/test commands.
41+
6. **Iterate & commit** - On failure, revise and re-verify (max 3 attempts). Once all pass, commit with a descriptive message.
42+
43+
## Examples
44+
45+
- "Fix the pipeline failure for build 6447834"
46+
- "My Python SDK CI is failing with a type error, fix it"
47+
- "Auto-fix the failing tests in my PR"
48+
49+
## Troubleshooting
50+
51+
- If the verify tools fail because the environment isn't ready, run `azsdk_verify_setup`, report its error, and stop — don't fall back to raw shell build/test commands.

0 commit comments

Comments
 (0)