Skip to content

Commit dd0ce31

Browse files
committed
Deduplicate and streamline skill docs
- Add Bash(pwsh:*) to allowed-tools for auto-approval - Replace inline categorization rules with pointer to failure-patterns.md - Sharpen "load when" guidance for reference files - Condense output format template from ~70 to ~15 lines - Add table of contents to failure-patterns.md 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
1 parent 1b97be5 commit dd0ce31

2 files changed

Lines changed: 37 additions & 157 deletions

File tree

.claude/skills/troubleshoot-ci-build/SKILL.md

Lines changed: 25 additions & 157 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: troubleshoot-ci-build
33
description: Troubleshoot CI failures in dd-trace-dotnet Azure DevOps pipeline. Use this skill whenever the user mentions a failing CI build, PR checks failing, Azure DevOps pipeline failures, test failures in CI, or when they share a build ID or PR number and want to understand what went wrong. Analyzes build failures, categorizes them (infrastructure/flaky/real), and provides actionable recommendations.
44
argument-hint: <pr NUMBER | build BUILD_ID>
55
user-invocable: true
6-
allowed-tools: WebFetch, Bash(gh pr checks:*), Bash(az devops invoke:*), Bash(az pipelines build list:*), Bash(az pipelines build show:*), Bash(az pipelines runs artifact list:*), Bash(az pipelines runs list:*), Bash(az pipelines runs show:*)
6+
allowed-tools: WebFetch, Bash(pwsh:*), Bash(gh pr checks:*), Bash(az devops invoke:*), Bash(az pipelines build list:*), Bash(az pipelines build show:*), Bash(az pipelines runs artifact list:*), Bash(az pipelines runs list:*), Bash(az pipelines runs show:*)
77
---
88

99
# Troubleshoot Azure DevOps Builds for dd-trace-dotnet
@@ -37,9 +37,9 @@ Troubleshoot Azure DevOps pipeline failures with automated analysis.
3737

3838
## Additional Resources
3939

40-
- **[failure-patterns.md](failure-patterns.md)** - Reference guide with known CI failure patterns, categorization rules, and decision trees. Load when you need to categorize a failure type or compare against historical patterns.
41-
- **[scripts-reference.md](scripts-reference.md)** - Documentation for `Get-AzureDevOpsBuildAnalysis.ps1` script (parameters, usage, output structure).
42-
- **[references/cli-reference.md](references/cli-reference.md)** - Azure DevOps API endpoints and Windows CLI pitfalls. Load when running Azure DevOps CLI commands directly (bypassing the PowerShell script).
40+
- **[failure-patterns.md](failure-patterns.md)** — Load ONLY during Phase 2 categorization or when the user asks about a specific failure type. Not needed for Phase 1 quick analysis.
41+
- **[scripts-reference.md](scripts-reference.md)** — Load ONLY if the PowerShell script fails, returns unexpected output, or you need to understand the output object shape.
42+
- **[references/cli-reference.md](references/cli-reference.md)** — Load ONLY if bypassing the PowerShell script entirely and running Azure DevOps CLI commands directly.
4343

4444
## Task
4545

@@ -150,168 +150,36 @@ The script automatically:
150150

151151
## Output Format
152152

153-
**Important**: Use actual Unicode emoji characters (e.g., ❌, 🔴, 🟡, 🔵, 🔍, ✅) in output, NOT markdown emoji codes (e.g., `:x:`, `:red_circle:`). Markdown emoji codes are not rendered in all contexts.
153+
**Important**: Use actual Unicode emoji characters (e.g., ❌, 🔴, 🟡, 🔵, 🔍, ✅), NOT markdown emoji codes (e.g., `:x:`).
154154

155-
### Phase 1: Quick Summary (Always Show This First)
155+
### Phase 1: Quick Summary
156156

157-
```markdown
158-
# CI Failure Analysis for Build <BUILD_ID>
159-
160-
**Status**: ❌ Failed
161-
**Build**: [<BUILD_NUMBER>](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=<BUILD_ID>)
162-
163-
**PR**: [#<PR_NUMBER>](https://github.com/DataDog/dd-trace-dotnet/pull/<PR_NUMBER>) _(if PR-triggered)_
164-
**Branch**: `<source_branch_name>` _(use `triggerInfo["pr.sourceBranch"]` for PR builds instead of `sourceBranch`)_
165-
**Commit**: `<commit_sha>`
166-
167-
## Quick Overview
168-
169-
**Failed Tasks** (<count>):
170-
- `<task_name_1>`
171-
- `<task_name_2>`
172-
- `<task_name_3>`
173-
...
174-
175-
**Failed Jobs** (<count> platforms affected):
176-
- `<job_name_1>` (e.g., Test alpine_net8.0_Tracer)
177-
- `<job_name_2>` (e.g., Win x86_net8.0_Tracer)
178-
...
179-
180-
**Failed Stages**:
181-
- integration_tests_linux
182-
- integration_tests_windows
183-
...
184-
185-
**Timed Out Jobs** (<count>, canceled after ~60 min):
186-
- `DockerTest alpine_netcoreapp3.0_group1 (60.3 min)`
187-
- `IntegrationTests Windows x64 net8.0 (58.7 min)`
188-
...
189-
_(Note: These are jobs with result="canceled" and duration >= 55 minutes)_
190-
191-
**Collateral Cancellations** (<count>, < 5 min):
192-
- `Dependent Job 1`
193-
- `Dependent Job 2`
194-
...
195-
_(Note: Jobs canceled quickly, likely due to parent stage failure)_
196-
197-
**Failed Tests** (<count>, if applicable):
198-
- `TestNamespace.TestClass.TestMethod1`
199-
- `TestNamespace.TestClass.TestMethod2`
200-
- `TestNamespace.TestClass.TestMethod3`
201-
...
202-
_(Note: List specific test names extracted from error messages, if test failures detected)_
203-
204-
**Snapshot Mismatches Detected** _(if applicable — show only when snapshot failures are detected)_
205-
206-
The following failed tests likely involve snapshot verification:
207-
- `TestClass.SubmitsTraces`
208-
- ...
209-
210-
To update snapshots from this build:
211-
- **Windows**: `./tracer/build.ps1 UpdateSnapshotsFromBuild --BuildId <BUILD_ID>`
212-
- **Linux/macOS**: `./tracer/build.sh UpdateSnapshotsFromBuild --BuildId <BUILD_ID>`
213-
214-
This downloads the `.received.txt` snapshot artifacts from CI and replaces your local `.verified.txt` files.
215-
216-
---
217-
218-
## 🔍 What would you like to investigate?
219-
220-
1. **Categorize failures** - Analyze failure types (infrastructure/flaky/real)
221-
2. **View specific logs** - Download logs for failed tasks
222-
3. **Show full analysis** - Run complete analysis with all details
223-
4. **Update snapshots** - Download and apply updated snapshots from this build _(shown only when snapshot failures detected)_
224-
```
225-
226-
**Resolving PR number**: When invoked with `build <BUILD_ID>`, extract the PR number from the build details JSON:
227-
- `triggerInfo["pr.number"]` — most direct
228-
- `sourceBranch` — parse from `refs/pull/<NUMBER>/merge` pattern
229-
- Only show the PR link if the build was PR-triggered (`reason == "pullRequest"`)
157+
Structure the output as:
230158

159+
1. **Header**: `# CI Failure Analysis for Build <BUILD_ID>`
160+
2. **Metadata**: Status, Build link (`https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=<BUILD_ID>`), PR link (if PR-triggered), Branch, Commit
161+
- For PR builds, use `triggerInfo["pr.sourceBranch"]` instead of `sourceBranch`
162+
- Extract PR number from `triggerInfo["pr.number"]` or parse from `refs/pull/<NUMBER>/merge`
163+
3. **Failed Tasks/Jobs/Stages**: List names with counts
164+
4. **Timed Out Jobs**: Jobs with `result="canceled"` and duration >= 55 min (show duration)
165+
5. **Collateral Cancellations**: Jobs canceled in < 5 min (parent stage failure cascade)
166+
6. **Failed Tests**: Specific test names extracted from error messages
167+
7. **Snapshot Mismatches** (if detected): List affected tests and show `UpdateSnapshotsFromBuild` command
168+
8. **Investigation menu**: Categorize failures / View logs / Full analysis / Update snapshots (if applicable)
231169

232170
### Phase 2: Detailed Output (Only If User Requests)
233171

234-
**If user requests log analysis**:
235-
```markdown
236-
## Log Analysis
172+
- **Log analysis**: List successfully retrieved vs failed downloads with error patterns
173+
- **Categorization**: Group failures into 🔴 Real / 🟡 Flaky / 🔵 Infrastructure (see [failure-patterns.md](failure-patterns.md))
237174

238-
Attempted to download logs for failed tasks:
175+
## Failure Categorization
239176

240-
✅ Successfully retrieved:
241-
- Task: `<name>` (Log ID: <id>)
242-
- Error pattern: <brief description>
243-
244-
❌ Failed to retrieve (HTTP 500):
245-
- Task: `<name>` (Log ID: <id>)
246-
- View manually: [Link](https://...)
247-
```
248-
249-
**If user requests categorization**:
250-
```markdown
251-
## Failure Categories
252-
253-
### 🔴 Real Failures (Action: Investigate)
254-
- `<task>` - <reason>
255-
256-
### 🟡 Flaky Tests (Action: Consider Retry)
257-
- `<task>` - <reason>
258-
259-
### 🔵 Infrastructure Issues (Action: Retry)
260-
- `<task>` - <reason>
261-
```
262-
263-
## Failure Categorization Rules
264-
265-
### 🔴 Real Failures (Action: Investigate)
266-
**Indicators**:
267-
- Test assertions: `Expected X but got Y`, `Assert.*failed`
268-
- Compilation errors: `error CS\d+`, `MSB\d+`
269-
- Segmentation faults: `SIGSEGV`, `Access Violation`
270-
- Missing spans: `Expected N spans but got M`
271-
272-
**Pattern examples**:
273-
```
274-
[FAIL] TestName
275-
Expected 21 spans but got 14
276-
Assert.Equal() Failure
277-
Expected: True
278-
Actual: False
279-
```
280-
281-
### 🟡 Flaky Tests (Action: Retry, May Investigate)
282-
**Indicators**:
283-
- Tests with `previousAttempts > 0` (already auto-retried by CI)
284-
- Stack walking failures: `Failed to walk N stacks for sampled exception: E_FAIL`
285-
- Known intermittent tests (reference `failure-patterns.md`)
286-
- Tests that pass/fail inconsistently across platforms
287-
- **Single-runtime failures**: Same test fails on only one .NET runtime but passes on others (especially net6+). Example: net6 pass, net8 pass, net10 fail → likely flaky, not a real regression.
288-
- **ARM64 single-platform timeout**: In an ARM64 stage (e.g., `unit_tests_arm64`), one runtime job is cancelled after ~60 min while all other runtimes complete normally in ~14 min → almost certainly a transient ARM64 infrastructure issue, not a code regression. Retry.
289-
290-
**Pattern examples**:
291-
```
292-
Stack walking failed with E_FAIL
293-
ThreadAbortException
294-
Timeout waiting for spans
295-
```
296-
297-
### 🔵 Infrastructure Failures (Action: Retry)
298-
**Indicators**:
299-
- Docker rate limiting: `toomanyrequests`, `pull rate limit exceeded`
300-
- Network timeouts: `TLS handshake timeout`, `Connection reset by peer`, `ECONNRESET`
301-
- Execution timeouts: `maximum execution time exceeded`, `Test timeout`
302-
- Timeout via cancellation: Job canceled with duration >= 55 minutes
303-
- Disk space: `No space left on device`, `ENOSPC`
304-
- Container failures: `docker: Error response from daemon`
305-
306-
**Pattern examples**:
307-
```
308-
pull access denied, repository does not exist or may require authentication
309-
TLS handshake timeout
310-
Connection reset by peer
311-
Build failed in XX:XX:XX (timeout)
312-
```
177+
For detailed categorization rules, pattern examples, and the decision tree, see [failure-patterns.md](failure-patterns.md).
313178

314-
**Recommendation**: Retry the build once. If the failure persists after 2 consecutive runs, investigate and alert the **#apm-dotnet** Slack channel.
179+
**Quick reference** — three categories:
180+
- 🔴 **Real Failures** — Test assertions, compilation errors, segfaults, span count mismatches → Investigate
181+
- 🟡 **Flaky Tests** — Auto-retried tests, single-runtime failures, Alpine stack walking, ARM64 timeouts → Retry
182+
- 🔵 **Infrastructure** — Docker rate limits, network timeouts, disk space, job cancellation >= 55 min → Retry
315183

316184
## Error Handling
317185

.claude/skills/troubleshoot-ci-build/failure-patterns.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,18 @@
22

33
Reference guide for categorizing and troubleshooting common CI failures in dd-trace-dotnet.
44

5+
## Table of Contents
6+
7+
- [Infrastructure Failures](#infrastructure-failures) — Docker rate limiting, network issues, timeouts, disk space
8+
- [Flaky Tests](#flaky-tests) — Stack walking, auto-retried tests, single-runtime failures, ASM init
9+
- [Real Failures](#real-failures) — Assertion failures, snapshot mismatches, compilation errors, segfaults, missing deps
10+
- [Platform-Specific Patterns](#platform-specific-patterns) — Windows-only, Linux-only, ARM64
11+
- [Framework-Specific Patterns](#framework-specific-patterns) — .NET Framework, .NET 6/7/8
12+
- [Test-Specific Patterns](#test-specific-patterns) — Azure Functions, integration tests, smoke tests
13+
- [Categorization Decision Tree](#categorization-decision-tree) — Flowchart for classifying failures
14+
- [Quick Reference Table](#quick-reference-table) — Pattern → Category → Action lookup
15+
- [When to Investigate vs Retry](#when-to-investigate-vs-retry)
16+
517
## Infrastructure Failures
618

719
These are typically transient issues with CI infrastructure, not code problems. **Recommendation: Retry the build. Alert #apm-dotnet if persistent after retries.**

0 commit comments

Comments
 (0)