detect canceled/timed-out jobs in build analysis, improve use of "jq"

Author: lucaspimentel

.claude/skills/troubleshoot-ci-build/SKILL.md

@@ -144,6 +144,18 @@ The script automatically:
 - integration_tests_windows
 ...
 
+**Timed Out Jobs** (<count>, canceled after ~60 min):
+- `DockerTest alpine_netcoreapp3.0_group1 (60.3 min)`
+- `IntegrationTests Windows x64 net8.0 (58.7 min)`
+...
+_(Note: These are jobs with result="canceled" and duration >= 55 minutes)_
+
+**Collateral Cancellations** (<count>, < 5 min):
+- `Dependent Job 1`
+- `Dependent Job 2`
+...
+_(Note: Jobs canceled quickly, likely due to parent stage failure)_
+
 **Failed Tests** (<count>, if applicable):
 - `TestNamespace.TestClass.TestMethod1`
 - `TestNamespace.TestClass.TestMethod2`
@@ -249,6 +261,7 @@ Timeout waiting for spans
 - Docker rate limiting: `toomanyrequests`, `pull rate limit exceeded`
 - Network timeouts: `TLS handshake timeout`, `Connection reset by peer`, `ECONNRESET`
 - Execution timeouts: `maximum execution time exceeded`, `Test timeout`
+- Timeout via cancellation: Job canceled with duration >= 55 minutes
 - Disk space: `No space left on device`, `ENOSPC`
 - Container failures: `docker: Error response from daemon`
 
@@ -411,6 +424,23 @@ jq '[.records[] | select(.result == "failed")] | .[0:10]'
 --query-parameters branchName=refs/heads/master '$top=10'
 ```
 
+### 5. jq Pitfalls with Nullable Fields
+
+**Problem**: Using `startswith()`, `contains()`, or `test()` on nullable fields causes errors when the field is null
+
+**Solution**: Guard with `!= null` or use `// ""` default value
+
+```bash
+# ❌ BAD - Fails if .name is null
+jq '.records[] | select(.name | startswith("Test"))'
+
+# ✅ GOOD - Guard with null check
+jq '.records[] | select(.name != null and (.name | startswith("Test")))'
+
+# ✅ GOOD - Use default value
+jq '.records[] | select((.name // "") | startswith("Test"))'
+```
+
 ## Error Handling
 
 ### Build Not Found
@@ -522,6 +552,21 @@ The Azure DevOps timeline contains a hierarchy:
 - Failed stages/jobs cascade down - focus on Task-level failures for specifics
 - Job `identifier` field reveals platform/variant info (e.g., `integration_tests_linux.Test.Job23`)
 
+### Canceled Jobs and Timeout Detection
+
+**Result values**: `"succeeded"`, `"failed"`, `"canceled"`, `"abandoned"`
+
+**Canceled vs Failed**: Jobs canceled due to timeout have `result == "canceled"`, NOT `"failed"`.
+
+**Duration-based Classification**:
+- **Timeout** (>= 55 min): Job likely hit Azure DevOps 60-minute timeout
+- **Collateral** (< 5 min): Job canceled quickly due to parent stage failure
+- **Unknown** (5-55 min): Could be manual cancellation or other cause
+
+**Key Rule**: If a stage shows `result == "failed"` but no jobs have `result == "failed"`, check for canceled jobs with duration >= 55 minutes. These are likely timeouts that caused the stage failure.
+
+**Example**: Build 195486 - `integration_tests_linux` stage failed with no failed jobs, but `DockerTest alpine_netcoreapp3.0_group1` was canceled after 60.3 minutes (timeout).
+
 ### Understanding Test Results
 
 1. **Build warnings vs test failures**:

.claude/skills/troubleshoot-ci-build/failure-patterns.md

@@ -57,6 +57,32 @@ Test run timed out after 600000 ms
 - Retry the build
 - If persistent, investigate test performance
 
+#### Timeout via Cancellation
+
+**Detection**: Jobs with `result == "canceled"` and duration >= 55 minutes
+
+**Why not "failed"?**: Azure DevOps marks timed-out jobs as "canceled" rather than "failed".
+
+**Differentiation**:
+| Duration | Classification | Likely Cause |
+|----------|---------------|--------------|
+| >= 55 min | Timeout | Azure DevOps 60-min limit exceeded |
+| < 5 min | Collateral | Parent stage failure triggered cascade cancellation |
+| 5-55 min | Unknown | Could be manual cancellation or other cause |
+
+**Example**: Build 195486
+- Stage: `integration_tests_linux` - result: "failed"
+- Job: `DockerTest alpine_netcoreapp3.0_group1` - result: "canceled", duration: 60.3 min
+- Diagnosis: Job timed out, causing stage to fail
+- Action: Retry once; investigate if persistent (check test performance, resource contention)
+
+**Solution**:
+- Retry the build (may be transient infrastructure slowness)
+- If persistent after 2 runs, investigate:
+  - Check job logs for stuck tests or infinite loops
+  - Look for resource contention (CPU, memory, I/O)
+  - Compare with successful runs to identify anomalies
+
 ---
 
 ### Disk Space
@@ -359,15 +385,22 @@ Expected tracer log but found none
 ## Categorization Decision Tree
 
 ```
-Is the failure in this PR only (not in master)?
-├─ Yes → **Real Failure** (investigate)
-└─ No → Is it an infrastructure issue?
-    ├─ Yes → **Infrastructure** (retry)
-    └─ No → Does it have previousAttempts > 0?
-        ├─ Yes → **Flaky** (retry, monitor)
-        └─ No → Is it a known flaky test?
+Are there canceled jobs?
+├─ Yes → Check duration:
+│   ├─ >= 55 min → **Timeout** (infrastructure, retry)
+│   ├─ < 5 min → **Collateral Cancellation** (check parent failure)
+│   └─ 5-55 min → **Unknown** (review manually, could be manual cancellation)
+│
+└─ No canceled jobs or after classifying them →
+    Is the failure in this PR only (not in master)?
+    ├─ Yes → **Real Failure** (investigate)
+    └─ No → Is it an infrastructure issue?
+        ├─ Yes → **Infrastructure** (retry)
+        └─ No → Does it have previousAttempts > 0?
             ├─ Yes → **Flaky** (retry, monitor)
-            └─ No → **Pre-existing Real Failure** (investigate, may be blocking)
+            └─ No → Is it a known flaky test?
+                ├─ Yes → **Flaky** (retry, monitor)
+                └─ No → **Pre-existing Real Failure** (investigate, may be blocking)
 ```
 
 ---
@@ -379,6 +412,8 @@ Is the failure in this PR only (not in master)?
 | `toomanyrequests`, `rate limit` | Infrastructure | Retry | Low |
 | `TLS handshake`, `Connection reset` | Infrastructure | Retry | Low |
 | `maximum execution time` | Infrastructure | Retry | Medium |
+| Canceled job, duration >= 55 min | Infrastructure (Timeout) | Retry, investigate if persistent | Medium |
+| Canceled job, duration < 5 min | Collateral | Check parent failure cause | None |
 | `Failed to walk N stacks` | Flaky | Retry, monitor | Low |
 | `previousAttempts > 0` | Flaky | Retry | Low |
 | `Expected X but got Y` (new) | Real | Investigate | **High** |

.claude/skills/troubleshoot-ci-build/scripts-reference.md

@@ -108,6 +108,9 @@ When using `-OutputFormat json` or capturing the returned object, the following
 | `FailedTasks` | string[] | Array of failed task names |
 | `FailedJobs` | string[] | Array of failed job names |
 | `FailedStages` | string[] | Array of failed stage names |
+| `CanceledJobs` | string[] | Canceled job names |
+| `TimedOutJobs` | string[] | Jobs canceled after ~60 min (format: "name (XX.X min)") |
+| `CollateralCanceled` | string[] | Jobs canceled in < 5 min |
 | `FailedTests` | string[] | Extracted test names (via regex patterns) |
 | `ErrorMessages` | string[] | Raw error messages from failed tasks |
 | `Comparison` | object | Comparison data (see below) or `null` |
@@ -165,6 +168,16 @@ Comparison uses PowerShell's `Where-Object` with `-notin` operator:
 
 Replaces bash `comm -13` for Windows compatibility.
 
+#### Canceled Job Detection
+
+Jobs with `result == "canceled"` are classified by duration:
+
+- **Timeout** (>= 55 min): Job likely hit Azure DevOps 60-minute timeout. Threshold is 55 minutes (not 60) to account for Azure DevOps timing variance and early cancellation.
+- **Collateral** (< 5 min): Job canceled quickly, likely due to parent stage failure triggering cascade cancellation.
+- **Unknown** (5-55 min): Could be manual cancellation or other causes.
+
+Duration is calculated from `startTime` and `finishTime` fields in the timeline record.
+
 ### Exit Codes
 
 - **0:** Success (build analyzed, data returned)
@@ -186,6 +199,7 @@ Use `-Verbose` to see:
 2. **Log URL Availability:** Some tasks may not have `.log.url` populated
 3. **Master Build Lookup:** Checks only last 10 builds; may fail if no successful build in range
 4. **Performance:** Downloads full timeline JSON (can be large for multi-stage pipelines)
+5. **Canceled Job Classification:** Duration-based heuristic may misclassify manual cancellations that occur between 5-55 minutes. In practice, most manual cancellations happen quickly (< 5 min) or timeouts occur at ~60 min, so this range is rare.
 
 ### Related Documentation
 

tracer/tools/Get-AzureDevOpsBuildAnalysis.ps1

@@ -189,6 +189,48 @@ function Get-FailedRecords {
     }
 }
 
+function Get-CanceledRecords {
+    param(
+        [object]$Timeline,
+        [string]$Type
+    )
+
+    $canceledRecords = $Timeline.records | Where-Object {
+        $_.result -eq 'canceled' -and $_.type -eq $Type
+    }
+
+    $results = @()
+    foreach ($record in $canceledRecords) {
+        $durationMinutes = 0
+        $classification = 'unknown'
+
+        if ($record.startTime -and $record.finishTime) {
+            $startTime = [datetime]::Parse($record.startTime)
+            $finishTime = [datetime]::Parse($record.finishTime)
+            $durationMinutes = ($finishTime - $startTime).TotalMinutes
+        }
+
+        # Classify based on duration
+        # >= 55 min: likely timeout (Azure DevOps has 60 min default)
+        # < 5 min: likely collateral damage from parent cancellation
+        # 5-55 min: unknown (could be manual cancellation or other)
+        if ($durationMinutes -ge 55) {
+            $classification = 'timeout'
+        }
+        elseif ($durationMinutes -lt 5) {
+            $classification = 'collateral'
+        }
+
+        $results += [PSCustomObject]@{
+            Name             = $record.name
+            DurationMinutes  = [math]::Round($durationMinutes, 1)
+            Classification   = $classification
+        }
+    }
+
+    return $results
+}
+
 #endregion
 
 try {
@@ -232,6 +274,10 @@ try {
     $failedJobs = @(Get-FailedRecords -Timeline $timeline -Type 'Job')
     $failedStages = @(Get-FailedRecords -Timeline $timeline -Type 'Stage')
 
+    $canceledJobs = @(Get-CanceledRecords -Timeline $timeline -Type 'Job')
+    $timedOutJobs = @($canceledJobs | Where-Object { $_.Classification -eq 'timeout' })
+    $collateralCanceledJobs = @($canceledJobs | Where-Object { $_.Classification -eq 'collateral' })
+
     $errorMessages = @($failedTasks | ForEach-Object {
         $_.issues | Where-Object { $_.type -eq 'error' } | ForEach-Object { $_.message }
     })
@@ -251,6 +297,9 @@ try {
         FailedTasks     = @($failedTasks | ForEach-Object { $_.name })
         FailedJobs      = @($failedJobs | ForEach-Object { $_.name })
         FailedStages    = @($failedStages | ForEach-Object { $_.name })
+        CanceledJobs         = @($canceledJobs | ForEach-Object { $_.Name })
+        TimedOutJobs         = @($timedOutJobs | ForEach-Object { "$($_.Name) ($($_.DurationMinutes) min)" })
+        CollateralCanceled   = @($collateralCanceledJobs | ForEach-Object { $_.Name })
         FailedTests     = $failedTests
         ErrorMessages   = $errorMessages
         Comparison      = $null
@@ -364,6 +413,8 @@ try {
         Write-Host "  Failed Jobs:   " -NoNewline; Write-Host $result.FailedJobs.Count -ForegroundColor Green
         Write-Host "  Failed Tasks:  " -NoNewline; Write-Host $result.FailedTasks.Count -ForegroundColor Green
         Write-Host "  Failed Tests:  " -NoNewline; Write-Host $result.FailedTests.Count -ForegroundColor Green
+        Write-Host "  Timed Out:     " -NoNewline; Write-Host $result.TimedOutJobs.Count -ForegroundColor Yellow
+        Write-Host "  Canceled:      " -NoNewline; Write-Host $result.CanceledJobs.Count -ForegroundColor DarkGray
 
         if ($result.FailedStages.Count -gt 0) {
             Write-Host "`nFailed Stages:" -ForegroundColor Red
@@ -380,6 +431,16 @@ try {
             $result.FailedTasks | ForEach-Object { Write-Host "  - $_" -ForegroundColor DarkGray }
         }
 
+        if ($result.TimedOutJobs.Count -gt 0) {
+            Write-Host "`nTimed Out Jobs (canceled after ~60 min):" -ForegroundColor Yellow
+            $result.TimedOutJobs | ForEach-Object { Write-Host "  - $_" -ForegroundColor DarkGray }
+        }
+
+        if ($result.CollateralCanceled.Count -gt 0) {
+            Write-Host "`nCollateral Cancellations (< 5 min, likely parent failure):" -ForegroundColor DarkGray
+            $result.CollateralCanceled | ForEach-Object { Write-Host "  - $_" -ForegroundColor DarkGray }
+        }
+
         if ($result.FailedTests.Count -gt 0) {
             Write-Host "`nFailed Tests:" -ForegroundColor Red
             $result.FailedTests | ForEach-Object { Write-Host "  - $_" -ForegroundColor DarkGray }