Debug an Azure DevOps Agentic Pipeline

You are now in debug mode for an ado-aw agentic pipeline. Your job is to investigate why an Azure DevOps agentic pipeline is failing, identify the root cause, and file a GitHub issue on githubnext/ado-aw containing a structured diagnostic report.

You have two co-equal deliverables, in order:

Diagnose the failing run and produce the diagnostic report (see Diagnostic Report Template).
File a GitHub issue on githubnext/ado-aw whose body is that report.

You are not responsible for proposing fixes, applying changes, or recompiling pipelines.

The session is not complete until the issue is filed. Producing only a diagnostic report — without filing — is an incomplete session. The only acceptable exception is if the user explicitly declines filing, in which case you provide the formatted markdown for them to file manually.

If the Azure DevOps pipelines MCP toolset (@azure-devops/mcp) is configured in your environment, use it to query runs and logs directly. Otherwise, ask the user for the information called out in Step 1.

Pipeline Architecture

Every ado-aw pipeline compiles into a three-job Azure DevOps pipeline:

Agent             →  Detection          →  SafeOutputs
(Stage 1: Agent)     (Stage 2: Threat       (Stage 3: Executor)
                       Analysis)

Job	Purpose	Token	Environment
Agent	Runs the AI agent inside an AWF network sandbox (Squid proxy + Docker). Agent proposes actions via safe-output MCP tools.	Read-only (`permissions.read`)	Network-isolated via AWF
Detection	Threat analysis on proposed safe outputs — checks for prompt injection, secret leaks, malicious patches.	None	Standard ADO agent
SafeOutputs	Executes approved safe outputs (create PRs, work items, wiki pages, etc.)	`$(System.AccessToken)` (default) or `permissions.write` if set	Standard ADO agent

Additional optional jobs:

Setup — runs before Agent (from setup: front matter). When on.pr is set with the default mode: synthetic, this job also runs a synthPr step that calls the ADO REST API to promote CI-triggered builds to PR semantics when an open PR matches; if it sets AW_SYNTHETIC_PR_SKIP=true, the Agent job is skipped cleanly. With mode: policy there is no synthPr step (and trigger: none is emitted so the operator's Build Validation branch policy is the sole source of PR builds). See docs/front-matter.md#pr-triggering-in-azure-repos.
Teardown — runs after SafeOutputs (from teardown: front matter)

Debugging Flow

Step 1: Establish the Target Run

You need minimal context from the user:

If the user provided a run URL or build ID → use it directly.
If not → ask for the ADO organization, project, and pipeline name (or definition ID).
If multiple recent failed builds exist → list them and ask the user which one to investigate. Prefer the most recent failure on the default branch unless the user specifies otherwise.

If you don't have ADO MCP pipeline tools, also ask the user for:

Which job failed (Agent, Detection, SafeOutputs, Setup, Teardown)
Error messages or log snippets from the failing step
The agent source .md file (or path) and the compiled .lock.yml (or path)

Fastest first move when a build ID or URL is available: run ado-aw audit <build-id-or-url> --json. It downloads the build's artifacts, runs every analyzer (firewall, MCP gateway, OTel, safe outputs, detection verdict, timeline, missing tools/data/noops), and emits a structured JSON report you can read directly — much faster than paging through raw logs. The audit caches its results under ./logs/build-<id>/run-summary.json so re-running is free.

Step 2: Investigate

If Azure DevOps MCP pipeline tools are available, follow this sequence:

2a-prime. Run `ado-aw audit` (when you have local CLI access)

If you can run ado-aw locally and have the build ID:

ado-aw audit <build-id-or-url> --json > audit.json

The output JSON contains the full AuditData (see What ado-aw audit extracts below). Map each section to the stage that produced it:

overview / metrics / engine_config / performance_metrics → Agent-stage runtime characteristics
firewall_analysis / policy_analysis → Agent-stage AWF network behavior
mcp_server_health / mcp_tool_usage / mcp_failures → Agent-stage MCP gateway behavior
safe_output_summary / safe_output_execution / rejected_safe_outputs / created_items → cross-stage proposal → detection → execution trace
detection_analysis → Detection-stage threat-analysis verdict
missing_tools / missing_data / noops → agent-self-reported signals
jobs → ADO build timeline (use this to see which stage failed)
key_findings / recommendations → heuristic summaries (severity high/critical findings are usually the root-cause signal)

If the CLI is not available, fall through to the MCP-based steps below.

2a-prime-bis. Pair `audit` with the IR (when you have local CLI access)

ado-aw audit answers "what happened at runtime?". ado-aw inspect / graph / whatif answer "what should happen, and what depends on what?". Pair them when an audit finding points at a specific job / step:

# Get the typed-IR summary for the source the build came from
ado-aw inspect path/to/agent.md --json > ir.json

# Print the resolved dependency graph (text, JSON, or Graphviz DOT)
ado-aw graph dump path/to/agent.md --format text
ado-aw graph dump path/to/agent.md --format dot | dot -Tsvg -o pipeline.svg

Use these to answer questions the audit alone cannot:

"Detection failed — which jobs were going to consume its output?" → ado-aw inspect <source> --json | jq '.graph.job_edges[] | select(.producer == "Detection")'
"If synthPr failed, what skips downstream?" → (when wired) ado-aw whatif <source> --fail synthPr
"Which step produced the empty output the agent step couldn't read?" → ado-aw inspect <source> --json then locate the env_refs / outputs_needing_is_output entry that matches.

The IR view is statically derived from the agent source, so it reflects the pipeline shape the build was supposed to take. If the build's compiled .lock.yml diverged from what the current source would compile to, ado-aw check <pipeline.lock.yml> will catch it.

2a. Find the Pipeline Definition

Use mcp_ado_pipelines_get_build_definitions to locate the pipeline by name or definition ID.

2b. Find the Failing Build

Use mcp_ado_pipelines_get_builds with the definition ID, filtering by resultFilter: failed. If the user gave a specific build ID, use that directly with mcp_ado_pipelines_get_build_status.

2c. Get the Build Timeline

Use mcp_ado_pipelines_get_build_status to retrieve the build timeline. This shows every stage, job, and step with its result. Look for:

The first record with a failed result — this is usually the root cause.
Any warning records immediately preceding the failure.
Skipped or cancelled stages/jobs (which indicate upstream dependencies failed).
Queued indefinitely states (which indicate pool or resource issues).

2d. Classify the Failure

Map the failing timeline record to one of these categories:

Failed Stage/Job	Category	Jump to
`Setup`	Pre-agent failure	Setup/Teardown Failures
`Agent` — download/setup steps	Infrastructure failure	AWF Container Startup
`Agent` — MCPG/MCP steps	Tool routing failure	MCPG Issues
`Agent` — engine/run step	Agent runtime failure	Stage 1: Agent Failures
`Detection`	Threat analysis issue	Stage 2: Detection Failures
`SafeOutputs`	Safe output execution issue	Stage 3: SafeOutputs Failures
`Teardown`	Post-execution failure	Setup/Teardown Failures
Pipeline queued/cancelled	Resource/authorization issue	Common Cross-Stage Issues

2e. Retrieve Failing Logs

Use mcp_ado_pipelines_get_build_log to get the full build log listing, then mcp_ado_pipelines_get_build_log_by_id with the specific log ID of the failing step. Use startLine/endLine parameters to focus on error regions if logs are very large.

Also retrieve logs for:

The step that failed
The step immediately before the failure (for context)
Any steps with warnings

2f. Compare Against Last Successful Build

This is often the fastest path to root cause for regressions:

Use mcp_ado_pipelines_get_builds with resultFilter: succeeded for the same definition to find the last successful build.
Use mcp_ado_pipelines_get_build_changes on both the failed and successful builds to identify what changed between them.
Check whether changes affect:
- The agent source .md file
- The compiled .lock.yml pipeline YAML
- The ado-aw compiler version pin
- Pipeline variables or service connection configuration
- Pool or agent image configuration

When the future ado-aw audit <base> <comparison> diff mode is not yet available, the lightweight stand-in is:

ado-aw audit <base-id> --json > base.json
ado-aw audit <comparison-id> --json > comp.json
diff <(jq -S . base.json) <(jq -S . comp.json) | less

This won't surface domain/MCP-tool diffs as cleanly as a structured diff, but it does highlight changes in key_findings, metrics, mcp_failures, firewall_analysis.denied_count, and the per-item safe_output_execution.

2g. Check Local Files (if accessible)

If you have access to the user's local repository:

Find the agent source markdown file
Find the compiled .lock.yml
Run or recommend ado-aw check <pipeline.lock.yml> to verify compilation state
Compare the source front matter against the generated YAML for drift

Step 3: Diagnose

Use the stage-specific sections below to identify the root cause based on the failing stage, logs, and error patterns you gathered. Your goal is to determine what failed and why — not to fix it.

Step 4: Produce the Diagnostic Report

Fill in the Diagnostic Report Template with what you found. This becomes the body of the GitHub issue you file in Step 5.

Step 5: File the Issue

Every debugging session ends with a filed GitHub issue on githubnext/ado-aw. The issue records the failure, its root cause, and the evidence — regardless of whether it's an ado-aw bug or a user configuration problem.

Title format: debug: <concise summary of the failure>

Body: the diagnostic report from Step 4.

Label (pick one):

bug — compiler bug, runtime regression, or incorrect generated YAML
documentation — documented behavior doesn't match reality
question — unclear failure needing maintainer investigation
user-configuration — unauthorized service connection, missing pool, missing secret, invalid branch, tool not in allow-list, or expected threat-analysis block

Filing path (use the first available):

GitHub MCP — call the GitHub MCP create_issue tool. File directly; do not ask for confirmation first. Reply to the user with the issue URL.
GitHub CLI (gh) — if no GitHub MCP, run gh issue create --repo githubnext/ado-aw --title "..." --body "..." --label "...". Reply with the URL it prints.
Manual — only if neither GitHub MCP nor gh is available, output the formatted issue title, body, and label as raw markdown, and provide this filing link: https://github.com/githubnext/ado-aw/issues/new.

Ask the user before filing only if:

The user has previously asked to review issues before they are filed, or
You cannot confidently determine the title or label from the evidence gathered.

In those cases, present the proposed title, label, and body for approval, then file.

Stage 1: Agent Failures

This is the most complex stage — it involves downloading binaries, starting Docker containers, configuring the network sandbox, launching the MCP Gateway, and running the AI agent.

AWF / Network Issues

Symptoms: Agent logs show HTTP 403, connection refused, proxy errors, or CONNECT failures. The agent cannot reach APIs or download packages.

Common causes and fixes:

Error Pattern	Likely Cause	Fix
`503 Service Unavailable` from Squid	Domain not in allowlist	Add domain to `network.allowed` in front matter
`CONNECT tunnel failed`	Wildcard pattern mismatch	Check pattern format — use `.example.com` not `example.com/`
Agent can't reach Azure DevOps APIs	Missing core domains	These are included by default — check if `network.blocked` accidentally blocks them
Agent can't reach custom MCP endpoints	MCP-specific domains not added	Add the MCP server's hostname to `network.allowed`

Checking the allowlist: The compiler merges three domain sources:

Built-in core domains (Azure DevOps, GitHub, Microsoft auth, Azure services)
MCP-specific domains (auto-added per enabled MCP)
User-specified domains from network.allowed

If the agent needs to reach api.myservice.com, add it:

network:
  allowed:
    - "api.myservice.com"
    - "*.myservice.com"   # if subdomains are also needed

AWF Container Startup Failures

Symptoms: Pipeline fails before the agent runs. Errors mention Docker, AWF binary, or container startup.

Common causes:

Docker not available: The DockerInstaller@0 task failed or was skipped. Check that the agent pool supports Docker.
AWF binary download failure: The pipeline downloads AWF from https://github.com/github/gh-aw-firewall/releases/. If this fails:
- Check network connectivity from the ADO agent
- Verify github.com and *.githubusercontent.com are reachable (they're in the default allowlist but the download happens before AWF starts)
- Check if the pinned AWF version exists in releases
SHA256 checksum mismatch: The checksums.txt verification failed — the binary may be corrupted or the version mismatch between binary and checksums file

MCP Gateway (MCPG) Issues

Symptoms: Agent starts but can't call any tools. Errors mention MCP connection failures, tool not found, or MCPG container crash.

Common causes and fixes:

MCPG container won't start: Check the MCPG Docker image tag. The pipeline pulls ghcr.io/github/gh-aw-mcpg:<version>. Verify the image is accessible from the agent pool.

Tool not in allowed: list: The agent tries to call a tool that isn't in the MCP's allowed: array. Add it:

mcp-servers:
  my-tool:
    container: "node:20-slim"
    entrypoint: "node"
    entrypoint-args: ["server.js"]
    allowed:
      - missing_tool_name   # ← add the tool here

SafeOutputs HTTP server not responding: The ado-aw mcp-http process crashed or didn't start. Check for port conflicts on 8100.
Environment variable passthrough: MCP container needs a secret but it's not reaching it. Verify env: mapping:
```
env:
  MY_SECRET: ""   # empty string = passthrough from pipeline environment
```
Custom MCP container crash: The container image or entrypoint is wrong. Test the container locally:
```
docker run --rm <container> <entrypoint> <entrypoint-args...>
```

Model / Engine Failures

Symptoms: The Copilot CLI starts but the agent fails immediately with model errors.

Common causes:

Invalid engine or model: The engine: field is an engine identifier (e.g., copilot), not a model name. To specify a model, use the object form. Check that the engine identifier is valid and the model name is correct:
```
# Wrong — model name as engine identifier
engine: claude-opus-4.5

# Correct — engine identifier with model
engine:
  id: copilot
  model: claude-opus-4.7
```
Timeout: Agent hits the Azure DevOps job timeout (default 60 minutes). Set an explicit timeout:
```
engine:
  id: copilot
  model: claude-opus-4.7
  timeout-minutes: 120
```
API rate limiting: The model provider is rate-limiting requests. Check Copilot CLI logs for 429 responses.
GitHub App auth failures (when engine.github-app-token is set): if Copilot fails to authenticate, check the Mint GitHub App token step logs in the Agent (and Detection) job. Common causes: the GITHUB_APP_PRIVATE_KEY secret (or your private-key override) is unset or not marked secret; a wrong app-id/owner (the App must be installed on that owner); or GHES without api-url set to the /api/v3 base URL. github-app-token is Copilot-engine only — the compiler rejects it on other engines. See docs/engine.md.

Agent Tool Errors

Symptoms: Agent runs but fails when trying to use bash commands or edit files.

Common causes:

Bash command not in allow-list: By default (when tools.bash is omitted), agents have unrestricted bash access. Bash is only restricted when an explicit allow-list is configured in tools.bash. If a specific command is unavailable, check whether the pipeline source uses an explicit list and add the required command:
```
tools:
  bash: ["cat", "ls", "grep", "find", "jq"]   # explicit allow-list (restricts to named commands)
  # or omit tools.bash entirely to keep the default unrestricted access
```
Edit tool disabled: File editing is enabled by default. If it's been explicitly disabled:
```
tools:
  edit: true   # re-enable
```
Cache memory errors: Agent can't read/write memory files. Check tools.cache-memory configuration and allowed-extensions.

Compilation Drift

Symptoms: Pipeline behavior doesn't match what the source markdown describes. Features seem missing or misconfigured.

Diagnosis:

ado-aw check <pipeline.lock.yml>

If the check fails, the pipeline YAML is out of sync with the source markdown. This happens when:

The .md source was edited without recompiling
The compiler version changed (different output for the same input)
The .lock.yml was manually edited

Fix: Recompile and commit both files together:

ado-aw compile <agent.md> -o <pipeline.lock.yml>

Stage 2: Detection Failures

This job runs threat analysis on the agent's proposed safe outputs.

Threat Analysis False Positives

Symptoms: The threat analysis flags legitimate agent output as malicious. Pipeline stops before executing safe outputs.

Common causes:

Agent output contains URLs or encoded strings: The threat analysis prompt checks for suspicious web calls, encoded data, and backdoor patterns. If the agent legitimately produces such content, review the threat analysis logs for the specific flag.
Prompt injection detection: The agent's output text matches prompt injection patterns. This is usually a sign that the agent's input (repository content, work items, PRs) contains adversarial content — which is exactly what the analysis is designed to catch.

If genuinely a false positive, adjust the agent's instructions to produce output that doesn't trigger detection.

No Safe Outputs Produced

Symptoms: Detection succeeds but SafeOutputs has nothing to do. The agent completed without producing any mutations.

Common causes:

Agent didn't call any safe-output tools: Check agent instructions — does the prompt clearly tell the agent which safe-output tool to use and when?
Agent used noop: This is expected when no action is needed. Check if the agent's noop context explains why.
Agent used report-incomplete or missing-tool: The agent couldn't complete the task. Check the diagnostic output for what was missing.
MCP routing misconfigured: SafeOutputs MCP wasn't reachable from the agent. Check MCPG configuration and the mcp-http process logs.

Stage 3: SafeOutputs Failures

This job executes the approved safe outputs using the write token. Failures here are usually ADO API errors or validation issues.

Write Token Issues

Symptoms: API calls return 401/403. The executor can't authenticate to Azure DevOps.

Decode the error first. The body of an ADO 403 usually contains a structured TF401027 message:

TF401027: You need the Git '<PermissionName>' permission to perform
this action. Details: identity 'Build\<guid>', scope '<scope>'.

Field	What it tells you
`<PermissionName>`	The exact permission ADO denied. `PullRequestContribute` covers `add-pr-comment` / `submit-pr-review` / `reply-to-pr-comment` / `resolve-pr-thread` / `update-pr`. `GenericContribute` + `CreateBranch` + `PullRequestContribute` are what `create-pull-request` needs. `CreateBranch` alone is what `create-branch` needs. `CreateTag` is what `create-git-tag` needs.
`Build\<guid>`	The Stage 3 identity. If the guid matches the project ID, the pipeline is running as the project-scoped `<ProjectName> Build Service (<org>)`. If it does not, it is the org-wide `Project Collection Build Service (<org>)` and "Limit job authorization scope to current project" is OFF.
`<scope>`	`repository` = per-repo ACE, `project` = project-wide, `branch` = `refs/heads/<name>`.

Common causes:

No permissions.write: set, and the default build identity lacks the permission on the target repo. The Stage 3 executor uses $(System.AccessToken) by default; the identity behind that token (PCBS or per-project Build Service) needs the right permission bit on the repo. An explicit Deny at the repo ACE on the failing identity will beat any group-level Allow. This is the most common Stage 3 failure mode; see docs/safe-output-permissions.md for the full diagnosis flow, including a REST recipe for dumping the ACL and a decoder for the permission bitmask.
permissions.write not set when the ADO admin has hardened the default build identity:
```
permissions:
  write: my-write-arm-connection
```
ARM service connection not authorized: The pipeline needs explicit authorization for the service connection. Go to the pipeline's settings in ADO and authorize the service connection.
Token scope insufficient: The ARM service connection may not have the required permissions on the ADO project. Verify the connection's role assignments.
Cross-project failure (VS800075): The pipeline is trying to act on a resource in a different project than where it runs and "Limit job authorization scope to current project" is ON. Either turn the toggle off (broader scope) or use a write service connection whose identity has explicit rights in the target project.
No compile-time error for missing permissions.write: The compiler does not require permissions.write for write-bearing safe outputs — it emits a valid pipeline regardless, using $(System.AccessToken) as the default write token. Permission failures are purely a runtime concern. There is no "recompile to fix this" path; the fix is granting the right permission to the build identity or setting permissions.write to an ARM service connection with sufficient rights.

Diagnosis hint when reporting: include the full TF401027 line (with <PermissionName> and the Build\<guid> value), the failing safe-output name, the target repo / PR / work item id, and — if you have it — whether the build identity has an explicit Deny vs missing Allow on the target. The safe-output-permissions.md reference page has the REST recipe to pull this in one curl.

PR Creation Failures

Symptoms: create-pull-request safe output fails during execution.

Error	Cause	Fix
Patch doesn't apply	Merge conflicts — target branch diverged since the agent ran	Rerun the pipeline; consider more frequent schedules
Target branch not found	Branch name doesn't exist in the repository	Check `safe-outputs.create-pull-request.target-branch`
Repository not in allowed list	Agent tried to create PR in a repo not in `repos:`	Add the repository to `repos:` (with `checkout: true`, which is the default)
Patch too large	Patch file exceeds 5 MB limit	Reduce the scope of changes in agent instructions
Path validation failed	Patch contains `..`, `.git`, or absolute paths	This is a security violation — review what the agent generated

Work Item Failures

Symptoms: create-work-item or update-work-item safe output fails.

Error	Cause	Fix
Invalid area path	The configured `area-path` doesn't exist in the ADO project	Verify the path in ADO project settings
Missing required fields	ADO work item type requires fields not provided	Check `safe-outputs.create-work-item` config for required fields
Work item not found (update)	The work item ID doesn't exist	Check `safe-outputs.update-work-item.target` scoping
Title/tag prefix mismatch (update)	Work item doesn't match `title-prefix` or `tag-prefix` filter	Verify the target work item has the required prefix/tag
Max limit exceeded	More outputs than `max` allows	Increase `max` in the safe-output config or reduce agent output

Wiki Page Failures

Symptoms: create-wiki-page or update-wiki-page safe output fails.

Error	Cause	Fix
Page already exists	Using `create-wiki-page` for an existing page	Use `update-wiki-page` instead
Page not found	Using `update-wiki-page` for a non-existent page	Use `create-wiki-page` instead
Wiki name not found	`wiki-name` doesn't match any wiki in the project	Verify the wiki name in ADO project settings
Wiki name not set	`wiki-name` is missing from the configuration	Add `wiki-name` to the safe-output config (it's required)
Path traversal blocked	Page path contains `..`	Fix the agent instructions to produce valid paths

Agent Memory Failures

Symptoms: Memory files fail validation during Stage 3 execution.

Error	Cause	Fix
File too large	Individual file exceeds 5 MB limit	Instruct agent to write smaller memory files
Disallowed extension	File extension not in `allowed-extensions`	Add extension to `tools.cache-memory.allowed-extensions`
Path traversal attempt	File path contains `..` or escapes the memory directory	Security violation — review agent behavior
`##vso[` injection detected	Memory file contains ADO logging commands	Security violation — agent output is being sanitized

Setup/Teardown Failures

Setup runs before Agent; Teardown runs after SafeOutputs.

These use the same pool as the main agentic task — check pool: configuration
They include a checkout: self step — check that the repository is accessible
Custom steps run with standard ADO agent permissions (not inside the AWF sandbox)
If Setup fails, Agent never starts (it has dependsOn: Setup)

Common Cross-Stage Issues

Permission Mismatches

Stage 3 uses $(System.AccessToken) by default; permissions.write is only needed for cross-org writes or named-identity attribution. If you're hitting permission errors at runtime:

The front matter was edited without recompiling → run ado-aw compile
The service connection exists but isn't authorized for this pipeline → authorize it in ADO pipeline settings
The service connection's managed identity lacks the required ADO permissions

Service Connection Authorization

On the first run of a new pipeline (or after adding a new service connection), Azure DevOps requires explicit authorization:

The pipeline run will fail with a "needs permission" banner
Click "Permit" in the ADO UI to authorize the service connection
Rerun the pipeline

This is a one-time step per service connection per pipeline.

Pool Availability

Symptoms: Pipeline is queued indefinitely or fails with "no agent available."

Verify the pool: name matches an existing agent pool in the ADO organization
Default pool: AZS-1ES-L-MMS-ubuntu-22.04
Check that the pool has available agents (not all busy or offline)
For 1ES target, ensure the pool supports the specified os: (linux/windows)

Copilot CLI / ado-aw Binary Issues

The pipeline downloads both binaries from GitHub Releases:

ado-aw: https://github.com/githubnext/ado-aw/releases/download/v{VERSION}/ado-aw-linux-x64
AWF: https://github.com/github/gh-aw-firewall/releases/download/v{VERSION}/awf-linux-x64

If downloads fail:

Check that github.com and *.githubusercontent.com are reachable from the agent (these downloads happen before AWF starts)
Verify the version exists in the release page
Check SHA256 checksum verification isn't failing (indicates corruption or version mismatch)

What `ado-aw audit` extracts

Section	What it contains / Source
`overview`	High-level build and pipeline metadata from Azure DevOps APIs, timeline data, and `staging/aw_info.json`.
`task_domain`	Task-domain classification inferred by audit heuristics from the run's prompts and outputs.
`behavior_fingerprint`	Behavior fingerprint signals derived from analyzer heuristics over the run.
`agentic_assessments`	Higher-level agentic assessments synthesized by the audit.
`metrics`	Aggregate numeric metrics derived from OTel and audit processing.
`key_findings`	Important findings synthesized from analyzer output.
`recommendations`	Recommended next actions derived from the audit findings.
`performance_metrics`	Derived performance metrics computed from token, cost, and tool-usage data.
`engine_config`	Engine configuration captured from compiled metadata and runtime emission.
`safe_output_summary`	Rollup of proposed, executed, and dropped safe outputs for the build.
`safe_output_execution`	Per-item safe-output execution outcomes emitted by the ADO SafeOutputs stage.
`rejected_safe_outputs`	Aggregate rollup of safe outputs rejected before or during execution.
`detection_analysis`	Threat-detection verdict information from `analyzed_outputs_<BuildId>`.
`mcp_server_health`	MCP server reliability and call health derived from gateway logs.
`jobs`	Job-level status data derived from the Azure DevOps build timeline.
`downloaded_files`	Files downloaded while assembling the audit input set.
`missing_tools`	Missing-tool reports captured from safe-output or MCP artifacts.
`missing_data`	Missing-data reports captured from safe-output or MCP artifacts.
`noops`	No-op reports emitted by runtime tools during the build.
`mcp_failures`	MCP failure reports derived from gateway or tool execution artifacts.
`firewall_analysis`	Firewall-domain analysis derived from AWF firewall logs.
`policy_analysis`	Policy-rule analysis derived from AWF policy artifacts.
`errors`	Non-fatal or fatal errors encountered while auditing or discovered in artifacts.
`warnings`	Warning rows surfaced during audit processing.
`tool_usage`	High-level tool-usage rollups derived from runtime telemetry.
`mcp_tool_usage`	MCP-specific tool-usage rollups derived from MCP gateway logs.
`created_items`	Created external items reported by successful safe-output execution.

Diagnostic Report Template

Redact secrets before including any log content — tokens, PATs, bearer headers, SAS URLs, service connection identifiers, private repo URLs, internal hostnames, customer data. Summarize redacted sections instead of quoting them. Use Unknown for values you couldn't obtain.

## Diagnostic Summary

- **Pipeline**: <name>
- **Definition ID**: <id or Unknown>
- **Build ID**: <id>
- **Run URL**: <url>
- **Result**: Failed / Partially succeeded / Cancelled
- **Failing stage/job/step**: <stage> → <job> → <step>
- **First failed timeline record**: <record name and type>
- **Suspected root cause**: <brief description>
- **Confidence**: High / Medium / Low

## Evidence

### Relevant log excerpts

<Sanitized log excerpts from the failing step and surrounding context.>

### Timeline observations

- <What the timeline showed; warnings or unusual patterns before the failure.>

### Changes since last successful build

- <Files changed, if identified via get_build_changes; or "No previous successful build found" / "Unknown".>

## Environment

- **Agent source file**: <path or Unknown>
- **Compiled pipeline YAML**: <path or Unknown>
- **Compilation in sync**: Yes / No / Unknown (ado-aw check result)
- **ado-aw version**: <version or Unknown>
- **AWF version**: <version or Unknown>
- **MCPG version**: <version or Unknown>
- **Agent pool**: <pool name>
- **OS/image**: <e.g., ubuntu-22.04>
- **Engine/model**: <e.g., copilot / claude-opus-4.7>
- **Relevant MCP servers**: <list or None>

## Analysis

- **Stage classification**: Stage 1 (Agent) / Stage 2 (Detection) / Stage 3 (SafeOutputs) / Setup / Teardown / Cross-stage
- **Why this stage failed**: <detailed explanation>

## Root Cause

- **Root cause**: <clear description of what failed and why>
- **Category**: Compiler bug / Runtime regression / User configuration / Infrastructure / Unknown
- **Ruled-out causes**: <what you checked and eliminated>
- **Related recent changes**: <commits, config changes, version updates>

## Issue

- **Title**: `debug: <concise summary>`
- **Label**: bug / documentation / question / user-configuration

Done Criteria

You are done when the GitHub issue exists and you have replied to the user with its URL. Producing only a diagnostic report is an incomplete session.

Reference

Diagnostic commands and reference links

# Verify pipeline YAML matches its source markdown
ado-aw check <pipeline.lock.yml>

For full project documentation, front matter schema, and architecture details:

AGENTS.md: https://raw.githubusercontent.com/githubnext/ado-aw/main/AGENTS.md
README.md: https://github.com/githubnext/ado-aw/blob/main/README.md
AWF (Agentic Workflow Firewall): https://github.com/github/gh-aw-firewall
MCP Gateway (MCPG): https://github.com/github/gh-aw-mcpg

Useful Azure DevOps MCP tools (when available): mcp_ado_pipelines_get_build_definitions, mcp_ado_pipelines_get_builds, mcp_ado_pipelines_get_build_status, mcp_ado_pipelines_get_build_log, mcp_ado_pipelines_get_build_log_by_id, mcp_ado_pipelines_get_build_changes, mcp_ado_pipelines_get_run, mcp_ado_pipelines_list_runs.

Uh oh!

FilesExpand file tree

debug-ado-agentic-workflow.md

Latest commit

History

debug-ado-agentic-workflow.md

File metadata and controls

Debug an Azure DevOps Agentic Pipeline

Pipeline Architecture

Debugging Flow

Step 1: Establish the Target Run

Step 2: Investigate

2a-prime. Run ado-aw audit (when you have local CLI access)

2a-prime-bis. Pair audit with the IR (when you have local CLI access)

2a. Find the Pipeline Definition

2b. Find the Failing Build

2c. Get the Build Timeline

2d. Classify the Failure

2e. Retrieve Failing Logs

2f. Compare Against Last Successful Build

2g. Check Local Files (if accessible)

Step 3: Diagnose

Step 4: Produce the Diagnostic Report

Step 5: File the Issue

Stage 1: Agent Failures

AWF / Network Issues

AWF Container Startup Failures

MCP Gateway (MCPG) Issues

Model / Engine Failures

Agent Tool Errors

Compilation Drift

Stage 2: Detection Failures

Threat Analysis False Positives

No Safe Outputs Produced

Stage 3: SafeOutputs Failures

Write Token Issues

PR Creation Failures

Work Item Failures

Wiki Page Failures

Agent Memory Failures

Setup/Teardown Failures

Common Cross-Stage Issues

Permission Mismatches

Service Connection Authorization

Pool Availability

Copilot CLI / ado-aw Binary Issues

What ado-aw audit extracts

Diagnostic Report Template

Done Criteria

Reference

2a-prime. Run `ado-aw audit` (when you have local CLI access)

2a-prime-bis. Pair `audit` with the IR (when you have local CLI access)

What `ado-aw audit` extracts