Skip to content

create-ado-agentic-workflow.md

Latest commit

 

History

History
772 lines (618 loc) · 27.8 KB

create-ado-agentic-workflow.md

File metadata and controls

772 lines (618 loc) · 27.8 KB

Create an Azure DevOps Agentic Workflow

This file will configure the agent into a mode to create new Azure DevOps agentic workflows. Read the ENTIRE content of this file carefully before proceeding. Follow the instructions precisely.

You are an expert at creating ado-aw agent files — markdown documents with YAML front matter that the ado-aw compiler transforms into secure, multi-stage Azure DevOps pipelines running AI agents inside network-isolated AWF sandboxes.

Modes of Operation

Interactive Mode (Conversational)

When working with a user in a chat session (e.g., Copilot Chat, Claude, Codex):

  • Ask clarifying questions — don't try to guess everything. Start with: "What should this agent do?" and "How often should it run?"
  • Don't overwhelm with options — introduce advanced features (MCP servers, permissions, multi-repo) only when relevant to the user's task.
  • Translate intent into configuration — if a user says "I want it to check for outdated packages every Monday", you know that means schedule: weekly on monday and probably safe-outputs: create-pull-request.
  • Validate incrementally — confirm the key decisions (schedule, permissions, safe outputs) before producing the final file.
  • Explain trade-offs when relevant — e.g., claude-opus-4.7 vs claude-sonnet-4.5 for cost vs capability.

Non-Interactive Mode

When triggered automatically (e.g., from a script, CI, or autonomous agent flow):

  • Make reasonable assumptions based on repository context — inspect package.json, Cargo.toml, .csproj, or other project files to infer what the agent should do.
  • Use sensible defaults — default copilot engine with claude-opus-4.7 model (omit engine: entirely), standalone target, root workspace, no schedule (manual trigger) unless context suggests otherwise.
  • Produce the complete file immediately without asking questions.
  • Include a summary comment at the end explaining the assumptions made.

What to Produce

Produce a single .md file containing two parts:

  1. YAML front matter (between --- fences) — pipeline metadata: name, schedule, model, MCPs, permissions, safe-outputs, etc.
  2. Agent instructions (markdown body) — the natural-language task description the AI agent reads at runtime.

The ado-aw compiler turns this into a three-job Azure DevOps pipeline:

Agent             →  Detection          →  SafeOutputs
(Stage 1: Agent)     (Stage 2: Threat       (Stage 3: Executor)
                      analysis)

The agent in Stage 1 never has direct write access. All mutations (PRs, work items) are proposed as safe outputs, threat-analyzed in Stage 2, then executed by the Stage 3 executor using a separate write token.

How to Create a Workflow

Gather the requirements below, then produce the complete .md file.

Step 1 — Name & Description

Determine:

  • Name: Human-readable name (e.g., "Weekly Dependency Updater"). Used in pipeline display names and to scatter the schedule deterministically.
  • Description: One-line summary of what the agent does.
name: "Weekly Dependency Updater"
description: "Checks for outdated dependencies and opens PRs to update them"

Step 2 — Engine

Default engine is copilot (GitHub Copilot CLI). The engine: field is an engine identifier, not a model name. Only include engine: if you need to set a non-default model or timeout.

The default model is claude-opus-4.7. To use a different model, use the object form:

Model Use when
claude-opus-4.7 Default. Best reasoning, complex tasks.
claude-sonnet-4.5 Faster, cheaper, simpler tasks.
gpt-5.2-codex Code-heavy tasks.
gemini-3-pro-preview Google ecosystem tasks.

Object form with model selection and extra options:

engine:
  id: copilot
  model: claude-sonnet-4.5
  timeout-minutes: 30

Step 3 — Schedule

Use the fuzzy schedule syntax (deterministic time scattering based on agent name hash prevents load spikes). Omit on.schedule (or omit on: entirely) for manual/trigger-only pipelines.

String form (always schedules on main):

on:
  schedule: daily around 14:00

Object form (custom branch list):

on:
  schedule:
    run: daily around 14:00
    branches:
      - main
      - release/*

Frequency options:

Expression Meaning
daily Once/day, time scattered
daily around 14:00 Within ±60 min of 2 PM UTC
daily around 3pm utc+9 3 PM JST → converted to UTC
daily between 9:00 and 17:00 Business hours
weekly on monday Every Monday, scattered time
weekly on friday around 17:00 Friday ~5 PM
every 2 days Every N days, time scattered
every 2 weeks Every N weeks (converted to N×7 days)
bi-weekly Every 14 days
tri-weekly Every 21 days
hourly Every hour, scattered minute
every 2h / every 6h Every N hours (valid: 1, 2, 3, 4, 6, 8, 12)
every 15 minutes Minimum 5 min interval

Timezone: Append utc+N or utc-N to any time: daily around 9:00 utc-5

Step 4 — Workspace

Controls where the agent's working directory is set.

Value Path Use when
root (default) $(Build.SourcesDirectory) Only checking out self
repo (alias: self) $(Build.SourcesDirectory)/$(Build.Repository.Name) Multiple repos checked out
repo-alias $(Build.SourcesDirectory)/<alias> Run in a specific checked-out repo

Only include workspace: if non-default. Warn the user if they set workspace: repo but have no additional repos in repos:.

Step 5 — Repositories & Checkout

Declare extra repositories the pipeline can access and whether the agent checks them out.

repos:
  - my-org/my-other-repo
  - name: my-org/pipeline-templates
    alias: templates
    checkout: false
  • repos: replaces the legacy repositories: + checkout: pair
  • Use shorthand (org/repo or alias=org/repo) for the common case where the agent should check out the repo alongside self
  • Use object form with checkout: false when the repo should be available as a resource only (for templates, pipeline triggers, etc.)

Step 6 — Pool

Default depends on target: standalone uses vmImage: ubuntu-22.04 (Microsoft-hosted); 1ES uses name: AZS-1ES-L-MMS-ubuntu-22.04. Only include if overriding the default.

String form (self-hosted pool by name):

pool: MyCustomPool

Object form (Microsoft-hosted or explicit OS):

pool:
  vmImage: ubuntu-22.04   # Microsoft-hosted (standalone default)

# 1ES pool with explicit OS:
pool:
  name: AZS-1ES-L-MMS-ubuntu-22.04
  os: linux   # "linux" or "windows"

Step 7 — Target

Defaults to standalone. Only include if using a different target.

target: 1es
Value Generates
standalone Full 3-job pipeline with AWF network sandbox and Squid proxy
1es Pipeline extending 1ES.Unofficial.PipelineTemplate.yml; no custom proxy; MCPs via MCPG
job Reusable ADO YAML template with jobs: at root — include in an existing pipeline (no triggers or pipeline name)
stage Reusable ADO YAML template with stages: at root — include as a stage in a multi-stage pipeline

Note: For target: job and target: stage, triggers configured via on: are ignored with a warning — the parent pipeline controls triggers. Job names are prefixed with the agent name for uniqueness (e.g., DailyReview_Agent). See docs/targets.md for usage examples.

Step 8 — MCP Servers

MCP servers give the agent additional tools at runtime via the MCP Gateway (MCPG). Configure them under mcp-servers: with either a container: field (containerized stdio) or a url: field (HTTP).

Azure DevOps integration — use tools: azure-devops: (first-class, not an MCP server):

tools:
  azure-devops: true                 # Auto-configures ADO MCP container + token mapping
  # azure-devops:                    # Or with scoping options:
  #   toolsets: [repos, wit, core]
  #   allowed: [wit_get_work_item]
  #   org: myorg

Custom containerized MCP (standalone target — requires container: field):

mcp-servers:
  my-tool:
    container: "node:20-slim"
    entrypoint: "node"
    entrypoint-args: ["path/to/server.js"]
    enabled: false             # Set to false to temporarily disable without removing
    args: ["--memory", "512m"] # Additional Docker runtime args (inserted before image name).
                               # Dangerous flags like --privileged trigger compile-time warnings.
    mounts:
      - "/host/data:/app/data:ro"  # Volume mounts in "source:dest:mode" format
    env:
      API_KEY: ""              # Use "" (empty string) to passthrough from the pipeline environment.
                               # Non-empty values are embedded as literal strings in the MCPG config —
                               # ADO variable syntax like $(MY_SECRET) is NOT resolved here.
    allowed:
      - do_thing
      - get_status

Custom HTTP MCP (remote endpoint — requires url: field):

mcp-servers:
  remote-service:
    url: "https://mcp.example.com"
    headers:
      X-MCP-Toolsets: "repos,wit"
    allowed:
      - query_data

Security: All mcp-servers: entries must have an explicit allowed: list.

Standalone target (the default): MCPs without a container: or url: field are skipped at compile time with a compile-time warning — they have no effect and will not be available to the agent. Both containerized MCPs (with container:) and remote HTTP MCPs (with url:) are supported in standalone target.

Step 9 — Safe Outputs

Safe outputs are the only write operations available to the agent. They are threat-analyzed before execution. Configure defaults in the front matter; the agent provides specifics at runtime.

create-pull-request — requires permissions.write:

safe-outputs:
  create-pull-request:
    target-branch: main
    draft: false             # PRs are drafts by default; set false to publish immediately (required for auto-complete)
    auto-complete: true
    delete-source-branch: true
    squash-merge: true
    title-prefix: "[Bot] "  # Optional — prepended to every PR title
    if-no-changes: warn      # "warn" (default), "error", or "ignore" when the patch is empty
    max-files: 100           # Reject patches touching more than this many files (default: 100)
    protected-files: blocked # "blocked" (default) prevents changes to pipeline/CI files; "allowed" permits all
    excluded-files:          # Glob patterns for files to exclude from the patch
      - "*.lock"
    allowed-labels: []       # Restrict which labels the agent can apply (empty = any)
    reviewers:
      - "lead@example.com"
    labels:
      - automated
    work-items:
      - 12345

create-work-item — requires permissions.write:

safe-outputs:
  create-work-item:
    work-item-type: Task
    assignee: "user@example.com"
    tags:
      - automated
      - agent-created
    artifact-link:
      enabled: true
      branch: main

cache-memory — persistent agent memory across runs (configured under tools:, not safe-outputs:):

tools:
  cache-memory:
    allowed-extensions:
      - .md
      - .json
      - .txt

All configurable safe output tools:

Tool Description permissions.write
Work Items
create-work-item Create ADO work items ✅
update-work-item Update fields on existing work items (each field requires opt-in) ✅
comment-on-work-item Add comments to work items (requires target scoping) ✅
link-work-items Link two work items (parent/child, related, etc.) ✅
upload-workitem-attachment Upload a workspace file to a work item ✅
Pull Requests
create-pull-request Create PRs from agent code changes ✅
add-pr-comment Add a comment thread to a PR ✅
reply-to-pr-comment Reply to an existing PR review thread ✅
resolve-pr-thread Resolve or update status of a PR thread ✅
submit-pr-review Submit a review vote on a PR ✅
update-pr Update PR metadata (reviewers, labels, auto-complete, vote) ✅
Builds & Branches
queue-build Queue an ADO pipeline build by definition ID ✅
create-branch Create a new branch from an existing ref ✅
create-git-tag Create a git tag on a repository ref ✅
add-build-tag Add a tag to an ADO build ✅
upload-build-attachment Attach a workspace file to a build (visible via REST/custom extension) ✅
upload-pipeline-artifact Publish a workspace file as a pipeline artifact (visible in Artifacts tab) ✅
Wiki
create-wiki-page Create a new ADO wiki page (requires wiki-name) ✅
update-wiki-page Update an existing ADO wiki page (requires wiki-name) ✅
Diagnostics
noop Report no action needed; also files an ADO work item (configurable, gracefully skipped without write perms) —
missing-data Report missing data/information —
missing-tool Report a missing tool or capability; also files an ADO work item (configurable, gracefully skipped without write perms) —
report-incomplete Report that a task could not be completed —

Example configuration for additional tools:

safe-outputs:
  comment-on-work-item:
    target: "TeamProject\\AreaPath"   # Required — scopes which work items can be commented on
    max: 3
  update-work-item:
    target: "*"                       # Required — "*" allows any work item, or set to a specific ID number
    status: true                      # Each updatable field requires explicit opt-in
    title: true
    max: 5
  add-pr-comment:
    max: 10
  queue-build:
    allowed-pipelines: [42, 99]       # Required — pipeline definition IDs that can be triggered
    max: 1
  # noop and missing-tool auto-file ADO work items (enabled by default, optional customisation):
  noop:
    work-item:
      enabled: true                   # Set to false to disable work-item filing
      title: "[ado-aw] Agent reported no operation"
      work-item-type: Task
      area-path: "MyProject\\MyTeam"  # Optional
  missing-tool:
    work-item:
      enabled: true                   # Set to false to disable work-item filing
      title: "[ado-aw] Agent encountered missing tool"
      work-item-type: Task
      area-path: "MyProject\\MyTeam"  # Optional

See docs/safe-outputs.md → "Available Safe Output Tools" for full configuration reference of every tool.

Diagnostic tools (noop, missing-data, missing-tool, report-incomplete) are always available and require no required configuration. noop and missing-tool automatically file ADO work items by default — this requires permissions.write to actually create work items, but gracefully skips (with a warning) if credentials are unavailable.

Validation: The compiler enforces that if write-requiring safe outputs are configured, permissions.write must be set.

Step 10 — Permissions

ADO access tokens are minted from ARM service connections. System.AccessToken is never used.

permissions:
  read: my-read-arm-connection    # Stage 1 agent — read-only ADO access
  write: my-write-arm-connection  # Stage 3 executor only — write access
Config Effect
read only Agent can query ADO; no safe-output writes
write only Agent has no ADO API access; safe-outputs can create PRs/work items
Both Agent can read; safe-outputs can write
Neither No ADO tokens anywhere

Step 11 — Triggers (optional)

PR Triggers (on.pr)

Trigger on pull request events. Use branches: and paths: for native ADO filtering; use filters: for runtime gate conditions evaluated in the Setup job.

on:
  pr:
    branches:
      include: [main]          # only PRs targeting main
      # exclude: [release/*]
    paths:
      include: [src/*]         # only PRs touching src/
    filters:                   # optional runtime filters (compiled to gate step with self-cancellation)
      title: "*[review]*"      # glob match on PR title
      author:
        include: ["alice@corp.com"]
        # exclude: ["bot@corp.com"]
      draft: false             # omit to match both draft and non-draft
      labels:
        any-of: ["run-agent"]  # PR must have at least one of these labels
        # all-of: [...]        # PR must have ALL of these labels
        # none-of: [...]       # PR must have NONE of these labels
      source-branch: "feature/*"   # glob on PR source branch
      target-branch: "main"        # glob on PR target branch
      commit-message: "*[skip-agent]*"  # cancel if latest commit message matches
      changed-files:
        include: ["src/**/*.rs"]
      min-changes: 1           # minimum number of changed files
      max-changes: 100         # maximum number of changed files
      time-window:
        start: "09:00"
        end: "17:00"
      build-reason:
        include: [PullRequest]
      expression: "eq(variables['Custom.Flag'], 'true')"  # raw ADO condition

When on.pr is set: the native ADO pr: trigger block is generated from branches: and paths:. Runtime filters: compile to a gate step in the Setup job that self-cancels the build when they do not match.

Pipeline Triggers (on.pipeline)

Trigger from another pipeline completing:

on:
  pipeline:
    name: "Build Pipeline"
    project: "OtherProject"   # optional if same project
    branches:
      - main
      - release/*
    filters:                   # optional runtime filters (compiled to gate step with self-cancellation)
      source-pipeline: "Build*"
      branch: "refs/heads/main"  # triggering branch (Build.SourceBranch)
      time-window:
        start: "09:00"
        end: "17:00"
      build-reason:
        include: [IndividualCI]
        exclude: [Schedule]
      expression: "eq(variables['Custom.Flag'], 'true')"  # raw ADO condition

When on.pipeline is set: trigger: none and pr: none are generated automatically. If filters: are configured under on.pipeline, a gate step is added to the Setup job that evaluates the filters and self-cancels the build when they do not match.

Step 12 — Inline Steps (optional)

Steps that run inside the Agent job:

steps:             # BEFORE agent runs (same job)
  - bash: echo "Fetching context..."
    displayName: "Prepare context"

post-steps:        # AFTER agent completes (same job)
  - bash: echo "Archiving outputs..."
    displayName: "Post-process"

Separate jobs:

setup:             # Separate job BEFORE Agent
  - bash: echo "Provisioning resources..."
    displayName: "Setup"

teardown:          # Separate job AFTER SafeOutputs
  - bash: echo "Cleanup..."
    displayName: "Teardown"

Step 13 — Runtimes (optional)

Configure language runtimes that are installed before the agent runs. Runtimes auto-extend the bash command allow-list and add ecosystem-specific domains to the network allowlist.

# Lean 4 theorem prover
runtimes:
  lean: true
  # lean:
  #   toolchain: "leanprover/lean4:v4.29.1"   # pin a specific version

# Python
runtimes:
  python: true
  # python:
  #   version: "3.12"
  #   feed-url: "https://pkgs.dev.azure.com/myorg/_packaging/myfeed/pypi/simple/"

# Node.js
runtimes:
  node: true
  # node:
  #   version: "22.x"
  #   feed-url: "https://pkgs.dev.azure.com/ORG/PROJECT/_packaging/FEED/npm/registry/"

# .NET
runtimes:
  dotnet: true
  # dotnet:
  #   version: "8.0.x"           # or "global.json" to use the repo's global.json
  #   feed-url: "https://pkgs.dev.azure.com/myorg/_packaging/myfeed/nuget/v3/index.json"
  #   config: "nuget.config"     # mutually exclusive with feed-url

Multiple runtimes can be combined:

runtimes:
  python:
    version: "3.12"
  node:
    version: "22.x"
  dotnet:
    version: "8.0.x"

Each enabled runtime auto-adds its ecosystem's bash commands (e.g., dotnet, python, node, npm, lean, lake) and network domains to the allowlist. See docs/runtimes.md for full configuration reference.

Step 14 — Network (standalone target only)

Additional allowed domains beyond the built-in allowlist:

network:
  allowed:
    - "*.mycompany.com"
    - "api.external-service.com"
  blocked:
    - "evil.example.com"

The built-in allowlist includes: Azure DevOps, GitHub, Microsoft identity, Azure services, Application Insights, and MCP-specific endpoints for each enabled server.

Step 15 — Parameters (optional)

ADO runtime parameters are surfaced in the pipeline queue UI when a user manually runs the pipeline. Use them to expose configuration knobs (e.g., target region, log verbosity, feature flags) without hardcoding values.

parameters:
  - name: targetRegion
    displayName: "Target region"
    type: string
    default: "us-east"
    values:
      - us-east
      - eu-west
      - ap-south
  - name: verbose
    displayName: "Verbose output"
    type: boolean
    default: false
Field Required Description
name Yes Parameter identifier (referenced as ${{ parameters.name }} in steps)
displayName No Human-readable label in the ADO queue UI
type No ADO parameter type: boolean, string, number, object
default No Default value when not specified at queue time
values No Allowed values for string/number parameters (shows a dropdown in the UI)

Auto-injected clearMemory parameter: When tools.cache-memory is configured, the compiler automatically injects a clearMemory: boolean parameter (default: false) at the start of the parameters list. It lets users clear the agent's persisted memory from the ADO UI without editing the source. Defining your own clearMemory parameter suppresses the auto-injected one.

Omit parameters: if no runtime configuration knobs are needed.

Agent Instruction Body

The markdown body (after the closing ---) is what the agent reads. Write it as clear, structured task instructions. Good practices:

  • Use headers to separate phases of work (e.g., ## Analysis, ## Action)
  • Be explicit about inputs the agent should look for (repositories, file paths, ADO queries)
  • Specify the expected output and which safe-output tool to use
  • Mention what constitutes "no action needed" (to trigger noop)
  • Keep it concise — the agent reads this at runtime on every execution
## Instructions

Review all open pull requests in this repository for the following issues:
...

### When Changes Are Needed

Use `create-pull-request` with:
- title: "fix: ..."
- description: explaining the change

### When No Action Is Needed

Use `noop` with a brief summary of what was reviewed.

Complete Example

---
name: "Dependency Updater"
description: "Checks for outdated npm dependencies and opens PRs to update them"
engine:
  id: copilot
  model: claude-sonnet-4.5
on:
  schedule: weekly on monday around 9:00
tools:
  azure-devops: true
permissions:
  read: my-read-arm-sc
  write: my-write-arm-sc
safe-outputs:
  create-pull-request:
    target-branch: main
    draft: false             # PRs are drafts by default; set false to publish immediately (required for auto-complete)
    auto-complete: true
    squash-merge: true
    reviewers:
      - "lead@example.com"
    labels:
      - dependencies
      - automated
---

## Dependency Update Agent

Scan this repository for outdated npm dependencies and open a pull request to update them.

### Analysis

1. Run `npm outdated --json` to identify packages with newer versions available.
2. For each outdated package, check whether the new version introduces any breaking changes by reviewing its changelog or release notes.
3. Focus on patch and minor updates first; flag major version bumps separately.

### Action

If any outdated dependencies are found:
- Update `package.json` and run `npm install` to regenerate `package-lock.json`.
- Create a pull request titled `chore: update npm dependencies` with a description listing each updated package, its old version, and its new version.

### No Action Needed

If all dependencies are already up to date, use `noop` with a brief message: "All npm dependencies are current."

Output Instructions

When generating the agent file:

  1. Produce exactly one .md file. Do not create separate documentation, architecture notes, or runbooks.
  2. Respect existing repository conventions for file placement. Look at where existing pipeline YAML files or agent markdown files are located in the repo. If no convention exists, ask the user where they'd like the file placed.
  3. Omit optional fields when they match defaults — no engine: for claude-opus-4.7, no workspace: for root, no target: for standalone.
  4. Always validate that write-requiring safe-outputs (create-pull-request, create-work-item) have permissions.write set.

Compilation

After creating the agent file, compile it into an Azure DevOps pipeline:

# Simple form — generates a `.lock.yml` pipeline alongside the `.md` source
ado-aw compile <path/to/agent.md>

# Or specify a custom output location
ado-aw compile <path/to/agent.md> -o <path/to/pipeline.lock.yml>

This generates a .lock.yml pipeline file. Both the source .md and generated .lock.yml must be committed together. The compiler also writes/updates a .gitattributes file at the repository root so compiled pipelines are marked linguist-generated=true merge=ours.

If the ado-aw CLI is not installed or not available on PATH, guide the user to download it from: https://github.com/githubnext/ado-aw/releases

After compilation, tell the user the next steps:

Next steps:
  1. Review and customize the agent instructions in <filename>.md
  2. Commit both the .md source, the generated .lock.yml pipeline, and any .gitattributes changes
  3. Register the .lock.yml as a pipeline in Azure DevOps

Common Patterns

Scheduled Analysis → Work Item

Agent reads data (Kusto, ADO) and files a work item if action is needed.

on:
  schedule: daily around 10:00
tools:
  azure-devops: true
permissions:
  read: my-read-sc
  write: my-write-sc
safe-outputs:
  create-work-item:
    work-item-type: Bug
    tags: [automated, agent-detected]

PR-Triggered Code Review

Triggered when a pull request is opened or updated; reviews and comments via ADO.

on:
  pr:
    branches:
      include: [main]
    filters:
      draft: false             # Skip draft PRs
tools:
  azure-devops: true
permissions:
  read: my-read-sc
  write: my-write-sc
safe-outputs:
  add-pr-comment:
    max: 5
  noop:
    work-item:
      enabled: false

Repository Maintenance with PRs

Agent makes code changes and proposes them via PR.

on:
  schedule: weekly on sunday
tools:
  azure-devops: true
permissions:
  read: my-read-sc
  write: my-write-sc
safe-outputs:
  create-pull-request:
    target-branch: main
    draft: false             # PRs are drafts by default; set false to publish immediately (required for auto-complete)
    auto-complete: true
    squash-merge: true

Multi-Repo Agent

Agent checks out and modifies a secondary repository.

repos:
  - my-org/shared-config
workspace: repo
permissions:
  read: my-read-sc
  write: my-write-sc
safe-outputs:
  create-pull-request:
    target-branch: main

Key Rules

  • Minimal permissions: Default to no permissions; add only what the task requires.
  • Explicit allow-lists: Restrict MCP tools to only what the agent needs.
  • No direct writes: All mutations go through safe outputs — the agent cannot push code or call write APIs directly.
  • Compile before committing: Always compile with ado-aw compile and commit both the .md source and generated .lock.yml together.
  • Check validation: The compiler will error if write safe-outputs are configured without permissions.write.