Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion LICENSE.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
## __The MIT License (MIT)__
Copyright © 2024 Jonathan Scholtes
Copyright © 2024–2026 Jonathan Scholtes

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

Expand Down
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

> Automated incident detection and remediation with Azure SRE Agent + GitHub Copilot

For background and architecture context, see the companion article:
[Beyond the Alert: Building Self-Healing Pipelines with Azure SRE Agent and GitHub Copilot](https://stochasticcoder.com/2026/04/29/beyond-the-alert-building-self-healing-pipelines-with-azure-sre-agent-and-github-copilot/)

![Self-Healing Application Pipeline](media/self-healing-pipeline.png)

---
Expand All @@ -20,7 +23,7 @@ This demo collapses that loop. [**Azure SRE Agent**](https://learn.microsoft.com
|------|------|-------------|
| 1 | Container Apps + Load Generator | Synthetic traffic hits a deliberately broken API |
| 2 | Application Insights / Log Analytics | Exception rate spike and failure metrics detected |
| 3 | Azure SRE Agent | Root-cause analysis; files an ADO WorkItem to track the incident |
| 3 | Azure SRE Agent | Root-cause analysis; files an Azure DevOps work item to track the incident |
| 4 | Azure SRE Agent | Creates a GitHub Issue with diagnostic context; assigns the Copilot coding agent |
| 5 | GitHub Copilot Coding Agent | Reads the GitHub Issue, opens a fix PR on a new branch |
| 6 | Human reviewer | Reviews the PR and merges — the human gate |
Expand All @@ -35,7 +38,7 @@ flowchart TD
ACA["Azure Container Apps\n+ Load Generator Job\n(5-min cron, synthetic traffic)"]
AI["Application Insights\nLog Analytics"]
SRE["Azure SRE Agent\n(investigates, deduplicates)"]
ADO["Azure DevOps\nWorkItem\n(tracks incident)"]
ADO["Azure DevOps\nWork Item\n(tracks incident)"]
GHI["GitHub Issue\n(assigned to Copilot)"]
COP["GitHub Copilot Agent\n(creates branch + fix PR)"]
HUM["Human Review + Merge\n— the human gate —"]
Expand Down Expand Up @@ -73,9 +76,9 @@ az login
.\deploy.ps1 -Subscription "<subscription-name-or-id>" -SetupGitHub
```

→ [Full deployment guide](docs/deployment.md) — infrastructure, GitHub OIDC, SRE Agent portal configuration, and branch protection setup.
→ [Full deployment guide](docs/deployment.md) — infrastructure, GitHub OIDC, SRE Agent portal configuration, branch protection setup, and troubleshooting tips.

→ [Walkthrough](docs/walkthrough.md) — injecting bugs, walkthrough, and reset instructions.
→ [Walkthrough](docs/walkthrough.md) — bug injection, live demo flow, and reset instructions.

---

Expand Down
80 changes: 62 additions & 18 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Step-by-step instructions for deploying the Azure SRE Agent closed-loop demo.
| [PowerShell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell) | >= 7 | |
| [GitHub CLI](https://cli.github.com/) | latest | Required for OIDC setup |
| Azure subscription | | Contributor + User Access Administrator |
| Azure DevOps organization | | Required for SRE Agent WorkItem filing |
| Azure DevOps organization | | Required for SRE Agent work item filing |
| GitHub Copilot Enterprise | | Required for coding agent PR creation |

> **Region constraint**: The SRE Agent resource (`Microsoft.App/agents`) is currently available only in `eastus2`, `swedencentral` and `australiaeast`. Set `location` in `terraform.tfvars` accordingly.
Expand Down Expand Up @@ -48,7 +48,7 @@ The script runs four phases automatically:
Subsequent runs skip the bootstrap step:

```powershell
.\deploy.ps1 -SkipBootstrap
.\deploy.ps1 -Subscription "<subscription-name-or-id>" -SkipBootstrap
```

---
Expand All @@ -58,7 +58,7 @@ Subsequent runs skip the bootstrap step:
Wire up OIDC federated credentials and GitHub repository secrets:

```powershell
.\deploy.ps1 -SetupGitHub
.\deploy.ps1 -Subscription "<subscription-name-or-id>" -SetupGitHub
# or standalone:
.\scripts\New-GitHubOidc.ps1
```
Expand All @@ -78,40 +78,70 @@ TF_STATE_STORAGE_ACCOUNT = <storage account name printed by deploy.ps1 phase 0>

---

## Step 4 — Connect SRE Agent to Azure DevOps (Azure portal)
## Step 4 — Configure incident platform and response plan (Azure portal)

In the SRE Agent resource, configure inbound incident routing from Azure Monitor:

1. Open the SRE Agent resource in the Azure portal
2. Go to **Builder → Incident Platform**
3. Select **Azure Monitor** as the incident platform and complete the connection
4. Go to **Builder → Incident response plans** and create (or customize) a plan for this demo
5. Set plan filters (for example severity/service/title) and set run mode to **Autonomous** for this demo
6. Save and enable the plan

Azure SRE leverages **Memories**, for resolution, tracking and routing intent (for example: create an Azure DevOps **Issue** and route code-fix work to **GitHub Copilot**). This helps keep behavior consistent across incidents.

> You can use the built-in SRE Agent chat to refine investigation and routing behavior **Memories**, then update the response plan configuration based on those results.

When incidents match the enabled response plan filters, the agent runs automatically from that incident context.

> References:
> - [Incident platforms](https://learn.microsoft.com/en-us/azure/sre-agent/incident-platforms)
> - [Automate incident response](https://learn.microsoft.com/en-us/azure/sre-agent/incident-response)
> - [Incident response plans](https://learn.microsoft.com/en-us/azure/sre-agent/incident-response-plans)
> - [Workflow automation](https://learn.microsoft.com/en-us/azure/sre-agent/workflow-automation)
> - [User memories](https://learn.microsoft.com/en-us/azure/sre-agent/memory#user-memories)

---

## Step 5 — Add Azure DevOps as an automated destination

After `terraform apply` completes:

1. Open the SRE Agent resource in the Azure portal
2. Go to **Workflows** → add a new workflow
3. Set the **trigger**: Log Analytics workspace connected to Application Insights — the agent queries exception rates and failure metrics
4. Set the **destination**: your ADO organization, project, and board; work item type **Issue**
5. Save — the agent will now create an ADO WorkItem automatically when an anomaly is detected, including the exception type, stack trace, and telemetry deep-links
2. Go to **Builder → Connectors** and create an **Azure DevOps connector** (org, project, board, auth)
3. Select the target Azure DevOps project/repository context for this demo
4. Save — when incidents match the enabled response plan, the agent automatically creates/updates Azure DevOps work items using this connector

---

## Step 5Connect SRE Agent to GitHub Issues (Azure portal)
## Step 6Add GitHub Issues as an automated destination

In the same or a parallel workflow in the SRE Agent portal:

1. Add a **GitHub** destination
2. Authenticate and select this repository
3. Set the assignee to **GitHub Copilot** (the coding agent)
4. Save — the agent will now open a GitHub Issue with diagnostic context and assign Copilot to resolve it
1. Go to **Builder → Connectors** and create a **GitHub connector** for this repository
2. Select this repository in the connector configuration
3. Save — for each matching incident, the agent automatically opens a GitHub Issue with diagnostic context and assigns GitHub Copilot to complete the fix



> Connector reference: [Azure SRE Agent connectors documentation](https://learn.microsoft.com/en-us/azure/sre-agent/connectors)

> Both the Azure DevOps work item and the GitHub Issue come from the same SRE Agent investigation. Azure DevOps tracks the incident operationally; the GitHub Issue is the work item Copilot acts on.


> Both the ADO WorkItem and the GitHub Issue come from the same SRE Agent investigation. ADO tracks the incident operationally; the GitHub Issue is the work item Copilot acts on.

---

## Step 6 — Enable branch protection on `main`
## Step 7 — Enable branch protection on `main`

This enforces the human gate — Copilot's fix PR cannot merge without a review.

1. Go to **Settings → Branches → Add branch ruleset**
2. Target: `main`
3. Enable:
- **Require a pull request before merging** (1 required approval)
- **Require status checks to pass** → add `Validate` (from `validate.yml`)
- **Require status checks to pass** → add the check run from `.github/workflows/validate.yml` (typically `Lint`)
- **Block force pushes**

---
Expand All @@ -127,8 +157,10 @@ Copy `infra/terraform.tfvars.example` to `infra/terraform.tfvars` and fill in yo
| `project_name` | Yes | Used in resource naming |
| `location` | Yes | Azure region — see region constraint above |
| `resource_token` | No | Unique suffix; auto-generated if empty |
| `enable_sre_agent` | No | Defaults to `true` |
| `sre_agent_access_level` | No | `High` (Reader + Contributor) or `Low` (Reader only) |
| `enable_sre_agent` | No | Deploy the SRE Agent resource (default: `false`) |
| `sre_agent_name` | No | Name of the SRE Agent resource (default: `sre-agent`) |
| `sre_agent_access_level` | No | `High` (Reader + Contributor) or `Low` (Reader only; default: `High`) |
| `sre_agent_target_resource_groups` | No | Additional resource groups for the agent to monitor |

---

Expand All @@ -139,3 +171,15 @@ Copy `infra/terraform.tfvars.example` to `infra/terraform.tfvars` and fill in yo
```

> The Terraform remote state storage account is in a separate resource group (`rg-tfstate-sre`) and is **not** deleted by the above command. Delete it manually when no longer needed.

---

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `deploy.ps1` fails before Terraform | Not authenticated to the correct subscription | Run `az login`, then rerun with `-Subscription` |
| GitHub OIDC setup fails on `gh secret set` | GitHub CLI is not authenticated | Run `gh auth login` and retry Step 3 |
| SRE Agent does not create incidents from alerts | Azure Monitor incident platform or response plan is not enabled | Recheck Step 4 and confirm the response plan is turned on |
| SRE Agent does not create work items/issues | Azure DevOps/GitHub connector project or repository mapping is not configured correctly | Recheck Steps 5 and 6 and verify connector scope |
| Branch rule blocks merge unexpectedly | Required check name mismatch | Use the check run generated by `.github/workflows/validate.yml` |
21 changes: 16 additions & 5 deletions docs/walkthrough.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

Instructions for injecting a failure, generating errors, and observing the self-healing pipeline end to end.

This guide is written as a companion reference for the article:
[Beyond the Alert: Building Self-Healing Pipelines with Azure SRE Agent and GitHub Copilot](https://stochasticcoder.com/2026/04/29/beyond-the-alert-building-self-healing-pipelines-with-azure-sre-agent-and-github-copilot/)

---

## Step 1 — Inject the bug and generate errors
Expand All @@ -10,7 +13,7 @@ Instructions for injecting a failure, generating errors, and observing the self-
.\tools\Start-SreDemo.ps1
```

This patches a realistic bug into the order management API, rebuilds and redeploys the container image via ACR, waits for the new revision to become healthy, then fires a burst of 20 orders to flood Application Insights with errors. Direct portal deep-links are printed for every presenter step.
This patches a realistic bug into the order-management API, rebuilds and redeploys the container image via ACR, waits for the new revision to become healthy, then fires a burst of 20 orders to flood Application Insights with errors. Direct portal deep-links are printed for each validation step.

```powershell
# Inject a specific bug type with a larger load burst
Expand All @@ -30,22 +33,30 @@ This patches a realistic bug into the order management API, rebuilds and redeplo

> **`-SkipDeploy`** patches the source file and writes `.chaos-state` but skips the ACR build. Commit and push to trigger the GitHub Actions workflow — useful for demonstrating the full CI/CD path.

Typical timeline after load starts:

- Application Insights failures appear in about 1-2 minutes
- SRE Agent investigation and ticketing usually follow shortly after
- Copilot PR timing depends on repository activity and queue depth

---

## Step 2 — Walkthrough
## Step 2 — Verify the closed-loop flow

Use this sequence to verify that each stage of the closed-loop flow is working correctly.

| Step | Where to look | What to show |
| Step | Where to look | Expected signal |
|------|---------------|-------------|
| 1 | Application Insights → **Failures** blade | Error rate spike, exception type, stack trace |
| 2 | [Azure SRE Agent](https://learn.microsoft.com/en-us/azure/sre-agent/overview?tabs=task) portal | Agent investigating the anomaly in real time |
| 3 | Azure DevOps Boards | ADO WorkItem auto-created with telemetry deep-link |
| 3 | Azure DevOps Boards | Azure DevOps work item auto-created with telemetry deep-link |
| 4 | GitHub Issues | Issue created by SRE Agent, assigned to GitHub Copilot |
| 5 | GitHub Pull Requests | Copilot's fix PR on a new branch |
| 6 | Merge the PR | CI/CD pipeline triggers; error rate returns to zero |

---

## Step 3 — Reset after the demo
## Step 3 — Reset the environment

```powershell
.\tools\Invoke-ChaosBug.ps1 -Revert `
Expand Down
4 changes: 3 additions & 1 deletion infra/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Copy-Item infra/terraform.tfvars.example infra/terraform.tfvars
| `environment_name` | Yes | e.g. `dev`, `prod` — used in resource naming |
| `project_name` | Yes | Short identifier used in resource naming |
| `resource_token` | No | Unique suffix (auto-generated if omitted) |
| `location` | Yes | Azure region — SRE Agent requires `eastus2`, `swedencentral`, `uksouth`, or `australiaeast` |
| `location` | Yes | Azure region — SRE Agent requires `eastus2`, `swedencentral`, or `australiaeast` |
| `enable_sre_agent` | No | Deploy the SRE Agent resource (default: `false`) |
| `sre_agent_name` | No | Name of the SRE Agent resource (default: `sre-agent`) |
| `sre_agent_access_level` | No | `High` or `Low` — controls RBAC granted to the agent (default: `High`) |
Expand All @@ -86,6 +86,8 @@ terraform apply tfplan

To enable the SRE Agent, add `-var="enable_sre_agent=true"` to the plan command, or set `enable_sre_agent = true` in `terraform.tfvars`.

When `enable_sre_agent = true`, use a supported region (`eastus2`, `swedencentral`, or `australiaeast`).

## Outputs

| Output | Description |
Expand Down
2 changes: 1 addition & 1 deletion infra/terraform.tfvars.example
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ environment_name = "dev" # Environment: dev,
project_name = "sre" # Project name (used in resource naming)
resource_token = "" # Unique suffix — leave empty to auto-generate
location = "eastus2" # Azure region
# SRE Agent requires: eastus2, swedencentral, uksouth, australiaeast
# SRE Agent requires: eastus2, swedencentral, australiaeast

# SRE Agent (set to true to deploy the Azure SRE Agent)
enable_sre_agent = true
Expand Down
2 changes: 1 addition & 1 deletion infra/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ variable "location" {
# ── SRE Agent ────────────────────────────────────────────────────────────────

variable "enable_sre_agent" {
description = "Deploy the Azure SRE Agent. Requires a location supported by the preview: eastus2, swedencentral, uksouth, australiaeast."
description = "Deploy the Azure SRE Agent. Requires a location supported by the preview: eastus2, swedencentral, australiaeast."
type = bool
default = false
}
Expand Down
Loading