diff --git a/LICENSE.md b/LICENSE.md index 9a6779b..2b5bd86 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -1,5 +1,5 @@ ## __The MIT License (MIT)__ -Copyright © 2024 Jonathan Scholtes +Copyright © 2024–2026 Jonathan Scholtes Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: diff --git a/README.md b/README.md index c507868..ca1ef05 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,9 @@ > Automated incident detection and remediation with Azure SRE Agent + GitHub Copilot +For background and architecture context, see the companion article: +[Beyond the Alert: Building Self-Healing Pipelines with Azure SRE Agent and GitHub Copilot](https://stochasticcoder.com/2026/04/29/beyond-the-alert-building-self-healing-pipelines-with-azure-sre-agent-and-github-copilot/) + ![Self-Healing Application Pipeline](media/self-healing-pipeline.png) --- @@ -20,7 +23,7 @@ This demo collapses that loop. [**Azure SRE Agent**](https://learn.microsoft.com |------|------|-------------| | 1 | Container Apps + Load Generator | Synthetic traffic hits a deliberately broken API | | 2 | Application Insights / Log Analytics | Exception rate spike and failure metrics detected | -| 3 | Azure SRE Agent | Root-cause analysis; files an ADO WorkItem to track the incident | +| 3 | Azure SRE Agent | Root-cause analysis; files an Azure DevOps work item to track the incident | | 4 | Azure SRE Agent | Creates a GitHub Issue with diagnostic context; assigns the Copilot coding agent | | 5 | GitHub Copilot Coding Agent | Reads the GitHub Issue, opens a fix PR on a new branch | | 6 | Human reviewer | Reviews the PR and merges — the human gate | @@ -35,7 +38,7 @@ flowchart TD ACA["Azure Container Apps\n+ Load Generator Job\n(5-min cron, synthetic traffic)"] AI["Application Insights\nLog Analytics"] SRE["Azure SRE Agent\n(investigates, deduplicates)"] - ADO["Azure DevOps\nWorkItem\n(tracks incident)"] + ADO["Azure DevOps\nWork Item\n(tracks incident)"] GHI["GitHub Issue\n(assigned to Copilot)"] COP["GitHub Copilot Agent\n(creates branch + fix PR)"] HUM["Human Review + Merge\n— the human gate —"] @@ -73,9 +76,9 @@ az login .\deploy.ps1 -Subscription "" -SetupGitHub ``` -→ [Full deployment guide](docs/deployment.md) — infrastructure, GitHub OIDC, SRE Agent portal configuration, and branch protection setup. +→ [Full deployment guide](docs/deployment.md) — infrastructure, GitHub OIDC, SRE Agent portal configuration, branch protection setup, and troubleshooting tips. -→ [Walkthrough](docs/walkthrough.md) — injecting bugs, walkthrough, and reset instructions. +→ [Walkthrough](docs/walkthrough.md) — bug injection, live demo flow, and reset instructions. --- diff --git a/docs/deployment.md b/docs/deployment.md index 9190ef4..dd23565 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -13,7 +13,7 @@ Step-by-step instructions for deploying the Azure SRE Agent closed-loop demo. | [PowerShell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell) | >= 7 | | | [GitHub CLI](https://cli.github.com/) | latest | Required for OIDC setup | | Azure subscription | | Contributor + User Access Administrator | -| Azure DevOps organization | | Required for SRE Agent WorkItem filing | +| Azure DevOps organization | | Required for SRE Agent work item filing | | GitHub Copilot Enterprise | | Required for coding agent PR creation | > **Region constraint**: The SRE Agent resource (`Microsoft.App/agents`) is currently available only in `eastus2`, `swedencentral` and `australiaeast`. Set `location` in `terraform.tfvars` accordingly. @@ -48,7 +48,7 @@ The script runs four phases automatically: Subsequent runs skip the bootstrap step: ```powershell -.\deploy.ps1 -SkipBootstrap +.\deploy.ps1 -Subscription "" -SkipBootstrap ``` --- @@ -58,7 +58,7 @@ Subsequent runs skip the bootstrap step: Wire up OIDC federated credentials and GitHub repository secrets: ```powershell -.\deploy.ps1 -SetupGitHub +.\deploy.ps1 -Subscription "" -SetupGitHub # or standalone: .\scripts\New-GitHubOidc.ps1 ``` @@ -78,32 +78,62 @@ TF_STATE_STORAGE_ACCOUNT = --- -## Step 4 — Connect SRE Agent to Azure DevOps (Azure portal) +## Step 4 — Configure incident platform and response plan (Azure portal) + +In the SRE Agent resource, configure inbound incident routing from Azure Monitor: + +1. Open the SRE Agent resource in the Azure portal +2. Go to **Builder → Incident Platform** +3. Select **Azure Monitor** as the incident platform and complete the connection +4. Go to **Builder → Incident response plans** and create (or customize) a plan for this demo +5. Set plan filters (for example severity/service/title) and set run mode to **Autonomous** for this demo +6. Save and enable the plan + +Azure SRE leverages **Memories**, for resolution, tracking and routing intent (for example: create an Azure DevOps **Issue** and route code-fix work to **GitHub Copilot**). This helps keep behavior consistent across incidents. + +> You can use the built-in SRE Agent chat to refine investigation and routing behavior **Memories**, then update the response plan configuration based on those results. + +When incidents match the enabled response plan filters, the agent runs automatically from that incident context. + +> References: +> - [Incident platforms](https://learn.microsoft.com/en-us/azure/sre-agent/incident-platforms) +> - [Automate incident response](https://learn.microsoft.com/en-us/azure/sre-agent/incident-response) +> - [Incident response plans](https://learn.microsoft.com/en-us/azure/sre-agent/incident-response-plans) +> - [Workflow automation](https://learn.microsoft.com/en-us/azure/sre-agent/workflow-automation) +> - [User memories](https://learn.microsoft.com/en-us/azure/sre-agent/memory#user-memories) + +--- + +## Step 5 — Add Azure DevOps as an automated destination After `terraform apply` completes: 1. Open the SRE Agent resource in the Azure portal -2. Go to **Workflows** → add a new workflow -3. Set the **trigger**: Log Analytics workspace connected to Application Insights — the agent queries exception rates and failure metrics -4. Set the **destination**: your ADO organization, project, and board; work item type **Issue** -5. Save — the agent will now create an ADO WorkItem automatically when an anomaly is detected, including the exception type, stack trace, and telemetry deep-links +2. Go to **Builder → Connectors** and create an **Azure DevOps connector** (org, project, board, auth) +3. Select the target Azure DevOps project/repository context for this demo +4. Save — when incidents match the enabled response plan, the agent automatically creates/updates Azure DevOps work items using this connector --- -## Step 5 — Connect SRE Agent to GitHub Issues (Azure portal) +## Step 6 — Add GitHub Issues as an automated destination In the same or a parallel workflow in the SRE Agent portal: -1. Add a **GitHub** destination -2. Authenticate and select this repository -3. Set the assignee to **GitHub Copilot** (the coding agent) -4. Save — the agent will now open a GitHub Issue with diagnostic context and assign Copilot to resolve it +1. Go to **Builder → Connectors** and create a **GitHub connector** for this repository +2. Select this repository in the connector configuration +3. Save — for each matching incident, the agent automatically opens a GitHub Issue with diagnostic context and assigns GitHub Copilot to complete the fix + + + +> Connector reference: [Azure SRE Agent connectors documentation](https://learn.microsoft.com/en-us/azure/sre-agent/connectors) + +> Both the Azure DevOps work item and the GitHub Issue come from the same SRE Agent investigation. Azure DevOps tracks the incident operationally; the GitHub Issue is the work item Copilot acts on. + -> Both the ADO WorkItem and the GitHub Issue come from the same SRE Agent investigation. ADO tracks the incident operationally; the GitHub Issue is the work item Copilot acts on. --- -## Step 6 — Enable branch protection on `main` +## Step 7 — Enable branch protection on `main` This enforces the human gate — Copilot's fix PR cannot merge without a review. @@ -111,7 +141,7 @@ This enforces the human gate — Copilot's fix PR cannot merge without a review. 2. Target: `main` 3. Enable: - **Require a pull request before merging** (1 required approval) - - **Require status checks to pass** → add `Validate` (from `validate.yml`) + - **Require status checks to pass** → add the check run from `.github/workflows/validate.yml` (typically `Lint`) - **Block force pushes** --- @@ -127,8 +157,10 @@ Copy `infra/terraform.tfvars.example` to `infra/terraform.tfvars` and fill in yo | `project_name` | Yes | Used in resource naming | | `location` | Yes | Azure region — see region constraint above | | `resource_token` | No | Unique suffix; auto-generated if empty | -| `enable_sre_agent` | No | Defaults to `true` | -| `sre_agent_access_level` | No | `High` (Reader + Contributor) or `Low` (Reader only) | +| `enable_sre_agent` | No | Deploy the SRE Agent resource (default: `false`) | +| `sre_agent_name` | No | Name of the SRE Agent resource (default: `sre-agent`) | +| `sre_agent_access_level` | No | `High` (Reader + Contributor) or `Low` (Reader only; default: `High`) | +| `sre_agent_target_resource_groups` | No | Additional resource groups for the agent to monitor | --- @@ -139,3 +171,15 @@ Copy `infra/terraform.tfvars.example` to `infra/terraform.tfvars` and fill in yo ``` > The Terraform remote state storage account is in a separate resource group (`rg-tfstate-sre`) and is **not** deleted by the above command. Delete it manually when no longer needed. + +--- + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---------|--------------|-----| +| `deploy.ps1` fails before Terraform | Not authenticated to the correct subscription | Run `az login`, then rerun with `-Subscription` | +| GitHub OIDC setup fails on `gh secret set` | GitHub CLI is not authenticated | Run `gh auth login` and retry Step 3 | +| SRE Agent does not create incidents from alerts | Azure Monitor incident platform or response plan is not enabled | Recheck Step 4 and confirm the response plan is turned on | +| SRE Agent does not create work items/issues | Azure DevOps/GitHub connector project or repository mapping is not configured correctly | Recheck Steps 5 and 6 and verify connector scope | +| Branch rule blocks merge unexpectedly | Required check name mismatch | Use the check run generated by `.github/workflows/validate.yml` | diff --git a/docs/walkthrough.md b/docs/walkthrough.md index e27c6fe..1af47d2 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -2,6 +2,9 @@ Instructions for injecting a failure, generating errors, and observing the self-healing pipeline end to end. +This guide is written as a companion reference for the article: +[Beyond the Alert: Building Self-Healing Pipelines with Azure SRE Agent and GitHub Copilot](https://stochasticcoder.com/2026/04/29/beyond-the-alert-building-self-healing-pipelines-with-azure-sre-agent-and-github-copilot/) + --- ## Step 1 — Inject the bug and generate errors @@ -10,7 +13,7 @@ Instructions for injecting a failure, generating errors, and observing the self- .\tools\Start-SreDemo.ps1 ``` -This patches a realistic bug into the order management API, rebuilds and redeploys the container image via ACR, waits for the new revision to become healthy, then fires a burst of 20 orders to flood Application Insights with errors. Direct portal deep-links are printed for every presenter step. +This patches a realistic bug into the order-management API, rebuilds and redeploys the container image via ACR, waits for the new revision to become healthy, then fires a burst of 20 orders to flood Application Insights with errors. Direct portal deep-links are printed for each validation step. ```powershell # Inject a specific bug type with a larger load burst @@ -30,22 +33,30 @@ This patches a realistic bug into the order management API, rebuilds and redeplo > **`-SkipDeploy`** patches the source file and writes `.chaos-state` but skips the ACR build. Commit and push to trigger the GitHub Actions workflow — useful for demonstrating the full CI/CD path. +Typical timeline after load starts: + +- Application Insights failures appear in about 1-2 minutes +- SRE Agent investigation and ticketing usually follow shortly after +- Copilot PR timing depends on repository activity and queue depth + --- -## Step 2 — Walkthrough +## Step 2 — Verify the closed-loop flow + +Use this sequence to verify that each stage of the closed-loop flow is working correctly. -| Step | Where to look | What to show | +| Step | Where to look | Expected signal | |------|---------------|-------------| | 1 | Application Insights → **Failures** blade | Error rate spike, exception type, stack trace | | 2 | [Azure SRE Agent](https://learn.microsoft.com/en-us/azure/sre-agent/overview?tabs=task) portal | Agent investigating the anomaly in real time | -| 3 | Azure DevOps Boards | ADO WorkItem auto-created with telemetry deep-link | +| 3 | Azure DevOps Boards | Azure DevOps work item auto-created with telemetry deep-link | | 4 | GitHub Issues | Issue created by SRE Agent, assigned to GitHub Copilot | | 5 | GitHub Pull Requests | Copilot's fix PR on a new branch | | 6 | Merge the PR | CI/CD pipeline triggers; error rate returns to zero | --- -## Step 3 — Reset after the demo +## Step 3 — Reset the environment ```powershell .\tools\Invoke-ChaosBug.ps1 -Revert ` diff --git a/infra/README.md b/infra/README.md index 62e75a5..c6a0f5e 100644 --- a/infra/README.md +++ b/infra/README.md @@ -59,7 +59,7 @@ Copy-Item infra/terraform.tfvars.example infra/terraform.tfvars | `environment_name` | Yes | e.g. `dev`, `prod` — used in resource naming | | `project_name` | Yes | Short identifier used in resource naming | | `resource_token` | No | Unique suffix (auto-generated if omitted) | -| `location` | Yes | Azure region — SRE Agent requires `eastus2`, `swedencentral`, `uksouth`, or `australiaeast` | +| `location` | Yes | Azure region — SRE Agent requires `eastus2`, `swedencentral`, or `australiaeast` | | `enable_sre_agent` | No | Deploy the SRE Agent resource (default: `false`) | | `sre_agent_name` | No | Name of the SRE Agent resource (default: `sre-agent`) | | `sre_agent_access_level` | No | `High` or `Low` — controls RBAC granted to the agent (default: `High`) | @@ -86,6 +86,8 @@ terraform apply tfplan To enable the SRE Agent, add `-var="enable_sre_agent=true"` to the plan command, or set `enable_sre_agent = true` in `terraform.tfvars`. +When `enable_sre_agent = true`, use a supported region (`eastus2`, `swedencentral`, or `australiaeast`). + ## Outputs | Output | Description | diff --git a/infra/terraform.tfvars.example b/infra/terraform.tfvars.example index 955131d..07b346b 100644 --- a/infra/terraform.tfvars.example +++ b/infra/terraform.tfvars.example @@ -7,7 +7,7 @@ environment_name = "dev" # Environment: dev, project_name = "sre" # Project name (used in resource naming) resource_token = "" # Unique suffix — leave empty to auto-generate location = "eastus2" # Azure region - # SRE Agent requires: eastus2, swedencentral, uksouth, australiaeast + # SRE Agent requires: eastus2, swedencentral, australiaeast # SRE Agent (set to true to deploy the Azure SRE Agent) enable_sre_agent = true diff --git a/infra/variables.tf b/infra/variables.tf index ef8c9c1..368f283 100644 --- a/infra/variables.tf +++ b/infra/variables.tf @@ -39,7 +39,7 @@ variable "location" { # ── SRE Agent ──────────────────────────────────────────────────────────────── variable "enable_sre_agent" { - description = "Deploy the Azure SRE Agent. Requires a location supported by the preview: eastus2, swedencentral, uksouth, australiaeast." + description = "Deploy the Azure SRE Agent. Requires a location supported by the preview: eastus2, swedencentral, australiaeast." type = bool default = false }