Skip to content

Commit 427f873

Browse files
github-actions[bot]github-actions[bot]
authored
[workshop] Add node: Audit and Monitor Your Agentic Workflows (#1371)
1 parent 4f4b7be commit 427f873

3 files changed

Lines changed: 173 additions & 0 deletions

File tree

‎workshop/14-next-steps.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,7 @@ You've reached the end of the main path — but the graph stays open. Come back
126126
- ➡️ [Make Your Workflows Resilient to Failure](22-error-handling-and-resilience.md) — add defensive briefs, timeouts, and fallback outputs so unattended runs stay reliable.
127127
- ➡️ [Test Your Prompt Ideas with A/B Experiments](23-ab-experiments.md) — compare prompt variants across runs and let data decide which one to keep.
128128
- ➡️ [Run Your Agentic Workflow on a Self-Hosted Runner](24-self-hosted-runners.md) — target your organisation's runner fleet instead of GitHub-hosted machines (enterprise teams).
129+
- ➡️ [Audit and Monitor Your Agentic Workflows](25-audit-and-observability.md) — read run artifacts, understand token usage, and build an audit trail for enterprise compliance.
129130

130131
## 📚 See Also
131132

‎workshop/25-audit-and-observability.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Audit and Monitor Your Agentic Workflows
2+
3+
> _Knowing what your agent did — and proving it — is what turns a useful automation into a trustworthy one._
4+
5+
## 🎯 What You'll Do
6+
7+
You will use the `gh aw logs` and `gh aw audit` commands to read the built-in artifacts that every agentic workflow run produces, understand token usage, and identify the audit trail your organisation needs for enterprise compliance. You will also learn to bring audit output to an agent using the `/agentic-workflows` Copilot skill to debug unexpected behavior and iterate on your workflow. By the end you will know exactly where to look when a run behaves unexpectedly or when a compliance review asks what the agent did.
8+
9+
## 📋 Before You Start
10+
11+
- Your workflow runs successfully (see [Test and Iterate on Your Workflow](12-test-and-iterate.md)).
12+
- `gh aw` is installed and authenticated (see [Install the gh-aw CLI Extension](06-install-gh-aw.md)).
13+
14+
## Steps
15+
16+
### Review recent runs with gh aw logs
17+
18+
`gh aw logs` downloads artifacts from your workflow's recent runs and prints a summary table showing duration, token usage, and cost in AI Credits (AIC).
19+
20+
Run it from inside your repository:
21+
22+
```bash
23+
gh aw logs <your-workflow-id>
24+
```
25+
26+
Replace `<your-workflow-id>` with the basename of your workflow file (for example, `daily-status` for `daily-status.md`).
27+
28+
The summary table shows one row per run. Key columns:
29+
30+
| Column | What it tells you |
31+
|---|---|
32+
| AIC | Total AI Credits consumed by the agent |
33+
| ⌖ AIC | Credits used by threat detection |
34+
| Model | The AI model that ran the agent |
35+
| Conclusion | Whether the run succeeded |
36+
37+
To download all artifacts — including the agent conversation, firewall log, and MCP tool calls — add `--artifacts all`:
38+
39+
```bash
40+
gh aw logs <your-workflow-id> --artifacts all
41+
```
42+
43+
Downloaded files land in `.github/aw/logs/<run-id>/` by default.
44+
45+
### Audit a specific run with gh aw audit
46+
47+
When you need a deeper look at one run — for debugging or compliance evidence — use `gh aw audit` with the run ID or URL from the Actions tab:
48+
49+
```bash
50+
gh aw audit <run-id>
51+
```
52+
53+
This downloads all artifacts for that run and generates a concise Markdown report covering:
54+
55+
- Run metadata: workflow, trigger, engine, model
56+
- Agent AIC and threat-detection AIC
57+
- MCP tool calls and any errors
58+
- Threat detection verdict (prompt injection, secret leak, malicious patch)
59+
- Safe outputs declared by the agent
60+
61+
To also parse the raw agent and firewall logs into readable Markdown:
62+
63+
```bash
64+
gh aw audit <run-id> --parse
65+
```
66+
67+
This writes `log.md` (agent conversation) and `firewall.md` (network access summary) to the output directory.
68+
69+
> [!TIP]
70+
> Pass the run URL directly from your browser's address bar — `gh aw audit` accepts both numeric run IDs and full GitHub Actions URLs.
71+
72+
### Debug with an agent and the /agentic-workflows skill
73+
74+
Once you have an audit report, the fastest way to interpret it and improve your workflow is to bring it to an agent. Open GitHub Copilot Chat, then paste the contents of the audit report and describe what puzzled you:
75+
76+
```text
77+
Here is the audit report for my last run. The agent called github.list_issues
78+
three times in a row and the total AIC was higher than I expected.
79+
Help me understand why and suggest how to reduce it.
80+
81+
<paste report here>
82+
```
83+
84+
For more targeted guidance, use the **`/agentic-workflows` skill** in the chat:
85+
86+
```text
87+
/agentic-workflows Here is my audit log. The firewall blocked calls to
88+
api.example.com. Can you help me add the correct network.allow entry?
89+
90+
<paste firewall audit here>
91+
```
92+
93+
The `/agentic-workflows` skill understands agentic workflow frontmatter and
94+
safe-output rules. It can validate your changes, suggest a more efficient prompt,
95+
or walk you through adding a `network.allow` entry — all without you leaving the chat.
96+
97+
> [!TIP]
98+
> For follow-up edits to the workflow file, ask the agent to make the changes directly rather than hand-editing every line. This way the agent runs `gh aw compile` to validate the result before committing.
99+
100+
### Read the agent artifact
101+
102+
The `agent` artifact — downloaded by both `gh aw logs --artifacts all` and `gh aw audit` — contains the full record of what the agent did. After downloading, look for:
103+
104+
| File (inside the agent artifact) | What it tells you |
105+
|---|---|
106+
| `safeoutputs.jsonl` | Every safe-output declaration the agent emitted |
107+
| `mcp-logs/` | One log file per MCP server, listing every tool call and result |
108+
| `sandbox/firewall/audit/` | Domain-level network access log |
109+
| `agent_usage.json` | Token usage for the agent turn |
110+
111+
### Understand the usage artifact
112+
113+
The `usage` artifact contains token consumption broken down by job. It is also summarised in the `gh aw logs` overview table under the **AIC** column.
114+
115+
Use this data to spot unexpectedly long runs, track cost trends over time, and set a budget baseline before enabling a scheduled trigger.
116+
117+
> [!NOTE]
118+
> AIC (AI Credits) is the billing unit for agentic workflow inference and is derived from token consumption. Exact billing figures appear in your GitHub billing dashboard.
119+
120+
### Check the firewall log
121+
122+
The firewall artifact records every outbound domain the agent attempted to reach and whether it was allowed or blocked.
123+
124+
After `gh aw audit <run-id> --parse`, open `firewall.md` for a formatted summary. To scan raw data, look inside `sandbox/firewall/audit/` in the downloaded agent artifact.
125+
126+
If a domain your workflow needs is blocked, add it to `network.allow` in your workflow frontmatter:
127+
128+
```yaml
129+
network:
130+
allow:
131+
- api.example.com
132+
```
133+
134+
> [!TIP]
135+
> Share the allowed-domains list from a successful run with your enterprise security team as a ready-made firewall allowlist.
136+
137+
### Browse artifacts in the GitHub UI
138+
139+
If you prefer not to use the CLI, every artifact is also available in the browser:
140+
141+
1. Go to the **Actions** tab in your repository.
142+
2. Click a completed workflow run.
143+
3. Scroll to the **Artifacts** section at the bottom of the run summary page and download the archive you need.
144+
145+
### Retention policy
146+
147+
GitHub retains artifacts for **90 days** by default. For enterprise compliance, ask your GitHub administrator whether a global retention policy overrides this, or whether you need to copy artifacts to external storage for longer-term audit requirements.
148+
149+
> [!NOTE]
150+
> On GitHub Enterprise Server, retention defaults may differ from github.com.
151+
152+
## ✅ Checkpoint
153+
154+
- [ ] You ran `gh aw logs <your-workflow-id>` and read the AIC summary for your workflow
155+
- [ ] You ran `gh aw audit <run-id>` and reviewed the generated report
156+
- [ ] You pasted an audit report into Copilot Chat with the `/agentic-workflows` skill and got actionable feedback
157+
- [ ] You located the `safeoutputs.jsonl` and `mcp-logs/` files in the `agent` artifact
158+
- [ ] You checked the firewall log for blocked domains
159+
- [ ] You can describe what a compliance reviewer would find in the `agent` artifact
160+
- [ ] You know your organisation's artifact retention policy (or know who to ask)
161+
162+
**Next:** [What's Next? Keep Exploring](14-next-steps.md)
163+
164+
## 📚 See Also
165+
166+
- [Overview of GitHub Agentic Workflows](https://github.github.com/gh-aw/introduction/overview/)
167+
- [Network reference](https://github.github.com/gh-aw/reference/network/)
168+
- [Safe Outputs reference](https://github.github.com/gh-aw/reference/safe-outputs/)
169+
- [Side Quest: Enterprise Setup Considerations](side-quest-enterprise-setup.md)
170+
- [Run Your Agentic Workflow on a Self-Hosted Runner](24-self-hosted-runners.md)
171+
- [Make Your Workflows Resilient to Failure](22-error-handling-and-resilience.md)

‎workshop/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ A hands-on workshop that takes you from zero to a fully automated, AI-powered wo
5656
| 22 | [22-error-handling-and-resilience.md](22-error-handling-and-resilience.md) | Make Your Workflows Resilient to Failure | ✅ |
5757
| 23 | [23-ab-experiments.md](23-ab-experiments.md) | Test Your Prompt Ideas with A/B Experiments | ✅ |
5858
| 24 | [24-self-hosted-runners.md](24-self-hosted-runners.md) | Run Your Agentic Workflow on a Self-Hosted Runner | ✅ |
59+
| 25 | [25-audit-and-observability.md](25-audit-and-observability.md) | Audit and Monitor Your Agentic Workflows | ✅ |
5960

6061
## Optional Side Quests
6162

0 commit comments

Comments
 (0)