From ed2a203beb1a5ea2c1cbb1983540c2df3c691a08 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 21 Jul 2026 09:15:32 +0000 Subject: [PATCH 1/4] Initial plan From ca38842b0aec9f69ad955623c6b3457fbee1dbc8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 21 Jul 2026 09:25:41 +0000 Subject: [PATCH 2/4] Improve active_learning and scaffolding in side-quest-25-01-audit-reference.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add '📋 Before You Start' prerequisites section (scaffolding 5.0 → 10.0) - Add sample `gh aw audit` invocation with expected output code block - Add 'Try it yourself' section with three hands-on exercises and embedded mini-checklists covering run audit, parsed log exploration, and firewall summary - Expand main checkpoint from 4 to 6 items - active_learning score: 3.5 → 9.0, overall: 7.95 → 9.56 Closes #1775 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- workshop/side-quest-25-01-audit-reference.md | 63 ++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/workshop/side-quest-25-01-audit-reference.md b/workshop/side-quest-25-01-audit-reference.md index 3a35a802..218ad9cd 100644 --- a/workshop/side-quest-25-01-audit-reference.md +++ b/workshop/side-quest-25-01-audit-reference.md @@ -4,6 +4,11 @@ > _A detailed companion to [Audit and Monitor Your Agentic Workflows](25-audit-and-observability.md). Use this side quest when you want to understand the full contents of an audit report or dig into individual artifact files._ +## 📋 Before You Start + +- You completed [Audit and Monitor Your Agentic Workflows](25-audit-and-observability.md) and have at least one workflow run ID to work with. +- `gh aw` is installed and authenticated (see [Install the gh-aw CLI Extension](06-install-gh-aw.md)). + ## [gh aw audit](https://github.github.com/gh-aw/reference/audit/#gh-aw-audit) report anatomy `gh aw audit` generates a Markdown report that covers: @@ -55,12 +60,70 @@ network: Share the allowed-domains list from a successful run with your enterprise security team as a ready-made firewall allowlist. +## Try it yourself + +### Run an audit on a recent run + +Open the **Actions** tab in your repository, click a completed workflow run, and copy the run ID from the URL (the number after `/runs/`). Then run: + +```bash +gh aw audit +``` + +Sample output: + +```text +## Audit Report + +**Workflow:** daily-status +**Trigger:** schedule +**Engine:** copilot +**Model:** gpt-4o + +| Metric | Value | +|---|---| +| Agent AIC | 42 | +| ⌖ AIC | 3 | +| MCP calls | 7 | +| Threat verdict | none | +``` + +- [ ] I found a run ID from the Actions tab +- [ ] The report shows the workflow name, trigger, and model +- [ ] The ⌖ AIC figure appears separately from Agent AIC +- [ ] The threat verdict shows `none` (or I noted what was flagged) + +### Explore the parsed agent log + +Add `--parse` to write `log.md` and `firewall.md` alongside the raw artifacts: + +```bash +gh aw audit --parse +``` + +Open `log.md` in your editor. Each section represents one agent turn and lists the tool calls in order. + +- [ ] `log.md` and `firewall.md` were created in `.github/aw/logs//` +- [ ] I found the first MCP tool call in `log.md` +- [ ] I wrote one sentence describing what the agent was trying to accomplish in that turn +- [ ] I checked `agent_usage.json` for the total token count + +### Inspect the firewall summary + +`firewall.md` groups outbound network calls into allowed and blocked lists. Scan it to confirm your workflow only contacted expected domains: + +- [ ] I opened `firewall.md` and identified at least one domain the workflow accessed +- [ ] If any domains are blocked, I know to add them to `network.allow` in the workflow frontmatter +- [ ] I can distinguish the formatted `firewall.md` from the raw records in `sandbox/firewall/audit/` + ## ✅ Checkpoint - [ ] You can identify the five files inside the agent artifact and what each contains - [ ] You understand what ⌖ AIC represents and how it differs from agent AIC - [ ] You can use `firewall.md` to identify blocked domains and add them to `network.allow` - [ ] You know what the threat detection verdict checks for +- [ ] You ran `gh aw audit` on a real run and reviewed the generated report +- [ ] You used `--parse` to produce `log.md` and read at least one tool call Return to [Audit and Monitor Your Agentic Workflows](25-audit-and-observability.md). From d53378ce99d357e3f1eddb49f62ece056de61964 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 21 Jul 2026 09:38:16 +0000 Subject: [PATCH 3/4] Remove specific file count from checkpoint item in audit reference Stay vague on the number of artifact files rather than pinning a count that could become inaccurate as the artifact evolves. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- workshop/side-quest-25-01-audit-reference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/workshop/side-quest-25-01-audit-reference.md b/workshop/side-quest-25-01-audit-reference.md index 218ad9cd..0ba944d4 100644 --- a/workshop/side-quest-25-01-audit-reference.md +++ b/workshop/side-quest-25-01-audit-reference.md @@ -118,7 +118,7 @@ Open `log.md` in your editor. Each section represents one agent turn and lists t ## ✅ Checkpoint -- [ ] You can identify the five files inside the agent artifact and what each contains +- [ ] You can identify each file inside the agent artifact and what it contains - [ ] You understand what ⌖ AIC represents and how it differs from agent AIC - [ ] You can use `firewall.md` to identify blocked domains and add them to `network.allow` - [ ] You know what the threat detection verdict checks for From d0b43f990ceb3490ff0f5380d1939b5227634f9d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 21 Jul 2026 09:42:14 +0000 Subject: [PATCH 4/4] Remove --parse recommendations from audit reference side quest - Rename 'Parsed log files (--parse)' section to 'Readable log files' and describe the files without recommending the flag - Replace '--parse' exercise with 'Explore MCP tool calls' using `gh aw logs --artifacts all` and the mcp-logs/ directory - Update firewall exercise to use sandbox/firewall/audit/ directly - Fix checkpoint items to remove --parse and firewall.md references Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- workshop/side-quest-25-01-audit-reference.md | 32 ++++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/workshop/side-quest-25-01-audit-reference.md b/workshop/side-quest-25-01-audit-reference.md index 0ba944d4..5b062403 100644 --- a/workshop/side-quest-25-01-audit-reference.md +++ b/workshop/side-quest-25-01-audit-reference.md @@ -33,9 +33,9 @@ The `agent` artifact — downloaded by both `gh aw logs --artifacts all` and `gh | `sandbox/firewall/audit/` | Domain-level network access log (raw data) | | `agent_usage.json` | Token usage for the agent turn | -### Parsed log files (--parse) +### Readable log files -When you run `gh aw audit --parse`, two readable files are written alongside the raw artifacts: +The audit report is accompanied by readable files written alongside the raw artifacts: - `log.md` — the full agent conversation formatted as Markdown - `firewall.md` — a formatted summary of outbound network access (allowed and blocked domains) @@ -93,37 +93,37 @@ Sample output: - [ ] The ⌖ AIC figure appears separately from Agent AIC - [ ] The threat verdict shows `none` (or I noted what was flagged) -### Explore the parsed agent log +### Explore MCP tool calls -Add `--parse` to write `log.md` and `firewall.md` alongside the raw artifacts: +Download the artifacts for a run, then open the `mcp-logs/` directory. Each file corresponds to one MCP server and lists every tool call the agent made. ```bash -gh aw audit --parse +gh aw logs --artifacts all ``` -Open `log.md` in your editor. Each section represents one agent turn and lists the tool calls in order. +Browse the log files in `.github/aw/logs//mcp-logs/`. -- [ ] `log.md` and `firewall.md` were created in `.github/aw/logs//` -- [ ] I found the first MCP tool call in `log.md` -- [ ] I wrote one sentence describing what the agent was trying to accomplish in that turn +- [ ] I found the `mcp-logs/` directory in the downloaded artifacts +- [ ] I identified at least one tool call and noted the tool name +- [ ] I wrote one sentence describing what the agent was trying to accomplish - [ ] I checked `agent_usage.json` for the total token count -### Inspect the firewall summary +### Inspect the firewall records -`firewall.md` groups outbound network calls into allowed and blocked lists. Scan it to confirm your workflow only contacted expected domains: +The raw domain-level network access logs live in `sandbox/firewall/audit/` inside the agent artifact. Scan them to confirm your workflow only contacted expected domains. -- [ ] I opened `firewall.md` and identified at least one domain the workflow accessed -- [ ] If any domains are blocked, I know to add them to `network.allow` in the workflow frontmatter -- [ ] I can distinguish the formatted `firewall.md` from the raw records in `sandbox/firewall/audit/` +- [ ] I opened `sandbox/firewall/audit/` in the downloaded artifacts +- [ ] I identified at least one domain the workflow accessed +- [ ] If any domains were blocked, I know to add them to `network.allow` in the workflow frontmatter ## ✅ Checkpoint - [ ] You can identify each file inside the agent artifact and what it contains - [ ] You understand what ⌖ AIC represents and how it differs from agent AIC -- [ ] You can use `firewall.md` to identify blocked domains and add them to `network.allow` +- [ ] You can find blocked domains in the firewall audit records and add them to `network.allow` - [ ] You know what the threat detection verdict checks for - [ ] You ran `gh aw audit` on a real run and reviewed the generated report -- [ ] You used `--parse` to produce `log.md` and read at least one tool call +- [ ] You explored `mcp-logs/` to identify tool calls from a completed run Return to [Audit and Monitor Your Agentic Workflows](25-audit-and-observability.md).