Skip to content

Commit bf34af3

Browse files
jamesadevineCopilot
andcommitted
feat(audit): add ado-aw audit <build-id-or-url> command
Single-run audit: download a build's artifacts, run every analyzer (firewall, MCP gateway, OTel, safe outputs, detection verdict, build timeline, missing tools/data/noops), and emit a Markdown or JSON report. ADO-side counterpart to `gh aw audit`. New module tree under `src/audit/`: - `model.rs` — `AuditData` (drift-compatible with gh-aw's top-level contract; adds ADO-specific `detection_analysis`, `safe_output_execution`, `rejected_safe_outputs` sections). - `url.rs` — parses bare IDs, dev.azure.com URLs, legacy visualstudio.com URLs, and on-prem Azure DevOps Server URLs (with optional `&j=`/`&t=`/`&s=` job/step anchors). - `cache.rs` — CLI-version-keyed `run-summary.json` with atomic writes. - `analyzers/{firewall,policy,mcp,otel,safe_outputs,detection,missing,jobs}.rs` — eight defensive NDJSON/REST analyzers. - `findings.rs` — eight heuristic rules emitting severity-rated findings + recommendations. - `render/{console,json}.rs` — two renderers; JSON shape is the public contract. - `cli.rs` — orchestration: URL parse → auth → metadata fetch → artifact download → analyzers → findings → cache → render. Unified rejection trace: when the aggregate `THREAT_DETECTION_RESULT` has any threat flag set, every proposal lands in `not_processed_due_to_aggregate_gate` carrying the aggregate `reasons[]`, exactly one severity-`high` `KeyFinding` is emitted, and a `rejected_safe_outputs` rollup appears at the top level. Pipeline-side runtime additions (so an `ado-aw audit` of an existing build has the data it needs): - `src/data/*-base.yml` (via `AdoAwMarkerExtension`): emits `staging/aw_info.json` at runtime with engine, model, agent name, source path, target, compiler version, and ADO build context. - `src/execute.rs`: writes a per-item `safe-outputs-executed.ndjson` in `<output-dir>` so the audit can show the proposed → detection → executed trace. CLI surface: ado-aw audit <build-id-or-url> -o, --output <dir> # default ./logs --json --org / --project / --pat --artifacts <agent,detection,safe-outputs> --no-cache New dependencies: `zip` (artifact unpack), `wiremock` (dev only — integration test mock server). Tests: 80 new audit unit tests + 3 integration tests against a fake ADO REST server (happy path, permission-denied, cache hit) using a thin `ADO_AW_TEST_ORG_URL` test seam. 1740 total tests pass. Docs: new `docs/audit.md`; updates to `docs/cli.md`, `README.md`, `AGENTS.md` index, and `prompts/debug-ado-agentic-workflow.md` (Step 1 first-move + new Step 2a-prime + `AuditData` reference + jq-diff fallback). Out of scope (explicit follow-ups): diff mode, cross-run trends, `--parse` log.md/firewall.md, job/step-anchored audit, MCP-exposed audit, per-item detection verdict (upstream coordination with gh-aw), partial-approval gating, AWF policy-manifest plumbing, AWF token-usage.jsonl, `audit-manifest.json` build inventory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 9c500e3 commit bf34af3

33 files changed

Lines changed: 10183 additions & 191 deletions

AGENTS.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,10 @@ index to jump to the right page.
227227
in `src/data/base.yml`, `src/data/1es-base.yml`, `src/data/job-base.yml`, and `src/data/stage-base.yml` and how it is replaced.
228228
- [`docs/cli.md`](docs/cli.md)`ado-aw` CLI commands (`init`, `compile`,
229229
`check`, `mcp`, `mcp-http`, `execute`, `secrets`, `enable`, `disable`,
230-
`remove`, `list`, `status`, `run`; `configure` is a deprecated hidden alias).
230+
`remove`, `list`, `status`, `run`, `audit`; `configure` is a deprecated hidden alias).
231+
- [`docs/audit.md`](docs/audit.md)`ado-aw audit`: accepted build-id / URL
232+
forms, artifact layout, cache behavior, rejection tracing, and `AuditData`
233+
report shape.
231234
- [`docs/mcp.md`](docs/mcp.md) — MCP server configuration (stdio containers,
232235
HTTP servers, env passthrough).
233236
- [`docs/mcpg.md`](docs/mcpg.md) — MCP Gateway architecture and pipeline

Cargo.lock

Lines changed: 154 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ base64 = "0.22.1"
3535
glob-match = "0.2.1"
3636
similar = "3.1.0"
3737
sha2 = "0.11.0"
38+
zip = { version = "8.6.0", default-features = false, features = ["deflate"] }
3839

3940
[dev-dependencies]
4041
reqwest = { version = "0.12", features = ["blocking"] }
42+
wiremock = "0.6"

README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -496,6 +496,8 @@ network:
496496

497497
## CLI Reference
498498

499+
- `audit <build-id-or-url>` - Audit a single Azure DevOps build: download artifacts, analyze logs, render Markdown or JSON report. See [`docs/audit.md`](docs/audit.md).
500+
499501
```
500502
ado-aw [OPTIONS] <COMMAND>
501503
@@ -513,6 +515,7 @@ Commands:
513515
list List matched ADO definitions with their latest-run state
514516
status Per-pipeline status block for matched ADO definitions
515517
run Queue builds for matched ADO definitions (optionally poll to completion)
518+
audit Audit a single Azure DevOps build: download artifacts, analyze logs, render a report
516519
517520
Options:
518521
-v, --verbose Enable info-level logging

0 commit comments

Comments
 (0)