[audit-workflows] 🔍 Agentic Workflow Audit — 2026-07-12 (HEALTHY 88.0%, all fails copilot-driver)

[github-actions[bot]]

🔍 Agentic Workflow Audit — 2026-07-12

Verdict: HEALTHY on baseline. 88.0% success (89.8% excluding 2 intentional credit-limit tests). Every real failure is a copilot-engine driver failure — for the 7th consecutive audit. Zero agent-logic failures, zero missing-tools/data, zero MCP failures, zero novel anomalies. Non-copilot engines were perfect.

Window: partial ~11.5h (09:30–21:05Z). 334 run dirs enumerated; 100 fully downloaded, 234 empty due to the bridge 120s cap (known partial-window pattern). Stats below are the 100 complete runs.

---

📊 Key Metrics

Metric	Value
Runs (complete)	100
Success / Failure	88 / 12
Success rate	88.0% (89.8% ex-2-intentional)
Missing tools / data / MCP / noop	0 / 0 / 0 / 0
Novel anomalies	0
AIC (cost units)	4,361
Tokens (in+out)	5.85M (5.12M in / 0.73M out)
Action minutes	1,006

Engine health

Engine	Runs	Fail	Rate
copilot	43	11	74.4% (76.2% ex-intentional)
pi	33	0	100%
claude	15	0	100%
codex	6	0	100%
gemini	1	0	100%
antigravity	1	0	100%

The failure surface is entirely copilot. Every other engine ran clean.

---

📈 Trends

Workflow Health (30 days)

Success rate holds in its stable ~86–90% band. Today's 88.0% sits squarely on the multi-week baseline — no degradation, no spike. The red (failure) bars remain small and, as every audit since 07-04, are composed exclusively of copilot 0-turn driver failures rather than agent-logic errors.

Token Usage (reporting-gap days omitted)

Daily token volume (5.85M this partial window) tracks the 7-day moving average with no outlier. Many historical days show gaps because metrics.TokenUsage reports 0 fleet-wide; the values plotted come from the reliable token_usage_summary fallback.

---

🧭 Failure clustering (all 12)

All failures map to existing known-issues — expand for detail

Intentional (2) — expected, excluded from health rate:

• Daily Max Ai Credits Test, Daily Credit Limit Test

copilot driver failures (10 real) — all 0-turn at a copilot driver step:

Workflow	Step	AIC	Known issue
Smoke CI ×2 (push/main)	Execute GitHub Copilot CLI	—	smoke-ci-copilot-cli-100pct-fail-on-push (recur 7)
Daily Security Observability Report	Execute GitHub Copilot CLI	141.8 (30m)	copilot-sdk-driver-failures
Daily Project Performance Summary	Execute GitHub Copilot CLI	156.2 (21m)	copilot-sdk-driver-failures
PR Triage Agent	Execute GitHub Copilot CLI	142.9	copilot-sdk-driver-failures
Daily Formal Spec Verifier	Execute GitHub Copilot CLI	52.2	copilot-sdk-driver-failures
Smoke Copilot Small	Execute GitHub Copilot CLI	—	copilot-sdk-driver-failures
Smoke Copilot Sub Agents	Execute GitHub Copilot CLI	—	copilot-sdk-driver-failures
Daily Issues Report Generator	Execute GitHub Copilot CLI	—	chroot-node-not-available (recur 12)
Smoke Copilot	Process Safe Outputs	100.1	safe-output-partial-failure-intolerance

The longrun failures (30m/21m, AIC 142–156) burned real budget — the agent did the work, then the Execute GitHub Copilot CLI step exited non-zero on export/finalize. This is the copilot-SDK driver signature, not an agent-logic problem.

---

⚠️ New infrastructure observation: turns-reporting-gap

metrics.Turns is now 0 for all 100 runs including successes — not just metrics.TokenUsage. This breaks the long-used "0-turn ⇒ pre-agent driver failure" heuristic, since Turns=0 is now indistinguishable between healthy and failed runs. This audit classified failures by failed-step + AIC instead. token_usage_summary remains authoritative (5.85M). Logged under token-usage-reporting-gap.

---

✅ Recommendations

1. [medium] Fix the turns/token metric population. Populate metrics.Turns and metrics.TokenUsage from token_usage_summary (which is correct), or document that consumers must read token_usage_summary + failed-step — not metrics.*. Evidence: 100/100 runs have run.Turns=0 and metrics.TokenUsage=0 while token_usage_summary sums to 5.12M input.

2. [high] Root-cause the copilot CLI export/exit path. copilot is the sole failure source (76.2% ex-intentional) for the 7th straight audit; several fails burn AIC 52–156 before exiting non-zero. As mitigation, consider migrating the most-affected daily reporters to claude/pi (both 100% today). Evidence: copilot 32/42 vs pi 33/33, claude 15/15, codex 6/6.

3. [low] Smoke CI chronic red on push/main (smoke-ci-copilot-cli-100pct-fail-on-push, now recur 7) continues to color the health dashboard red on main with no signal value — either fix or suppress from the main-branch health view.

---

💰 Top cost runs (context)

All top-cost runs are claude read-only reporters at normal levels — no cost anomaly:
Daily Code Metrics (AIC 330), Claude Code User Docs Reviewer (267), GitHub MCP Remote Server Tools Report (265). PR Sous Chef ran 23/23 = 100%.

References:

• §29205558650 — Smoke Copilot Small (copilot driver fail)
• §29188648302 — PR Sous Chef (healthy sample)
• §29206052899 — Smoke Copilot Sub Agents (copilot driver fail)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔍 Agentic Workflow Audit Agent · 230.5 AIC · ⌖ 35.7 AIC · ⊞ 7.4K · ◷

• expires on Jul 13, 2026, 1:39 PM UTC-08:00