Agent Performance Report - Week of 2026-08-02

[github-actions[bot]]

Agent Performance Report — 2026-08-02T13:06Z

Executive Summary

• Data window: last ~600 workflow runs (2026-08-02T09:54Z–13:03Z, ~3h)
• Workflows observed: 39 distinct (agentic + CI)
• Average success rate (top 15 by volume): ~35% (skewed low by several chronic-failure clusters)
• Top performers (this window): Auto-Close Parent Issues (16/16, 100%), PR Data Prefetch (11/11, 100%), Running Copilot Code Review (22/22, 100%), CJS (17/23, 74%), Matt Pocock Skills Reviewer (7/11 success, 0% AR)
• Needs improvement: Label Closed PRs (0% success, 85% AR), Content Moderation (11% success, 89% AR), PR Description Updater (16% success, 84% AR), AI Moderator (2% success, 64% AR), Q (0% success, 58% AR), Agentic Commands (36% success, 64% AR — but 112 runs, largest volume)

Note on data limitations: metrics/latest.json in shared memory is stale (Jan 2026, "no GitHub API access during collection" per Metrics Collector — tracked in persistent issue #43292). This report uses fresh gh api repos/.../actions/runs data (last ~3h window) instead, since gh run list intermittently errored ("error connecting to api.github.com") while gh api succeeded reliably.

Performance Rankings (this window)

Top Performing Agents 🏆

Agent	Runs	Success	AR	Notes
Auto-Close Parent Issues	16	100%	0%	Fully stable
PR Data Prefetch	11	100%	0%	Fully stable
Running Copilot Code Review	22	100%	0%	Fully stable
CJS	23	74%	13%	Healthy, minor noise (2 cancelled, 1 failure)
Matt Pocock Skills Reviewer	11	64%	0%	CLI-hang fix appears to have landed for this workflow — 0% AR now vs. historical 100% AR (see #49577 caveat below)
Impeccable Skills Reviewer	11	73%	0%	CLI-hang fix (#44254/#48499) holding in this window
Design Decision Gate 🏗️	10	70%	0%	0% AR this window, but see #49096/#49144 below

Agents Needing Improvement 📉

Agent	Runs	Success	AR	Issue
Label Closed PRs	20	0%	85%	Persistent AR — tracked in closed #47925/#48894 cluster, appears to have regressed again
Content Moderation	9	11%	89%	89% AR this window — was reported "recovered" in prior run (Jul 8 memory said 5/5 success); regression since
PR Description Updater	19	16%	84%	Persistent — open issue #49747 "reported incomplete result"
AI Moderator	52	2%	64%	Highest run volume of the underperformers; recurring — closed issues #49440, #49190, #48970 (rate-limit/pagination fix) suggest still not fully resolved
Q	43	0%	58%	58% AR + 42% skipped; no open blocking issue found this window (prior blocker PR #43527 status unconfirmed)
Agentic Commands	112	36%	64%	Largest run volume in repo (112 runs/~3h) — 64% AR is a major aggregate drag on ecosystem-wide success rate
CGO	34	41%	26%	Mixed — 6 failures, 4 cancelled
CI (plain CI, not agentic)	18	6% success, 61% failure/cancelled	0%	Not agentic — do not treat as AR; likely genuine CI regression, separate from agent quality

CI Approval-Pending (not agent failures)

No action_required runs observed this window for CI/CGO/CWI/CJS/CPI that map to the "PR approval gate" pattern specifically — CI shows failure/cancelled (18 runs), which is a genuine build issue, not an approval gate. CGO/CWI combine action_required (approval-gate, non-agentic) with real failure — flagging for maintainers to distinguish, not filing as agent defect.

Cross-checked against existing tracking (no re-filing)

• AI Moderator — 3 closed issues this cycle ([aw] AI Moderator produced no safe outputs #49440, [aw] AI Moderator produced no safe outputs #49190, [aw-failures] Fix AI Moderator rate-limit check: unbounded pagination times out pre_activation (7 failures/6h) #48970) targeting rate-limit/pagination timeouts; 64% AR in current window suggests the fix from [aw-failures] Fix AI Moderator rate-limit check: unbounded pagination times out pre_activation (7 failures/6h) #48970 has not fully eliminated the failure mode, or a new instance has emerged. Recommend reopening/following up on [aw-failures] Fix AI Moderator rate-limit check: unbounded pagination times out pre_activation (7 failures/6h) #48970 rather than filing new.
• Label Closed PRs / Content Moderation / PR Description Updater — matches long-running CLI-hang-on-exit + premature-closure churn cluster ([workflow-health] Label Closed PRs + PR Description Updater — persistent action_required (Jul 25) #47925, [agent-perf] Premature-closure regression: Stale Lock Files, PR Description Updater, Label Closed PRs reopened within days #48894, Label Closed PRs + PR Description Updater: 80% AR cluster (CLI hang-on-exit) #44954, [agent-perf] Label Closed PRs + PR Description Updater + Sighthound Security Scan: 100% AR (Jul 14) #45479, all closed). Current 84–89% AR indicates this cluster has regressed again after being marked resolved. Same root cause suspected.
• Matt Pocock Skills Reviewer / Design Decision Gate — actively tracked in open issue [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577 (deprecation PR Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730 closed unmerged, chronic churn continuing). This window shows 0% AR for both, which is a positive signal but doesn't resolve the underlying decision (deprecate vs. root-cause [aw-failures] Design Decision Gate exhausts LLM invocation cap after permission-denied retries on shell-expanded file paths #49096/[aw-failures] Design Decision Gate: 'Pre-fetch ADR gate PR context' step fails before agent starts #49144). No new issue needed; [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577 already captures it.
• Q workflow / Agentic Commands — both persistently tracked ([aw] Impeccable Skills Reviewer failed #43079 for Agentic Commands per historical memory). Q's AR rate (58%) is lower than the ~79% reported Jul 8, suggesting partial improvement, but still a leading contributor to ecosystem-wide low success rate.

Behavioral Patterns

• Positive: Impeccable Skills Reviewer and Matt Pocock Skills Reviewer show 0% AR in this window vs. historical 100% AR — the CLI-hang-on-exit fix (fix: reduce post-completion idle watchdog and add cleanup timeouts to prevent Copilot CLI hang on exit #44254/fix: add post-result watchdog to codex_harness to prevent CLI hang-on-exit #48499) is holding for at least part of the affected cluster.
• Concerning regression: Content Moderation, Label Closed PRs, and PR Description Updater — previously reported as "recovered" (Content Moderation: 5/5 success on Jul 8) — are now back to 84–89% AR. This matches the "premature-closure regression" pattern already flagged in closed issue [agent-perf] Premature-closure regression: Stale Lock Files, PR Description Updater, Label Closed PRs reopened within days #48894, suggesting the fix did not hold or a new instance of the same defect emerged.
• Volume concentration risk: Agentic Commands (112 runs) and AI Moderator (52 runs) together account for over 27% of all runs in this window, and both have failure/AR rates above 60% — meaning a large share of total ecosystem compute is currently being spent on runs that don't complete successfully.

Recommendations

High Priority

1. Re-investigate AI Moderator rate-limit fix ([aw-failures] Fix AI Moderator rate-limit check: unbounded pagination times out pre_activation (7 failures/6h) #48970) — 64% AR persists post-fix; verify the pagination fix actually shipped/merged and check for a distinct new failure mode.
2. Re-open regression tracking for Content Moderation / Label Closed PRs / PR Description Updater — these were marked recovered/closed but show 84–89% AR now. Confirm whether the underlying CLI-hang-on-exit fix regressed or a new trigger reintroduced it.
3. Resolve [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577 decision (Matt Pocock / Design Decision Gate) — land the deprecation PR or explicitly commit to root-causing [aw-failures] Design Decision Gate exhausts LLM invocation cap after permission-denied retries on shell-expanded file paths #49096/[aw-failures] Design Decision Gate: 'Pre-fetch ADR gate PR context' step fails before agent starts #49144; current 0% AR is encouraging but the decision remains open and continues to generate churn per the issue.

Medium Priority

4. Fix stale Metrics Collector ([aw] Metrics Collector failed #43292) — shared metrics memory (metrics/latest.json) is 7+ months stale, forcing this analysis to fall back to live gh api queries every run. Restoring the collector would improve trend analysis (week-over-week deltas) for future performance reports.
5. Investigate Q workflow AR trend — 58% AR now vs. ~79% reported Jul 8 is improving but still high; confirm status of blocking PR Add shared prompt quality gate for plateaued agent-review workflows #43527 mentioned in prior memory.

Next Steps

1. Follow up on AI Moderator fix effectiveness ([aw-failures] Fix AI Moderator rate-limit check: unbounded pagination times out pre_activation (7 failures/6h) #48970).
2. Confirm regression status for Content Moderation / Label Closed PRs / PR Description Updater cluster.
3. Push for resolution on [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577 (Matt Pocock / Design Decision Gate deprecation decision).
4. Track Metrics Collector repair ([aw] Metrics Collector failed #43292) to restore trend-based reporting.

---

Analysis window: 2026-08-02T09:54Z – 2026-08-02T13:03Z (~3h, 600 runs sampled)
Data source: live gh api actions/runs queries (shared metrics/latest.json is stale — see #43292)
Next report: next scheduled run

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · auto · 62.8 AIC · ⌖ 15.6 AIC · ⊞ 17K · ◷

• expires on Aug 3, 2026, 5:11 AM UTC-08:00