[audit-workflows] Daily Audit — 2026-05-29: cache-memory git infra bug (2 failures) + 9-day codex routing issue RESOLVED

[github-actions[bot]]

Overview

Audit of agentic workflow runs in the last ~24h. This was a partial mid-cycle window of 21 runs (17 success, 2 failure, 2 in-progress) — a smaller sample than a full day, so absence of a prior issue is not proof it is fixed.

The headline is mixed-but-positive: a single new infrastructure bug caused both of today's failures, while several long-standing issues cleared — most notably the 9-day-stuck codex alpha-model routing failure, now resolved.

Summary

Metric	Value
Runs (window)	21 (17 ✅ / 2 ❌ / 2 ⏳)
Completion rate	89.5% (17/19 completed)
Total tokens	14.55M (effective 85.25M)
Action minutes	157
Turns	317
Measured cost	~$7.18 (claude-only; copilot/codex cost not reported)
Missing tools	3 (2 in Smoke Claude, 1 in Smoke Codex)
Engines	copilot 11 · claude 5 · codex 2 · antigravity 1 · gemini 1 · pi 1

🔴 Critical — NEW: cache-memory git setup bug (both failures)

Both failures today share one root cause in setup_cache_memory_git.sh:

• Chaos PR Bundle Fuzzer (§26662502294) — agent never ran (turns=0)
• Copilot PR Prompt Pattern Analysis (§26662652590)

The setup step logs a cache hit, then fails:

Cache hit detected: git repository found (restored from a previous run)
=== Cache directory: non-git files present after restore ===
./cache-hit-history.json
fatal: not in a git directory
##[error]Process completed with exit code 128.

The restored cache contains the data files but no valid .git, yet cache-hit detection asserts a repo was found and runs git commands anyway. The same fault then re-fails the downstream Commit cache-memory changes step. For Chaos this aborted before the agent even executed.

Fix (single script, restores 2 workflows): after restore, verify with git -C "$GH_AW_CACHE_DIR" rev-parse --git-dir (or test .git); when files are present but it is not a valid git repo, git init + re-attach remote/branch instead of trusting the cache-hit flag. Add a regression guard for the "non-git files present after restore" branch.

✅ Resolved / Recovered this cycle

• codex-model-not-found-alpha — RESOLVED after 9 days. Changeset Generator (§26662945784) now reads model: gpt-5.4 and succeeded. The one-line frontmatter edit recommended daily for 9 days was applied.
• Daily Safe Output Tool Optimizer cost — REVERSED. Dropped $11.33 → $3.07 (54.5M → 3.1M eff-tokens), breaking the +87% 3-day escalation.
• Smoke Copilot succeeded (was failing on gpt-5.3-codex routing).
• safe-output target=* failures did NOT recur — Contribution Check, Smoke Claude, and Smoke Codex all passed today (they failed this way on 05-28). Partial window; not yet confirmed fixed.

🟡 Capability gaps (runs passed, coverage reduced)

Missing / broken tools

• Smoke Claude (§26662945808):
  • serena find_symbol / get_symbols_overview return MCP error 0 EOF on first query — the Go LSP backend crashes after a successful activate_project. Symbol-search coverage silently lost.
  • mcpscripts-gh passthrough absent (only github_pr_query/issue/discussion/go/make); the test self-corrected with github_pr_query.
• Smoke Codex (§26662945780): web-fetch MCP tool not available in environment.

Action: provision the tool or update the smoke-test prompt to stop expecting it, per tool. The serena Go-LSP crash is the most worth a dedicated reliability fix.

⏸️ Carried / could not evaluate in this window

• Avenger max-turns (5-day open rec, raise --max-turns to ~50): Avenger did not run in this window — unverified, recommendation stays open.
• copilot-CLI 'all retries exhausted no category': no instances today (Smoke Copilot passed; PR Sous Chef in-progress; Auto-Triage absent). Keep harness error-classification rec open.

📊 Trends (last ~12 days)

Success rate sits at 89.5% in today's partial window, comfortably inside the normal band and well above the 05-23 dip (41.6%). Failure counts are low (2) and both trace to one infrastructure bug rather than model/agent behavior, so the underlying agent reliability is healthy. The dashed 05-29 bar is a small mid-cycle sample — treat its absolute counts as partial.

Daily cost tracks token volume closely and the 7-day moving average is flat-to-declining, with no runaway escalation — the previously-rising Safe Output Optimizer cost has fallen back into line. Note today's cost (~$7.18) is claude-engine only (copilot/codex/gemini estimated cost is not emitted in run_summary.json), so it understates true spend and is not directly comparable to full-day historical figures.

Recommendations (priority order)

1. [critical] Fix setup_cache_memory_git.sh to re-init when a restored cache has files but no valid .git — restores Chaos PR Bundle Fuzzer + Copilot PR Prompt Pattern Analysis.
2. [high] Re-verify Avenger max-turns next cycle it runs; keep the max-turns: ~50 edit queued.
3. [medium] Investigate serena Go-LSP EOF crash in Smoke Claude (capture gopls stderr; check toolchain compatibility).
4. [low] Decide provisioning vs prompt-trim for web-fetch (Smoke Codex) and mcpscripts-gh (Smoke Claude).
5. [low] Confirm Changeset Generator stays green for 2 cycles, then fully close the codex-alpha issue.

References:

• §26662502294 — Chaos PR Bundle Fuzzer (cache-memory git fail)
• §26662652590 — Copilot PR Prompt Pattern Analysis (cache-memory git fail)
• §26662945784 — Changeset Generator (resolved, gpt-5.4)

Generated by 🔍 Agentic Workflow Audit Agent · opus48 3.1M · ◷

• expires on May 30, 2026, 9:58 PM UTC

[github-actions[bot]]

---

Warning

The push_to_pull_request_branch operation failed: Incremental diff size (6057 KB) exceeds maximum allowed size (1024 KB). Bundle size: 1 KB.. The code changes were not applied.

💥 KA-POW! 🦸 The Smoke Test Agent BLAZED through here in a streak of light! ⚡ WHOOSH! 🌪️ Every gadget tested, every system GREEN! ✅ "Claude engine nominal, citizen!" 🚀 ...and with a final FWOOSH, the agent vanished into the CI clouds! 💨✨

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · opus48 1.1M · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #36002.