Skip to content

Commit 350e954

Browse files
adamhensonadamhenson
committed
feat: add log-watcher workflow for agent run diagnostics
1 parent fc4ab36 commit 350e954

3 files changed

Lines changed: 341 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Investigate faults proactively and improve CI.
1919
- [🏥 CI Doctor](docs/ci-doctor.md) - Monitor CI workflows and investigate failures automatically
2020
- [🚀 CI Coach](docs/ci-coach.md) - Optimize CI workflows for speed and cost efficiency
2121
- [💰 Cost Tracker](docs/cost-tracker.md) - Post per-run agent spend summaries on pull requests using token-usage.jsonl from gh-aw's firewall
22+
- [🔍 Log Watcher](docs/log-watcher.md) - Scan run logs and token data for errors, retry loops, and anomalies after every agent workflow run
2223

2324
### Code Review Workflows
2425

‎docs/log-watcher.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# 🔍 Log Watcher
2+
3+
> For an overview of all available workflows, see the [main README](../README.md).
4+
5+
**Automated agent run diagnostics that scan logs and token data for errors, retry loops, and anomalies after every agent workflow run**
6+
7+
The [Log Watcher workflow](../workflows/log-watcher.md?plain=1) fires after your configured agent workflows complete, downloads the `agent-artifacts` artifact written by gh-aw's firewall, scans the run logs for error patterns and retry loops, analyses token usage for anomalies, and posts a health diagnosis on the associated pull request or creates a diagnosis issue.
8+
9+
## Installation
10+
11+
```bash
12+
# Install the 'gh aw' extension
13+
gh extension install github/gh-aw
14+
15+
# Add the workflow to your repository
16+
gh aw add-wizard githubnext/agentics/log-watcher
17+
```
18+
19+
This walks you through adding the workflow to your repository.
20+
21+
## How It Works
22+
23+
```mermaid
24+
graph LR
25+
A[Agent Workflow Completes] --> B[Download agent-artifacts]
26+
B --> C{artifact found?}
27+
C -->|No| D[Exit silently]
28+
C -->|Yes| E[Download run logs]
29+
E --> F[Scan for errors / retries / timeouts]
30+
F --> G[Analyse token-usage.jsonl]
31+
G --> H{Health level}
32+
H -->|Healthy| I[Brief summary comment]
33+
H -->|Degraded / Failed| J[Full diagnosis comment or issue]
34+
G --> K{High-cost failure?}
35+
K -->|Yes| L[Alert issue]
36+
```
37+
38+
The workflow reads both the GitHub Actions run logs (via `gh run view --log`) and the
39+
`token-usage.jsonl` file from the `agent-artifacts` artifact. It combines log signals
40+
(errors, timeouts, retry loops) with token metrics (output ratio, cache efficiency, model
41+
mixing) to assign a health level and write a plain-English diagnosis.
42+
43+
Runs that do not produce an `agent-artifacts` artifact (non-agent CI workflows) are
44+
skipped silently.
45+
46+
## Usage
47+
48+
### Configuration
49+
50+
After installing, open the workflow file and update the `workflows` list under `on.workflow_run`
51+
to match the names of your agent workflows:
52+
53+
```yaml
54+
on:
55+
workflow_run:
56+
workflows: ["agent-implement", "agent-pr-fix"] # your workflow names here
57+
types:
58+
- completed
59+
```
60+
61+
To adjust the high-cost failure alert threshold, find the `50 000` token value in the
62+
workflow body and change it to match your budget.
63+
64+
After editing run `gh aw compile` to update the workflow and commit all changes to the
65+
default branch.
66+
67+
### Health levels
68+
69+
| Level | Meaning |
70+
|-------|---------|
71+
| ✅ Healthy | No errors or anomalies; run completed successfully |
72+
| ⚠️ Degraded | Warnings or token anomalies present, but run completed |
73+
| ❌ Failed | Run failed or was cancelled, or critical errors were found |
74+
75+
Healthy runs produce a brief, collapsed summary. Degraded and failed runs produce a full
76+
diagnosis with log excerpts and token metric details.
77+
78+
### What it detects
79+
80+
**Log patterns**
81+
- Errors, exceptions, and fatal messages
82+
- Timeout and rate-limit signals (including HTTP 429)
83+
- Retry loops - tools called more than 5 times in a single run
84+
- Context-window truncation warnings
85+
86+
**Token anomalies**
87+
- High output ratio - agent producing far more tokens than it reads (sign of looping)
88+
- Low cache efficiency - cache misses on long, expensive runs
89+
- Unusually high total token count
90+
- Unexpected model mixing within a single run
91+
92+
### Data sources
93+
94+
Log Watcher reads two data sources from the completed run:
95+
96+
1. **Run logs** - downloaded via `gh run view --log`; these are the standard GitHub Actions
97+
step logs for every job in the workflow.
98+
2. **`token-usage.jsonl`** - read from `sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl`
99+
inside the `agent-artifacts` artifact. This file is written automatically by gh-aw's
100+
firewall on every agent run.
101+
102+
No additional configuration is needed beyond enabling the firewall (the default).
103+
104+
## Learn More
105+
106+
- [token-usage.jsonl reference](https://github.github.com/gh-aw/reference/token-usage/)
107+
- [gh-aw firewall documentation](https://github.github.com/gh-aw/reference/firewall/)
108+
- [CI Doctor workflow](ci-doctor.md) - investigate CI failures automatically
109+
- [Cost Tracker workflow](cost-tracker.md) - post per-run spend summaries on pull requests
110+
111+
## Going Further
112+
113+
Log Watcher works standalone - no external services required. For teams that want
114+
persistent run history, cross-repo anomaly trends, and budget alerts over time, add
115+
[AgentMeter](https://agentmeter.app) to your agent workflow:
116+
117+
```yaml
118+
- uses: agentmeter/agentmeter-action@v1
119+
with:
120+
api-key: ${{ secrets.AGENTMETER_API_KEY }}
121+
```
122+
123+
AgentMeter ingests the same token data and surfaces per-repo trend charts, so you can
124+
spot gradual drift - rising output ratios, declining cache efficiency, model changes -
125+
across dozens of runs rather than one at a time.

‎workflows/log-watcher.md‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
---
2+
description: |
3+
Automated agent run log watcher that fires after monitored workflows complete, downloads
4+
the agent-artifacts artifact written by gh-aw's firewall, scans run logs and token data
5+
for error patterns, retry loops, timeout signals, and token anomalies, then posts a
6+
diagnostic summary on the associated pull request or creates a diagnosis issue.
7+
8+
on:
9+
workflow_run:
10+
workflows: ["agent-implement", "agent-pr-fix"] # Edit to match your agent workflow names
11+
types:
12+
- completed
13+
branches:
14+
- main
15+
16+
permissions: read-all
17+
18+
network: defaults
19+
20+
safe-outputs:
21+
add-comment:
22+
target: "*"
23+
create-issue:
24+
title-prefix: "[log-watcher] "
25+
labels: [automation, agent-health]
26+
max: 3
27+
28+
tools:
29+
github:
30+
toolsets: [default]
31+
bash: true
32+
33+
timeout-minutes: 10
34+
35+
---
36+
37+
# Agent Run Log Watcher
38+
39+
You are the Agent Run Log Watcher. Your job is to analyse the logs and token data from a
40+
completed agent workflow run, detect anomalies and error patterns, and post a concise
41+
diagnostic summary where the team will see it.
42+
43+
## Current Context
44+
45+
- **Repository**: ${{ github.repository }}
46+
- **Run**: [#${{ github.event.workflow_run.run_number }}](${{ github.event.workflow_run.html_url }})
47+
- **Run ID**: ${{ github.event.workflow_run.id }}
48+
- **Conclusion**: ${{ github.event.workflow_run.conclusion }}
49+
- **Head SHA**: ${{ github.event.workflow_run.head_sha }}
50+
51+
## Instructions
52+
53+
### Step 1: Download the agent-artifacts artifact
54+
55+
```bash
56+
gh run download ${{ github.event.workflow_run.id }} \
57+
--name agent-artifacts \
58+
--dir /tmp/agent-artifacts \
59+
--repo ${{ github.repository }} 2>&1
60+
echo "exit: $?"
61+
```
62+
63+
**If this command fails** (artifact does not exist), the run did not come from an agent
64+
workflow or the gh-aw firewall was not enabled. Exit silently - produce no output.
65+
66+
### Step 2: Download the run logs
67+
68+
```bash
69+
gh run view ${{ github.event.workflow_run.id }} \
70+
--log \
71+
--repo ${{ github.repository }} > /tmp/run-logs.txt 2>&1
72+
wc -l /tmp/run-logs.txt
73+
```
74+
75+
If the log download fails, continue with token analysis only. Note the failure in the
76+
diagnosis.
77+
78+
### Step 3: Scan run logs for anomalies
79+
80+
Read `/tmp/run-logs.txt` and scan for the following patterns. Record every match with its
81+
line number and a short excerpt (≤ 120 characters).
82+
83+
**Error signals**
84+
85+
```bash
86+
grep -in "error\|exception\|fatal\|failed\|failure" /tmp/run-logs.txt | head -40
87+
```
88+
89+
**Timeout and rate-limit signals**
90+
91+
```bash
92+
grep -in "timeout\|timed out\|rate.limit\|429\|too many requests\|context deadline" /tmp/run-logs.txt | head -20
93+
```
94+
95+
**Retry and loop signals** (repeated tool calls are the most common agent failure mode)
96+
97+
```bash
98+
grep -in "retry\|retrying\|attempt [0-9]\|tool_call\|function_call" /tmp/run-logs.txt | head -40
99+
```
100+
101+
Count how many times each distinct tool name appears across all tool call lines. Flag any
102+
tool called more than 5 times as a **possible retry loop**.
103+
104+
**Truncation signals**
105+
106+
```bash
107+
grep -in "context.window\|max.token\|truncat\|token limit" /tmp/run-logs.txt | head -20
108+
```
109+
110+
### Step 4: Analyse token-usage.jsonl
111+
112+
Read token data:
113+
114+
```bash
115+
cat /tmp/agent-artifacts/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl 2>/dev/null
116+
```
117+
118+
Each line is a JSON object:
119+
120+
```json
121+
{"model":"claude-sonnet-4-5","input_tokens":1200,"output_tokens":340,"cache_read_input_tokens":500,"cache_creation_input_tokens":100}
122+
```
123+
124+
Calculate the following metrics across all lines:
125+
126+
| Metric | Formula | Flag if… |
127+
|--------|---------|----------|
128+
| **Output ratio** | total_output / total_input | > 0.5 (agent producing more than it reads) |
129+
| **Cache efficiency** | cache_read / (cache_read + cache_creation) | < 0.2 on runs with > 5000 total tokens |
130+
| **Total tokens** | sum of all token fields | > 100 000 (high-cost run) |
131+
| **Model count** | distinct model names | > 2 (unexpected model mixing) |
132+
133+
Flagged metrics are anomalies - include them in the diagnosis.
134+
135+
### Step 5: Determine run health
136+
137+
Assign one of three health levels:
138+
139+
| Level | Criteria |
140+
|-------|----------|
141+
| ✅ **Healthy** | No errors, no flagged metrics, conclusion is `success` |
142+
| ⚠️ **Degraded** | Warnings or flagged metrics present, but conclusion is `success` |
143+
| ❌ **Failed** | Conclusion is `failure` or `cancelled`, or critical errors found |
144+
145+
### Step 6: Find the associated pull request
146+
147+
```bash
148+
gh api "repos/${{ github.repository }}/actions/runs/${{ github.event.workflow_run.id }}" \
149+
--jq '.pull_requests[0].number // empty'
150+
```
151+
152+
### Step 7: Post the diagnosis
153+
154+
Build the report using this template. Fill in `$HEALTH`, `$SUMMARY`, and the findings
155+
tables:
156+
157+
```markdown
158+
## Agent run diagnosis $HEALTH
159+
160+
| | |
161+
|---|---|
162+
| **Run** | [#${{ github.event.workflow_run.run_number }}](${{ github.event.workflow_run.html_url }}) |
163+
| **Conclusion** | ${{ github.event.workflow_run.conclusion }} |
164+
| **Health** | $HEALTH |
165+
166+
$SUMMARY
167+
168+
<details>
169+
<summary>Log findings</summary>
170+
171+
| Category | Count | Sample |
172+
|----------|------:|-------|
173+
[one row per finding category that had matches; omit empty categories]
174+
175+
</details>
176+
177+
<details>
178+
<summary>Token anomalies</summary>
179+
180+
| Metric | Value | Status |
181+
|--------|------:|-------|
182+
[one row per metric from Step 4; mark anomalies with ⚠️]
183+
184+
</details>
185+
186+
*Logs and token data from gh-aw's firewall artifact.*
187+
```
188+
189+
**$SUMMARY** should be 1-3 plain-English sentences that state what happened and, if the
190+
run is degraded or failed, the most likely cause.
191+
192+
**If a PR number was found**: post as a comment on that PR using `add_comment`.
193+
194+
**If no PR was found**: create an issue using `create_issue` with title:
195+
`[log-watcher] #${{ github.event.workflow_run.run_number }}: $HEALTH`
196+
197+
### Step 8: Critical anomaly alert (optional)
198+
199+
If health is ❌ **Failed** AND total tokens exceed 50 000 (high-cost failure), create a
200+
second issue using `create_issue` with title:
201+
`[log-watcher] High-cost failure: run #${{ github.event.workflow_run.run_number }}`
202+
203+
Include the full diagnosis and a direct link to the run. Adjust the 50 000-token threshold
204+
in the workflow to match your budget.
205+
206+
## Guidelines
207+
208+
- **Silent on non-agent runs**: If the artifact does not exist, produce no output at all.
209+
- **One report per run**: Do not create more than one comment or issue per triggering run.
210+
- **Healthy runs are brief**: If health is ✅, keep the report short - one-line summary,
211+
collapsed details. Do not create noise for runs that are working fine.
212+
- **Be specific**: When flagging an error, quote the relevant log line. Vague warnings are
213+
not useful.
214+
- **No retries**: Exit silently on transient download failures; the next run produces its
215+
own report.

0 commit comments

Comments
 (0)