Skip to content

Commit e8a4d8a

Browse files
adamhensonAdam Hensondsyme
authored
Add cost-tracker workflow for agent spend observability (#319)
* Add cost-tracker workflow for agent spend observability * docs: add AgentMeter as optional persistent history upgrade path * docs: add agentmeter-action YAML snippet to Going Further section --------- Co-authored-by: Adam Henson <adam@agentmeter.app> Co-authored-by: Don Syme <dsyme@users.noreply.github.com>
1 parent e16ae50 commit e8a4d8a

3 files changed

Lines changed: 266 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Investigate faults proactively and improve CI.
1818

1919
- [🏥 CI Doctor](docs/ci-doctor.md) - Monitor CI workflows and investigate failures automatically
2020
- [🚀 CI Coach](docs/ci-coach.md) - Optimize CI workflows for speed and cost efficiency
21+
- [💰 Cost Tracker](docs/cost-tracker.md) - Post per-run agent spend summaries on pull requests using token-usage.jsonl from gh-aw's firewall
2122

2223
### Code Review Workflows
2324

‎docs/cost-tracker.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# 💰 Cost Tracker
2+
3+
> For an overview of all available workflows, see the [main README](../README.md).
4+
5+
**Automated agent cost reporter that posts a spend summary after every agent workflow run**
6+
7+
The [Cost Tracker workflow](../workflows/cost-tracker.md?plain=1) fires after your configured agent workflows complete, downloads the `token-usage.jsonl` data written by gh-aw's firewall, calculates per-model spend, and posts a cost breakdown on the associated pull request or creates a cost report issue.
8+
9+
## Installation
10+
11+
```bash
12+
# Install the 'gh aw' extension
13+
gh extension install github/gh-aw
14+
15+
# Add the workflow to your repository
16+
gh aw add-wizard githubnext/agentics/cost-tracker
17+
```
18+
19+
This walks you through adding the workflow to your repository.
20+
21+
## How It Works
22+
23+
```mermaid
24+
graph LR
25+
A[Agent Workflow Completes] --> B[Download agent-artifacts]
26+
B --> C{artifact found?}
27+
C -->|No| D[Exit silently]
28+
C -->|Yes| E[Parse token-usage.jsonl]
29+
E --> F[Calculate per-model cost]
30+
F --> G{PR associated?}
31+
G -->|Yes| H[Post comment on PR]
32+
G -->|No| I[Create cost report issue]
33+
F --> J{Cost over threshold?}
34+
J -->|Yes| K[Create high-spend alert issue]
35+
```
36+
37+
The workflow reads `token-usage.jsonl` from the `agent-artifacts` artifact written by
38+
gh-aw's firewall on every agent run. It calculates cost using a built-in per-model
39+
pricing table and posts the result where it is most useful — the PR that triggered the
40+
agent run, or a new issue when there is no PR.
41+
42+
Runs that do not produce an `agent-artifacts` artifact (non-agent CI workflows) are
43+
skipped silently.
44+
45+
## Usage
46+
47+
### Configuration
48+
49+
After installing, open the workflow file and update the `workflows` list under `on.workflow_run`
50+
to match the names of your agent workflows:
51+
52+
```yaml
53+
on:
54+
workflow_run:
55+
workflows: ["agent-implement", "agent-pr-fix"] # your workflow names here
56+
types:
57+
- completed
58+
```
59+
60+
To adjust the high-spend alert threshold, find the `$1.00` value in the workflow body
61+
and change it to your preferred limit.
62+
63+
After editing run `gh aw compile` to update the workflow and commit all changes to the
64+
default branch.
65+
66+
### Token data source
67+
68+
Cost Tracker reads `sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl` from the
69+
`agent-artifacts` artifact. This file is written automatically by gh-aw's firewall on
70+
every agent run. No additional configuration is needed to produce it — the data is
71+
already there if you are running gh-aw with firewall enabled (the default).
72+
73+
## Learn More
74+
75+
- [token-usage.jsonl reference](https://github.github.com/gh-aw/reference/token-usage/)
76+
- [gh-aw firewall documentation](https://github.github.com/gh-aw/reference/firewall/)
77+
- [CI Doctor workflow](ci-doctor.md) — investigate CI failures automatically
78+
79+
## Going Further
80+
81+
Cost Tracker works standalone — no external services required. For teams that want
82+
persistent run history, per-repo spend trends, and budget alerts across multiple repos,
83+
add [AgentMeter](https://agentmeter.app) to your agent workflow:
84+
85+
```yaml
86+
- uses: agentmeter/agentmeter-action@v1
87+
with:
88+
api-key: ${{ secrets.AGENTMETER_API_KEY }}
89+
```
90+
91+
AgentMeter ingests the same token data and surfaces it in a shared dashboard with
92+
per-repo trend charts and budget alerts.

‎workflows/cost-tracker.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
---
2+
description: |
3+
Automated agent cost tracker that fires after monitored workflows complete, downloads
4+
the agent-artifacts artifact written by gh-aw's firewall, parses token-usage.jsonl,
5+
calculates per-model spend, and posts a cost summary on the associated pull request.
6+
Creates a cost report issue when no pull request is found. Optionally creates a
7+
high-spend alert issue when a single run exceeds a configurable threshold.
8+
9+
on:
10+
workflow_run:
11+
workflows: ["agent-implement", "agent-pr-fix"] # Edit to match your agent workflow names
12+
types:
13+
- completed
14+
15+
permissions: read-all
16+
17+
network: defaults
18+
19+
safe-outputs:
20+
add-comment:
21+
target: "*"
22+
create-issue:
23+
title-prefix: "[cost-tracker] "
24+
labels: [automation, cost]
25+
max: 2
26+
27+
tools:
28+
github:
29+
toolsets: [default]
30+
bash: true
31+
32+
timeout-minutes: 10
33+
34+
---
35+
36+
# Agent Cost Tracker
37+
38+
You are the Agent Cost Tracker. Your job is to read the token usage data written by
39+
gh-aw's firewall after an agent workflow completes and report back what that run cost.
40+
41+
## Current Context
42+
43+
- **Repository**: ${{ github.repository }}
44+
- **Triggered by**: ${{ github.event.workflow_run.name }}
45+
- **Run**: [#${{ github.event.workflow_run.run_number }}](${{ github.event.workflow_run.html_url }})
46+
- **Run ID**: ${{ github.event.workflow_run.id }}
47+
- **Conclusion**: ${{ github.event.workflow_run.conclusion }}
48+
- **Head SHA**: ${{ github.event.workflow_run.head_sha }}
49+
50+
## Instructions
51+
52+
### Step 1: Download the agent-artifacts artifact
53+
54+
Use bash to download the `agent-artifacts` artifact from the triggering run:
55+
56+
```bash
57+
gh run download ${{ github.event.workflow_run.id }} \
58+
--name agent-artifacts \
59+
--dir /tmp/agent-artifacts \
60+
--repo ${{ github.repository }} 2>&1
61+
echo "exit: $?"
62+
```
63+
64+
**If this command fails** (the artifact does not exist), the triggering workflow did not
65+
produce cost data — it was not an agent run or the firewall was not enabled. Exit without
66+
creating any issue or comment. Do not report an error.
67+
68+
### Step 2: Parse token-usage.jsonl
69+
70+
Read the token usage file:
71+
72+
```bash
73+
cat /tmp/agent-artifacts/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl 2>/dev/null
74+
```
75+
76+
Each line is a JSON object. Example:
77+
78+
```json
79+
{"model":"claude-sonnet-4-5","input_tokens":1200,"output_tokens":340,"cache_read_input_tokens":500,"cache_creation_input_tokens":100}
80+
```
81+
82+
If the file is missing or empty, exit without creating any output.
83+
84+
### Step 3: Calculate cost
85+
86+
Aggregate token counts by model across all lines. Use this pricing table (USD per 1M tokens):
87+
88+
| Model | Input | Output | Cache write | Cache read |
89+
|-------|-------|--------|-------------|------------|
90+
| claude-opus-4-5 | $15.00 | $75.00 | $18.75 | $1.50 |
91+
| claude-sonnet-4-5 | $3.00 | $15.00 | $3.75 | $0.30 |
92+
| claude-haiku-4-5 | $0.80 | $4.00 | $1.00 | $0.08 |
93+
| claude-3-opus | $15.00 | $75.00 | $18.75 | $1.50 |
94+
| claude-3-5-sonnet | $3.00 | $15.00 | $3.75 | $0.30 |
95+
| claude-3-5-haiku | $0.80 | $4.00 | $1.00 | $0.08 |
96+
| gpt-4o | $2.50 | $10.00 | — | $1.25 |
97+
| gpt-4o-mini | $0.15 | $0.60 | — | $0.075 |
98+
| gpt-4.1 | $2.00 | $8.00 | — | $0.50 |
99+
| o3 | $10.00 | $40.00 | — | $2.50 |
100+
| o4-mini | $1.10 | $4.40 | — | $0.275 |
101+
| gemini-1.5-pro | $1.25 | $5.00 | — | — |
102+
| gemini-2.0-flash | $0.10 | $0.40 | — | — |
103+
104+
For any model not in this table, use $3.00 input / $15.00 output as a conservative fallback.
105+
106+
Cost per model = (input_tokens * input_rate + output_tokens * output_rate
107+
+ cache_creation_input_tokens * cache_write_rate
108+
+ cache_read_input_tokens * cache_read_rate) / 1_000_000
109+
110+
Total cost = sum across all models.
111+
112+
Format costs with 4 decimal places: `$0.0123`. Use `< $0.0001` if below that threshold.
113+
114+
### Step 4: Find the associated pull request
115+
116+
Check whether the triggering run is linked to a pull request:
117+
118+
```bash
119+
gh api "repos/${{ github.repository }}/actions/runs/${{ github.event.workflow_run.id }}" \
120+
--jq '.pull_requests[0].number // empty'
121+
```
122+
123+
Store the PR number if one is returned.
124+
125+
### Step 5: Post the cost report
126+
127+
Build a report using this template. Fill in `$TOTAL_COST` and the breakdown table:
128+
129+
```markdown
130+
## Agent run cost
131+
132+
| | |
133+
|---|---|
134+
| **Workflow** | ${{ github.event.workflow_run.name }} |
135+
| **Run** | [#${{ github.event.workflow_run.run_number }}](${{ github.event.workflow_run.html_url }}) |
136+
| **Conclusion** | ${{ github.event.workflow_run.conclusion }} |
137+
| **Total cost** | $TOTAL_COST |
138+
139+
<details>
140+
<summary>Token breakdown by model</summary>
141+
142+
| Model | Input | Output | Cache write | Cache read | Cost |
143+
|-------|------:|------:|------------:|-----------:|-----:|
144+
[one row per model with actual token counts and per-model cost]
145+
146+
</details>
147+
148+
*Data from [token-usage.jsonl](https://github.github.com/gh-aw/reference/token-usage/) written by gh-aw's firewall.*
149+
```
150+
151+
**If a PR number was found**: post this as a comment on that PR using the `add_comment`
152+
GitHub tool.
153+
154+
**If no PR was found**: create an issue using the `create_issue` GitHub tool. Use this
155+
title format:
156+
`[cost-tracker] ${{ github.event.workflow_run.name }} #${{ github.event.workflow_run.run_number }}: $TOTAL_COST`
157+
158+
### Step 6: High-spend alert (optional)
159+
160+
If the total cost exceeds **$1.00**, create a second issue using the `create_issue`
161+
GitHub tool with title:
162+
`[cost-tracker] High spend alert: $TOTAL_COST for ${{ github.event.workflow_run.name }}`
163+
164+
Include the full breakdown and a link to the run. The $1.00 threshold is a conservative
165+
starting point. Edit this workflow to raise or lower it to match your budget.
166+
167+
## Guidelines
168+
169+
- **Silent on non-agent runs**: If the artifact does not exist, produce no output at all.
170+
- **One report per run**: Do not create more than one comment or issue per triggering run.
171+
- **Accurate math**: Double-check token counts and cost calculations before posting.
172+
- **No retries**: If the artifact download fails with a transient error, exit silently
173+
rather than retrying — the next run will generate its own report.

0 commit comments

Comments
 (0)