Skip to content

Commit b924ab0

Browse files
Copilotmnkiefer
andauthored
fix: consolidate token workflows with upstream AIC updates
Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>
1 parent 864a83e commit b924ab0

9 files changed

Lines changed: 201 additions & 193 deletions

.github/workflows/agentic-token-audit.lock.yml

Lines changed: 39 additions & 39 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/agentic-token-audit.md

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
description: Daily audit of AI credit spend across all agentic workflows with historical trend tracking
2+
description: Daily audit of AI Credit (AIC) usage across all agentic workflows with historical trend tracking
33
on:
44
schedule:
55
- cron: "daily around 12:00 on weekdays"
@@ -9,6 +9,10 @@ permissions:
99
actions: read
1010
issues: read
1111
pull-requests: read
12+
network:
13+
allowed:
14+
- defaults
15+
- python
1216
tracker-id: agentic-token-audit
1317
safe-outputs:
1418
create-issue:
@@ -25,12 +29,12 @@ tools:
2529
- "*"
2630
repo-memory:
2731
branch-name: "memory/token-audit"
28-
description: "Historical daily workflow token usage snapshots (shared with agentic-token-optimizer)"
32+
description: "Historical daily workflow AIC snapshots (shared with agentic-token-optimizer)"
2933
file-glob: ["*.json", "*.jsonl", "*.csv", "*.md"]
3034
max-file-size: 102400
3135
max-patch-size: 51200
3236
steps:
33-
- name: Setup Python runtime
37+
- name: Setup Python
3438
uses: actions/setup-python@v6.2.0
3539
with:
3640
python-version: "3.12"
@@ -71,15 +75,15 @@ steps:
7175
timeout-minutes: 25
7276
---
7377

74-
# Daily Agentic Workflow AI Credit Spend Audit
78+
# Daily Agentic Workflow AIC Usage Audit
7579

76-
You are the Agentic Workflow Auditor — a workflow that tracks daily AI credit spend and token consumption across all agentic workflows in this repository and maintains a historical record for trend analysis.
80+
You are the Agentic Workflow Auditor — a workflow that tracks daily AI Credit (AIC) spend and token consumption across all agentic workflows in this repository and maintains a historical record for trend analysis.
7781

7882
## Mission
7983

80-
1. Parse the pre-downloaded agentic workflow logs and compute per-workflow AI credit spend and token usage metrics.
84+
1. Parse the pre-downloaded agentic workflow logs and compute per-workflow AIC spend and token usage metrics.
8185
2. Persist today's snapshot to repo-memory so the optimizer (and future runs of this audit) can read historical data.
82-
3. Publish a concise audit issue summarizing today's AI credit spend and trend highlights.
86+
3. Publish a concise audit issue summarizing today's AIC spend and trend highlights.
8387

8488
## Data Sources
8589

@@ -103,9 +107,9 @@ Each element of `.runs` is a `RunData` object with (among others):
103107
|---|---|---|
104108
| `workflow_name` | string | Human-readable name |
105109
| `workflow_path` | string | `.github/workflows/....lock.yml` |
106-
| `ai_credits` | float | AI credits consumed (primary billing metric; 1 AIC = $0.01 USD) |
110+
| `aic` | float | AI Credits (AIC) consumed (primary billing metric; 1 AIC = $0.01 USD) |
107111
| `token_usage` | int | Total tokens (`omitempty` — treat missing/null as 0) |
108-
| `effective_tokens` | int | Legacy normalized token metric (deprecated; use `ai_credits` for billing) |
112+
| `effective_tokens` | int | Legacy normalized token metric (deprecated; use `aic` for billing) |
109113
| `action_minutes` | float | Billable GitHub Actions minutes |
110114
| `turns` | int | Number of agent turns |
111115
| `duration` | string | Human-readable duration |
@@ -163,7 +167,7 @@ Write a Python script to `/tmp/gh-aw/token-audit/process_audit.py` and run it. T
163167
}
164168
```
165169

166-
Handle null/missing `ai_credits` and `token_usage` by treating them as 0.
170+
Handle null/missing `aic` and `token_usage` by treating them as 0.
167171

168172
## Phase 2 — Persist Snapshot to Repo-Memory
169173

@@ -258,7 +262,7 @@ Summarize AI credit, token, and active-workflow changes from `rolling-summary.js
258262

259263
## Important Notes
260264

261-
- Use `// 0` (null coalescing) in jq and `.get(field, 0)` in Python for nullable numeric fields (`ai_credits`, `token_usage`).
265+
- Use `// 0` (null coalescing) in jq and `.get(field, 0)` in Python for nullable numeric fields (`aic`, `token_usage`).
262266
- Distinguish between these two cases in the issue:
263267
- the raw `.runs` array is empty
264268
- the raw `.runs` array is non-empty but none of the runs are `status == "completed"`

.github/workflows/agentic-token-optimizer.lock.yml

Lines changed: 30 additions & 30 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/agentic-token-optimizer.md

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
description: Daily optimizer that identifies high-AI-credit-spend agentic workflows, audits their runs, and recommends efficiency improvements including inline sub-agent refactors when warranted
2+
description: Daily optimizer that identifies a high-AIC agentic workflow, audits its runs, and recommends efficiency improvements including inline sub-agent refactors when warranted
33
on:
44
schedule:
55
- cron: "daily around 14:00 on weekdays"
@@ -18,7 +18,7 @@ tools:
1818
- "*"
1919
repo-memory:
2020
branch-name: "memory/token-audit"
21-
description: "Historical daily workflow token usage snapshots (shared with agentic-token-audit)"
21+
description: "Historical daily workflow AIC snapshots (shared with agentic-token-audit)"
2222
file-glob: ["*.json", "*.jsonl", "*.csv", "*.md"]
2323
max-file-size: 102400
2424
max-patch-size: 51200
@@ -58,7 +58,7 @@ steps:
5858
echo '{"runs":[],"summary":{}}' > /tmp/gh-aw/token-audit/all-runs.json
5959
fi
6060
61-
- name: Aggregate top workflows by token usage
61+
- name: Aggregate top workflows by AIC usage
6262
run: |
6363
set -euo pipefail
6464
mkdir -p /tmp/gh-aw/token-audit
@@ -71,7 +71,7 @@ steps:
7171
| select(.status == "completed")
7272
| {
7373
workflow_name: .workflow_name,
74-
ai_credits: (.ai_credits // 0),
74+
ai_credits: (.aic // 0),
7575
tokens: (.token_usage // 0),
7676
turns: (.turns // 0),
7777
action_minutes: (.action_minutes // 0)
@@ -110,14 +110,14 @@ steps:
110110
fi
111111
---
112112

113-
# Agentic Workflow AI Credit Spend Optimizer
113+
# Agentic Workflow AIC Usage Optimizer
114114

115-
You are the Agentic Workflow Optimizer. Pick one high-AI-credit-spend workflow, audit recent runs, and create a conservative optimization issue with measurable improvements. Your recommendations may include prompt, tool, reliability, setup-prefix, and inline sub-agent improvements when the evidence supports them.
115+
You are the Agentic Workflow Optimizer. Pick one high-AIC workflow, audit recent runs, and create a conservative optimization issue with measurable improvements. Your recommendations may include prompt, tool, reliability, setup-prefix, and inline sub-agent improvements when the evidence supports them.
116116

117117
## Objectives
118118

119119
1. Select one workflow using repo-memory and pre-aggregated data.
120-
2. Analyze tokens, turns, errors, tool usage patterns, and prompt structure across multiple runs.
120+
2. Analyze AIC, tokens, turns, errors, tool usage patterns, and prompt structure across multiple runs.
121121
3. Propose safe, high-impact optimizations with evidence, including inline sub-agent refactors only when they are a clear fit.
122122
4. Publish one issue and update optimization history.
123123

@@ -151,11 +151,11 @@ Prefer `--jq` on `gh api` calls over a separate `| jq` step when the filter is s
151151
## Data Inputs
152152

153153
- `/tmp/gh-aw/token-audit/all-runs.json`: full 7-day run data (`gh aw logs --json`).
154-
- `/tmp/gh-aw/token-audit/top-workflows.json`: pre-aggregated top 10 workflows by total AI credits.
154+
- `/tmp/gh-aw/token-audit/top-workflows.json`: pre-aggregated top 10 workflows by total AIC.
155155
- `/tmp/gh-aw/repo-memory/default/YYYY-MM-DD.json`: daily audit snapshots.
156156
- `/tmp/gh-aw/repo-memory/default/optimization-log.json`: prior optimizations (if present).
157157

158-
Treat missing numeric fields (`ai_credits`, `token_usage`, `turns`, `action_minutes`) as `0`.
158+
Treat missing numeric fields (`aic`, `token_usage`, `turns`, `action_minutes`) as `0`.
159159

160160
## Phase 1 — Select Target
161161

@@ -168,7 +168,7 @@ Treat missing numeric fields (`ai_credits`, `token_usage`, `turns`, `action_minu
168168
Then collect run-level data for the selected workflow:
169169

170170
- run count
171-
- total and average AI credits
171+
- total and average AIC
172172
- total and average tokens
173173
- total and average turns
174174
- conclusions/error patterns
@@ -180,8 +180,8 @@ Use this compact analysis matrix:
180180
| Area | Required checks | Output |
181181
|---|---|---|
182182
| Tool usage | Compare configured tools from workflow source vs observed usage across multiple runs | Keep / Consider removing / Remove |
183-
| AI credit spend | Evaluate AI credits, token totals, cache efficiency, turns | Top spend drivers |
184-
| Reliability | Repeated errors, warnings, retries, missing tools | AI credit waste from failures |
183+
| AI credit spend | Evaluate AIC, token totals, cache efficiency, turns | Top spend drivers |
184+
| Reliability | Repeated errors, warnings, retries, missing tools | AIC waste from failures |
185185
| Prompt efficiency | Redundant instructions, overlong sections, avoidable iteration | Prompt reduction opportunities |
186186
| Structural optimization | Repeated setup/tool-call prefixes and sections suited for inline sub-agents | Extract setup / Add sub-agent / Keep in main agent |
187187

@@ -283,10 +283,10 @@ Create one issue with:
283283

284284
- **Target workflow + reason selected**
285285
- **Analysis period + runs analyzed**
286-
- **Spend profile table** (total AI credits, avg AI credits/run, total tokens, avg turns/run, cache efficiency)
286+
- **Spend profile table** (total AIC, avg AIC/run, total tokens, avg turns/run, cache efficiency)
287287
- **Ranked recommendations** with:
288288
- title
289-
- estimated AI credit savings per run
289+
- estimated AIC savings per run
290290
- concrete action
291291
- evidence from observed runs
292292
- **Optional structural optimizations** for shared setup prefixes and inline sub-agents when supported by the analysis
@@ -304,7 +304,7 @@ Create one issue with:
304304

305305
Append one entry to `/tmp/gh-aw/repo-memory/default/optimization-log.json`:
306306

307-
`{"date":"YYYY-MM-DD","workflow_name":"...","total_ai_credits_analyzed":F,"total_tokens_analyzed":N,"runs_audited":N,"recommendations_count":N,"subagent_candidates":N,"estimated_ai_credit_savings_per_run":F}`
307+
`{"date":"YYYY-MM-DD","workflow_name":"...","total_ai_credits_analyzed":F,"total_tokens_analyzed":N,"runs_audited":N,"recommendations_count":N,"subagent_candidates":N,"estimated_aic_savings_per_run":F}`
308308

309309
Use `subagent_candidates` for the count of inline sub-agent candidates you actually recommend in the issue body.
310310

README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ It is built for platform engineers, developer productivity teams, and repository
1010
## Key Features
1111

1212
- **Clear operational visibility** with a daily audit that captures AI credit spend, usage trends, and workflow-level hotspots.
13-
- **Actionable optimization guidance** that identifies high-AI-credit-spend workflows and proposes safe, conservative improvements.
13+
- **Actionable optimization guidance** that identifies high-AIC workflows and proposes safe, conservative improvements.
1414
- **Faster efficiency improvements** by helping teams find waste before it becomes recurring operational overhead.
1515
- **Built for real GitHub workflows** using GitHub Agentic Workflows, so installation and adoption fit naturally into existing repositories.
1616
- **Useful historical context** through shared snapshots that support trend analysis instead of one-off debugging.
@@ -50,8 +50,8 @@ Included workflows:
5050

5151
| Workflow | What it does |
5252
| ----- | --- |
53-
| [`Daily Agentic Workflow AI Credit Spend Audit`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-audit.md?plain=1) | Collects recent agentic workflow usage and creates a daily AI credit spend snapshot. |
54-
| [`Agentic Workflow AI Credit Spend Optimizer`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-optimizer.md?plain=1) | Analyzes high-AI-credit-spend workflows and proposes conservative efficiency changes, including inline sub-agent opportunities when they are a strong fit. |
53+
| [`Daily Agentic Workflow AIC Usage Audit`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-audit.md?plain=1) | Collects recent agentic workflow usage and creates a daily AIC spend snapshot. |
54+
| [`Agentic Workflow AIC Usage Optimizer`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-optimizer.md?plain=1) | Analyzes high-AIC workflows and proposes conservative efficiency changes, including inline sub-agent opportunities when they are a strong fit. |
5555

5656
## License
5757

0 commit comments

Comments
 (0)