Skip to content

Commit 772a6b1

Browse files
lpcoxCopilot
andcommitted
ci(smoke): add token-usage sanity checks to smoke workflows
Add a verify_token_usage job to smoke-copilot, smoke-claude, and smoke-codex that runs after the agent job on the downloaded agent artifact and fails the workflow when token accounting looks wrong. The checker (scripts/ci/check-token-usage.js) enforces two invariants: - Internal consistency: the sum of per-response records in token-usage.jsonl must exactly equal the aggregated agent_usage.json (input/output/cache_read/cache_write). This is engine-independent. - cache_read_tokens must not be 0 across multiple responses, which is the symptom of the cached-token normalization bug. ai_credits/ambient_context drift is reported as warnings only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 8c46503 commit 772a6b1

8 files changed

Lines changed: 713 additions & 22 deletions

File tree

.github/workflows/smoke-claude.lock.yml

Lines changed: 52 additions & 20 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/smoke-claude.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,25 @@ sandbox:
2323
mcp:
2424
version: v0.3.1
2525
strict: false
26+
jobs:
27+
verify_token_usage:
28+
needs: agent
29+
if: always() && needs.agent.result != 'skipped' && needs.agent.result != 'cancelled'
30+
runs-on: ubuntu-latest
31+
permissions:
32+
contents: read
33+
steps:
34+
- name: Checkout repository
35+
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
36+
with:
37+
persist-credentials: false
38+
- name: Download agent artifact
39+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
40+
with:
41+
name: agent
42+
path: /tmp/gh-aw-agent
43+
- name: Token-usage sanity check
44+
run: node scripts/ci/check-token-usage.js --artifact-root /tmp/gh-aw-agent --engine claude
2645
tools:
2746
bash:
2847
- bash

.github/workflows/smoke-codex.lock.yml

Lines changed: 32 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/smoke-codex.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,25 @@ sandbox:
2222
mcp:
2323
version: latest
2424
strict: false
25+
jobs:
26+
verify_token_usage:
27+
needs: agent
28+
if: always() && needs.agent.result != 'skipped' && needs.agent.result != 'cancelled'
29+
runs-on: ubuntu-latest
30+
permissions:
31+
contents: read
32+
steps:
33+
- name: Checkout repository
34+
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
35+
with:
36+
persist-credentials: false
37+
- name: Download agent artifact
38+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
39+
with:
40+
name: agent
41+
path: /tmp/gh-aw-agent
42+
- name: Token-usage sanity check
43+
run: node scripts/ci/check-token-usage.js --artifact-root /tmp/gh-aw-agent --engine codex
2544
imports:
2645
- shared/gh.md
2746
- shared/reporting.md

.github/workflows/smoke-copilot.lock.yml

Lines changed: 32 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.github/workflows/smoke-copilot.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,25 @@ sandbox:
4545
mcp:
4646
version: v0.3.1
4747
strict: false
48+
jobs:
49+
verify_token_usage:
50+
needs: agent
51+
if: always() && needs.agent.result != 'skipped' && needs.agent.result != 'cancelled'
52+
runs-on: ubuntu-latest
53+
permissions:
54+
contents: read
55+
steps:
56+
- name: Checkout repository
57+
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
58+
with:
59+
persist-credentials: false
60+
- name: Download agent artifact
61+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
62+
with:
63+
name: agent
64+
path: /tmp/gh-aw-agent
65+
- name: Token-usage sanity check
66+
run: node scripts/ci/check-token-usage.js --artifact-root /tmp/gh-aw-agent --engine copilot
4867
steps:
4968
- name: Pre-compute smoke test data
5069
id: smoke-data

0 commit comments

Comments
 (0)