⚡ Claude Token Optimization2026-07-04 — smoke-claude

[github-actions]

Target Workflow: smoke-claude

Source report: (no claude-token-usage-report issue found; analysis based on local run data at .github/aw/logs/)
Runs analyzed: 11 runs over ~3 days (2026-07-01 to 2026-07-04)
Average AIC per run: 49.35 (range: 35.5–55.7)
Total tokens per run: ~357,500 (avg: 3,299 input + 841 output + 294,293 cache_read + 52,050 cache_write)
Cache write rate: ~14.6% of total token volume — but 67.8% of estimated cost
Cache read rate: 98.9% hit rate (excellent prefix caching)
LLM turns per run: 1 (single request; max-turns: 8 is over-provisioned)
Model configured: claude-haiku-4-5 — Model actually used: claude-opus-4-8 ⚠️
Estimated cost per run (opus-4-8 pricing): ~$0.57
Estimated cost per run (haiku-4-5 pricing): ~$0.11

Current Configuration

Setting	Value
Tools loaded	1 (bash)
Tools actually used	bash (reads pre-computed JSON, calls safeoutputs)
github:	false (correct)
Network groups	defaults (60+ domains, only api.anthropic.com actually used)
Pre-agent steps	✅ Yes — smoke file, PR fetch, GitHub reachability, result JSON all pre-computed
Prompt size	7,092 bytes (~1,800 tokens)
max-turns	8 (only 1 turn ever used)
Ambient context	5,786 tokens (main branch AGENTS.md) vs 3,289 tokens (PR branches)

Recommendations

1. Investigate why claude-haiku-4-5 is being served as claude-opus-4-8

Estimated savings: 80% cost reduction ($0.46/run, ~$6.44/week at 14 runs/week)

The workflow frontmatter specifies model: claude-haiku-4-5 and the lock file confirms GH_AW_INFO_MODEL: "claude-haiku-4-5" and ANTHROPIC_MODEL: claude-haiku-4-5. However, every run's agent_usage.json shows "primary_model": "claude-opus-4-8".

This is the single largest cost driver. The models: section in the AWF config lists "claude": ["agent"] → "agent": ["sonnet-6x", "gpt-5.5", ..., "any"] — this model steering configuration appears to override the workflow's explicit haiku request and upgrade to opus.

Investigation steps:

1. Check AWF config models.agent — currently set to ["sonnet-6x", "gpt-5.5", "gpt-5.4", "gpt-5.3", "gemini-pro", "any"]. The "any" entry permits any model and may be causing haiku requests to be re-routed.
2. Verify if .github/aw/models.json or GH_AW_MODELS_JSON_PATH overrides the model at runtime.
3. Add model_override_check to the verify_token_usage job to assert primary_model contains haiku.

If haiku were used, pricing would drop from opus rates ($5/M input, $25/M output, $6.25/M cache_write) to haiku rates ($1/M input, $5/M output, $1.25/M cache_write).

2. Reduce AGENTS.md ambient context to lower cache write cost

Estimated savings: 36K tokens cache_write/run ($0.22/run at opus, ~$0.04/run at haiku)

The ambient_context field in agent_usage.json shows 5,786 tokens on main branch runs vs 3,289 tokens on PR-branch runs — a 2,497-token difference. This directly inflates cache_write by ~35,000 tokens per run (the ambient context is written to cache on every cold start).

AGENTS.md is currently 20,345 bytes / 397 lines — much of it is CI/CD internals, container architecture details, and debugging instructions that a smoke-test agent running a single bash command does not need.

Implementation:
Add a .agents/smoke-claude/ override directory (or agentsignore equivalent) with a minimal AGENTS.md stub for this workflow, OR reduce the main AGENTS.md by moving implementation-detail sections (container architecture, traffic flow diagrams, DNS config details, iptables rules, log formats) to a separate doc referenced by link.

Alternatively, use agents-context: false or a scoped AGENTS.md to exclude the full file from this workflow's context.

Evidence: Three PR-branch runs (28663868157, 28663845911, 28663732543) on SHA abd385b1 achieved:

• cache_write: 26,940 tokens (vs 61,820 on main)
• aic: 35.5 (vs 55.6 on main)

This 36% AIC reduction came entirely from the smaller ambient context on that branch.

3. Reduce max-turns from 8 to 1

Estimated savings: Minor overhead reduction; improves intent documentation

Every run completes in exactly 1 LLM turn (confirmed across all 11 runs). The workflow is fully deterministic: pre-steps compute all data, agent reads /tmp/gh-aw/agent/final-result.json, calls noop or add_comment, and stops. There is no scenario requiring a second turn.

Setting max-turns: 1 in smoke-claude.md:

max-turns: 1  # was: 8

Benefits:

• Eliminates the overhead of turn-count tracking and guardrail processing for turns 2–8
• Makes the workflow's single-turn contract explicit (documented intent)
• Prevents accidental multi-turn drift if the prompt is modified

4. Verify github: false prevents the blocked MCP probe

No token savings; prevents wasted network attempt

Every run shows 1 blocked request to awmg-mcpg:8080. This is the agent attempting to call an MCP tool (likely from the safe-outputs schema probe that the prompt explicitly warns against). With github: false, no GitHub tools are loaded, but the MCP gateway is still reachable.

The prompt already warns: "Never call add_comment or add_labels with empty arguments or as a schema probe." The blocked request appears to be the agent calling safeoutputs via MCP instead of CLI. This is already handled correctly — no change needed — but worth monitoring.

Cache Analysis (Anthropic-Specific)

All runs complete in exactly 1 LLM turn (1 API request), so there is no Turn 2 amortization.

Run	Input	Output	Cache Read	Cache Write	AIC	Note
28690081250	3,298	721	233,629	60,322	51.2	main, schedule
28682360061	3,300	858	296,098	61,820	55.6	main, schedule
28682081717	3,300	865	296,032	61,809	55.6	main, schedule
28676779124	3,300	873	296,735	61,953	55.7	main, schedule
28671133963	3,300	870	296,028	61,831	55.6	main, schedule
28668175485	3,300	852	295,986	61,814	55.6	main, schedule
28663868157	3,300	863	330,989	26,940	35.5	PR branch, smaller AGENTS.md
28663845911	3,300	872	330,940	26,933	35.6	PR branch, smaller AGENTS.md
28663732543	3,300	881	331,086	26,977	35.6	PR branch, smaller AGENTS.md
28663698562	3,300	883	296,073	61,832	55.7	main, schedule
28662229580	3,298	718	233,629	60,321	51.2	main, schedule

Cache write amortization: With 1 turn/run, cache writes are never amortized within a session. They benefit the next run (5-minute Anthropic cache TTL means consecutive schedule runs 12h apart do NOT hit cache from prior runs — cache writes are effectively wasted unless two runs occur within 5 minutes).

Cache cost vs benefit: Cache reads (294K tokens × $0.50/M = $0.147) are still cheaper than re-computing input (294K × $5/M = $1.47), so cache reads are valuable. But cache writes at $6.25/M for 62K tokens = $0.387 per run — 67.8% of total cost — for context that expires before the next run. This is the primary inefficiency.

Recommendation: Since the Anthropic cache TTL is ~5 minutes and this workflow runs every 12 hours, nearly all cache writes expire unused. The cache write cost is unavoidable (Anthropic charges for all context on first request), but reducing the context size (via AGENTS.md trimming) directly reduces this cost.

Expected Impact

Metric	Current	Projected (fix #1 only)	Projected (fix #1+#2)	Savings
Model	claude-opus-4-8	claude-haiku-4-5	claude-haiku-4-5	—
Cache write tokens/run	~62K	~62K	~27K	-35K
Estimated cost/run	~$0.57	~$0.11	~$0.07	-88%
AIC/run	49.4 avg	~18 (est)	~13 (est)	-74%
LLM turns	1	1	1	0
Session time	~5.4m	~5m	~5m	~-8%
Weekly cost (14 runs)	~$8.00	~$1.60	~$0.98	-88%

Implementation Checklist

• Improve links in readme to AW project #1 (highest impact): Investigate why claude-haiku-4-5 is being served as claude-opus-4-8
  • Check AWF models.agent steering config — "any" entry may be overriding haiku
  • Check if .github/aw/models.json overrides model at runtime
  • Add primary_model assertion in verify_token_usage job
• Secret proxying #2: Reduce AGENTS.md ambient context size
  • Move implementation-detail sections (container arch, traffic flow, DNS, iptables, log formats) to linked docs
  • Target: reduce from ~5,786 ambient tokens to ~3,289 (matching PR-branch baseline)
  • Verify by checking ambient_context in next run's agent_usage.json
• feat: add integration test for rostbuness #3: Set max-turns: 1 in smoke-claude.md (was max-turns: 8)
• After changes: recompile with gh aw compile .github/workflows/smoke-claude.md
• Post-process: npx tsx scripts/ci/postprocess-smoke-workflows.ts
• Verify CI passes on PR
• Compare agent_usage.json from new run vs baseline (target: AIC < 20, cost < $0.15/run)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Daily Claude Token Optimization Advisor · 167.9 AIC · ⊞ 6.3K · ◷