|
| 1 | +<!-- page-journey: all --> |
| 2 | +<!-- page-adventure: side-quest --> |
| 3 | +# Side Quest: Observe and Reduce Token Costs |
| 4 | + |
| 5 | +> _Use this activity when you want to move from “my workflow costs something” to “I know why it costs that much, and I can lower it on purpose.”_ |
| 6 | +
|
| 7 | +## 📋 Before You Start |
| 8 | + |
| 9 | +- You completed [Build Your First Event-Driven Workflow: PR Auto-Reviewer](13-pr-reviewer-workflow.md). |
| 10 | +- You have a working PR reviewer workflow or another workflow with at least 5 completed runs so you can compare before-and-after usage. |
| 11 | +- If you want extra background on AIC, audit artifacts, or budget guardrails, continue later to [Audit and Monitor Your Agentic Workflows](25-audit-and-observability.md) and [Manage Costs and AI Credit Budgets](26-manage-costs-and-budgets.md). |
| 12 | + |
| 13 | +## Build a cost baseline |
| 14 | + |
| 15 | +Start by measuring your current pattern before you change anything: |
| 16 | + |
| 17 | +```bash |
| 18 | +gh aw logs <your-workflow-id> --count 5 |
| 19 | +``` |
| 20 | + |
| 21 | +Record three things from the last five runs: |
| 22 | + |
| 23 | +| Signal | What to record | Why it matters | |
| 24 | +|---|---|---| |
| 25 | +| AIC | Average and highest run | Shows your baseline and worst case | |
| 26 | +| Conclusion | Success or failure | Failed runs still spend credits | |
| 27 | +| Model | Which model ran | Helps explain differences between runs | |
| 28 | + |
| 29 | +If one run is much higher than the others, audit it: |
| 30 | + |
| 31 | +```bash |
| 32 | +gh aw audit <run-id> --parse |
| 33 | +``` |
| 34 | + |
| 35 | +Then inspect: |
| 36 | + |
| 37 | +- `log.md` for long agent turns or repeated reasoning |
| 38 | +- `agent_usage.json` for token totals |
| 39 | +- `mcp-logs/` for repeated tool calls |
| 40 | +- `firewall.md` for blocked domains that may have forced retries or fallback behavior |
| 41 | + |
| 42 | +## Match cost symptoms to likely causes |
| 43 | + |
| 44 | +Use your baseline to decide what to change first: |
| 45 | + |
| 46 | +| If you observe this | Check for this cause | First fix to try | |
| 47 | +|---|---|---| |
| 48 | +| AIC grows after you add more repository data | Too much raw context in the brief | Pre-filter the data in a deterministic step before passing it to the agent | |
| 49 | +| One run is much higher than the others | The agent explored too broadly or retried tool calls | Tighten the brief and remove tools the task does not need | |
| 50 | +| Every run costs about the same and feels high | The brief is longer than it needs to be | Shorten instructions, examples, and repeated boilerplate | |
| 51 | +| Costs spike after adding a new schedule | The workflow runs more often than the value it creates | Reduce the schedule frequency or add conditions so no-op runs skip the agent | |
| 52 | +| The workflow keeps talking about the same items | The agent re-processes unchanged data every run | Add [persistent memory](20-persistent-memory.md) or a deterministic diff step | |
| 53 | + |
| 54 | +## Use the highest-leverage reduction techniques |
| 55 | + |
| 56 | +Apply one change at a time so you can see which technique helped. |
| 57 | + |
| 58 | +### Pass less context to the model |
| 59 | + |
| 60 | +The cheapest token is the one you never send. |
| 61 | + |
| 62 | +- Replace raw issue or PR dumps with a deterministic summary step. |
| 63 | +- Pass only the fields the agent needs. |
| 64 | +- Limit history windows when a full backlog is unnecessary. |
| 65 | + |
| 66 | +### Make the brief more specific |
| 67 | + |
| 68 | +Vague prompts often cost more because the agent explores, retries, or writes too much. |
| 69 | + |
| 70 | +- State the exact output shape you want. |
| 71 | +- Tell the agent what not to do. |
| 72 | +- Remove duplicate instructions and long examples once the pattern is clear. |
| 73 | + |
| 74 | +### Reduce unnecessary runs |
| 75 | + |
| 76 | +If the workflow does not need to run, the cheapest run is zero AIC. |
| 77 | + |
| 78 | +- Lower the schedule frequency. |
| 79 | +- Add `if:` conditions around setup steps so you only call the agent when new data exists. |
| 80 | +- Use `workflow_dispatch` for occasional manual analysis instead of a frequent schedule. |
| 81 | + |
| 82 | +### Avoid re-processing unchanged work |
| 83 | + |
| 84 | +Repeated work is repeated cost. |
| 85 | + |
| 86 | +- Use [cache-memory or repo-memory](20-persistent-memory.md) to remember what was already handled. |
| 87 | +- Store identifiers, timestamps, or hashes so the agent can skip items it has already seen. |
| 88 | +- Combine memory with a deterministic pre-filter for the biggest savings. |
| 89 | + |
| 90 | +### Keep tool usage narrow |
| 91 | + |
| 92 | +Extra tool calls can increase cost indirectly by extending the turn and adding more reasoning. |
| 93 | + |
| 94 | +- Expose only the tools the workflow needs. |
| 95 | +- Prefer one targeted MCP query over several broad ones. |
| 96 | +- When possible, fetch structured data in a deterministic step and let the agent interpret it. |
| 97 | + |
| 98 | +### Compare quality before choosing a more expensive setup |
| 99 | + |
| 100 | +Higher cost is only justified when it improves the outcome enough to matter. |
| 101 | + |
| 102 | +- Compare prompt variants with [A/B experiments](23-ab-experiments.md). |
| 103 | +- If your organisation supports multiple models, compare a lower-cost option against your current workflow before standardising on the more expensive one. |
| 104 | + |
| 105 | +## Add hard guardrails |
| 106 | + |
| 107 | +After you reduce cost, keep it reduced: |
| 108 | + |
| 109 | +- Use `max-ai-credits` to cap a single run. |
| 110 | +- Use `max-daily-ai-credits` to cap 24-hour usage. |
| 111 | +- Use `timeout-minutes` to stop unusually long runs. |
| 112 | +- Use [gh aw forecast](side-quest-26-01-forecast-costs.md) to size the guardrails from real history instead of guessing. |
| 113 | + |
| 114 | +## Try it yourself |
| 115 | + |
| 116 | +### Run one optimization cycle |
| 117 | + |
| 118 | +1. Pick your PR reviewer workflow (or another workflow) and copy the average AIC from your last five runs. |
| 119 | +2. Choose one technique from this page. |
| 120 | +3. Make exactly one change to your workflow. |
| 121 | +4. Compile your workflow: |
| 122 | + |
| 123 | + ```bash |
| 124 | + gh aw compile |
| 125 | + ``` |
| 126 | + |
| 127 | +5. Run the workflow at least two more times. |
| 128 | +6. Compare the new average AIC with your baseline. |
| 129 | +7. Keep the change only if quality still meets your bar. |
| 130 | + |
| 131 | +Use this quick notes table: |
| 132 | + |
| 133 | +| Baseline average AIC | Change you made | New average AIC | Quality stayed acceptable? | |
| 134 | +|---|---|---|---| |
| 135 | +| | | | | |
| 136 | + |
| 137 | +### Ask an agent to suggest the next optimization |
| 138 | + |
| 139 | +Open Copilot Chat or the Agents tab in your practice repository and send: |
| 140 | + |
| 141 | +```text |
| 142 | +/agentic-workflows Review my workflow brief and this audit summary. |
| 143 | +Identify the single change most likely to reduce AIC without hurting output quality. |
| 144 | +Explain why that change is the best next step, then apply it and run gh aw compile. |
| 145 | +``` |
| 146 | + |
| 147 | +Paste the relevant excerpt from your `gh aw audit --parse` output below the prompt. |
| 148 | + |
| 149 | +## ✅ Checkpoint |
| 150 | + |
| 151 | +- [ ] You collected a five-run AIC baseline for one workflow |
| 152 | +- [ ] You audited at least one unusually expensive run |
| 153 | +- [ ] You identified whether your biggest cost driver was context size, run frequency, repeated work, or tool usage |
| 154 | +- [ ] You applied exactly one optimization technique and re-ran the workflow |
| 155 | +- [ ] You compared the new AIC average with your baseline |
| 156 | +- [ ] You added or confirmed `max-ai-credits`, `max-daily-ai-credits`, or `timeout-minutes` |
| 157 | +- [ ] You can name the next optimization you would test if cost is still too high |
| 158 | + |
| 159 | +<!-- journey: all --> |
| 160 | +Return to [Build Your First Event-Driven Workflow: PR Auto-Reviewer](13-pr-reviewer-workflow.md). |
| 161 | +<!-- /journey --> |
0 commit comments