Skip to content

Commit 0817302

Browse files
feat(tools): add tier-advisor CLI (cnighswonger#244)
Recommends cache TTL tier upgrades/downgrades from measured session behavior rather than guesswork. Verified locally merged onto current main: full suite 1487/0. Closes cnighswonger#63
1 parent 81f47a5 commit 0817302

9 files changed

Lines changed: 1855 additions & 0 deletions

README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,10 @@ All proxy settings are controlled via environment variables. Set them before sta
241241
| `CACHE_FIX_DEBUG` | `0` | Enable debug logging |
242242
| `CACHE_FIX_HOT_RELOAD` | unset | Set to `on` to enable in-process extension hot-reload. Off by default as of v4.0.0 — see [Upgrading from v3.x](#upgrading-from-v3x) for details and the supervisor restart flow. |
243243
| `CACHE_FIX_READ_DEDUPE` | unset | Set to `1` to dedupe repeat `Read` tool results that re-appear unchanged across turns. Keeps the first occurrence intact; replaces later byte-identical ones (keyed on `file_path` + content + `offset` + `limit`) with a stable pointer line. Default-off; opt in per session to validate before broader rollout. See [extension impact guide](docs/extension-impact-guide.md). |
244+
| `CACHE_FIX_ADVISOR_PLAN` | unset | Plan override for `tools/tier-advisor.mjs` — one of `max-5x`, `max-20x`, `pro`. Bypasses heuristic plan detection. See [Tier advisor](docs/tier-advisor.md). |
245+
| `CACHE_FIX_ADVISOR_UPGRADE_THRESHOLD` | `80` | Projected-Q7d percent that triggers an `UPGRADE` recommendation from tier-advisor. |
246+
| `CACHE_FIX_ADVISOR_DOWNGRADE_THRESHOLD` | `20` | Projected-Q7d percent that triggers a `DOWNGRADE` recommendation from tier-advisor (paired with the `DOWNGRADE_WEEKS` consecutive-weeks gate). |
247+
| `CACHE_FIX_ADVISOR_DOWNGRADE_WEEKS` | `2` | Consecutive completed weeks under the downgrade threshold required before tier-advisor recommends downgrade. Single-week dips never trigger; single-week spikes DO trigger upgrade (cost-of-being-throttled asymmetry). |
244248

245249
### Corporate environments (proxies, custom CAs)
246250

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Review: PR #244 tier-advisor implementation
2+
3+
Date: 2026-06-27
4+
Reviewed: PR #244 (`feature/tier-advisor-impl`) at `21a1d48e91bed2eaf55e2527fc76bcb738399587`
5+
Round: 2
6+
Label applied: changes-requested
7+
8+
## What Is Correct
9+
10+
The round 1 fallback blocker is fixed for the high-burn cases that failed previously. In the stale `account.json` path, the advisor now exits `1`, reports `burn_rate_source: "log"`, sets `current_q7d_pct` from the weighted log sum (`98.0392156862745` in my reproduction), projects above the upgrade threshold, and recommends `upgrade`. In the missing `account.json` path, the same high-burn fixture also exits `1`, reports `burn_rate_source: "log"`, sets `current_q7d_pct` from the log, and recommends `upgrade`.
11+
12+
Actual round 1 reproduction rerun against this HEAD:
13+
14+
```text
15+
stale account.json + high usage.jsonl:
16+
exit=1
17+
current_q7d_pct=98.0392156862745
18+
burn_rate_source=log
19+
projected_q7d_at_reset=114.37901568682078
20+
recommendation=upgrade
21+
22+
missing account.json + high usage.jsonl:
23+
exit=1
24+
current_q7d_pct=98.0392156862745
25+
burn_rate_source=log
26+
projected_q7d_at_reset=99.51359014876492
27+
recommendation=upgrade
28+
```
29+
30+
The single-source state persistence path is also fixed for fallback runs that cross a week boundary. A stale-header/log-fallback run wrote a completed `weeks[]` entry with `q7d_actual_at_reset: 98.0392156862745`, `under_downgrade: false`, and `tier_assumed: "max-5x"` rather than carrying the stale header's `5%` value into state (`tools/tier-advisor.mjs:519-528`).
31+
32+
The round 1 attention items are addressed coherently. `parseArgs` no longer has `opts.week` or `--week` parsing (`tools/tier-advisor.mjs:57-74`), the help text only lists shipped flags (`tools/tier-advisor.mjs:661-666`), and the docs now say historical-week analysis is deferred (`docs/tier-advisor.md:49-59`). The `recentQ5hBudgetTokens()` stub is now documented honestly in code and docs (`tools/tier-advisor.mjs:310-318`, `docs/tier-advisor.md:140-147`). The stray FIRST-keeper phrase is gone from `docs/monitoring.md:23-27`.
33+
34+
Targeted edge cases requested in this round:
35+
36+
```text
37+
fallback state persistence after boundary:
38+
exit=1
39+
state.last_recommendation=tier:upgrade
40+
state.weeks[0].q7d_actual_at_reset=98.0392156862745
41+
42+
missing account.json + zero-entry usage.jsonl:
43+
exit=0
44+
current_q7d_pct=0
45+
burn_rate_source=log
46+
recommendation=ok
47+
state.last_recommendation=tier:ok
48+
49+
missing account.json + high usage.jsonl + CACHE_FIX_ADVISOR_PLAN=pro:
50+
exit=3
51+
recommendation=unknown
52+
state_exists=no
53+
```
54+
55+
Verification passed:
56+
57+
```text
58+
npm test
59+
tests 1392
60+
pass 1392
61+
fail 0
62+
```
63+
64+
## Blockers
65+
66+
1. `tier:unknown` / `pro` recommendations return before state persistence, so the documented statusline integration cannot display them.
67+
68+
`tools/tier-advisor.mjs:462-465` and `tools/tier-advisor.mjs:484-485` call `emitUnknown()` before the state load/persist block at `tools/tier-advisor.mjs:494-579`. That means an advisor run that exits `3` never writes `last_recommendation: "tier:unknown"`. The statusline only reads persisted state (`tools/quota-statusline.sh:238-250`), and the docs promise that it appends `tier:upgrade` / `tier:downgrade` / `tier:unknown` from `~/.claude/tier-advisor-state.json` after the advisor runs (`docs/tier-advisor.md:74-78`).
69+
70+
I reproduced this with missing `account.json`, high `usage.jsonl`, and `CACHE_FIX_ADVISOR_PLAN=pro`: the CLI correctly returned exit `3` with JSON `recommendation: "unknown"`, but no state file was written. The same early-return behavior also applies to `planRes.plan === "unknown"`, and the current test only checks stdout/exit for that case (`test/tier-advisor.test.mjs:569-582`), not state persistence.
71+
72+
Fix options: either persist `last_run` and `last_recommendation: "tier:unknown"` for exit-3 runs when `--no-state` is not set, or explicitly remove `tier:unknown` from the statusline/docs contract. Given the existing statusline allowlist includes `tier:unknown`, persisting the state is the smaller behavioral fix.
73+
74+
## What Needs Attention
75+
76+
None beyond the blocker above.
77+
78+
## Bloat / Non-Functional
79+
80+
None. The fallback fix stays local to the advisor math/state path, and the new regression coverage is targeted.
81+
82+
## Recommendations
83+
84+
Add regression coverage for both unknown exits: no plan override plus available inputs, and `CACHE_FIX_ADVISOR_PLAN=pro`. The assertions should cover exit `3`, JSON `recommendation: "unknown"`, and the persisted state behavior chosen by the contract. If the intended behavior is to suppress statusline `tier:unknown`, update `tools/quota-statusline.sh` and `docs/tier-advisor.md` together so the CLI and prompt contracts do not diverge.
85+
86+
## Bottom Line
87+
88+
REQUEST_CHANGES. The original fallback math bug is fixed and the requested tests pass, but the newly checked fallback/pro/unknown path exposes a state-persistence gap: `tier:unknown` is documented and wired into the statusline reader, yet exit-3 runs never write the state needed for the statusline to show it.
89+
90+
— Codex review
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Review: PR #244 tier-advisor implementation
2+
3+
Date: 2026-06-27
4+
Reviewed: PR #244 (`feature/tier-advisor-impl`) at `c85a3f4ac60dc97869b0d9282478940790df4627`
5+
Round: 3
6+
Label applied: approved-by-codex-agent
7+
8+
## What Is Correct
9+
10+
The round 2 blocker is fixed. Both exit-3 paths now call `emitUnknown(opts, writeOutput, planRes, statePath)`, so the state writer can update `last_run` and `last_recommendation: "tier:unknown"` before returning (`tools/tier-advisor.mjs:462-485`, `tools/tier-advisor.mjs:637-676`).
11+
12+
I reran the prior reproductions against `c85a3f4`:
13+
14+
```text
15+
heuristic unknown path:
16+
exit=3
17+
recommendation=unknown
18+
state.last_recommendation=tier:unknown
19+
loadAdvisorState(...).last_recommendation=tier:unknown
20+
21+
pro plan via log fallback:
22+
exit=3
23+
recommendation=unknown
24+
state.last_recommendation=tier:unknown
25+
loadAdvisorState(...).last_recommendation=tier:unknown
26+
27+
--no-state unknown path:
28+
exit=3
29+
recommendation=unknown
30+
state_exists=false
31+
```
32+
33+
The new regression tests cover the important contract points: heuristic unknown persists, pro-plan log fallback persists, and `--no-state` still avoids state writes (`test/tier-advisor.test.mjs:592-645`). The state loader accepts `tier:unknown` cleanly because `loadAdvisorState()` preserves `last_recommendation` as a string and does not restrict it to upgrade/downgrade/ok (`tools/tier-advisor.mjs:199-209`). The statusline allowlist already includes `tier:unknown` (`tools/quota-statusline.sh:238-250`).
34+
35+
Full verification passed:
36+
37+
```text
38+
npm test
39+
tests 1395
40+
pass 1395
41+
fail 0
42+
```
43+
44+
## Blockers
45+
46+
None.
47+
48+
## What Needs Attention
49+
50+
The malformed-state edge is intentionally tolerant on the unknown path. I verified that if `CACHE_FIX_ADVISOR_STATE` points at malformed JSON and the advisor otherwise resolves to `tier:unknown`, the CLI exits `3`, prints the unknown recommendation, and leaves the malformed file untouched:
51+
52+
```text
53+
malformed state + unknown path:
54+
exit=3
55+
recommendation=unknown
56+
state_still_malformed=true
57+
```
58+
59+
That differs from normal recommendation paths, where malformed state still returns the hard-error exit `4` (`tools/tier-advisor.mjs:494-504`). I am not treating this as a blocker because this PR's stated round 3 behavior deliberately keeps unknown-path persistence failures non-fatal, and this case is not worse than the previous exit-3 behavior that did not read or write state at all. The next run with a known plan still hits the normal hard-error path and forces the malformed state file to be repaired.
60+
61+
## Bloat / Non-Functional
62+
63+
None. The fix is scoped to the existing `emitUnknown()` helper and the regression tests are targeted.
64+
65+
## Recommendations
66+
67+
No required changes before merge. If the team wants the docs to be exact about this edge, add a note that malformed state is exit `4` for normal analysis paths, while the plan-undetectable exit `3` path reports the plan problem first and treats state persistence as best-effort.
68+
69+
## Bottom Line
70+
71+
APPROVE. The round 2 state-persistence blocker is resolved, `tier:unknown` survives reload through the state schema, the `--no-state` guard is preserved, and the full test suite passes with 1395 tests.
72+
73+
— Codex review
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Review: PR #244 tier-advisor implementation
2+
3+
Date: 2026-06-25
4+
Reviewed: PR #244 (`feature/tier-advisor-impl`) at `22a8175cbc67a4ec696c512c7560f790169c5d78`
5+
Round: 1
6+
Label applied: changes-requested
7+
8+
## What Is Correct
9+
10+
The implementation lands in the requested shape: `tools/tier-advisor.mjs` is a standalone CLI with no proxy extension, and `tools/quota-statusline.sh` only reads the persisted advisor state and appends a single `tier:` token. The statusline change preserves the existing heredoc security model: hook input is still read from `CC_INPUT` via `os.environ`, and the new tier-advisor state read is a fixed path with no shell interpolation.
11+
12+
The modern quota path pin is correct. The advisor defaults to `~/.claude/quota-status/account.json` and does not read the preload-era `~/.claude/quota-status.json`; tests override the path with `CACHE_FIX_ADVISOR_QUOTA_STATUS`.
13+
14+
The focused tests cover the prior directive blockers at the helper/API level: single exit-code mapping across default / `--json` / `--quiet`, bounded newest-first `weeks[]`, no duplicate same-week records, projection cap at 200%, additive overage-warning enrichment, and statusline omission for `tier:ok`. Test 30's structural comparison that ignores sub-millisecond float drift is acceptable because those fields are intentionally `now`-dependent.
15+
16+
## Blockers
17+
18+
1. `usage.jsonl` fallback does not produce a valid projection and can silently return `tier:ok` on high burn.
19+
20+
In `tools/tier-advisor.mjs:425-442`, the fallback branch computes a log-derived `burnRate` and sets `burnRateSource = "log"`, but `tools/tier-advisor.mjs:456-459` only projects when `q7dPct` came from `account.json`. If `account.json` is missing, projection becomes `0`; if `account.json` is stale, the projection combines stale header Q7d percent with log-derived burn. Both cases violate the directive's binary source rule: primary = fresh header, fallback = usage log, never blend.
21+
22+
Reproduction against this PR:
23+
24+
```bash
25+
tmp=$(mktemp -d)
26+
node -e 'console.log(JSON.stringify({ts: new Date().toISOString(), usage:{input_tokens:200000000, cache_creation_input_tokens:0, cache_read_input_tokens:0}}))' > "$tmp/usage.jsonl"
27+
CACHE_FIX_ADVISOR_QUOTA_STATUS="$tmp/missing-account.json" \
28+
CACHE_FIX_ADVISOR_USAGE_LOG="$tmp/usage.jsonl" \
29+
CACHE_FIX_ADVISOR_STATE="$tmp/state.json" \
30+
CACHE_FIX_ADVISOR_PLAN=max-5x \
31+
node tools/tier-advisor.mjs --json
32+
```
33+
34+
The output records `burn_rate_source: "log"` and `burn_rate_per_hour: 0.92...`, but `current_q7d_pct: null`, `projected_q7d_at_reset: 0`, `recommendation: "ok"`, and exit `0`. A 200M weighted-token current-week log on a 204M plan is not a hold signal. The fallback path needs to derive the current-week consumed percent from the same log token sum used for burn rate, then project from that value and persist week observations from that same source. Add a test where `account.json` is absent or stale, `usage.jsonl` alone implies an upgrade, and the recommendation/exit code are `upgrade`/`1`.
35+
36+
## What Needs Attention
37+
38+
- `--week` is documented as historical analysis (`tools/tier-advisor.mjs:628`) and parsed (`tools/tier-advisor.mjs:72-73`), but the parsed value is not used by `runAdvisor`. The current test only asserts parse acceptance, despite the test name saying it affects analysis. If historical analysis is still in scope, implement it; otherwise remove the behavior claim from help/docs and make the test name honest.
39+
40+
- Runtime heuristic plan detection is currently a stub: `recentQ5hBudgetTokens()` always returns `null`, so the CLI's actual plan order is CLI override, env override, fallback unknown. That can be acceptable only if the fallback is intentionally the v1 behavior, but it does not match the directive checklist's stated heuristic step. Either wire a real heuristic from available quota/usage data or document the limitation consistently in the PR docs and tests.
41+
42+
- `docs/monitoring.md:27` mentions a "FIRST-keeper byte-stability guarantee" that belongs to read-dedupe, not the tier-advisor state history. This is a docs copy/paste artifact and should be corrected while the PR is already open.
43+
44+
## Bloat / Non-Functional
45+
46+
None. The CLI is intentionally self-contained and uses Node built-ins only; the size is mostly tests and explicit formatting/state helpers, not unnecessary abstraction.
47+
48+
## Recommendations
49+
50+
Keep the fallback math single-source: have the usage-log path return both `currentPct` and `burnRate`, set `current_q7d_pct` from the log-derived percent, and avoid carrying stale/missing header utilization into projection or state. That will also make the `burn_rate_source` field auditably true.
51+
52+
After the fallback fix, add a regression test for stale `account.json` plus high `usage.jsonl` to prove the stale header percent is not blended with log burn. The existing missing-both-inputs test does not exercise the fallback recommendation path.
53+
54+
## Bottom Line
55+
56+
REQUEST_CHANGES. The implementation is close and the statusline/security pieces look sound, but the fallback burn-rate path currently violates the load-bearing single-source contract and can suppress an upgrade recommendation exactly when `account.json` is unavailable or stale.
57+
58+
— Codex review

docs/monitoring.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,12 @@ On the first API call, the interceptor reads `~/.claude.json` and logs the curre
2020

2121
Response headers are parsed for `anthropic-ratelimit-unified-5h-utilization` and `7d-utilization`, saved for consumption by status line hooks or other tools. Proxy mode (v3.5.0+, via `cache-telemetry` extension) splits state into `~/.claude/quota-status/account.json` (account-global facts) plus `~/.claude/quota-status/sessions/<filename>.json` (per-session cache facts), so multi-agent users no longer see cross-session contamination. Preload mode keeps the legacy single-file `~/.claude/quota-status.json` (single-session by construction).
2222

23+
## Tier upgrade/downgrade advisor
24+
25+
`tools/tier-advisor.mjs` is a CLI tool that consumes the proxy-written `account.json` snapshot and the per-call `usage.jsonl` log, projects this week's Q7d burn forward to the weekly reset, and emits a tier-change recommendation (`tier:upgrade` / `tier:downgrade` / `tier:ok` / `tier:unknown`). Designed to be run on a cron / shell alias; the cache-fix statusline picks up the persisted recommendation and appends a single token to the user's prompt.
26+
27+
See [`docs/tier-advisor.md`](tier-advisor.md) for the full reference: exit codes, CLI flags, env vars (`CACHE_FIX_ADVISOR_*`), cron setup, state-file shape, and the calendar-week-scoped history semantics. Directive: [`docs/directives/proxy-tier-advisor.md`](directives/proxy-tier-advisor.md) (PR #93, issue #63).
28+
2329
## Peak hour detection
2430

2531
Anthropic applies elevated quota drain rates during weekday peak hours (13:00–19:00 UTC, Mon–Fri). The interceptor detects peak windows and writes `peak_hour: true/false` to the quota-status payload (`account.json` in proxy mode, `quota-status.json` in preload mode). See `docs/peak-hours-reference.md` for sources and details.

0 commit comments

Comments
 (0)