feat: v0.16.0 — atomic fetch locks (multi-instance 429 stampede)

Launching N Claude Code instances together produced 429 bursts: each
CLI fires 2 usage requests of its own on boot, and the statusline's
check-then-touch lock — with the touch landing hundreds of ms after
the check, behind a slow token read — let every instance through to
fetch concurrently on top of that.

acquire_lock makes acquisition atomic (noclobber create) with stale-
lock reaping, and it happens BEFORE the token read. Applied to usage,
prepaid, and OAuth refresh locks — the last one matters most: with
rotating refresh tokens, two concurrent refreshes can invalidate each
other.

Regression test: 8 concurrent fetches with a slow token read make
exactly one API call (fails on the old code).

Also: docs/api/oauth-usage.md — observed /api/oauth/usage contract
synced with Claude Code v2.1.201, captured via local mitm proxy,
sanitized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: lroolle

CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## v0.16.0 — 2026-07-06 — atomic fetch locks (multi-instance 429 stampede)
+
+Launching several Claude Code instances together produced 429 bursts. Two
+compounding causes, confirmed against a mitm capture of CLI v2.1.201:
+
+1. **Each CLI instance already fires 2 usage requests on boot** (two internal
+   clients: `claude-cli/... (external, cli)` and `claude-code/...`). N
+   instances = 2N requests before any statusline fetch.
+2. **The statusline's lock was check-then-touch, not atomic** — and the
+   `touch` happened *after* `get_oauth_token`, hundreds of ms after the
+   check. Every freshly launched instance passed the check inside that
+   window and fetched concurrently, adding up to N more requests to the
+   same burst.
+
+The statusline can only fix its own share:
+
+- New `acquire_lock`: noclobber (`set -C`) create — atomic acquire-or-fail,
+  with stale-lock reaping (orphaned locks from crashed fetchers).
+- Lock is acquired **before** the token read in `fetch_usage_for_session`
+  and `fetch_prepaid_balance`; released on every early-return path.
+- `refresh_oauth_credentials_file` uses it too: two concurrent OAuth
+  refreshes are worse than a missed one — with rotating refresh tokens the
+  loser's grant can invalidate the winner's.
+- Regression test: 8 concurrent fetches with a deliberately slow token read
+  make exactly 1 API call (fails on the old code). Plus a 20-contender
+  single-winner lock test.
+
+New: `docs/api/oauth-usage.md` — the observed `/api/oauth/usage` wire
+contract (request headers, both CLI client forms, full response schema,
+`limits[]` semantics, 429 obligations), synced with Claude Code v2.1.201.
+Sanitized; no credentials or account identifiers.
+
+5 new tests (254 total).
+
 ## v0.15.1 — 2026-07-05 — scoped quota badge moves next to the model
 
 `fb[67%]` now renders right after the model+context block instead of at the

README.md

@@ -109,7 +109,7 @@ red, matching its Claude Code TUI color) = model family; everything else is
 | OAuth + macOS Keychain | -- | Yes |
 | Quota bump flash (+N) | -- | Yes |
 | Model-scoped weekly quota (`fb`/`op`/`sn`) | -- | Yes |
-| 249 bats tests + CI | -- | Yes |
+| 254 bats tests + CI | -- | Yes |
 
 ## Configuration
 
@@ -282,7 +282,7 @@ run. Setting only `CLAUDE_CACHE_DIR` keeps the legacy single-dir behavior.
 npm exec --yes bats -- t/
 ```
 
-249 tests across `t/statusline.bats` (237 statusline + integration) and
+254 tests across `t/statusline.bats` (242 statusline + integration) and
 `t/install.bats` (12 installer). CI runs on push and PR to `main`.
 
 ## Project Structure
@@ -298,6 +298,7 @@ t/helpers.bash               Sources real functions from statusline.sh
 CHANGELOG.md                 Release notes
 CONTRIBUTING.md              Contribution guide
 docs/devlog/                 Implementation history
+docs/api/oauth-usage.md      Observed /api/oauth/usage contract (synced: CLI v2.1.201)
 ```
 
 ## Contributing

docs/api/oauth-usage.md

@@ -0,0 +1,191 @@
+# `/api/oauth/usage` — HTTP contract
+
+The endpoint this statusline polls for 5h / 7d / per-model quota. This file
+documents the observed wire contract so parser changes can be checked against
+ground truth instead of guesswork.
+
+- Synced with: **Claude Code v2.1.201** (captured 2026-07-06 via a local
+  mitm reverse proxy in front of the official CLI)
+- All credentials, org IDs, and request IDs below are redacted or fabricated.
+- Fields marked *reserved* were observed only as `null`; names are as sent by
+  the server.
+
+## Request
+
+```
+GET https://api.anthropic.com/api/oauth/usage
+```
+
+| Header | Value | Notes |
+|--------|-------|-------|
+| `Authorization` | `Bearer <oauth access token>` | The `claudeAiOauth.accessToken` from Claude Code credentials |
+| `Content-Type` | `application/json` | |
+| `User-Agent` | see below | |
+| `anthropic-beta` | `oauth-2025-04-20` | sent by one of the CLI's two internal clients, not required |
+
+No query parameters. No request body.
+
+### The CLI itself calls this twice per launch
+
+One `claude` v2.1.201 session was observed issuing **two** usage requests on
+boot, from two distinct internal clients:
+
+```
+user-agent: claude-cli/2.1.201 (external, cli)   + anthropic-beta: oauth-2025-04-20
+user-agent: claude-code/2.1.201
+```
+
+Both received 200 with identical schema. This matters for rate-limit math:
+launching N Claude Code instances produces 2N usage requests from the CLIs
+alone before any statusline fetch happens. The statusline mimics the second
+form (`claude-code/<version>`) and adds at most one request per account per
+TTL window (shared cache + atomic lock, see below).
+
+## Response — 200
+
+`Content-Type: application/json`. Notable response headers:
+`anthropic-organization-id: <org uuid>` (redacted), `request-id: req_...`.
+
+```json
+{
+  "five_hour": {
+    "utilization": 96.0,
+    "resets_at": "2026-07-06T10:59:59.800387+00:00",
+    "limit_dollars": null,
+    "used_dollars": null,
+    "remaining_dollars": null
+  },
+  "seven_day": {
+    "utilization": 65.0,
+    "resets_at": "2026-07-08T15:59:59.800406+00:00",
+    "limit_dollars": null,
+    "used_dollars": null,
+    "remaining_dollars": null
+  },
+  "limits": [
+    {
+      "kind": "session",
+      "group": "session",
+      "percent": 96,
+      "severity": "critical",
+      "resets_at": "2026-07-06T10:59:59.800387+00:00",
+      "scope": null,
+      "is_active": true
+    },
+    {
+      "kind": "weekly_all",
+      "group": "weekly",
+      "percent": 65,
+      "severity": "normal",
+      "resets_at": "2026-07-08T15:59:59.800406+00:00",
+      "scope": null,
+      "is_active": false
+    },
+    {
+      "kind": "weekly_scoped",
+      "group": "weekly",
+      "percent": 83,
+      "severity": "warning",
+      "resets_at": "2026-07-08T15:59:59.800706+00:00",
+      "scope": {
+        "model": { "id": null, "display_name": "Fable" },
+        "surface": null
+      },
+      "is_active": false
+    }
+  ],
+  "extra_usage": {
+    "is_enabled": false,
+    "monthly_limit": 20000,
+    "used_credits": 0.0,
+    "utilization": 0.0,
+    "currency": "USD",
+    "decimal_places": 2,
+    "disabled_reason": "out_of_credits",
+    "daily": null,
+    "weekly": null
+  },
+  "spend": {
+    "used":  { "amount_minor": 0,     "currency": "USD", "exponent": 2 },
+    "limit": { "amount_minor": 20000, "currency": "USD", "exponent": 2 },
+    "percent": 0,
+    "severity": "normal",
+    "enabled": false,
+    "disabled_reason": "out_of_credits",
+    "cap": { "money": null, "credits": { "amount_minor": 20000, "exponent": 2 } },
+    "balance": null,
+    "auto_reload": null,
+    "disclaimer": "Usage credits cover you when you hit your plan limits. [...]",
+    "can_purchase_credits": false,
+    "can_toggle": false
+  },
+  "member_dashboard_available": false,
+
+  "seven_day_opus": null,
+  "seven_day_sonnet": null,
+  "seven_day_oauth_apps": null,
+  "seven_day_cowork": null,
+  "seven_day_omelette": null,
+  "omelette_promotional": null,
+  "tangelo": null,
+  "iguana_necktie": null,
+  "nimbus_quill": null,
+  "cinder_cove": null,
+  "amber_ladder": null
+}
+```
+
+### Field notes
+
+**`five_hour` / `seven_day`** — the two account-wide windows. `utilization`
+is a float percentage; `resets_at` is ISO-8601 with microseconds and offset.
+The `*_dollars` fields have only been observed `null` on subscription plans.
+
+**`limits[]`** — the newer, generic limit contract (this is what the
+statusline parses first):
+
+| Field | Values observed | Meaning |
+|-------|-----------------|---------|
+| `kind` | `session`, `weekly_all`, `weekly_scoped` | which limit |
+| `group` | `session`, `weekly` | window family |
+| `percent` | integer | utilization |
+| `severity` | `normal`, `warning`, `critical` | server-side level |
+| `resets_at` | ISO-8601 | window end |
+| `scope` | `null` or `{model:{id,display_name},surface}` | `weekly_scoped` carries the model, e.g. `display_name: "Fable"`; `id` observed `null` |
+| `is_active` | bool | whether this is the currently binding limit |
+
+A `weekly_scoped` limit shares its `resets_at` with `weekly_all` — it is the
+same 7-day window, scoped to one model.
+
+**Legacy per-model fields** — `seven_day_opus` / `seven_day_sonnet` used to
+carry `{utilization, resets_at}`; since the `limits[]` array appeared they
+arrive `null`. Parsers should prefer `limits[]` and keep the old fields only
+as a fallback for cached data.
+
+**Reserved fields** — `tangelo`, `iguana_necktie`, `omelette_promotional`,
+`nimbus_quill`, `cinder_cove`, `amber_ladder`, `seven_day_oauth_apps`,
+`seven_day_cowork`, `seven_day_omelette`: observed only as `null`. Do not
+depend on them; listed so a future non-null appearance is recognized as a
+contract change.
+
+## Response — 429 (rate limited)
+
+Not present in the proxy capture (the CLI wasn't throttled that run), but
+observed regularly by this statusline's own fetches when several instances
+run: plain 429 with a `Retry-After: <seconds>` header (value `300` observed).
+
+Client obligations, as implemented here:
+
+- Serve the cached response while throttled; never retry inside the
+  `Retry-After` window.
+- Escalate the retry gap on consecutive failures (this repo: 120s → 240s →
+  480s → 600s cap, `Retry-After` wins when longer).
+- Coordinate across processes: one shared cache + one atomic lock per
+  account dir, so N concurrent statusline renders produce at most one fetch.
+
+## Related endpoints used by this statusline (not in this capture)
+
+| Endpoint | Purpose |
+|----------|---------|
+| `GET /api/oauth/profile` | account display name, org UUID, tier |
+| `GET /api/oauth/organizations/<org_uuid>/prepaid/credits` | prepaid balance for the extra-usage badge (requires `x-organization-uuid` header) |

statusline.sh

@@ -407,12 +407,9 @@ refresh_oauth_credentials_file() {
 
     mkdir -p "$CLAUDE_ACCOUNT_DIR"
     local lock_file="$CLAUDE_ACCOUNT_DIR/oauth_refresh.lock"
-    if [ -f "$lock_file" ]; then
-        local lock_age=$(($(date +%s) - $(stat -c %Y "$lock_file" 2>/dev/null || stat -f %m "$lock_file" 2>/dev/null || echo 0)))
-        [ "$lock_age" -lt 30 ] && return 1
-        rm -f "$lock_file"
-    fi
-    touch "$lock_file"
+    # Atomic: two concurrent refreshes are worse than a missed one — with
+    # rotating refresh tokens the loser's grant can invalidate the winner's.
+    acquire_lock "$lock_file" 30 || return 1
 
     scope_string=$(jq -r '.claudeAiOauth.scopes // [] | join(" ")' "$cred_file" 2>/dev/null)
     if [ -z "$scope_string" ]; then
@@ -677,25 +674,25 @@ fetch_usage_for_session() {
         return 0
     fi
 
-    if [ -f "$lock_file" ]; then
-        local lock_age=$(($(date +%s) - $(stat -c %Y "$lock_file" 2>/dev/null || stat -f %m "$lock_file" 2>/dev/null || echo 0)))
-        if [ $lock_age -lt 10 ]; then
-            debug_log "fetch_usage_for_session: lock active, skip"
-            [ -f "$cache_file" ] && cat "$cache_file"
-            return 0
-        fi
-        rm -f "$lock_file"
+    # Acquire the lock BEFORE the token read: get_oauth_token can take
+    # hundreds of ms (file reads, possibly a refresh round-trip), and N
+    # instances launched together all sat in that gap under the old
+    # check-then-touch, then fetched concurrently — the main source of 429
+    # bursts (each CLI already fires 2 usage requests of its own on boot).
+    if ! acquire_lock "$lock_file" 10; then
+        debug_log "fetch_usage_for_session: lock contended, skip"
+        [ -f "$cache_file" ] && cat "$cache_file"
+        return 0
     fi
 
     local token=$(get_oauth_token)
     if [ -z "$token" ]; then
         debug_log "fetch_usage_for_session: no token available"
+        rm -f "$lock_file"
         [ -f "$cache_file" ] && cat "$cache_file"
         return 1
     fi
 
-    touch "$lock_file"
-
     # Use CLI version from input for User-Agent, fall back to generic
     local ua="claude-code/${cli_version:-2.1.170}"
 
@@ -775,24 +772,20 @@ fetch_prepaid_balance() {
         return 0
     fi
 
-    if [ -f "$lock_file" ]; then
-        local lock_age=$(($(date +%s) - $(stat -c %Y "$lock_file" 2>/dev/null || stat -f %m "$lock_file" 2>/dev/null || echo 0)))
-        if [ $lock_age -lt 10 ]; then
-            [ -f "$cache_file" ] && cat "$cache_file"
-            return 0
-        fi
-        rm -f "$lock_file"
+    # Same atomic-acquire-before-token-read as fetch_usage_for_session.
+    if ! acquire_lock "$lock_file" 10; then
+        [ -f "$cache_file" ] && cat "$cache_file"
+        return 0
     fi
 
     local token=$(get_oauth_token)
     if [ -z "$token" ]; then
         debug_log "fetch_prepaid_balance: no token available"
+        rm -f "$lock_file"
         [ -f "$cache_file" ] && cat "$cache_file"
         return 1
     fi
 
-    touch "$lock_file"
-
     local ua="claude-code/${cli_version:-2.1.170}"
     local response=$(curl -s -w "\n%{http_code} %header{retry-after}" -X GET \
         "https://api.anthropic.com/api/oauth/organizations/${org_uuid}/prepaid/credits" \
@@ -1240,6 +1233,29 @@ get_adaptive_ttl() {
     fi
 }
 
+# Atomically acquire a lock file, or return 1 if another process holds a
+# fresh one. noclobber (set -C) makes create-if-absent a single atomic step —
+# the old check-then-touch pattern left a window (hundreds of ms when a slow
+# token read sat between check and touch) where N freshly launched instances
+# all passed the check and stampeded the API with concurrent fetches.
+# A lock older than $2 is presumed orphaned (fetcher died), reaped, and
+# re-contended once; losing that race just skips this cycle — the winner is
+# fetching anyway.
+acquire_lock() {
+    local lock_file="$1" max_age="${2:-10}"
+    if (set -C; : >"$lock_file") 2>/dev/null; then
+        return 0
+    fi
+    local mtime age
+    mtime=$(stat -c %Y "$lock_file" 2>/dev/null || stat -f %m "$lock_file" 2>/dev/null || echo 0)
+    age=$(($(date +%s) - mtime))
+    if [ "$age" -ge "$max_age" ] 2>/dev/null; then
+        rm -f "$lock_file"
+        (set -C; : >"$lock_file") 2>/dev/null && return 0
+    fi
+    return 1
+}
+
 # Remove a lock left behind by a fetch that died before cleanup. Without this,
 # a single orphaned *.lock permanently freezes the cache (the fetch gate skips
 # launching while the lock exists, and never reaps it). $2 = max age seconds.

t/helpers.bash

@@ -39,7 +39,7 @@ debug_log() {
 # Source individual functions by extracting them from statusline.sh.
 # This is deliberate: we test the actual production code, not copies.
 eval "$(awk '
-    /^(abbreviate_model_id|get_runtime_model|format_reset_relative|format_reset_absolute|get_reset_seconds|format_duration|should_show_extra|get_cache_health|infer_cache_ttl_class|format_cache_active_time|build_cache_indicator|get_usage_color|get_seven_day_color|seven_day_elapsed|seven_day_pace|weekend_secs_ahead|get_adaptive_ttl|reap_stale_lock|merge_stdin_rate_limits|rotate_usage_log|build_seven_day_profile|seven_day_forecast|premium_band_level|abbrev_effort|effort_color|_epoch_from_ts|_fmt_epoch|render_bar|format_money_minor|oauth_token_expired|refresh_oauth_credentials_file|is_default_1m_family|get_context_limit|is_1m_model|rotate_debug_log|build_display_path|quota_bump_notice|record_fetch_error|fetch_error_remaining|fetch_error_badge|model_scope_abbrev|build_scoped_quota_display|build_usage_display|build_extra_usage_display|build_user_info|get_user_tier)\(\)/ { capture=1 }
+    /^(abbreviate_model_id|get_runtime_model|format_reset_relative|format_reset_absolute|get_reset_seconds|format_duration|should_show_extra|get_cache_health|infer_cache_ttl_class|format_cache_active_time|build_cache_indicator|get_usage_color|get_seven_day_color|seven_day_elapsed|seven_day_pace|weekend_secs_ahead|get_adaptive_ttl|acquire_lock|reap_stale_lock|fetch_usage_for_session|merge_stdin_rate_limits|rotate_usage_log|build_seven_day_profile|seven_day_forecast|premium_band_level|abbrev_effort|effort_color|_epoch_from_ts|_fmt_epoch|render_bar|format_money_minor|oauth_token_expired|refresh_oauth_credentials_file|is_default_1m_family|get_context_limit|is_1m_model|rotate_debug_log|build_display_path|quota_bump_notice|record_fetch_error|fetch_error_remaining|fetch_error_badge|model_scope_abbrev|build_scoped_quota_display|build_usage_display|build_extra_usage_display|build_user_info|get_user_tier)\(\)/ { capture=1 }
     capture { print }
     capture && /^}$/ { capture=0 }
 ' "$SCRIPT_DIR/statusline.sh")"