Skip to content

stacks/profile 429s: add Hiro client resilience (retry/backoff/Retry-After/cache) + verify prod key #124

Description

@whoabuddy

Summary

The /stacks/profile/:address endpoint 429s on roughly an hourly cadence in prod (x402-api-host). A single hourly request shouldn't be able to 429 a properly-keyed client — so this is a symptom of two real gaps: the Hiro client has no rate-limit resilience, and the prod Hiro key situation needs verification.

Evidence (worker-logs, last 72h, prod)

  • 50 errors in the window, all on one path: /stacks/profile/SP2PABAF9FTAJYNFZH93XENAJ8FVY99RRM50D2JG9, timestamps clustered at HH:00:0X (the hourly cron test).
  • Errors cascade from Hiro 429: Hiro API error (Failed to fetch block/balance/account info: 429) → Failed to fetch current block / Failed to fetch BNS names / account info / balance.
  • Payment settlement failed with errorReason: conflicting_nonce (separate concern, see Scheduled job sends concurrent settlements causing batch nonce conflicts #86).

Root cause

  1. No resilience in the Hiro clientsrc/services/hiro.ts uses raw fetch() with no retry, no backoff, no Retry-After handling, no rate-limit header reading, and no caching. Any 429 fails the whole request immediately.
  2. Burst fan-outsrc/endpoints/stacks/profile.ts fires 3–4 Hiro calls per request (BNS lookup + Promise.all of getAccountBalance / getAccountInfo / getCurrentBlock), hitting the per-second limiter harder than a serial call.
  3. Retry amplification — when a call 429s the endpoint returns 500, and the cron test harness retries the whole request up to 3×, turning one profile test into ~16 Hiro calls and self-inflicting more 429s.
  4. Key/budget question — the client supports x-api-key from c.env.HIRO_API_KEY, but .env carries a shared dev key. Whether the prod secret is set, and whether it shares a budget with landing-page (whose competition sweep is draining the same Hiro account — see landing-page companion issue), is unverified.

Proposal

  • Verify prod HIRO_API_KEY secret is set (wrangler secret list) and decide shared-vs-separate key/budget vs landing-page. (fast, high-leverage — may resolve most of this on its own)
  • Adopt the resilient Hiro-fetch pattern already standardized in landing-page (lib/stacks-api-fetch.ts): 429-specific exponential backoff, honor Retry-After, read x-ratelimit-*-stacks-month headers, emit stacksApi.* telemetry. Ideal long-term: extract a shared package.
  • Add a short cache for profile sub-calls — block height ~60s, balance/account ~10–30s, BNS ~1h.
  • Stop the test harness from retrying 429-origin 500s (avoid the amplification loop).

Reference

  • landing-page/lib/stacks-api-fetch.ts — the standard to mirror (Retry-After, monthly-quota header parsing, stacksApi.* events)

Priority: Medium (low volume, single path, not user-facing — but it's masking a missing-resilience defect)


🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions