Commit 01e579e
refactor(waterdata): Replace static max_chunks/safety_floor with dynamic rate-limit gate
Addresses PR #283 review feedback. The static caps
(``_DEFAULT_MAX_CHUNKS=1000``, ``_DEFAULT_QUOTA_SAFETY_FLOOR=50``) and
the matching ``max_chunks`` / ``quota_safety_floor`` decorator
parameters are replaced by a quota check that runs after the first
sub-request, using the real ``x-ratelimit-remaining`` value rather
than a guessed cap.
Behavior:
- After the first sub-request the wrapper reads
``x-ratelimit-remaining``. If the rest of the plan won't fit in
the current rate-limit window, it raises a new
``RequestExceedsQuota(ValueError)`` carrying ``planned_chunks``,
``available``, and ``deficit`` so the message reports exactly how
far over budget the call is. The first chunk has already been
issued; the wrapper stops there rather than burn the rest of the
quota on a call that will fail mid-way.
- ``QuotaExhausted`` is now triggered only when an actual HTTP 429
propagates from a sub-request (detected by walking ``__cause__``
for ``RuntimeError("429: ...")``, the shape ``_raise_for_non_200``
produces and ``_walk_pages`` wraps). A single-process caller
should not normally see this — ``RequestExceedsQuota``
short-circuits in chunk 1; arrival here implies a concurrent
consumer drained the bucket faster than predicted. Carries the
partial frame for resume. ``partial_response`` becomes ``None``
when the 429 hits chunk 0 (no banked responses).
- A non-429 ``RuntimeError`` (e.g. 500) propagates unchanged so the
real cause surfaces to the caller.
- When the server doesn't echo ``x-ratelimit-remaining``,
``_read_remaining`` returns ``_QUOTA_UNKNOWN``; the wrapper skips
the post-first-chunk quota check (no signal → don't synthesize a
block).
Planner: ``_plan_list_chunks`` / ``_plan_joint`` no longer carry a
``max_chunks`` cap. ``RequestTooLarge`` fires only when nothing more
can be split (the genuine URL-byte floor). The rate-limit gate
replaces the static cap.
Module docstring rewritten to summarize the current design (joint
planning + dynamic quota gate); historical PR 233 / two-decorator
references dropped.
Tests: ten obsolete cap/floor tests removed; eight new tests added
covering ``RequestExceedsQuota`` after chunk 0, deficit reporting,
the no-header skip path, mid-call 429 → ``QuotaExhausted`` with
partial frame, the first-chunk 429 (partial_response=None) edge
case, and non-429 ``RuntimeError`` pass-through.
``_fetch_once`` in ``utils.py`` calls the decorator with defaults
only, so no call-site changes are needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent eeba277 commit 01e579e
2 files changed
Lines changed: 270 additions & 308 deletions
0 commit comments