Commit 7deedd7
refactor(waterdata): collapse sync/async chunker into one async core
The chunker carried full sync/async twins. Collapse to a single async
implementation behind a synchronous facade (the public getters and
`resume()` stay sync, same signatures/returns, by driving the async core
through the anyio blocking portal).
Removed twins: sync `_paginate`, `_walk_pages`, `_retry_sync`,
`_client_for`, `get_active_client`, the `_chunked_client` ContextVar, and
the `_fan_out_async`/`_execute_in_parallel`/`resume_async` split (folded
into `ChunkedCall.resume()` -> `_run`). `multi_value_chunked` now decorates
an `async def` fetcher (drops the `fetch_async=` param). `get_stats_data`
drives `_paginate_async` through the portal.
Concurrency is now bounded purely by the httpx connection pool
(`httpx.Limits(max_connections=N)`) — the explicit `asyncio.Semaphore` is
gone; `gather` dispatches every pending sub-request and the pool throttles
(N=1 is a single-connection gather, total<=1 a one-element gather).
Behavior note: because execution is now `gather(..., return_exceptions=True)`
over all pending sub-requests, an interruption completes every non-failing
sub-request before surfacing (even at concurrency=1) rather than stopping at
the first failure; `resume()` then re-issues only the still-failed chunks.
The public API, `resume()` contract, ChunkInterrupted/partials, finalize
hook, max_rows cap, retries, and progress reporting are unchanged.
Net ~-216 lines. Offline suite (265) + live getter suite (63) green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent f094d42 commit 7deedd7
7 files changed
Lines changed: 578 additions & 794 deletions
File tree
- dataretrieval/waterdata
- tests
Large diffs are not rendered by default.
0 commit comments