Skip to content

Commit 7deedd7

Browse files
thodson-usgsclaude
andcommitted
refactor(waterdata): collapse sync/async chunker into one async core
The chunker carried full sync/async twins. Collapse to a single async implementation behind a synchronous facade (the public getters and `resume()` stay sync, same signatures/returns, by driving the async core through the anyio blocking portal). Removed twins: sync `_paginate`, `_walk_pages`, `_retry_sync`, `_client_for`, `get_active_client`, the `_chunked_client` ContextVar, and the `_fan_out_async`/`_execute_in_parallel`/`resume_async` split (folded into `ChunkedCall.resume()` -> `_run`). `multi_value_chunked` now decorates an `async def` fetcher (drops the `fetch_async=` param). `get_stats_data` drives `_paginate_async` through the portal. Concurrency is now bounded purely by the httpx connection pool (`httpx.Limits(max_connections=N)`) — the explicit `asyncio.Semaphore` is gone; `gather` dispatches every pending sub-request and the pool throttles (N=1 is a single-connection gather, total<=1 a one-element gather). Behavior note: because execution is now `gather(..., return_exceptions=True)` over all pending sub-requests, an interruption completes every non-failing sub-request before surfacing (even at concurrency=1) rather than stopping at the first failure; `resume()` then re-issues only the still-failed chunks. The public API, `resume()` contract, ChunkInterrupted/partials, finalize hook, max_rows cap, retries, and progress reporting are unchanged. Net ~-216 lines. Offline suite (265) + live getter suite (63) green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent f094d42 commit 7deedd7

7 files changed

Lines changed: 578 additions & 794 deletions

File tree

dataretrieval/waterdata/chunking.py

Lines changed: 165 additions & 357 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)