Commit 9bc342b
committed
refactor(waterdata): Unify list and filter chunkers into one joint planner
Replaces the two-decorator stack (@multi_value_chunked outside
@filters.chunked) with a single planner that allocates URL byte
budget across list dims and filter clauses together. Same correctness
guarantees, fewer sub-requests when the previous design forced the
inner filter chunker to bail at its singleton-clause floor while the
outer list chunker held the bulk of the URL budget.
Algorithm:
- Enumerate filter chunk counts k = 1, 2, 4, ..., n_clauses.
- For each k, partition clauses into k balanced groups joined by OR
and identify the worst (longest URL-encoded) group.
- Substitute the worst group as the filter and plan the list dims
with greedy halving against the remaining byte budget.
- Pick the candidate whose list_count × k is smallest.
Net code shrinks: -50 lines in filters.py (retired the chunked
decorator and _effective_filter_budget), +30 in chunking.py for the
joint planner (offset by removing _filter_aware_probe_args and the
bail-floor coordination machinery), unstack the decorator pair on
_fetch_once. Two existing cross-decorator coordination tests collapse
into joint-planner tests (mismatched-clause-length probing was the
hardest-to-explain artefact of the old design — gone now).
New regression test: ``test_joint_planner_url_construction_long_filter_and_long_sites``
exercises the planner with 500 USGS site IDs + 20 datetime OR-clauses
using the real ``_construct_api_requests`` builder. Confirms every
sub-request URL stays under 8000 bytes, filter partitions cover every
clause exactly once, list partitions cover every site exactly once,
and the total sub-request count beats the naive bail-floor-style
worst case by ≥4×.
Live API verified: Ohio Stream sites (2888) → daily discharge (P7D)
chunks into 12 sub-requests with canonical URL preserved and
cumulative query_time accurate.1 parent 46335b6 commit 9bc342b
4 files changed
Lines changed: 459 additions & 471 deletions
0 commit comments