Skip to content

Commit 9bc342b

Browse files
committed
refactor(waterdata): Unify list and filter chunkers into one joint planner
Replaces the two-decorator stack (@multi_value_chunked outside @filters.chunked) with a single planner that allocates URL byte budget across list dims and filter clauses together. Same correctness guarantees, fewer sub-requests when the previous design forced the inner filter chunker to bail at its singleton-clause floor while the outer list chunker held the bulk of the URL budget. Algorithm: - Enumerate filter chunk counts k = 1, 2, 4, ..., n_clauses. - For each k, partition clauses into k balanced groups joined by OR and identify the worst (longest URL-encoded) group. - Substitute the worst group as the filter and plan the list dims with greedy halving against the remaining byte budget. - Pick the candidate whose list_count × k is smallest. Net code shrinks: -50 lines in filters.py (retired the chunked decorator and _effective_filter_budget), +30 in chunking.py for the joint planner (offset by removing _filter_aware_probe_args and the bail-floor coordination machinery), unstack the decorator pair on _fetch_once. Two existing cross-decorator coordination tests collapse into joint-planner tests (mismatched-clause-length probing was the hardest-to-explain artefact of the old design — gone now). New regression test: ``test_joint_planner_url_construction_long_filter_and_long_sites`` exercises the planner with 500 USGS site IDs + 20 datetime OR-clauses using the real ``_construct_api_requests`` builder. Confirms every sub-request URL stays under 8000 bytes, filter partitions cover every clause exactly once, list partitions cover every site exactly once, and the total sub-request count beats the naive bail-floor-style worst case by ≥4×. Live API verified: Ohio Stream sites (2888) → daily discharge (P7D) chunks into 12 sub-requests with canonical URL preserved and cumulative query_time accurate.
1 parent 46335b6 commit 9bc342b

4 files changed

Lines changed: 459 additions & 471 deletions

File tree

0 commit comments

Comments
 (0)