Commit df8943e
refactor(waterdata): decompose the 2033-LOC utils.py god-module into cohesive modules
waterdata/utils.py had grown to 2033 LOC spanning ~6 unrelated domains (request
building, response parsing, result finalization, pagination/async, stats
post-processing, validation) plus constants and the public engines -- changing
one domain meant navigating all of them.
Split it into six cohesive private modules under dataretrieval/waterdata/,
moving every definition VERBATIM (no signature/logic changes):
_constants.py URLs, _OUTPUT_ID_BY_SERVICE, regexes, param sets
_http.py headers, _error_body, _raise_for_non_200, retry-after
_validate.py arg normalization/validation (_get_args, _check_*)
_requests.py request building (_construct_api_requests, CQL2, dates)
_responses.py geometry-agnostic parsing/finalization/stats shaping
_engine.py pagination/async driver (_paginate, _run_sync, ...)
utils.py (2033 -> 651 LOC) becomes a thin facade that re-exports the package's
internal API (explicit __all__, 56 names) so every existing
`from dataretrieval.waterdata.utils import ...` and
`mock.patch("dataretrieval.waterdata.utils.<name>")` keeps working unchanged --
no import sites or tests were touched.
Seven functions stay physically defined in the facade because the test suite
monkeypatches them (or `gpd`) by their module-global name (get_ogc_data,
_fetch_once, get_stats_data, _get_resp_data, _ogc_parse_response, _walk_pages,
_handle_stats_nesting): a function's global lookups resolve in its defining
module, so they must live in utils.py for the patches to take effect. The
geopandas import probe stays with them; the pagination logger keeps the name
"dataretrieval.waterdata.utils".
Behavior-preserving: 56 top-level definitions moved AST-identically (none lost
or duplicated); 469 tests pass, 2 skipped; ruff clean; chunking.py untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 1adf174 commit df8943e
7 files changed
Lines changed: 1861 additions & 1553 deletions
File tree
- dataretrieval/waterdata
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
0 commit comments