Skip to content

Commit 551aa5e

Browse files
thodson-usgsclaude
andcommitted
Isolate filter/chunking code in a dedicated filters.py module
Move everything ``filter`` / ``filter_lang`` / CQL-chunking-related out of ``utils.py`` into a new ``dataretrieval/waterdata/filters.py``: - ``FILTER_LANG`` type alias (previously in ``types.py``) - Constants: ``_CQL_FILTER_CHUNK_LEN``, ``_WATERDATA_URL_BYTE_LIMIT``, ``_NON_FILTER_URL_HEADROOM`` - Regexes: ``_NUM`` / ``_IDENT`` / ``_OP`` subpatterns, ``_NUMERIC_COMPARE_RE``, ``_IN_NUMERIC_RE``, ``_BETWEEN_NUMERIC_RE`` - Splitter: ``_iter_or_boundaries``, ``_split_top_level_or`` - Chunker: ``_chunk_cql_or`` - Per-request URL budget: ``_effective_filter_budget`` - Lexicographic-pitfall guard: ``_check_numeric_filter_pitfall``, ``_raise_pitfall`` - Fan-out orchestration: ``_plan_filter_chunks``, ``_fetch_chunks``, ``_combine_chunk_frames``, ``_aggregate_response_metadata`` ``utils.get_ogc_data`` now calls one entry point — ``filters.fetch_combined(args, build_request=..., walk_pages=...)`` — which returns ``(combined_df, metadata)``. The ``build_request`` / ``walk_pages`` callables are injected so ``filters.py`` doesn't depend on ``utils.py`` (breaks the potential circular import). Tests that exercise filter helpers move to a new ``tests/waterdata_filters_test.py``; ``tests/waterdata_utils_test.py`` keeps only the handful of non-filter tests (``_get_args``, the ``_walk_pages`` mock suite). The motivation is isolation: if the filter feature is rolled back later, the rollback is: 1. Delete ``dataretrieval/waterdata/filters.py`` and ``tests/waterdata_filters_test.py``. 2. Replace the single ``filters.fetch_combined(...)`` call in ``utils.get_ogc_data`` with an inline ``_construct_api_requests(**args)`` + ``_walk_pages(...)`` (~3 lines). 3. Remove ``FILTER_LANG`` from ``waterdata/__init__.py`` (one line) and ``api.py`` (one import, plus the eight ``filter`` / ``filter_lang`` kwargs on the OGC getters). No behaviour change. All 162 non-live tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 9314d34 commit 551aa5e

7 files changed

Lines changed: 1005 additions & 905 deletions

File tree

dataretrieval/waterdata/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,9 @@
2626
get_stats_por,
2727
get_time_series_metadata,
2828
)
29+
from .filters import FILTER_LANG
2930
from .types import (
3031
CODE_SERVICES,
31-
FILTER_LANG,
3232
PROFILE_LOOKUP,
3333
PROFILES,
3434
SERVICES,

dataretrieval/waterdata/api.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,9 @@
1616
from requests.models import PreparedRequest
1717

1818
from dataretrieval.utils import BaseMetadata, to_str
19+
from dataretrieval.waterdata.filters import FILTER_LANG
1920
from dataretrieval.waterdata.types import (
2021
CODE_SERVICES,
21-
FILTER_LANG,
2222
METADATA_COLLECTIONS,
2323
PROFILES,
2424
SERVICES,

0 commit comments

Comments
 (0)