Skip to content

Commit 8cb9ea1

Browse files
committed
refactor(ogc): stabilize phase 1 boundaries
1 parent 0e2c16b commit 8cb9ea1

24 files changed

Lines changed: 1065 additions & 838 deletions

NEWS.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
**08/02/2026:** Phase 1 OGC boundary stabilization: the `dataretrieval.ogc` package now exposes a deliberate, small facade (`OgcDialect`, `prepare_request_args`, `get_ogc_data`, `fetch_ogc_request`) that service adapters (NGWMN, Water Data's generic wrapper) import. Internal request-construction helpers moved to a new `ogc.requests` module; the dialect type and endpoint constants live in the leaf `ogc.policy` module. `ogc.shaping` no longer depends on `ogc.engine` at all, and the complete runtime OGC import graph is now acyclic. `_default_headers` now accepts a target URL and adds `X-Api-Key` only for `api.waterdata.usgs.gov`; shared sync and async clients also strip the key before following any cross-host redirect, including external rating-asset downloads. `waterdata.utils` no longer bulk-re-exports private OGC symbols, and consumers import implementation helpers from their canonical modules. No public API, return-value, or deprecation changes.
2+
13
**08/02/2026:** Fixed source-distribution and wheel package discovery so the `dataretrieval.ogc` and `dataretrieval.waterdata` subpackages are included in installed artifacts. CI now builds and installs the wheel outside the source checkout before importing the core service modules. Added an architecture baseline, initial decision records, and executable dependency-direction guardrails for the existing modular-monolith boundaries.
24

35
**06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected.

dataretrieval/ngwmn.py

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
The NGWMN exposes its data through a dedicated OGC API
44
(``https://api.waterdata.usgs.gov/ngwmn/ogcapi``) with five collections:
55
``sites``, ``waterLevelObs``, ``lithologyObs``, ``constructionObs``, and
6-
``providers``. Each getter below delegates to the shared OGC engine
7-
(:func:`~dataretrieval.ogc.engine.get_ogc_data`) with
6+
``providers``. Each getter below delegates to the shared OGC facade
7+
(:func:`~dataretrieval.ogc.get_ogc_data`) with
88
``base_url=NGWMN_OGC_API_URL``, so multi-value chunking, pagination,
99
retry/resume, and result shaping all behave exactly as they do for the main
1010
Water Data getters.
@@ -24,9 +24,12 @@
2424
import pandas as pd
2525

2626
from dataretrieval.codes.states import apply_state
27-
from dataretrieval.ogc.engine import BASE_URL, OgcDialect, _get_args, get_ogc_data
27+
from dataretrieval.ogc import OgcDialect, get_ogc_data, prepare_request_args
2828
from dataretrieval.utils import BaseMetadata
2929

30+
# The Water Data API base URL, defined locally to avoid importing policy internals.
31+
BASE_URL = "https://api.waterdata.usgs.gov"
32+
3033
# The National Ground-Water Monitoring Network exposes its own OGC API at a
3134
# separate, unversioned base.
3235
NGWMN_OGC_API_URL = f"{BASE_URL}/ngwmn/ogcapi"
@@ -72,15 +75,15 @@
7275

7376

7477
def _get(service: str, local_vars: dict[str, Any]) -> tuple[pd.DataFrame, BaseMetadata]:
75-
"""Marshal a getter's arguments and dispatch to the shared OGC engine.
78+
"""Marshal a getter's arguments and dispatch to the shared OGC facade.
7679
7780
Every NGWMN getter ends with this same call; centralizing it keeps the
7881
NGWMN base URL, output id, and dialect wired up in exactly one place.
7982
"""
8083
queryable = _STATE_QUERYABLE.get(service)
8184
if queryable is not None:
8285
apply_state(local_vars, to=queryable["to"], into=queryable["into"])
83-
args = _get_args(local_vars)
86+
args = prepare_request_args(local_vars)
8487
return get_ogc_data(
8588
args,
8689
service,

dataretrieval/ogc/__init__.py

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,26 @@
1-
"""Generic OGC API engine shared by the Water Data and NGWMN getters."""
1+
"""Generic OGC API engine shared by the Water Data and NGWMN getters.
2+
3+
The public facade exposes only the minimal service-adapter seam:
4+
5+
- :class:`OgcDialect` — per-API request/response quirks.
6+
- :func:`prepare_request_args` — normalize caller kwargs for the engine.
7+
- :func:`get_ogc_data` — full orchestrated OGC fetch (chunking + pagination).
8+
- :func:`fetch_ogc_request` — execute a pre-built request with pagination.
9+
10+
Service adapters (NGWMN, Water Data's generic wrapper) import from this
11+
facade rather than reaching into engine internals. The engine module remains
12+
available for lower-level orchestration needs (e.g. ``_paginate``,
13+
``_run_sync``) that sibling modules like ``wateruse`` use under the accepted
14+
temporary variance.
15+
"""
16+
17+
from dataretrieval.ogc.engine import fetch_ogc_request, get_ogc_data
18+
from dataretrieval.ogc.policy import OgcDialect
19+
from dataretrieval.ogc.requests import prepare_request_args
20+
21+
__all__ = [
22+
"OgcDialect",
23+
"fetch_ogc_request",
24+
"get_ogc_data",
25+
"prepare_request_args",
26+
]

dataretrieval/ogc/chunking.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@
8383
import pandas as pd
8484
from anyio.from_thread import start_blocking_portal
8585

86-
from dataretrieval.utils import HTTPX_DEFAULTS, Ambient, _require_positive_int
86+
from dataretrieval.utils import HTTPX_ASYNC_DEFAULTS, Ambient, _require_positive_int
8787

8888
from . import progress as _progress
8989
from .combining import (
@@ -600,7 +600,7 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]:
600600
The semaphore, not the pool, is deliberately the throttle. If the
601601
pool throttled instead, the excess sub-requests would queue
602602
*inside* httpx waiting for a connection, and that wait counts
603-
against the pool-acquire timeout (60 s, from ``HTTPX_DEFAULTS``).
603+
against the pool-acquire timeout (60 s, from ``HTTPX_ASYNC_DEFAULTS``).
604604
A batch of slow pages that keeps every connection busy past that
605605
window would then trip ``httpx.PoolTimeout`` on the queued tail —
606606
a purely client-side failure that consumes the retry budget and
@@ -650,7 +650,7 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]:
650650
self.plan.total if max_concurrent is None else max_concurrent
651651
)
652652

653-
async with httpx.AsyncClient(limits=limits, **HTTPX_DEFAULTS) as client:
653+
async with httpx.AsyncClient(limits=limits, **HTTPX_ASYNC_DEFAULTS) as client:
654654
with _chunked_client(client):
655655
reporter = _progress.current()
656656
if reporter is not None:

0 commit comments

Comments
 (0)