feat(errors): unify HTTP status->exception across all request paths

Adds a single classifier so every HTTP error response raises the correct typed
DataRetrievalError, replacing four divergent paths (legacy query handled only
400/404/414/5xx; waterdata mapped other 4xx to a bare RuntimeError; nldi raised
a bare ValueError; nadp/streamstats raised httpx.HTTPStatusError).

- exceptions.py: add `error_for_status(status, message, *, retry_after)` -- the
  one status->type map (400->BadRequestError, 404->NotFoundError,
  413/414->URLTooLong, 429->RateLimited, 5xx->ServiceUnavailable, other
  4xx->new ClientError) -- plus `ClientError(DataRetrievalError, ValueError)`.
- Route every path through it: utils._raise_for_status (legacy query, now also
  429 + generic 4xx), waterdata._raise_for_non_200 (drops the bare RuntimeError),
  nldi._query_nldi, and nadp/streamstats (were raise_for_status()).
- A given status now surfaces as the same type everywhere, and a 429 is a
  RateLimited rather than a fatal ValueError on the single-shot paths too.

The type classifies the HTTP *condition* only -- it does NOT imply the path
retries it. Auto-retry/resume remains a Water Data chunker feature; the
single-shot query/nadp/streamstats paths raise the typed error without retrying
(a caller can read retry_after and re-issue). Connection-level failures
(timeouts/DNS) still surface as httpx errors on those paths. The general
exceptions module documents none of the chunker's resume internals -- that stays
in waterdata/chunking (the correct dependency direction).

mypy --strict clean; ruff clean; suite green (nwis live-test network flakes are
environmental, pass in isolation).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: thodson-usgs

dataretrieval/__init__.py

@@ -17,10 +17,10 @@
 ``nldi`` requires geopandas (``pip install dataretrieval[nldi]``) and is
 imported on demand: ``from dataretrieval import nldi``.
 
-Most request failures raise a subclass of :class:`dataretrieval.DataRetrievalError`
-(the taxonomy lives in ``dataretrieval.exceptions``); a few paths -- ``nadp``,
-``streamstats``, and some ``nldi`` / ``waterdata`` status codes -- are not
-migrated yet.
+When a request gets an HTTP error response it raises a subclass of
+:class:`dataretrieval.DataRetrievalError` (the taxonomy lives in
+``dataretrieval.exceptions``). Connection-level failures (timeouts, DNS) still
+surface as ``httpx`` exceptions on the single-shot service paths.
 """
 
 from importlib.metadata import PackageNotFoundError, version
@@ -32,6 +32,7 @@
 
 from dataretrieval.exceptions import (
     BadRequestError,
+    ClientError,
     DataRetrievalError,
     NoSitesError,
     NotFoundError,
@@ -67,6 +68,7 @@
     # so callers can ``except dataretrieval.DataRetrievalError``
     "exceptions",
     "BadRequestError",
+    "ClientError",
     "DataRetrievalError",
     "NoSitesError",
     "NotFoundError",

dataretrieval/exceptions.py

@@ -1,19 +1,16 @@
 """Exception taxonomy for ``dataretrieval``.
 
-Most request failures raise a subclass of :class:`DataRetrievalError` -- the
-legacy ``query`` path (``nwis`` / ``wqp`` / ``nldi``) and the modern ``waterdata``
-chunker both do -- so a caller can handle them with one
-``except dataretrieval.DataRetrievalError``. A few paths are not migrated yet:
-``nadp`` and ``streamstats`` raise ``httpx.HTTPStatusError``, and some ``nldi`` /
-``waterdata`` status codes still raise a bare ``ValueError`` / ``RuntimeError``.
+When a request gets an HTTP error response, the service modules (``nwis``,
+``wqp``, ``nldi``, ``waterdata``, ``nadp``, ``streamstats``) raise a subclass of
+:class:`DataRetrievalError`, so a caller can handle any of them with one
+``except dataretrieval.DataRetrievalError``. Connection-level failures (timeouts,
+DNS, refused connections) surface as ``httpx`` exceptions on the single-shot
+request paths.
 
 Two intermediate bases let a caller catch a whole family: :class:`RequestTooLarge`
-(the request can't fit, however it was issued) and :class:`TransientError` (a raw
-retryable transport failure). Note that after the Water Data chunker exhausts its
-own retries it raises a resumable
-:class:`~dataretrieval.waterdata.chunking.ChunkInterrupted` subclass
-(``QuotaExhausted`` / ``ServiceInterrupted``); those are *not* ``TransientError``
-instances -- catch ``ChunkInterrupted`` to resume a chunked call.
+(the request can't fit, however it was issued) and :class:`TransientError` (a
+429 or 5xx the server may serve on a later try). These classify the HTTP
+condition; they do not by themselves retry the request.
 
 This module imports only ``httpx`` (the package's core HTTP dependency, always
 installed) -- not pandas/geopandas -- so it stays cheap to import and free of
@@ -28,29 +25,33 @@
     "DataRetrievalError",
     "BadRequestError",
     "NotFoundError",
+    "ClientError",
     "RequestTooLarge",
     "URLTooLong",
     "Unchunkable",
     "NoSitesError",
     "TransientError",
     "RateLimited",
     "ServiceUnavailable",
+    "error_for_status",
 ]
 
 
 class DataRetrievalError(Exception):
     """Base class for errors raised when a request to a USGS or EPA web
     service fails.
 
-    The legacy ``query`` path (``nwis`` / ``wqp`` / ``nldi``) and the modern
-    ``waterdata`` chunker raise a subclass of this on a failed request, so a
-    caller can handle those uniformly::
+    Service modules raise a subclass of this when a request gets an HTTP error
+    response, so a caller can handle them uniformly::
 
         try:
             df, md = dataretrieval.wqp.get_results(...)
         except dataretrieval.DataRetrievalError:
             ...
 
+    (Connection-level failures still surface as ``httpx`` exceptions on the
+    single-shot paths.)
+
     Subclasses also inherit from the built-in exception this package has
     historically raised for the condition's *kind* -- :class:`ValueError` for a
     request that can't succeed as written (bad params, too large), and
@@ -73,6 +74,12 @@ class NotFoundError(DataRetrievalError, ValueError):
     """The requested resource was not found; often an empty query (HTTP 404)."""
 
 
+class ClientError(DataRetrievalError, ValueError):
+    """The service rejected the request with a 4xx not covered by a more
+    specific type (e.g. 401 Unauthorized, 403 Forbidden, 405). Fatal -- retrying
+    it unchanged won't help."""
+
+
 class RequestTooLarge(DataRetrievalError, ValueError):
     """The request is too large for the service to satisfy.
 
@@ -119,18 +126,19 @@ def __str__(self) -> str:
 
 # --- Transient transport errors ------------------------------------------
 # The service was reachable but temporarily refused the request; the same call
-# may succeed if retried. Each is also a ``RuntimeError`` (the built-in the
-# waterdata path has always raised). The Water Data chunker recognizes them via
-# ``isinstance(exc, TransientError)`` and wraps them as resumable
-# ``ChunkInterrupted`` subclasses.
+# may succeed on a later try. Each is also a ``RuntimeError`` for backward
+# compatibility. Whether a transient is actually retried is up to the calling
+# path.
 
 
 class TransientError(DataRetrievalError, RuntimeError):
-    """Base for transient HTTP failures that are worth an automatic retry.
+    """A 429 or 5xx the server may serve on a later try (:class:`RateLimited`
+    for 429, :class:`ServiceUnavailable` for 5xx).
 
-    One subclass per recoverable HTTP status family (429 -> :class:`RateLimited`,
-    5xx -> :class:`ServiceUnavailable`); the Water Data chunker recognizes them
-    by this shared base and wraps them as resumable interruptions.
+    This classifies the HTTP condition; it does not by itself retry the request.
+    Whether a transient is retried is up to the calling path -- a single-shot
+    request raises it for the caller to handle (e.g. wait :attr:`retry_after`
+    and re-issue).
 
     Parameters
     ----------
@@ -157,3 +165,29 @@ class ServiceUnavailable(TransientError):
     Raised by both the legacy ``query`` path and the Water Data path, so a 5xx
     surfaces as one type regardless of which subsystem issued the request.
     """
+
+
+def error_for_status(
+    status: int, message: str, *, retry_after: float | None = None
+) -> DataRetrievalError:
+    """Return the typed :class:`DataRetrievalError` for an HTTP error *status*.
+
+    The single status-to-type mapping shared by every request path (the legacy
+    ``query`` path, ``waterdata``, ``nadp`` / ``streamstats``), so a given status
+    surfaces as the same type everywhere. ``message`` is used
+    verbatim; ``retry_after`` is attached only to the transient
+    (:class:`TransientError`) types. The returned type classifies the HTTP
+    condition only -- it does not imply the caller's path retries it (see
+    :class:`TransientError`).
+    """
+    if status == 400:
+        return BadRequestError(message)
+    if status == 404:
+        return NotFoundError(message)
+    if status in (413, 414):
+        return URLTooLong(message)
+    if status == 429:
+        return RateLimited(message, retry_after=retry_after)
+    if 500 <= status < 600:
+        return ServiceUnavailable(message, retry_after=retry_after)
+    return ClientError(message)

dataretrieval/nadp.py

@@ -45,7 +45,7 @@
 
 import httpx
 
-from dataretrieval.utils import HTTPX_DEFAULTS
+from dataretrieval.utils import HTTPX_DEFAULTS, _raise_for_status
 
 _DEPRECATION_MESSAGE = (
     "The `nadp` module is deprecated and will be removed from `dataretrieval` "
@@ -230,7 +230,7 @@ def get_zip(url: str, filename: str) -> NADP_ZipFile:
     _warn_deprecated()
 
     req = httpx.get(url + filename, **HTTPX_DEFAULTS)
-    req.raise_for_status()
+    _raise_for_status(req)
 
     # z = zipfile.ZipFile(io.BytesIO(req.content))
     z = NADP_ZipFile(io.BytesIO(req.content))

dataretrieval/nldi.py

@@ -3,6 +3,7 @@
 from json import JSONDecodeError
 from typing import Any, Literal, cast
 
+from dataretrieval.exceptions import error_for_status
 from dataretrieval.utils import query
 
 try:
@@ -24,7 +25,10 @@ def _query_nldi(
     # A helper function to query the NLDI API
     response = query(url, payload=query_params)
     if response.status_code != 200:
-        raise ValueError(f"{error_message}. Error reason: {response.reason_phrase}")
+        raise error_for_status(
+            response.status_code,
+            f"{error_message} Error reason: {response.reason_phrase}",
+        )
 
     response_data: dict[str, Any] | list[Any] = {}
     try:

dataretrieval/streamstats.py

@@ -12,7 +12,7 @@
 
 import httpx
 
-from dataretrieval.utils import HTTPX_DEFAULTS
+from dataretrieval.utils import HTTPX_DEFAULTS, _raise_for_status
 
 
 def download_workspace(workspaceID: str, format: str = "") -> httpx.Response:
@@ -39,7 +39,7 @@ def download_workspace(workspaceID: str, format: str = "") -> httpx.Response:
 
     r = httpx.get(url, params=payload, **HTTPX_DEFAULTS)
 
-    r.raise_for_status()
+    _raise_for_status(r)
     return r
     # data = r.raw.read()
 
@@ -146,7 +146,7 @@ def get_watershed(
 
     r = httpx.get(url, params=payload, **HTTPX_DEFAULTS)
 
-    r.raise_for_status()
+    _raise_for_status(r)
 
     if format == "geojson":
         return r

dataretrieval/utils.py

@@ -14,11 +14,9 @@
 import dataretrieval
 from dataretrieval.codes import tz
 from dataretrieval.exceptions import (
-    BadRequestError,
     NoSitesError,
-    NotFoundError,
-    ServiceUnavailable,
     URLTooLong,
+    error_for_status,
 )
 
 # Typed as ``dict[str, Any]`` (not the inferred ``dict[str, object]``) so that
@@ -290,31 +288,22 @@ def _url_too_long_error(detail: str) -> URLTooLong:
 
 
 def _raise_for_status(response: httpx.Response) -> None:
-    """Map an unsuccessful HTTP status to a typed :class:`DataRetrievalError`;
+    """Raise the typed :class:`DataRetrievalError` for an HTTP error response;
     return ``None`` on success.
 
-    Shared by the legacy :func:`query` path. The 4xx types stay
-    :class:`ValueError`-compatible (this path's historical contract), but a 5xx
-    raises the transient :class:`ServiceUnavailable` (a :class:`RuntimeError`),
-    since a server failure is retryable rather than a bad request.
+    Shared by the legacy :func:`query` path (and ``nadp`` / ``streamstats``).
+    Delegates the status-to-type mapping to
+    :func:`dataretrieval.exceptions.error_for_status`; a 414 keeps the richer
+    "split your query" guidance via :func:`_url_too_long_error`.
     """
     status = response.status_code
-    if status == 400:
-        raise BadRequestError(
-            f"Bad Request, check that your parameters are correct. URL: {response.url}"
-        )
-    elif status == 404:
-        raise NotFoundError(
-            "Page Not Found Error. May be the result of an empty query. "
-            f"URL: {response.url}"
-        )
-    elif status == 414:
+    if status < 400:
+        return
+    if status in (413, 414):
         raise _url_too_long_error(f"API response reason: {response.reason_phrase}")
-    elif 500 <= status < 600:
-        raise ServiceUnavailable(
-            f"Service Unavailable: {status} {response.reason_phrase}. "
-            f"The service at {response.url} may be down or experiencing issues."
-        )
+    raise error_for_status(
+        status, f"{response.reason_phrase} (HTTP {status}). URL: {response.url}"
+    )
 
 
 def query(
@@ -348,13 +337,19 @@ def query(
     Raises
     ------
     DataRetrievalError
-        On failure: :class:`~dataretrieval.exceptions.BadRequestError` (400),
+        On an HTTP error response, the typed subclass for the status (via
+        :func:`dataretrieval.exceptions.error_for_status`) -- e.g.
+        :class:`~dataretrieval.exceptions.BadRequestError` (400),
         :class:`~dataretrieval.exceptions.NotFoundError` (404),
-        :class:`~dataretrieval.exceptions.URLTooLong` (414 or a client-side
-        over-long URL), :class:`~dataretrieval.exceptions.ServiceUnavailable`
-        (5xx), or :class:`~dataretrieval.exceptions.NoSitesError` (no sites/data
-        matched). The 4xx types are also :class:`ValueError`;
-        ``ServiceUnavailable`` is a :class:`RuntimeError`.
+        :class:`~dataretrieval.exceptions.URLTooLong` (413/414 or a client-side
+        over-long URL), :class:`~dataretrieval.exceptions.RateLimited` (429),
+        :class:`~dataretrieval.exceptions.ServiceUnavailable` (5xx), or
+        :class:`~dataretrieval.exceptions.ClientError` for any other 4xx -- or
+        :class:`~dataretrieval.exceptions.NoSitesError` when a 200 response
+        reports no sites/data matched. The 4xx types are also
+        :class:`ValueError`; the transient 429/5xx types are
+        :class:`RuntimeError`. Connection-level failures (timeouts, DNS) instead
+        surface as ``httpx`` exceptions.
     """
 
     for key, value in payload.items():