refactor(errors)!: a lean, idiomatic DataRetrievalError taxonomy

Every request failure raises a subclass of DataRetrievalError, so a caller can
handle any of them with a single `except dataretrieval.DataRetrievalError`. The
taxonomy stays small -- it adds only what the underlying httpx exceptions can't
express:

  DataRetrievalError(Exception)
  |- HTTPError                   # .status_code -- the server returned an error status
  |   '- TransientError          # .retry_after -- retryable (429 / 5xx)
  |       |- RateLimited         #   429
  |       '- ServiceUnavailable  #   5xx
  |- RequestTooLarge             # the request can't fit
  |   |- URLTooLong              #   414 / client-side over-long URL
  |   '- Unchunkable             #   the Water Data chunker can't split the call
  '- NoDataError                 # a 200 response with no data

One factory -- error_for_status(status, message, *, retry_after) -- maps a
status to its type, and every request path routes through it (the legacy
`query` path, the Water Data chunker, nldi, nadp, streamstats), so a given
status surfaces as the same type everywhere. A fatal 4xx is a generic HTTPError
carrying .status_code (inspect the code rather than a class per code). The
chunker keys retry/resume on TransientError; connection-level failures
(timeouts, DNS) surface as httpx exceptions on the single-shot paths. The typed
errors are picklable, so they survive a pickle / deepcopy back from a
multiprocessing / lithops worker (a chunk-interruption error sheds its live
resume handle to make the trip).

A too-long-URL status (413 / 414) on the legacy `query` path keeps the
actionable "split your query" remediation message (the same one the client-side
over-long-URL case raises), rather than degrading to a bare HTTP-status line.

BREAKING CHANGES
- Request failures raise typed DataRetrievalError subclasses instead of bare
  ValueError / RuntimeError / httpx.HTTPStatusError. The exceptions root only at
  DataRetrievalError(Exception) and no longer also inherit ValueError /
  RuntimeError -- catch DataRetrievalError (or a subclass), not the builtins.
- A fatal 4xx raises HTTPError (read .status_code); there are no per-code types.
- The empty-result error is renamed NoSitesError -> NoDataError (it is raised
  from the shared query path for any module, not just NWIS "sites"). NoSitesError
  stays as a deprecated alias and will be removed in a future release.

Also adds a dataretrieval.exceptions API docs page and a NEWS.md changelog entry.

mypy --strict clean; ruff clean; full suite green (487 passed, 2 skipped); the
Water Data chunker's resume tests pass unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: thodson-usgs

NEWS.md

@@ -1,3 +1,5 @@
+**06/03/2026:** The request-error hierarchy is now unified. Every module (`nwis`, `wqp`, `nldi`, `waterdata`, `nadp`, `streamstats`) raises a subclass of `dataretrieval.DataRetrievalError` on a failed request, so a single `except dataretrieval.DataRetrievalError` spans them all. An HTTP error status surfaces as an `HTTPError` carrying `.status_code` (inspect it to branch on a specific code); the retryable 429/5xx subset is `TransientError` (`RateLimited` / `ServiceUnavailable`, carrying `.retry_after`); and a request too large to satisfy is a `RequestTooLarge` (`URLTooLong` for an over-long single request, `Unchunkable` when the Water Data chunker cannot split a call small enough). Connection-level failures (timeouts, DNS) still surface as `httpx` exceptions on the single-shot paths. **Breaking change:** these exceptions no longer multiply-inherit a built-in — code that caught request failures with `except ValueError` or `except RuntimeError` should switch to `except dataretrieval.DataRetrievalError` (or a specific subclass). The error raised on a 200-but-empty result, formerly `NoSitesError`, is renamed `NoDataError` (the old name leaked NWIS-era "sites" terminology and the condition is general); `NoSitesError` remains as a deprecated alias and will be removed in a future release.
+
 **05/17/2026:** The OGC `waterdata` getters (`get_daily`, `get_continuous`, `get_field_measurements`, and the rest of the multi-value-capable functions) now transparently chunk requests whose URLs would otherwise exceed the server's ~8 KB byte limit.
 
 **05/16/2026:** Fixed silent truncation in the paginated `waterdata` request loops (`_walk_pages` and `get_stats_data`). Mid-pagination failures (HTTP 429, 5xx, network error) were previously swallowed — pagination would quietly stop and the function would return whatever rows it had collected, leaving callers with truncated DataFrames they had no way to detect. The loops now status-check every page like the initial request and raise `RuntimeError` on any failure, with the upstream exception chained as `__cause__` and a short menu of recovery actions (wait and retry, reduce the request, or obtain an API token) in the message. **Behavior change**: callers that previously consumed partial DataFrames on transient upstream blips will now see an exception; retry the call (possibly with a smaller `limit` or narrower query).

dataretrieval/__init__.py

@@ -17,8 +17,10 @@
 ``nldi`` requires geopandas (``pip install dataretrieval[nldi]``) and is
 imported on demand: ``from dataretrieval import nldi``.
 
-Every request failure raises a subclass of :class:`dataretrieval.DataRetrievalError`;
-the taxonomy lives in ``dataretrieval.exceptions``.
+When a request gets an HTTP error response it raises a subclass of
+:class:`dataretrieval.DataRetrievalError` (the taxonomy lives in
+``dataretrieval.exceptions``). Connection-level failures (timeouts, DNS) still
+surface as ``httpx`` exceptions on the single-shot service paths.
 """
 
 from importlib.metadata import PackageNotFoundError, version
@@ -29,10 +31,10 @@
     __version__ = "version-unknown"
 
 from dataretrieval.exceptions import (
-    BadRequestError,
     DataRetrievalError,
+    HTTPError,
+    NoDataError,
     NoSitesError,
-    NotFoundError,
     RateLimited,
     RequestTooLarge,
     ServiceUnavailable,
@@ -64,10 +66,10 @@
     # error taxonomy (canonical home: ``dataretrieval.exceptions``), re-exported
     # so callers can ``except dataretrieval.DataRetrievalError``
     "exceptions",
-    "BadRequestError",
     "DataRetrievalError",
-    "NoSitesError",
-    "NotFoundError",
+    "HTTPError",
+    "NoDataError",
+    "NoSitesError",  # deprecated alias for NoDataError
     "RateLimited",
     "RequestTooLarge",
     "ServiceUnavailable",

dataretrieval/exceptions.py

@@ -1,89 +1,202 @@
 """Exception taxonomy for ``dataretrieval``.
 
-A failed request from any service module (``nwis``, ``wqp``, ``waterdata``,
-``nldi``, ...) raises a subclass of :class:`DataRetrievalError`, so a caller can
-handle any request failure with a single ``except dataretrieval.DataRetrievalError``.
-
-The tree has two intermediate bases a caller can catch to span a whole family:
-:class:`RequestTooLarge` (the request can't fit, however it was issued) and
-:class:`TransientError` (a temporary failure worth retrying).
-
-This module deliberately has no third-party dependencies, so any module can
-import it without pulling in pandas/httpx.
+Every service module (``nwis``, ``wqp``, ``nldi``, ``waterdata``, ``nadp``,
+``streamstats``) raises a subclass of :class:`DataRetrievalError` when a request
+fails, so one ``except dataretrieval.DataRetrievalError`` catches them all.
+Connection-level failures (timeouts, DNS, refused connections) still surface as
+``httpx`` exceptions on the single-shot request paths.
+
+Most failures are an :class:`HTTPError` carrying the response ``.status_code``,
+of which :class:`TransientError` (429 / 5xx) is the retryable subset. The rest
+aren't a plain status: :class:`RequestTooLarge` (with :class:`URLTooLong` /
+:class:`Unchunkable`) and :class:`NoDataError`. :func:`error_for_status` is the
+one place that maps a status to its type.
+
+This module has no third-party runtime dependencies -- ``httpx`` is imported only
+for type checking -- so any module can import it without pulling in pandas / httpx
+and without risking an import cycle.
 """
 
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any, ClassVar
 
 if TYPE_CHECKING:
     import httpx
 
 __all__ = [
     "DataRetrievalError",
-    "BadRequestError",
-    "NotFoundError",
-    "RequestTooLarge",
-    "URLTooLong",
-    "Unchunkable",
-    "NoSitesError",
+    "HTTPError",
     "TransientError",
     "RateLimited",
     "ServiceUnavailable",
+    "RequestTooLarge",
+    "URLTooLong",
+    "Unchunkable",
+    "NoDataError",
+    "NoSitesError",  # deprecated alias for NoDataError
+    "error_for_status",
 ]
 
 
 class DataRetrievalError(Exception):
-    """Base class for errors raised when a request to a USGS or EPA web
-    service fails.
+    """Base class for every failed-request error in ``dataretrieval``.
 
-    Every service module (``nwis``, ``wqp``, ``waterdata``, ``nldi``, ...)
-    raises a subclass of this when a request fails, so a caller can handle any
-    request failure uniformly::
+    Catch it to handle any USGS or EPA service failure uniformly::
 
         try:
             df, md = dataretrieval.wqp.get_results(...)
         except dataretrieval.DataRetrievalError:
             ...
 
-    Subclasses also inherit from the built-in exception this package has
-    historically raised for the condition's *kind* -- :class:`ValueError` for a
-    request that can't succeed as written (bad params, too large), and
-    :class:`RuntimeError` for a transient transport failure -- so existing
-    ``except ValueError`` / ``except RuntimeError`` handlers keep working.
+    Connection-level failures (timeouts, DNS) still surface as ``httpx``
+    exceptions on the single-shot request paths.
     """
 
+    def __reduce__(self) -> tuple[Any, ...]:
+        # The status subclasses take keyword-only fields (status_code /
+        # retry_after) that the default ``BaseException.__reduce__`` can't
+        # round-trip: it rebuilds via ``cls(*self.args)``, which raises TypeError
+        # before the saved state is applied. Rebuild from args + state via
+        # ``__new__`` instead so every subclass survives a pickle / deepcopy back
+        # from a multiprocessing / lithops worker. A subclass holding
+        # un-picklable live state (e.g. the chunker's ``ChunkInterrupted.call``)
+        # overrides ``_pickle_state`` to shed it.
+        return (_rebuild_error, (self.__class__, self.args, self._pickle_state()))
+
+    def _pickle_state(self) -> dict[str, Any]:
+        """Instance state to serialize in :meth:`__reduce__`.
+
+        Defaults to ``__dict__``; override to drop attributes that can't pickle.
+        """
+        return self.__dict__
+
+
+def _rebuild_error(
+    cls: type[DataRetrievalError],
+    args: tuple[Any, ...],
+    state: dict[str, Any],
+) -> DataRetrievalError:
+    """Rebuild a :class:`DataRetrievalError` without calling ``__init__``.
+
+    See :meth:`DataRetrievalError.__reduce__`.
+    """
+    err = cls.__new__(cls)
+    err.args = args
+    err.__dict__.update(state)
+    return err
 
-# --- Fatal client errors -------------------------------------------------
-# The request can't succeed as written; retrying it unchanged won't help. Each
-# is also a ``ValueError`` -- the built-in the legacy ``query`` path has always
-# raised -- so existing ``except ValueError`` handlers keep working.
 
+# --- HTTP status errors --------------------------------------------------
 
-class BadRequestError(DataRetrievalError, ValueError):
-    """The service rejected the request parameters (HTTP 400)."""
 
+class HTTPError(DataRetrievalError):
+    """The service returned an error HTTP status.
 
-class NotFoundError(DataRetrievalError, ValueError):
-    """The requested resource was not found; often an empty query (HTTP 404)."""
+    The numeric status is on :attr:`status_code`; branch on it, e.g.
+    ``except HTTPError as e: ... if e.status_code == 404``. :class:`TransientError`
+    (429 / 5xx) is the retryable subset, and is itself an ``HTTPError``. The one
+    exception to "a status is an ``HTTPError``" is a request the service rejects
+    as too long: it surfaces as :class:`URLTooLong` (a :class:`RequestTooLarge`),
+    *not* an ``HTTPError`` -- so catch :class:`DataRetrievalError` to be certain
+    of spanning every failure. See :func:`error_for_status` for the full mapping.
+
+    Parameters
+    ----------
+    message : str
+        Human-readable error message.
+    status_code : int
+        The HTTP status the service returned.
+    """
+
+    def __init__(self, message: str, *, status_code: int) -> None:
+        super().__init__(message)
+        self.status_code = status_code
+
+
+class TransientError(HTTPError):
+    """A 429 or 5xx the server may serve on a later try -- :class:`RateLimited`
+    for 429, :class:`ServiceUnavailable` for 5xx.
+
+    This only classifies the condition; it does not itself retry. Whether to
+    retry is up to the calling path: a single-shot request raises it for the
+    caller to handle (e.g. wait :attr:`retry_after` seconds, then re-issue),
+    while the Water Data chunker retries and resumes automatically.
+
+    Parameters
+    ----------
+    message : str
+        Human-readable error message.
+    status_code : int, optional
+        The HTTP status the service returned. Defaults to the leaf's canonical
+        code (429 / 503) when omitted; :func:`error_for_status` always passes the
+        real status.
+    retry_after : float, optional
+        Seconds to wait before retrying, parsed from the ``Retry-After`` response
+        header; ``None`` when the header is absent or unparseable.
+    """
+
+    #: Canonical status a concrete transient stamps when built without an
+    #: explicit ``status_code`` (:class:`RateLimited` = 429,
+    #: :class:`ServiceUnavailable` = 503). ``TransientError`` itself is abstract
+    #: and sets none, so constructing it bare requires ``status_code``.
+    _DEFAULT_STATUS: ClassVar[int]
+
+    def __init__(
+        self,
+        message: str,
+        *,
+        status_code: int | None = None,
+        retry_after: float | None = None,
+    ) -> None:
+        if status_code is None:
+            status_code = getattr(self, "_DEFAULT_STATUS", None)
+        if status_code is None:
+            raise TypeError(
+                f"{type(self).__name__} requires status_code "
+                "(only the RateLimited / ServiceUnavailable leaves default it)"
+            )
+        super().__init__(message, status_code=status_code)
+        self.retry_after = retry_after
+
+
+class RateLimited(TransientError):
+    """A request was rejected with HTTP 429 (too many requests)."""
 
+    _DEFAULT_STATUS = 429
 
-class RequestTooLarge(DataRetrievalError, ValueError):
+
+class ServiceUnavailable(TransientError):
+    """A request was rejected with a server error (HTTP 5xx).
+
+    Raised by both the legacy ``query`` path and the Water Data path, so a 5xx
+    surfaces as one type whichever subsystem issued the request. ``.status_code``
+    holds the actual 5xx; it falls back to 503 only on a bare hand-construction.
+    """
+
+    _DEFAULT_STATUS = 503
+
+
+# --- Request can't fit (not necessarily an HTTP status) ------------------
+
+
+class RequestTooLarge(DataRetrievalError):
     """The request is too large for the service to satisfy.
 
-    A base for the two ways a request can exceed what the service accepts;
-    catch it to handle either. The concrete subclasses are :class:`URLTooLong`
-    (a single request the server rejected) and :class:`Unchunkable` (the Water
-    Data chunker could not split the call small enough to fit).
+    Base for the two ways that happens; catch it to handle either:
+    :class:`URLTooLong` (a single request rejected for length) and
+    :class:`Unchunkable` (a Water Data call the chunker could not split small
+    enough to fit).
     """
 
 
 class URLTooLong(RequestTooLarge):
-    """A single request URL exceeded the service's limit (HTTP 414, or rejected
-    client-side before it was sent).
+    """A single request URL was too long for the service.
 
-    Raised by the legacy ``query`` path, which issues one request without
-    chunking. Remediation: query fewer sites, or split the call manually.
+    Raised on the legacy ``query`` path (which sends one un-chunked request),
+    whether the URL is rejected client-side before sending or by the server
+    (see :func:`error_for_status`). Remediation: query fewer sites, or split the
+    call manually.
     """
 
 
@@ -99,56 +212,57 @@ class Unchunkable(RequestTooLarge):
     """
 
 
-class NoSitesError(DataRetrievalError):
-    """The selection criteria matched no sites/data."""
+# --- Empty result --------------------------------------------------------
+
+
+class NoDataError(DataRetrievalError):
+    """The request succeeded (HTTP 200) but the selection criteria matched
+    no data."""
 
     def __init__(self, url: httpx.URL) -> None:
         self.url = url
 
     def __str__(self) -> str:
         return (
-            "No sites/data found using the selection criteria specified in "
-            f"url: {self.url}"
+            f"No data found using the selection criteria specified in url: {self.url}"
         )
 
 
-# --- Transient transport errors ------------------------------------------
-# The service was reachable but temporarily refused the request; the same call
-# may succeed if retried. Each is also a ``RuntimeError`` (the built-in the
-# waterdata path has always raised). The Water Data chunker recognizes them via
-# ``isinstance(exc, TransientError)`` and wraps them as resumable
-# ``ChunkInterrupted`` subclasses.
-
-
-class TransientError(DataRetrievalError, RuntimeError):
-    """Base for transient HTTP failures that are worth an automatic retry.
-
-    One subclass per recoverable HTTP status family (429 -> :class:`RateLimited`,
-    5xx -> :class:`ServiceUnavailable`); the Water Data chunker recognizes them
-    by this shared base and wraps them as resumable interruptions.
-
-    Parameters
-    ----------
-    message : str
-        Human-readable error message.
-    retry_after : float, optional
-        Seconds to wait before retrying, parsed from the ``Retry-After``
-        response header; stored on the :attr:`retry_after` attribute (``None``
-        when the header is absent or unparseable).
-    """
-
-    def __init__(self, message: str, *, retry_after: float | None = None) -> None:
-        super().__init__(message)
-        self.retry_after = retry_after
+#: Deprecated alias for :class:`NoDataError`. The original name leaked NWIS-era
+#: "sites" terminology; it is retained so existing ``except NoSitesError``
+#: handlers keep working, and will be removed in a future release.
+NoSitesError = NoDataError
 
 
-class RateLimited(TransientError):
-    """A request was rejected with HTTP 429 (too many requests)."""
+def error_for_status(
+    status: int, message: str, *, retry_after: float | None = None
+) -> DataRetrievalError:
+    """Return the typed :class:`DataRetrievalError` for an HTTP error *status*.
 
+    The one status-to-type mapping every request path shares (the legacy
+    ``query`` path, ``waterdata``, ``nadp`` / ``streamstats``), so a given status
+    becomes the same type everywhere:
 
-class ServiceUnavailable(TransientError):
-    """A request was rejected with a server error (HTTP 5xx).
+    * **413, 414** -> :class:`URLTooLong` (a :class:`RequestTooLarge`) -- the
+      "too long" semantic is more actionable than a bare status, and it matches
+      the client-side over-long-URL case
+    * **429** -> :class:`RateLimited`
+    * **5xx** -> :class:`ServiceUnavailable`
+    * **anything else** -> :class:`HTTPError`
 
-    Raised by both the legacy ``query`` path and the Water Data path, so a 5xx
-    surfaces as one type regardless of which subsystem issued the request.
+    ``message`` is used verbatim; ``retry_after`` is attached only to the
+    transient (:class:`TransientError`) types. *status* must be an error status
+    (``>= 400``) -- classifying a success or redirect is a usage error and raises
+    :class:`ValueError`.
     """
+    if status < 400:
+        raise ValueError(
+            f"error_for_status expects an HTTP error status (>= 400), got {status}"
+        )
+    if status in (413, 414):
+        return URLTooLong(message)
+    if status == 429:
+        return RateLimited(message, status_code=status, retry_after=retry_after)
+    if 500 <= status < 600:
+        return ServiceUnavailable(message, status_code=status, retry_after=retry_after)
+    return HTTPError(message, status_code=status)