Auto-retry live-API tests on transient upstream errors

#273 surfaced a transient HTTP 502 from upstream on its merge-to-main
CI run that pre-PR-273 would have been silently swallowed by
_walk_pages and turned into an empty DataFrame. The status-code-aware
behavior is correct for users, but it makes every test that hits the
live USGS Water Data API susceptible to flaking on a transient blip.

This PR adds:

- pytest-rerunfailures to the `test` optional dependency set.
- tests/conftest.py that (a) registers a `live` pytest marker; (b)
  auto-applies it to every test that does not take `requests_mock` as
  a fixture (the existing mock-driven convention in this repo); and
  (c) configures live-marked tests to retry up to twice on a 5-second
  backoff — but ONLY when the failure trace matches one of a narrow
  set of transient-upstream patterns: `429:` / `5xx:` prefixes from
  `_raise_for_non_200`, `ConnectionError` shapes, and timeout strings
  from the requests/urllib3 stack.

Library bugs raising other exception types are not retried — the
`only_rerun` pattern set deliberately matches on the failure-text
shape produced by `_raise_for_non_200` rather than on generic
`RuntimeError`, so a real bug fails on the first attempt.

Verified: pytest collection identifies 59 of 61 tests in
waterdata_test.py as live (the two `*_mock_*` tests correctly stay
unmarked). Offline tests (`_get_args`, `_check_*`, `_normalize_str_*`,
`_construct_api_requests_*`) pass unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: thodson-usgs

NEWS.md

@@ -1,3 +1,5 @@
+**05/15/2026:** Live-API tests are now auto-retried on transient upstream errors. PR #273 (above) intentionally surfaces mid-pagination HTTP failures to callers instead of silently truncating results, which made any CI test that hits the live USGS Water Data API susceptible to flaking on transient 429/5xx/connection blips — a 502 Bad Gateway from upstream broke CI on the #273 merge to main. A new `tests/conftest.py` registers a `live` pytest marker, auto-applies it to every test that does not take `requests_mock` (the existing mock-driven convention), and via `pytest-rerunfailures` retries those tests up to twice on a 5-second backoff — but only when the failure trace matches a narrow transient-upstream pattern (`429:` / `5xx:` prefixes from `_raise_for_non_200`, plus `ConnectionError` / timeout shapes). Library bugs raising other exception types still fail on the first attempt.
+
 **05/14/2026:** Fixed two latent bugs in the paginated `waterdata` request loop (`_walk_pages` and `get_stats_data`). Previously, when `requests.Session.request(...)` itself raised mid-pagination (network error, timeout), the except block called `_error_body()` on the *prior page's* response, so the logged "error" described the wrong request and could itself crash on non-JSON bodies. Separately, no status-code check was performed on subsequent paginated responses, so a 5xx body that didn't include `numberReturned` was silently treated as an empty page — pagination quietly stopped and the user got truncated data with no error logged. The loop now status-checks each page like the initial request and reports the actual exception. The "best-effort" behavior (return whatever pages were collected) is preserved.
 
 **05/07/2026:** Bumped the declared minimum Python version from **3.8** to **3.9** (`pyproject.toml`'s `requires-python` and the ruff target). This brings the manifest in line with what was already being tested — CI's matrix has long covered only 3.9, 3.13, and 3.14, the `waterdata` test module already skipped itself on Python < 3.10, and several modules already use 3.9-only stdlib (e.g. `zoneinfo`). Users on 3.8 will no longer be able to install the package; please upgrade.

pyproject.toml

@@ -35,6 +35,7 @@ dataretrieval = ["py.typed"]
 test = [
   "pytest > 5.0.0",
   "pytest-cov[all]",
+  "pytest-rerunfailures",
   "coverage",
   "requests-mock",
   "ruff",

tests/conftest.py

@@ -0,0 +1,71 @@
+"""Test infrastructure shared across all test modules.
+
+After #273 the paginated ``waterdata`` getters surface mid-walk HTTP
+errors (429 / 5xx / connection drops) to the caller instead of silently
+truncating the result. That's the correct behavior for users — but it
+makes any CI test that hits the live USGS Water Data API susceptible to
+flaking on a transient upstream blip (e.g. the 502 Bad Gateway that
+broke CI on #273's merge to main).
+
+This file:
+
+* registers a ``live`` pytest marker;
+* auto-applies it to every test that does **not** take ``requests_mock``
+  as a fixture — the existing convention in this repo for mock-driven
+  tests, so the marker tracks "this test hits the network" without
+  needing to decorate 35 functions by hand;
+* via ``pytest-rerunfailures``, configures live-marked tests to retry
+  up to twice (5-second backoff) ONLY when the failure trace matches a
+  transient-upstream pattern: ``429:`` / ``5xx:`` prefixes that
+  ``_raise_for_non_200`` produces, plus ``ConnectionError`` / timeout
+  shapes from the ``requests`` library.
+
+Library bugs that raise unrelated exception types are NOT retried —
+the regex set deliberately omits generic ``RuntimeError`` matches.
+"""
+
+import pytest
+
+# Match anywhere in the failure traceback. Anchor-free because the
+# trace embeds the exception message after a long preamble.
+_TRANSIENT_PATTERNS = [
+    r"RuntimeError:\s*(?:429|5\d\d):",  # _raise_for_non_200 output
+    r"ConnectionError",  # requests/urllib3
+    r"ReadTimeout|ConnectTimeout|Timeout",
+    r"timed out",
+    r"Bad Gateway|Service Unavailable|Gateway Timeout",
+]
+
+_MAX_RERUNS = 2
+_RERUN_DELAY_SEC = 5
+
+
+def pytest_configure(config):
+    """Register the ``live`` marker so pytest doesn't warn about it."""
+    config.addinivalue_line(
+        "markers",
+        "live: marks tests that hit the live USGS Water Data API; "
+        "retried up to twice on transient upstream HTTP errors.",
+    )
+
+
+def pytest_collection_modifyitems(config, items):
+    """Auto-mark live-API tests with ``live`` + ``flaky`` retry config.
+
+    Heuristic: any test that does NOT request the ``requests_mock``
+    fixture is treated as live. This catches every test in
+    ``waterdata_test.py`` and similar modules without needing to
+    decorate them individually, and it auto-tracks new tests written
+    in the same style.
+    """
+    for item in items:
+        if "requests_mock" in getattr(item, "fixturenames", ()):
+            continue
+        item.add_marker(pytest.mark.live)
+        item.add_marker(
+            pytest.mark.flaky(
+                reruns=_MAX_RERUNS,
+                reruns_delay=_RERUN_DELAY_SEC,
+                only_rerun=_TRANSIENT_PATTERNS,
+            )
+        )