Merge remote-tracking branch 'origin/main' into worktree-waterdata-progress-bar

# Conflicts:
#	README.md
#	dataretrieval/waterdata/filters.py
#	dataretrieval/waterdata/utils.py
#	tests/waterdata_utils_test.py

Author: thodson-usgs

NEWS.md

@@ -1,4 +1,6 @@
-**05/14/2026:** Fixed two latent bugs in the paginated `waterdata` request loop (`_walk_pages` and `get_stats_data`). Previously, when `requests.Session.request(...)` itself raised mid-pagination (network error, timeout), the except block called `_error_body()` on the *prior page's* response, so the logged "error" described the wrong request and could itself crash on non-JSON bodies. Separately, no status-code check was performed on subsequent paginated responses, so a 5xx body that didn't include `numberReturned` was silently treated as an empty page — pagination quietly stopped and the user got truncated data with no error logged. The loop now status-checks each page like the initial request and reports the actual exception. The "best-effort" behavior (return whatever pages were collected) is preserved.
+**05/17/2026:** The OGC `waterdata` getters (`get_daily`, `get_continuous`, `get_field_measurements`, and the rest of the multi-value-capable functions) now transparently chunk requests whose URLs would otherwise exceed the server's ~8 KB byte limit.
+
+**05/16/2026:** Fixed silent truncation in the paginated `waterdata` request loops (`_walk_pages` and `get_stats_data`). Mid-pagination failures (HTTP 429, 5xx, network error) were previously swallowed — pagination would quietly stop and the function would return whatever rows it had collected, leaving callers with truncated DataFrames they had no way to detect. The loops now status-check every page like the initial request and raise `RuntimeError` on any failure, with the upstream exception chained as `__cause__` and a short menu of recovery actions (wait and retry, reduce the request, or obtain an API token) in the message. **Behavior change**: callers that previously consumed partial DataFrames on transient upstream blips will now see an exception; retry the call (possibly with a smaller `limit` or narrower query).
 
 **05/07/2026:** Bumped the declared minimum Python version from **3.8** to **3.9** (`pyproject.toml`'s `requires-python` and the ruff target). This brings the manifest in line with what was already being tested — CI's matrix has long covered only 3.9, 3.13, and 3.14, the `waterdata` test module already skipped itself on Python < 3.10, and several modules already use 3.9-only stdlib (e.g. `zoneinfo`). Users on 3.8 will no longer be able to install the package; please upgrade.
 
@@ -36,4 +38,4 @@
 
 **03/01/2024:** USGS data availability and format have changed on Water Quality Portal (WQP). Since March 2024, data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. All USGS data (up to and beyond March 2024) are available using the new WQP beta services. You can access the beta services by setting `legacy=False` in the functions in the `wqp` module.
 
-To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html
+To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html

README.md

@@ -4,29 +4,6 @@
 ![Conda Version](https://img.shields.io/conda/v/conda-forge/dataretrieval)
 ![Downloads](https://static.pepy.tech/badge/dataretrieval)
 
-## Latest Announcements
-
-**02/24/2026** The `get_gwlevels`, `get_discharge_measurements` functions in the `nwis` module are defunct and have been replaced with the `get_field_measurements` function in the `waterdata` module. The `get_pmcodes` function in the `nwis` module has been replaced with the `get_reference_table(collection='parameter_code)` function. Finally, the `get_water_use` function in the `nwis` module is defunct with no current replacement.
-
-:mega: **01/16/2026:** `dataretrieval` now features the `waterdata` module,
-which provides access to USGS's modernized [Water Data
-APIs](https://api.waterdata.usgs.gov/). The Water Data API endpoints include
-daily values, instantaneous values, field measurements, time series metadata, statistics,
-and discrete water quality data from the [Samples database](https://waterdata.usgs.gov/download-samples/#dataProfile=site). This new module replaces the `nwis` module, which provides access to the legacy [NWIS
-Water Services](https://waterservices.usgs.gov/). Take a look at the new [`waterdata` module demo notebook](demos/WaterData_demo.ipynb), which walks through an extended example using a majority of the available `waterdata` functions.
-
-Check out the [NEWS](NEWS.md) file for all updates and announcements.
-
-**Important:** Users of the Water Data APIs are strongly encouraged to obtain an
-API key for higher rate limits and greater access to USGS data. [Register for
-an API key](https://api.waterdata.usgs.gov/signup/) and set it as an
-environment variable:
-
-```python
-import os
-os.environ["API_USGS_PAT"] = "your_api_key_here"
-```
-
 ## What is dataretrieval?
 
 `dataretrieval` simplifies the process of loading hydrologic data into Python.
@@ -36,6 +13,8 @@ U.S. Geological Survey (USGS) hydrology data types available on the Web, as well
 as data from the Water Quality Portal (WQP) and Network Linked Data Index
 (NLDI).
 
+Check the [NEWS](NEWS.md) for all updates and announcements.
+
 ## Installation
 
 Install dataretrieval using pip:
@@ -44,13 +23,13 @@ Install dataretrieval using pip:
 pip install dataretrieval
 ```
 
-Or using conda:
+Or conda:
 
 ```bash
 conda install -c conda-forge dataretrieval
 ```
 
-To install the "main" branch directly from GitHub, use:
+Or directly from GitHub:
 
 ```bash
 pip install git+https://github.com/DOI-USGS/dataretrieval-python.git
@@ -60,11 +39,20 @@ pip install git+https://github.com/DOI-USGS/dataretrieval-python.git
 
 ### Water Data API (Recommended - Modern USGS Data)
 
-The `waterdata` module provides access to modern USGS Water Data APIs.
+Access USGS water-monitoring data.
 
-Some basic usage examples include retrieving daily streamflow data for a
-specific monitoring location, where the `/` in the `time` argument indicates
-the desired range:
+**Important:** Users are strongly encouraged to obtain an API key for higher
+rate limits. [Register for an API key](https://api.waterdata.usgs.gov/signup/)
+and set it as an environment variable:
+
+```python
+import os
+os.environ["API_USGS_PAT"] = "your_api_key_here"
+```
+
+The following example retrieves daily streamflow data for a specific
+monitoring location. The `/` in the `time` argument separates the start and
+end of the desired range:
 
 ```python
 from dataretrieval import waterdata
@@ -80,7 +68,7 @@ print(f"Retrieved {len(df)} records")
 print(f"Site: {df['monitoring_location_id'].iloc[0]}")
 print(f"Mean discharge: {df['value'].mean():.2f} {df['unit_of_measure'].iloc[0]}")
 ```
-Retrieving streamflow at multiple locations from October 1, 2024 to the present:
+Retrieve streamflow at multiple locations from October 1, 2024 to the present:
 
 ```python
 df, metadata = waterdata.get_daily(
@@ -91,8 +79,8 @@ df, metadata = waterdata.get_daily(
 
 print(f"Retrieved {len(df)} records")
 ```
-Retrieving location information for all monitoring locations categorized as
-stream sites in the state of Maryland:
+Retrieve location information for all monitoring locations categorized as
+stream sites in Maryland:
 
 ```python
 # Get monitoring location information
@@ -103,8 +91,9 @@ df, metadata = waterdata.get_monitoring_locations(
 
 print(f"Found {len(df)} stream monitoring locations in Maryland")
 ```
-Finally, retrieving continuous (a.k.a. "instantaneous") data
-for one location. We *strongly advise* breaking up continuous data requests into smaller time periods and collections to avoid timeouts and other issues:
+Finally, retrieve continuous (a.k.a. "instantaneous") data for one location.
+We *strongly advise* breaking continuous data requests into smaller time
+windows to avoid timeouts and other issues:
 
 ```python
 # Get continuous data for a single monitoring location and water year
@@ -118,14 +107,15 @@ print(f"Retrieved {len(df)} continuous gage height measurements")
 
 Visit the
 [API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html)
-for more information and examples on available services and input parameters. 
+for more information and examples on available services and input parameters.
 
 **Tracking progress:** Paginated and chunked `waterdata` queries report their
 progress on a single, self-updating line on `stderr` — showing the chunk and
-page counts, rows retrieved so far, and the API requests remaining for the hour:
+page counts, rows retrieved so far, and the API requests remaining (with the
+time until the hourly limit resets, when the server reports it):
 
 ```text
-Progress: chunk 2/5 · 14 pages · 8,421 rows · 4,870 requests left
+Progress: chunk 2/5 · 14 pages · 8,421 rows · 4,870 requests remaining, resets in 47m
 ```
 
 The line appears automatically when `stderr` is an interactive terminal.
@@ -139,29 +129,6 @@ import logging
 logging.basicConfig(level=logging.DEBUG)
 ```
 
-### Legacy NWIS Services (Deprecated but still functional)
-
-The `nwis` module accesses legacy NWIS Water Services:
-
-```python
-from dataretrieval import nwis
-
-# Get site information
-info, metadata = nwis.get_info(sites='01646500')
-    
-print(f"Site name: {info['station_nm'].iloc[0]}")
-
-# Get daily values
-dv, metadata = nwis.get_dv(
-  sites='01646500',
-  start='2024-10-01',
-  end='2024-10-02',
-  parameterCd='00060',
-)
-    
-print(f"Retrieved {len(dv)} daily values")
-```
-
 ### Water Quality Portal (WQP)
 
 Access water quality data from multiple agencies:
@@ -247,13 +214,13 @@ print(f"Found {len(flowlines)} upstream tributaries within 50km")
 ## More Examples
 
 Explore additional examples in the
-[`demos`](https://github.com/USGS-python/dataretrieval/tree/main/demos)
+[`demos`](https://github.com/DOI-USGS/dataretrieval-python/tree/main/demos)
 directory, including Jupyter notebooks demonstrating advanced usage patterns.
 
 ## Getting Help
 
-- **Issue tracker**: Report bugs and request features at https://github.com/USGS-python/dataretrieval/issues
-- **Documentation**: Full API documentation available in the source code docstrings
+- **Issue tracker**: Report bugs and request features at https://github.com/DOI-USGS/dataretrieval-python/issues
+- **Documentation**: https://doi-usgs.github.io/dataretrieval-python/
 
 ## Contributing
 

dataretrieval/waterdata/_progress.py

@@ -6,7 +6,7 @@
 and ``utils.get_stats_data``). This module surfaces that work as one line on
 stderr, rewritten in place as data arrives::
 
-    Progress: chunk 2/5 · 14 pages · 8,421 rows · 4,870 requests left
+    Progress: chunk 2/5 · 14 pages · 8,421 rows · 4,870 requests remaining
 
 It replaces the per-page ``logger.info`` calls that previously narrated the same
 events one line at a time.
@@ -27,10 +27,24 @@
 import contextvars
 import os
 import sys
+import time
 from collections.abc import Iterator
 from contextlib import contextmanager
 from typing import TextIO
 
+
+def _format_duration(seconds: float) -> str:
+    """Compact human duration: ``45s``, ``12m``, ``1h03m`` (clamped at 0)."""
+    secs = int(max(0, seconds))
+    if secs < 60:
+        return f"{secs}s"
+    if secs < 3600:
+        return f"{secs // 60}m"
+    hours, rem = divmod(secs, 3600)
+    minutes = rem // 60
+    return f"{hours}h{minutes:02d}m" if minutes else f"{hours}h"
+
+
 # The reporter active for the current query. A ContextVar (not a module global)
 # so concurrent queries — threads or async tasks sharing a client — each track
 # their own progress line.
@@ -74,6 +88,9 @@ def __init__(
         self.pages = 0
         self.rows = 0
         self.rate_remaining: str | None = None
+        # Absolute epoch second when the rate-limit window resets, derived from
+        # the server's reset header so the rendered countdown stays live.
+        self._reset_at: float | None = None
         self._last_len = 0
         self._closed = False
 
@@ -82,24 +99,42 @@ def set_chunks(self, total: int) -> None:
         self.total_chunks = max(int(total), 1)
 
     def start_chunk(self, index: int) -> None:
-        """Mark the start of chunk ``index`` (1-based) and redraw."""
+        """Mark the start of chunk ``index`` (1-based) and redraw.
+
+        Only redraws when actually chunking (``total_chunks > 1``); a
+        single-chunk plan has nothing chunk-specific to show yet, so it
+        avoids a premature "0 pages" frame before the first page arrives.
+        """
         self.current_chunk = index
-        self._render()
+        if self.total_chunks > 1:
+            self._render()
 
     def add_page(self, rows: int = 0) -> None:
         """Record one fetched page carrying ``rows`` rows and redraw."""
         self.pages += 1
         self.rows += int(rows)
         self._render()
 
-    def set_rate_remaining(self, value: str | int | None) -> None:
-        """Update the remaining-requests count from an ``x-ratelimit-remaining`` header.
-
-        Ignores empty/missing values so a page that omits the header doesn't
-        blank out the last known count.
+    def set_rate_remaining(
+        self, value: str | int | None, reset: str | int | None = None
+    ) -> None:
+        """Update the rate-limit display from the response headers.
+
+        ``value`` is ``x-ratelimit-remaining``; ``reset`` is the optional
+        ``x-ratelimit-reset`` companion. Empty/missing values are ignored so a
+        page that omits a header doesn't blank out the last known value. The
+        reset value is interpreted as an absolute epoch second when large
+        (the conventional form) and as seconds-until-reset otherwise; either
+        way it's stored as an absolute deadline so the countdown stays live.
         """
         if value not in (None, ""):
             self.rate_remaining = str(value)
+        if reset not in (None, ""):
+            try:
+                secs = float(reset)
+            except (TypeError, ValueError):
+                return
+            self._reset_at = secs if secs > 1_000_000 else time.time() + secs
 
     def _format(self) -> str:
         parts: list[str] = []
@@ -114,7 +149,11 @@ def _format(self) -> str:
             # alone is True for non-decimal unicode digits that ``int`` rejects.)
             rate = self.rate_remaining
             rate = f"{int(rate):,}" if rate.isascii() and rate.isdigit() else rate
-            parts.append(f"{rate} requests left")
+            segment = f"{rate} requests remaining"
+            if self._reset_at is not None:
+                eta = _format_duration(self._reset_at - time.time())
+                segment += f", resets in {eta}"
+            parts.append(segment)
         return "Progress: " + " · ".join(parts)
 
     def _render(self) -> None:

dataretrieval/waterdata/api.py

@@ -113,7 +113,7 @@ def get_daily(
         data are released on the condition that neither the USGS nor the United
         States Government may be held liable for any damages resulting from its
         use. This field reflects the approval status of each record, and is either
-        "Approved", meaining processing review has been completed and the data is
+        "Approved", meaning processing review has been completed and the data is
         approved for publication, or "Provisional" and subject to revision. For
         more information about provisional data, go to:
         https://waterdata.usgs.gov/provisional-data-statement/.
@@ -230,6 +230,21 @@ def get_daily(
         ...     parameter_code="00060",
         ...     last_modified="P7D",
         ... )
+
+        >>> # Chain queries: pull all stream sites in a state, then their
+        >>> # daily discharge for the last week. The site list can be hundreds
+        >>> # of values long — the request is transparently chunked across
+        >>> # multiple sub-requests so the URL stays under the server's byte
+        >>> # limit. Combined output looks like a single query.
+        >>> sites_df, _ = dataretrieval.waterdata.get_monitoring_locations(
+        ...     state_name="Ohio",
+        ...     site_type="Stream",
+        ... )
+        >>> df, md = dataretrieval.waterdata.get_daily(
+        ...     monitoring_location_id=sites_df["monitoring_location_id"].tolist(),
+        ...     parameter_code="00060",
+        ...     time="P7D",
+        ... )
     """
     service = "daily"
     output_id = "daily_id"
@@ -259,7 +274,7 @@ def get_continuous(
     convert_type: bool = True,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """
-    Continuous data provide instantanous water conditions.
+    Continuous data provide instantaneous water conditions.
 
     This is an early version of the continuous endpoint that is feature-complete
     and is being made available for limited use.  Geometries are not included
@@ -320,7 +335,7 @@ def get_continuous(
         data are released on the condition that neither the USGS nor the United
         States Government may be held liable for any damages resulting from its
         use. This field reflects the approval status of each record, and is either
-        "Approved", meaining processing review has been completed and the data is
+        "Approved", meaning processing review has been completed and the data is
         approved for publication, or "Provisional" and subject to revision. For
         more information about provisional data, go to:
         https://waterdata.usgs.gov/provisional-data-statement/.
@@ -1254,7 +1269,7 @@ def get_latest_continuous(
         data are released on the condition that neither the USGS nor the United
         States Government may be held liable for any damages resulting from its
         use. This field reflects the approval status of each record, and is either
-        "Approved", meaining processing review has been completed and the data is
+        "Approved", meaning processing review has been completed and the data is
         approved for publication, or "Provisional" and subject to revision. For
         more information about provisional data, go to:
         https://waterdata.usgs.gov/provisional-data-statement/.
@@ -1451,7 +1466,7 @@ def get_latest_daily(
         data are released on the condition that neither the USGS nor the United
         States Government may be held liable for any damages resulting from its
         use. This field reflects the approval status of each record, and is either
-        "Approved", meaining processing review has been completed and the data is
+        "Approved", meaning processing review has been completed and the data is
         approved for publication, or "Provisional" and subject to revision. For
         more information about provisional data, go to:
         https://waterdata.usgs.gov/provisional-data-statement/.
@@ -1633,7 +1648,7 @@ def get_field_measurements(
         data are released on the condition that neither the USGS nor the United
         States Government may be held liable for any damages resulting from its
         use. This field reflects the approval status of each record, and is either
-        "Approved", meaining processing review has been completed and the data is
+        "Approved", meaning processing review has been completed and the data is
         approved for publication, or "Provisional" and subject to revision. For
         more information about provisional data, go to:
         https://waterdata.usgs.gov/provisional-data-statement/.