Skip to content

Commit c7d7bab

Browse files
davetapleydavetapley
authored
Merge branch 'DOI-USGS:main' into py-typed
2 parents 560fa82 + 98f4ddd commit c7d7bab

29 files changed

Lines changed: 3053 additions & 58184 deletions

‎.github/workflows/python-package.yml‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
fail-fast: false
1717
matrix:
1818
os: [ubuntu-latest, windows-latest]
19-
python-version: [3.8, 3.9, '3.10', 3.11, 3.12]
19+
python-version: [3.9, 3.13, 3.14]
2020

2121
steps:
2222
- uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871
@@ -36,7 +36,5 @@ jobs:
3636
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
3737
- name: Test with pytest and report coverage
3838
run: |
39-
cd tests
40-
coverage run -m pytest
39+
coverage run -m pytest tests/
4140
coverage report -m
42-
cd ..

‎.github/workflows/sphinx-docs.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,9 @@ jobs:
1818
shell: bash -l {0}
1919
run: |
2020
python -m pip install --upgrade pip
21-
pip install .[doc]
21+
pip install .[doc,nldi]
2222
ipython kernel install --name "python3" --user
2323
sudo apt update -y && sudo apt install -y latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended dvipng pandoc
24-
(cd docs && make docs)
2524
(cd docs && make html)
2625
- name: Debug
2726
run: |

‎NEWS.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
**12/04/2025:** The `get_continuous()` function was added to the `waterdata` module, which provides access to measurements collected via automated sensors at a high frequency (often 15 minute intervals) at a monitoring location. This is an early version of the continuous endpoint and should be used with caution as the API team improves its performance. In the future, we anticipate the addition of an endpoint(s) specifically for handling large data requests, so it may make sense for power users to hold off on heavy development using the new continuous endpoint.
2+
3+
**11/24/2025:** `dataretrieval` is pleased to offer a new module, `waterdata`, which gives users access USGS's modernized [Water Data APIs](https://api.waterdata.usgs.gov/). The Water Data API endpoints include daily values, instantaneous values, field measurements (modernized groundwater levels service), time series metadata, and discrete water quality data from the Samples database. Though there will be a period of overlap, the functions within `waterdata` will eventually replace the `nwis` module, which currently provides access to the legacy [NWIS Water Services](https://waterservices.usgs.gov/). More example workflows and functions coming soon. Check `help(waterdata)` for more information.
4+
5+
**09/03/2024:** The groundwater levels service has switched endpoints, and `dataretrieval` was updated accordingly in [`v1.0.10`](https://github.com/DOI-USGS/dataretrieval-python/releases/tag/v1.0.10). Older versions using the discontinued endpoint will return 503 errors for `nwis.get_gwlevels` or the `service='gwlevels'` argument. Visit [Water Data For the Nation](https://waterdata.usgs.gov/blog/wdfn-waterservices-2024/) for more information.
6+
7+
**03/01/2024:** USGS data availability and format have changed on Water Quality Portal (WQP). Since March 2024, data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. All USGS data (up to and beyond March 2024) are available using the new WQP beta services. You can access the beta services by setting `legacy=False` in the functions in the `wqp` module.
8+
9+
To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html

‎README.md‎

Lines changed: 234 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -4,123 +4,281 @@
44
![Conda Version](https://img.shields.io/conda/v/conda-forge/dataretrieval)
55
![Downloads](https://static.pepy.tech/badge/dataretrieval)
66

7-
:warning: USGS data availability and format have changed on Water Quality Portal (WQP). Since March 2024, data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. All USGS data (up to and beyond March 2024) are available using the new WQP beta services. You can access the beta services by setting `legacy=False` in the functions in the `wqp` module.
7+
## Latest Announcements
88

9-
To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html
9+
:mega: **12/04/2025:** `dataretrieval` now features the new `waterdata` module,
10+
which provides access to USGS's modernized [Water Data
11+
APIs](https://api.waterdata.usgs.gov/). The Water Data API endpoints include
12+
daily values, **instantaneous values**, field measurements, time series metadata,
13+
and discrete water quality data from the Samples database. This new module will
14+
eventually replace the `nwis` module, which provides access to the legacy [NWIS
15+
Water Services](https://waterservices.usgs.gov/).
1016

11-
:mega: **09/03/2024:** The groundwater levels service has switched endpoints, and `dataretrieval` was updated accordingly in [`v1.0.10`](https://github.com/DOI-USGS/dataretrieval-python/releases/tag/v1.0.10). Older versions using the discontinued endpoint will return 503 errors for `nwis.get_gwlevels` or the `service='gwlevels'` argument. Visit [Water Data For the Nation](https://waterdata.usgs.gov/blog/wdfn-waterservices-2024/) for more information.
17+
Check out the [NEWS](NEWS.md) file for all updates and announcements.
18+
19+
**Important:** Users of the Water Data APIs are strongly encouraged to obtain an
20+
API key for higher rate limits and greater access to USGS data. [Register for
21+
an API key](https://api.waterdata.usgs.gov/signup/) and set it as an
22+
environment variable:
23+
24+
```python
25+
import os
26+
os.environ["API_USGS_PAT"] = "your_api_key_here"
27+
```
1228

1329
## What is dataretrieval?
14-
`dataretrieval` was created to simplify the process of loading hydrologic data into the Python environment.
15-
Like the original R version [`dataRetrieval`](https://github.com/DOI-USGS/dataRetrieval),
16-
it is designed to retrieve the major data types of U.S. Geological Survey (USGS) hydrology
17-
data that are available on the Web, as well as data from the Water
18-
Quality Portal (WQP), which currently houses water quality data from the
19-
Environmental Protection Agency (EPA), U.S. Department of Agriculture
20-
(USDA), and USGS. Direct USGS data is obtained from a service called the
21-
National Water Information System (NWIS).
2230

23-
Note that the python version is not a direct port of the original: it attempts to reproduce the functionality of the R package,
24-
though its organization and interface often differ.
31+
`dataretrieval` simplifies the process of loading hydrologic data into Python.
32+
Like the original R version
33+
[`dataRetrieval`](https://github.com/DOI-USGS/dataRetrieval), it retrieves major
34+
U.S. Geological Survey (USGS) hydrology data types available on the Web, as well
35+
as data from the Water Quality Portal (WQP) and Network Linked Data Index
36+
(NLDI).
37+
38+
## Installation
2539

26-
If there's a hydrologic or environmental data portal that you'd like dataretrieval to
27-
work with, raise it as an [issue](https://github.com/USGS-python/dataretrieval/issues).
40+
Install dataretrieval using pip:
41+
42+
```bash
43+
pip install dataretrieval
44+
```
45+
46+
Or using conda:
47+
48+
```bash
49+
conda install -c conda-forge dataretrieval
50+
```
2851

29-
Here's an example using `dataretrieval` to retrieve data from the National Water Information System (NWIS).
52+
To install the "main" branch directly from GitHub, use:
53+
54+
```bash
55+
pip install git+https://github.com/DOI-USGS/dataretrieval-python.git
56+
```
57+
58+
## Usage Examples
59+
60+
### Water Data API (Recommended - Modern USGS Data)
61+
62+
The `waterdata` module provides access to modern USGS Water Data APIs.
63+
64+
Some basic usage examples include retrieving daily streamflow data for a
65+
specific monitoring location, where the `/` in the `time` argument indicates
66+
the desired range:
3067

3168
```python
32-
# first import the functions for downloading data from NWIS
33-
import dataretrieval.nwis as nwis
69+
from dataretrieval import waterdata
70+
71+
# Get daily streamflow data (returns DataFrame and metadata)
72+
df, metadata = waterdata.get_daily(
73+
monitoring_location_id='USGS-01646500',
74+
parameter_code='00060', # Discharge
75+
time='2024-10-01/2025-09-30'
76+
)
77+
78+
print(f"Retrieved {len(df)} records")
79+
print(f"Site: {df['monitoring_location_id'].iloc[0]}")
80+
print(f"Mean discharge: {df['value'].mean():.2f} {df['unit_of_measure'].iloc[0]}")
81+
```
82+
Retrieving streamflow at multiple locations from October 1, 2024 to the present:
3483

35-
# specify the USGS site code for which we want data.
36-
site = '03339000'
84+
```python
85+
df, metadata = waterdata.get_daily(
86+
monitoring_location_id=["USGS-13018750","USGS-13013650"],
87+
parameter_code='00060',
88+
time='2024-10-01/..'
89+
)
3790

38-
# get instantaneous values (iv)
39-
df = nwis.get_record(sites=site, service='iv', start='2017-12-31', end='2018-01-01')
91+
print(f"Retrieved {len(df)} records")
92+
```
93+
Retrieving location information for all monitoring locations categorized as
94+
stream sites in the state of Maryland:
4095

41-
# get basic info about the site
42-
df2 = nwis.get_record(sites=site, service='site')
96+
```python
97+
# Get monitoring location information
98+
df, metadata = waterdata.get_monitoring_locations(
99+
state_name='Maryland',
100+
site_type_code='ST' # Stream sites
101+
)
102+
103+
print(f"Found {len(df)} stream monitoring locations in Maryland")
43104
```
44-
Services available from NWIS include:
45-
- instantaneous values (iv)
46-
- daily values (dv)
47-
- statistics (stat)
48-
- site info (site)
49-
- discharge peaks (peaks)
50-
- discharge measurements (measurements)
51-
52-
Water quality data are available from:
53-
- [Samples](https://waterdata.usgs.gov/download-samples/#dataProfile=site) - Discrete USGS water quality data only
54-
- [Water Quality Portal](https://www.waterqualitydata.us/) - Discrete water quality data from USGS and EPA. Older data are available in the legacy WQX version 2 format; all data are available in the beta WQX3.0 format.
55-
56-
To access the full functionality available from NWIS web services, nwis.get record appends any additional kwargs into the REST request. For example, this function call:
105+
Finally, retrieving continuous (a.k.a. "instantaneous") data
106+
for one location. We *strongly advise* breaking up continuous data requests into smaller time periods and collections to avoid timeouts and other issues:
107+
57108
```python
58-
nwis.get_record(sites='03339000', service='dv', start='2017-12-31', parameterCd='00060')
109+
# Get continuous data for a single monitoring location and water year
110+
df, metadata = waterdata.get_continuous(
111+
monitoring_location_id='USGS-01646500',
112+
parameter_code='00065', # Gage height
113+
time='2024-10-01/2025-09-30'
114+
)
115+
print(f"Retrieved {len(df)} continuous gage height measurements")
59116
```
60-
...will download daily data with the parameter code 00060 (discharge).
61117

62-
## Accessing the "Internal" NWIS
63-
If you're connected to the USGS network, dataretrieval call pull from the internal (non-public) NWIS interface.
64-
Most dataretrieval functions pass kwargs directly to NWIS's REST API, which provides simple access to internal data; simply specify "access='3'".
65-
For example
118+
Visit the
119+
[API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html)
120+
for more information and examples on available services and input parameters.
121+
122+
**NEW:** This new module implements
123+
[logging](https://docs.python.org/3/howto/logging.html#logging-basic-tutorial)
124+
in which users can view the URL requests sent to the USGS Water Data APIs
125+
and the number of requests they have remaining each hour. These messages can
126+
be helpful for troubleshooting and support. To enable logging in your python
127+
console or notebook:
128+
66129
```python
67-
nwis.get_record(sites='05404147',service='iv', start='2021-01-01', end='2021-3-01', access='3')
130+
import logging
131+
logging.basicConfig(level=logging.INFO)
132+
```
133+
To log messages to a file, you can specify a filename in the
134+
`basicConfig` call:
135+
136+
```python
137+
logging.basicConfig(filename='waterdata.log', level=logging.INFO)
68138
```
69139

70-
More services and documentation to come!
140+
### NWIS Legacy Services (Deprecated but still functional)
141+
142+
The `nwis` module accesses legacy NWIS Water Services:
143+
144+
```python
145+
from dataretrieval import nwis
146+
147+
# Get site information
148+
info, metadata = nwis.get_info(sites='01646500')
149+
150+
print(f"Site name: {info['station_nm'].iloc[0]}")
151+
152+
# Get daily values
153+
dv, metadata = nwis.get_dv(
154+
sites='01646500',
155+
start='2024-10-01',
156+
end='2024-10-02',
157+
parameterCd='00060',
158+
)
159+
160+
print(f"Retrieved {len(dv)} daily values")
161+
```
71162

72-
## Quick start
163+
### Water Quality Portal (WQP)
73164

74-
dataretrieval can be installed using pip:
75-
76-
$ python3 -m pip install -U dataretrieval
165+
Access water quality data from multiple agencies:
77166

78-
or conda:
167+
```python
168+
from dataretrieval import wqp
79169

80-
$ conda install -c conda-forge dataretrieval
170+
# Find water quality monitoring sites
171+
sites = wqp.what_sites(
172+
statecode='US:55', # Wisconsin
173+
siteType='Stream'
174+
)
81175

82-
More examples of use are include in [`demos`](https://github.com/USGS-python/dataretrieval/tree/main/demos).
176+
print(f"Found {len(sites)} stream monitoring sites in Wisconsin")
83177

84-
## Issue tracker
178+
# Get water quality results
179+
results = wqp.get_results(
180+
siteid='USGS-05427718',
181+
characteristicName='Temperature, water'
182+
)
85183

86-
Please report any bugs and enhancement ideas using the dataretrieval issue
87-
tracker:
184+
print(f"Retrieved {len(results)} temperature measurements")
185+
```
88186

89-
https://github.com/USGS-python/dataretrieval/issues
187+
### Network Linked Data Index (NLDI)
90188

91-
Feel free to also ask questions on the tracker.
189+
Discover and navigate hydrologic networks:
92190

191+
```python
192+
from dataretrieval import nldi
93193

94-
## Contributing
194+
# Get watershed basin for a stream reach
195+
basin = nldi.get_basin(
196+
feature_source='comid',
197+
feature_id='13293474' # NHD reach identifier
198+
)
95199

96-
Any help in testing, development, documentation and other tasks is welcome.
97-
For more details, see the file [CONTRIBUTING.md](CONTRIBUTING.md).
200+
print(f"Basin contains {len(basin)} feature(s)")
98201

202+
# Find upstream flowlines
203+
flowlines = nldi.get_flowlines(
204+
feature_source='comid',
205+
feature_id='13293474',
206+
navigation_mode='UT', # Upstream tributaries
207+
distance=50 # km
208+
)
99209

100-
## Need help?
210+
print(f"Found {len(flowlines)} upstream tributaries within 50km")
211+
```
101212

102-
The Water Mission Area of the USGS supports the development and maintenance of `dataretrieval`. Any questions can be directed to the Computational Tools team at
103-
comptools@usgs.gov.
213+
## Available Data Services
214+
215+
### Modern USGS Water Data APIs (Recommended)
216+
- **Daily values**: Daily statistical summaries (mean, min, max)
217+
- **Instantaneous values**: High-frequency continuous data
218+
- **Field measurements**: Discrete measurements from field visits
219+
- **Monitoring locations**: Site information and metadata
220+
- **Time series metadata**: Information about available data parameters
221+
- **Latest daily values**: Most recent daily statistical summary data
222+
- **Latest instantaneous values**: Most recent high-frequency continuous data
223+
- **Samples data**: Discrete USGS water quality data
224+
225+
### Legacy NWIS Services (Deprecated)
226+
- **Daily values (dv)**: Legacy daily statistical data
227+
- **Instantaneous values (iv)**: Legacy continuous data
228+
- **Site info (site)**: Basic site information
229+
- **Statistics (stat)**: Statistical summaries
230+
- **Discharge peaks (peaks)**: Annual peak discharge events
231+
- **Discharge measurements (measurements)**: Direct flow measurements
232+
233+
### Water Quality Portal
234+
- **Results**: Water quality analytical results from USGS, EPA, and other agencies
235+
- **Sites**: Monitoring location information
236+
- **Organizations**: Data provider information
237+
- **Projects**: Sampling project details
238+
239+
### Network Linked Data Index (NLDI)
240+
- **Basin delineation**: Watershed boundaries for any point
241+
- **Flow navigation**: Upstream/downstream network traversal
242+
- **Feature discovery**: Find monitoring sites, dams, and other features
243+
- **Hydrologic connectivity**: Link data across the stream network
244+
245+
## More Examples
246+
247+
Explore additional examples in the
248+
[`demos`](https://github.com/USGS-python/dataretrieval/tree/main/demos)
249+
directory, including Jupyter notebooks demonstrating advanced usage patterns.
250+
251+
## Getting Help
252+
253+
- **Issue tracker**: Report bugs and request features at https://github.com/USGS-python/dataretrieval/issues
254+
- **Documentation**: Full API documentation available in the source code docstrings
104255

105-
Resources are available primarily for maintenance and responding to user questions.
106-
Priorities on the development of new features are determined by the `dataretrieval` development team.
256+
## Contributing
107257

258+
Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for
259+
development guidelines.
108260

109261
## Acknowledgments
110-
This material is partially based upon work supported by the National Science Foundation (NSF) under award 1931297.
111-
Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
262+
263+
This material is partially based upon work supported by the National Science
264+
Foundation (NSF) under award 1931297. Any opinions, findings, conclusions, or
265+
recommendations expressed in this material are those of the authors and do not
266+
necessarily reflect the views of the NSF.
112267

113268
## Disclaimer
114269

115-
This software is preliminary or provisional and is subject to revision.
116-
It is being provided to meet the need for timely best science.
117-
The software has not received final approval by the U.S. Geological Survey (USGS).
118-
No warranty, expressed or implied, is made by the USGS or the U.S. Government as to the functionality of the software and related material nor shall the fact of release constitute any such warranty.
119-
The software is provided on the condition that neither the USGS nor the U.S. Government shall be held liable for any damages resulting from the authorized or unauthorized use of the software.
270+
This software is preliminary or provisional and is subject to revision. It is
271+
being provided to meet the need for timely best science. The software has not
272+
received final approval by the U.S. Geological Survey (USGS). No warranty,
273+
expressed or implied, is made by the USGS or the U.S. Government as to the
274+
functionality of the software and related material nor shall the fact of release
275+
constitute any such warranty. The software is provided on the condition that
276+
neither the USGS nor the U.S. Government shall be held liable for any damages
277+
resulting from the authorized or unauthorized use of the software.
120278

121279
## Citation
122280

123-
Hodson, T.O., Hariharan, J.A., Black, S., and Horsburgh, J.S., 2023, dataretrieval (Python): a Python package for discovering
124-
and retrieving water data available from U.S. federal hydrologic web services:
125-
U.S. Geological Survey software release,
126-
https://doi.org/10.5066/P94I5TX3.
281+
Hodson, T.O., Hariharan, J.A., Black, S., and Horsburgh, J.S., 2023,
282+
dataretrieval (Python): a Python package for discovering and retrieving water
283+
data available from U.S. federal hydrologic web services: U.S. Geological Survey
284+
software release, https://doi.org/10.5066/P94I5TX3.

0 commit comments

Comments
 (0)