chore: add AGENTS.md and switch pre-commit to ruff-only (DOI-USGS#293)

thodson-usgs · claude · web-flow · commit 906c85927d82 · 2026-05-28T13:02:31.000-05:00
* chore(lint): replace flake8/prettier configs with ruff-only pre-commit

CI already lints with ruff exclusively (python-package.yml), so .flake8
and .prettierrc.toml were dead config. Rewrite .pre-commit-config.yaml
to drop black, blackdoc, flake8, isort, prettier, pyupgrade, and
double-quote-string-fixer (all superseded by ruff check + ruff format)
and add astral-sh/ruff-pre-commit so local pre-commit matches CI.

Exclude tests/data/ from trailing-whitespace, end-of-file-fixer, and
mixed-line-ending; those fixtures are byte-exact API response captures
(RDB/TSV with significant trailing tabs) and must not be normalized.

Co-Authored-By: Claude Opus 4.7 (1M context) &lt;noreply@anthropic.com&gt;

* docs: add AGENTS.md with repo scope, notebook index, and waterdata notes

Brief guide for agents working in this repo: scope and excluded paths,
example notebook index (demos/*.ipynb plus demos/hydroshare/*.ipynb),
environment/lint/test/docs commands, testing gotchas (pytest-httpx
fixture, Python &lt;3.10 skips), and waterdata implementation notes
(httpx client, kwarg-to-API spelling translation, OGC byte-limit
auto-chunking in dataretrieval/waterdata/chunking.py).

Co-Authored-By: Claude Opus 4.7 (1M context) &lt;noreply@anthropic.com&gt;

* chore: fix latent trailing whitespace and missing EOF newlines

Auto-fixes from the new pre-commit hooks (trailing-whitespace,
end-of-file-fixer). Touched files are docs (.rst, .nblink), the README
code snippets, .gitignore, the nwqn Dockerfile, and the py.typed
marker. No behavior changes; this clears the slate so future commits
don't trip the hook on day one.

Co-Authored-By: Claude Opus 4.7 (1M context) &lt;noreply@anthropic.com&gt;

---------

Co-authored-by: Claude Opus 4.7 (1M context) &lt;noreply@anthropic.com&gt;
diff --git a/.flake8 b/.flake8
diff --git a/.gitignore b/.gitignore
@@ -111,4 +111,4 @@ ENV/
 .mypy_cache/
 
 # macOS
-*.DS_Store
+*.DS_Store
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -6,44 +6,23 @@ repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.4.0
     hooks:
+      # tests/data/ holds byte-exact API response captures (TSV/RDB with
+      # significant trailing tabs, plus pinned JSON). Skip whitespace and
+      # EOL normalization there so we don't silently change fixture bytes.
       - id: trailing-whitespace
+        exclude: ^tests/data/
       - id: end-of-file-fixer
+        exclude: ^tests/data/
+      - id: mixed-line-ending
+        exclude: ^tests/data/
       - id: check-docstring-first
       - id: check-json
       - id: check-yaml
-      - id: double-quote-string-fixer
       - id: debug-statements
-      - id: mixed-line-ending
-
-  - repo: https://github.com/asottile/pyupgrade
-    rev: v3.3.1
-    hooks:
-      - id: pyupgrade
-        args:
-          - '--py38-plus'
-
-  - repo: https://github.com/psf/black
-    rev: 23.3.0
-    hooks:
-      - id: black
-      - id: black-jupyter
-
-  - repo: https://github.com/keewis/blackdoc
-    rev: v0.3.8
-    hooks:
-      - id: blackdoc
-
-  - repo: https://github.com/PyCQA/flake8
-    rev: 6.0.0
-    hooks:
-      - id: flake8
-
-  - repo: https://github.com/PyCQA/isort
-    rev: 5.12.0
-    hooks:
-      - id: isort
 
-  - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: v3.0.0-alpha.6
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.15
     hooks:
-      - id: prettier
+      - id: ruff-check
+        args: [--fix]
+      - id: ruff-format
diff --git a/.prettierrc.toml b/.prettierrc.toml
diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,35 @@
+# AGENTS.md
+
+## Scope
+- Python code is in `dataretrieval/`; `dataretrieval/waterdata/` is the modern USGS Water Data API, `dataretrieval/nwis.py` is legacy/deprecated.
+- `R/dataRetrieval/` is the R project copy; leave it alone unless the task asks for R work.
+- Exclude `.claude/worktrees/` from searches and edits; it contains stale worktrees that pollute results.
+
+## Example Notebooks
+- `demos/*.ipynb` — top-level Water Data tour: `USGS_WaterData_Introduction_Examples.ipynb` is the entry point; `_ContinuousData_`, `_DailyStatistics_`, `_DiscreteSamples_`, `_ReferenceLists_` cover individual collections; `WaterData_demo.ipynb`, `peak_streamflow_trends.ipynb`, and `R Python Vignette equivalents.ipynb` are standalone walkthroughs.
+- `demos/hydroshare/*.ipynb` — per-service HydroShare examples (NLDI, NWIS WaterUse, and Water Data DailyValues / GroundwaterLevels / Measurements / ParameterCodes / Peaks / Ratings / Samples / SiteInfo / SiteInventory / Statistics / UnitValues). Mirror these when adding examples for a new collection.
+- `demos/nwqn_data_pull/` — non-notebook example: a lithops/Docker batch pipeline (`retrieve_nwqn_samples.py`, `retrieve_nwqn_streamflow.py`) with its own `README.md`.
+- Any `Untitled*.ipynb`, `*_test.ipynb`, or notebooks not listed here are untracked local scratch; ignore them.
+
+## Environment
+- Use `pip install .[test,nldi]` (CI uses pip, not uv despite `uv.lock`). Docs: `pip install .[doc,nldi]`.
+
+## Commands
+- Lint: `ruff check .` and `ruff format --check .`.
+- Tests: `coverage run -m pytest tests/ && coverage report -m`, or focused like `pytest tests/waterdata_test.py::test_mock_get_samples`.
+- Docs: install docs deps, `ipython kernel install --name "python3" --user`, then `make html` from `docs/`. `make docs` adds doctest+linkcheck (network-dependent).
+
+## Testing Gotchas
+- Tests mock HTTP with `pytest-httpx`'s `httpx_mock` fixture and fixtures under `tests/data/`; keep new API tests offline. `tests/conftest.py` relaxes the fixture's strict-mode defaults (unused mocks and unmocked requests are tolerated) so rerun-on-failure works.
+- `tests/nwis_test.py::test_nwis_service_live` hits live NWIS.
+- `tests/nadp_test.py` is module-skipped (NADP deprecated).
+- `tests/waterdata_test.py` and `tests/waterdata_ratings_test.py` skip on Python <3.10, so a 3.9 run does not cover them.
+
+## Implementation Notes
+- HTTP client is `httpx` (migrated from `requests` in #289); new code should use `httpx` and tests should mock with `httpx_mock`.
+- Public download helpers return `(DataFrame, metadata)`.
+- `dataretrieval/__init__.py` star-imports service modules; `dataretrieval/waterdata/__init__.py` controls Water Data exports via `__all__`.
+- `dataretrieval.waterdata.utils._default_headers()` adds `X-Api-Key` from `API_USGS_PAT`; never hard-code tokens in examples or tests.
+- Water Data request builders translate Python kwargs to API spellings (`skip_geometry` -> `skipGeometry`, `filter_lang` -> `filter-lang`); tests assert exact URLs/query params.
+- Multi-value OGC params are comma-joined GETs, except `monitoring-locations` which POSTs CQL2 JSON. The OGC edge WAF caps total request bytes (URL + body) at ~8200, so `dataretrieval/waterdata/chunking.py` auto-splits oversized queries across sub-requests (both GET and POST paths); preserve this when adding new list-shaped kwargs.
+- NLDI requires `geopandas` at import time (`pip install .[nldi]`); other modules fall back to pandas when geopandas is absent.
diff --git a/README.md b/README.md
@@ -59,7 +59,7 @@ from dataretrieval import waterdata
 
 # Get daily streamflow data (returns DataFrame and metadata)
 df, metadata = waterdata.get_daily(
-    monitoring_location_id='USGS-01646500', 
+    monitoring_location_id='USGS-01646500',
     parameter_code='00060',  # Discharge
     time='2024-10-01/2025-09-30'
 )
@@ -98,7 +98,7 @@ windows to avoid timeouts and other issues:
 ```python
 # Get continuous data for a single monitoring location and water year
 df, metadata = waterdata.get_continuous(
-    monitoring_location_id='USGS-01646500', 
+    monitoring_location_id='USGS-01646500',
     parameter_code='00065',  # Gage height
     time='2024-10-01/2025-09-30'
 )
@@ -152,7 +152,7 @@ from dataretrieval import nldi
 # Get watershed basin for a stream reach
 basin = nldi.get_basin(
     feature_source='comid',
-    feature_id='13293474'  # NHD reach identifier  
+    feature_id='13293474'  # NHD reach identifier
 )
 
 print(f"Basin contains {len(basin)} feature(s)")
@@ -184,7 +184,7 @@ print(f"Found {len(flowlines)} upstream tributaries within 50km")
 ### Legacy NWIS Services (Deprecated)
 - **Daily values (dv)**: Legacy daily statistical data
 - **Instantaneous values (iv)**: Legacy continuous data
-- **Site info (site)**: Basic site information  
+- **Site info (site)**: Basic site information
 - **Statistics (stat)**: Statistical summaries
 - **Discharge peaks (peaks)**: Annual peak discharge events
 
diff --git a/dataretrieval/py.typed b/dataretrieval/py.typed
@@ -1 +0,0 @@
-
diff --git a/demos/nwqn_data_pull/Dockerfile_dataretrieval b/demos/nwqn_data_pull/Dockerfile_dataretrieval
@@ -56,4 +56,4 @@ COPY requirements.txt requirements.txt
 RUN pip install --no-cache-dir -r requirements.txt
 
 ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
-CMD [ "handler.entry_point.lambda_handler" ]
+CMD [ "handler.entry_point.lambda_handler" ]
diff --git a/docs/source/examples/USGS_WaterData_DailyValues_Examples.nblink b/docs/source/examples/USGS_WaterData_DailyValues_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_DailyValues_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_GroundwaterLevels_Examples.nblink b/docs/source/examples/USGS_WaterData_GroundwaterLevels_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_GroundwaterLevels_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_Measurements_Examples.nblink b/docs/source/examples/USGS_WaterData_Measurements_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_Measurements_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_ParameterCodes_Examples.nblink b/docs/source/examples/USGS_WaterData_ParameterCodes_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_ParameterCodes_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_Peaks_Examples.nblink b/docs/source/examples/USGS_WaterData_Peaks_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_Peaks_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_Ratings_Examples.nblink b/docs/source/examples/USGS_WaterData_Ratings_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_Ratings_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_Samples_Examples.nblink b/docs/source/examples/USGS_WaterData_Samples_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_Samples_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_SiteInfo_Examples.nblink b/docs/source/examples/USGS_WaterData_SiteInfo_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_SiteInfo_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_SiteInventory_Examples.nblink b/docs/source/examples/USGS_WaterData_SiteInventory_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_SiteInventory_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_Statistics_Examples.nblink b/docs/source/examples/USGS_WaterData_Statistics_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_Statistics_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/USGS_WaterData_UnitValues_Examples.nblink b/docs/source/examples/USGS_WaterData_UnitValues_Examples.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/hydroshare/USGS_WaterData_UnitValues_Examples.ipynb"
-}
+}
diff --git a/docs/source/examples/WaterData_demo.nblink b/docs/source/examples/WaterData_demo.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/WaterData_demo.ipynb"
-}
+}
diff --git a/docs/source/examples/index.rst b/docs/source/examples/index.rst
@@ -92,4 +92,4 @@ Duplicating the R ``dataRetrieval`` vignettes functionality
 .. toctree::
     :maxdepth: 2
 
-    rvignettes
+    rvignettes
diff --git a/docs/source/examples/rvignettes.nblink b/docs/source/examples/rvignettes.nblink
@@ -1,3 +1,3 @@
 {
     "path": "../../../demos/R Python Vignette equivalents.ipynb"
-}
+}
diff --git a/docs/source/meta/contributing.rst b/docs/source/meta/contributing.rst
@@ -156,4 +156,4 @@ Acknowledgements
 ----------------
 This document was adapted from the ``cookiecutter`` project's CONTRIBUTING file, which resides at
 https://github.com/cookiecutter/cookiecutter/blob/main/CONTRIBUTING.md
-Thank you to the ``cookiecutter`` team for helping streamline open-source development for the masses.
+Thank you to the ``cookiecutter`` team for helping streamline open-source development for the masses.
diff --git a/docs/source/meta/installing.rst b/docs/source/meta/installing.rst
@@ -86,4 +86,4 @@ You can also build the documentation locally by running the following commands:
 This both tests the documentation (runs code blocks and checks links), and also
 locally *builds* the documentation, placing the HTML files within the
 ``docs/build/html`` directory. You can then open the ``index.html`` file in
-your browser to view the documentation.
+your browser to view the documentation.
diff --git a/docs/source/meta/license.rst b/docs/source/meta/license.rst
@@ -9,4 +9,4 @@ Interior. For more information, see the `LICENSE.md`_ file. See the
 
 .. _LICENSE.md: https://github.com/DOI-USGS/dataretrieval-python/blob/main/LICENSE.md
 
-.. _Disclaimer.md: https://github.com/DOI-USGS/dataretrieval-python/blob/main/DISCLAIMER.md
+.. _Disclaimer.md: https://github.com/DOI-USGS/dataretrieval-python/blob/main/DISCLAIMER.md
diff --git a/docs/source/reference/nadp.rst b/docs/source/reference/nadp.rst
@@ -5,4 +5,4 @@ dataretrieval.nadp
 
 .. automodule:: dataretrieval.nadp
     :members:
-    :special-members:
+    :special-members:
diff --git a/docs/source/reference/nldi.rst b/docs/source/reference/nldi.rst
@@ -5,4 +5,4 @@ dataretrieval.nldi
 
 .. automodule:: dataretrieval.nldi
     :members:
-    :special-members:
+    :special-members:
diff --git a/docs/source/reference/nwis.rst b/docs/source/reference/nwis.rst
@@ -5,4 +5,4 @@ dataretrieval.nwis
 
 .. automodule:: dataretrieval.nwis
     :members:
-    :special-members:
+    :special-members:
diff --git a/docs/source/reference/streamstats.rst b/docs/source/reference/streamstats.rst
@@ -5,4 +5,4 @@ dataretrieval.streamstats
 
 .. automodule:: dataretrieval.streamstats
     :members:
-    :special-members:
+    :special-members:
diff --git a/docs/source/reference/utils.rst b/docs/source/reference/utils.rst
@@ -5,4 +5,4 @@ dataretrieval.utils
 
 .. automodule:: dataretrieval.utils
     :members:
-    :special-members:
+    :special-members:
diff --git a/docs/source/reference/waterdata.rst b/docs/source/reference/waterdata.rst
@@ -5,4 +5,4 @@ dataretrieval.waterdata
 
 .. automodule:: dataretrieval.waterdata
     :members:
-    :special-members:
+    :special-members:
diff --git a/docs/source/reference/wqp.rst b/docs/source/reference/wqp.rst
@@ -5,4 +5,4 @@ dataretrieval.wqp
 
 .. automodule:: dataretrieval.wqp
     :members:
-    :special-members:
+    :special-members:

Original file line number	Diff line number	Diff line change
`@@ -1,3 +1,3 @@`
`1`	`1`	`{`
`2`	`2`	`"path": "../../../demos/hydroshare/USGS_WaterData_DailyValues_Examples.ipynb"`
`3`		`-}`
	`3`	`+}`