Update documentation.

Author: Eli

README.md

@@ -1,21 +1,38 @@
 # dms_datastore
 
-Delta Modeling Section Datastore provides tools for downloading and managing continuous data. This repository is a work in progress. 
-
-## Table of Contents
-- [Overview](#overview)
-- [Data Repository Structure](#data-repository-structure)
-- [Data Quality and Flags](#data-quality-and-flags)
-- [Data Screening and Error Detection Methods](#data-screening-and-error-detection-methods)
-- [Metadata and Station Concepts](#metadata-and-station-concepts)
-- [File Naming Conventions](#file-naming-conventions)
-- [Units and Standardization](#units-and-standardization)
-- [Data Fetching and Priority](#data-fetching-and-priority)
-- [Configuration System](#configuration-system)
-- [Accessing Datastore Data](#accessing-datastore-data)
-- [Challenges and Exceptions](#challenges-and-exceptions)
-- [Installation](#installation)
-- [CLI Commands](#cli-commands)
+Delta Modeling Section Datastore — tools for downloading, formatting, screening, and
+managing continuous time-series data from water-quality and hydrological agencies
+(USGS, CDEC, NOAA, NCRO, DES, and others).
+
+**Full documentation:** https://cadwrdeltamodeling.github.io/dms_datastore/
+
+## Quick Install
+
+```bash
+git clone https://github.com/CADWRDeltaModeling/dms_datastore
+conda env create -f environment.yml
+conda activate dms_datastore
+```
+
+Development install after the above:
+
+```bash
+pip install --no-deps -e .
+```
+
+## Quick Start
+
+```python
+from dms_datastore.read_multi import read_ts_repo
+
+# Read EC data for station "dsj"
+data = read_ts_repo("dsj", "ec")
+```
+
+For the complete data model, CLI reference, configuration system, and API docs see
+the [documentation site](https://cadwrdeltamodeling.github.io/dms_datastore/) or the
+[`docsrc/`](docsrc/) folder for the Sphinx source.
+
 
 ## Overview
 
@@ -170,7 +187,7 @@ The Dropbox Data Processing System provides a mechanism for importing ad-hoc or
 
 The system uses a YAML configuration file (`dropbox_spec.yaml`) to define data sources, collection patterns, and metadata handling rules. The `dropbox_data.py` script processes these configurations to locate, transform, and store the data in the standardized repository format.
 
-See [README-dropbox.md](README-dropbox.md) for detailed documentation on this system.
+See the Dropbox documentation in the project docs (docsrc/dropbox.rst) for detailed information on this system.
 
 ## Configuration System
 

README.rst

@@ -1,41 +1,25 @@
-===============================
+===============
 dms_datastore
-===============================
+===============
 
-Downloading tools and data repository management. This repository is a work in progress. It is not recommended for any purpose while it is under construction.
+Delta Modeling Section Datastore — tools for downloading, formatting, screening, and
+managing continuous time-series data from water-quality and hydrological agencies
+(USGS, CDEC, NOAA, NCRO, DES, and others).
 
+**Full documentation:** https://cadwrdeltamodeling.github.io/dms_datastore/
 
-
-There are improvements that are needed to the downloading system tools.
-
-1. The downloading scripts should be called download_XXX.py where XXX is noaa, nwis, cdec etc.
-2. The API should be made uniform between these scripts. 
-   a. It should be able to use our new id or the agency_id with the new id preferred
-   b. It should be able to use our variable name or the agency variable code
-   c. It should produce files that are named according to the file naming convention in the data plan: http://msb-confluence/display/DMKB/Strawman+Data+Organization+Plan:
-
-usgs_sjj@bgc_11337190_turbidity_2021.csv
-
-This is all potentially destabilizing, so perhaps it should be done on a shortlived branch
-
-The station files don't have a uniform format. I prefer all look like this:
-id,agency_id,subloc,variable
-sjj,11337190,bgc,turbidity
-
-The agency_id column is optional. 
-
-===============================
 Installation
-===============================
+============
 
 .. code-block:: bash
-   
+
    git clone https://github.com/CADWRDeltaModeling/dms_datastore
-   conda env create -f environment.yml # should create a dms_datastore and pip install the package
-   # alternatively, pip install -e . after running the above command if you want to develop the package
+   conda env create -f environment.yml
    conda activate dms_datastore
 
+Development install after the above:
 
+.. code-block:: bash
 
-
+   pip install --no-deps -e .
 

docsrc/commands.rst

@@ -0,0 +1,137 @@
+dms_datastore Command Reference
+===============================
+
+This page collects CLI command help pointers and workflow examples for the commands
+exposed by the package entry points.
+
+Main Entrypoint
+---------------
+
+Use the `dms` grouped CLI (or call commands directly by script name).
+
+.. code-block:: bash
+
+   # grouped help
+   dms --help
+
+   # subcommand help (example)
+   dms download_ncro --help
+
+Command Help Shortcuts
+-----------------------
+
+.. code-block:: bash
+
+   dms --help
+   download_noaa --help
+   download_hycom --help
+   download_hrrr --help
+   download_cdec --help
+   download_wdl --help
+   download_nwis --help
+   download_des --help
+   download_ncro --help
+   download_mokelumne --help
+   download_ucdipm --help
+   download_cimis --help
+   download_dcc --help
+   download_montezuma_gates --help
+   download_smscg --help
+   compare_directories --help
+   populate_repo --help
+   station_info --help
+   reformat --help
+   auto_screen --help
+   inventory --help
+   usgs_multi --help
+   delete_from_filelist --help
+   data_cache --help
+   merge_files --help
+   dropbox --help
+   coarsen --help
+   update_repo --help
+   update_flagged_data --help
+   rationalize_time_partitions --help
+
+Workflow A: Repository Build Pipeline (Download → Reformat → Auto Screen)
+-----------------------------------------------------------------------
+
+Stage 1: Download into raw/staging
+
+.. code-block:: bash
+
+   # helper pattern used by all downloaders
+   download_noaa --help
+   download_nwis --help
+   download_des --help
+   download_ncro --help
+
+Examples
+~~~~~~~~
+
+.. code-block:: bash
+
+   # NOAA
+   download_noaa --start 2024-01-01 --end 2024-01-31 --param water_level --stations ccc --dest <raw_dir>
+
+   # NWIS
+   download_nwis --start 2024-01-01 --end 2024-01-31 --stations sjj --param 00060 --dest <raw_dir>
+
+   # DES
+   download_des --start 2024-01-01 --end 2024-01-31 --stations cll --param flow --dest <raw_dir>
+
+   # NCRO timeseries
+   download_ncro --start 2024-01-01 --end 2024-12-31 --stations orm --param elev --dest <raw_dir>
+
+   # NCRO inventory only
+   download_ncro --inventory-only
+
+   # CDEC
+   download_cdec --start 2024-01-01 --end 2024-01-31 --stations cse --param elev --dest <raw_dir>
+
+   # HYCOM
+   download_hycom --sdate 2024-01-01 --edate 2024-01-31 --raw_dest <hycom_raw_dir> --processed_dest <hycom_processed_dir>
+
+   # HRRR
+   download_hrrr --sdate 2024-01-01 --edate 2024-01-03 --dest <hrrr_raw_dir>
+
+   # UCD IPM (positional dates)
+   download_ucdipm 2024-01-01 2024-01-31 --stnkey 281
+
+Stage 2: Reformat raw → formatted
+
+.. code-block:: bash
+
+   reformat --inpath <raw_dir> --outpath <formatted_dir>
+
+Stage 2b: USGS multivariate cleanup
+
+.. code-block:: bash
+
+   usgs_multi --fpath <formatted_dir>
+
+Stage 3: Auto screen formatted → screened
+
+.. code-block:: bash
+
+   auto_screen --fpath <formatted_dir> --dest <screened_dir>
+
+Workflow B: Dropbox Ingest (separate workflow)
+----------------------------------------------
+
+.. code-block:: bash
+
+   dropbox --input dms_datastore/config_data/dropbox_spec.yaml
+
+Workflow C: Staging → Repo update and utilities
+------------------------------------------------
+
+.. code-block:: bash
+
+   update_repo <staging_formatted_dir> <repo_formatted_dir> --plan --out-actions update_plan.csv
+   update_repo <staging_formatted_dir> <repo_formatted_dir> --apply
+
+Additional Utilities
+--------------------
+
+See the documentation for details and examples for `station_info`, `merge_files`, `coarsen`, and other utilities.