Clean documentation and remove orphaned scripts in preparation for public release.

Author: njhenry

.claude/CLAUDE.md

@@ -9,7 +9,9 @@ make build_env       # Create conda env from environment.yml (name: openpois, Py
 make install_package # pip install -e . (editable install)
 ```
 
-Python executable: `/home/nathenry/miniforge3/envs/openpois/bin/python`.
+Python executable: `$CONDA_PREFIX/bin/python` after `conda activate openpois`.
+
+> **Note on data paths.** Examples in this directory show resolved paths under `~/data/openpois/...`, which is the maintainer's configured `directories.*.path` value in `config.yaml`. Substitute your own `directories.*.path` root when reading them; the layout under that root is set by `config.yaml`.
 
 ## Common commands
 

.claude/docs/data-sources.md

@@ -12,7 +12,7 @@ Reference for every external data source openpois ingests. For the workflow that
 - **Auth**: OAuth — any OSM account works. Produce a Netscape-format cookie jar (browser export or Geofabrik's `oauth_cookie_client.py`). Path: `download.osm.history_cookie_file` (default `~/data/openpois/.creds/geofabrik_cookies.txt`).
 - **Pipeline**: `osmium tags-filter --omit-referenced` → `osmium time-filter` → pyosmium streams to `osm_versions.parquet` + `osm_changes.parquet`.
 - **Entry**: [src/openpois/io/osm_history_pbf.py](../../src/openpois/io/osm_history_pbf.py) (`download_osm_history`).
-- **Config**: `download.osm.start_date`, `end_date`, `date_interval_days`, `filter_keys`, `extract_keys`.
+- **Config**: `download.osm.start_date`, `end_date`, `filter_keys`, `extract_keys`.
 
 ## OSM snapshot (Geofabrik standard PBFs)
 
@@ -55,11 +55,3 @@ Reference for every external data source openpois ingests. For the workflow that
 - **Returns**: `(boundary_gdf, coarse_bboxes)` — single-row dissolved+buffered polygon (EPSG:4326) plus a list of bboxes for predicate pushdown.
 - **Antimeridian**: Aleutians split into two bboxes (Near Islands at +172°E vs. rest of AK at negative longitudes).
 
-## Legacy: Overpass-based OSM history
-
-Still wired up but superseded by the PBF pipeline. Queries Overpass API for element IDs in a bbox, then fetches per-element histories from the OSM API.
-
-- **Config**: `download.osm.history_bbox` (Seattle-scoped; Overpass can't serve US-wide histories).
-- **Entry**: [src/openpois/io/osm_history.py](../../src/openpois/io/osm_history.py) (`download_element_histories`).
-- **Script**: `scripts/osm_data/download.py`.
-- **When to use**: city-scale testing, or if Geofabrik OAuth is unavailable.

.claude/skills/conflate-snapshots/SKILL.md

@@ -18,7 +18,7 @@ upload for web consumption.
 > credentials (`aws_access_key_id` starting with `ASIA…`). **Before** running
 > step 7, ask the user to regenerate them at
 > <https://source.coop/repositories/henryspatialanalysis/openpois/manage>
-> and overwrite `~/repos/openpois/.env.json`. The upload script will warn if
+> and overwrite `.env.json` at the repo root. The upload script will warn if
 > the file looks stale, but it cannot tell whether the token itself has
 > expired until it actually fails.
 

Makefile

@@ -20,7 +20,7 @@ CONDA_PYTHON := $(shell conda run -n openpois which python 2>/dev/null || echo p
 CONDA_BIN := $(dir $(CONDA_PYTHON))
 
 lint:
-	@$(CONDA_BIN)flake8 src/ exploratory/ tests/
+	@$(CONDA_BIN)flake8 src/ scripts/ tests/
 	@$(CONDA_BIN)pylint src/openpois/
 
 # Build the site for production

config.yaml

@@ -22,14 +22,6 @@ download:
   osm:
     start_date: 2016-01-01
     end_date: 2025-12-31
-    date_interval_days: 7
-    # Seattle-area bbox used only by the Overpass-based historical download.
-    # Overpass cannot serve US-wide histories; keep this scoped to a city.
-    history_bbox:
-      xmin: -122.45
-      ymin: 47.50
-      xmax: -122.25
-      ymax: 47.70
     pbf_url: "https://download.geofabrik.de/north-america/us-latest.osm.pbf"
     pr_pbf_url: "https://download.geofabrik.de/north-america/us/puerto-rico-latest.osm.pbf"
     # Full-history PBFs live on Geofabrik's OAuth-protected internal server.
@@ -156,9 +148,6 @@ directories:
       us_changes: us_osm_changes.parquet
       pr_versions: pr_osm_versions.parquet
       pr_changes: pr_osm_changes.parquet
-      # Legacy Overpass-based pipeline (still used by scripts/osm_data/download.py)
-      osm_elements: osm_elements.csv
-      osm_failed_elements: osm_failed_elements.csv
       # Modelling-ready observations (one row per POI version × shared_label)
       osm_observations: osm_observations.parquet
   model_output:
@@ -208,7 +197,6 @@ directories:
 
 # Settings for POI conflation
 conflation:
-  osm_confidence_weight: 0.8
   overture_confidence_weight: 0.7
   min_match_score: 0.50
   max_radius_m: 200

docs/api.rst

@@ -51,15 +51,16 @@ used for type-agreement scoring.
 io
 --
 
-openpois.io.osm_history
-~~~~~~~~~~~~~~~~~~~~~~~
+openpois.io.osm_history_pbf
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Download OpenStreetMap element change histories via the Overpass and OSM APIs.
-Builds Overpass queries across a configured date range to collect element IDs,
-then fetches the full version history of each element, producing per-version
-and per-change tables suitable for the change-rate model.
+Download Geofabrik full-history PBFs (US + Puerto Rico), filter to POI tags
+with ``osmium tags-filter``, time-window with ``osmium time-filter``, and parse
+with pyosmium into per-version and per-change Parquet tables suitable for the
+change-rate model. Uses an OAuth cookie jar against Geofabrik's internal
+server.
 
-.. automodule:: openpois.io.osm_history
+.. automodule:: openpois.io.osm_history_pbf
    :members:
    :undoc-members:
    :show-inheritance:
@@ -105,15 +106,27 @@ sorts rows within each partition by a finer geohash for spatial locality.
    :undoc-members:
    :show-inheritance:
 
-openpois.io.s3
-~~~~~~~~~~~~~~
+openpois.io.source_coop
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Upload a locally partitioned dataset to Source Cooperative's S3-compatible
+storage. Walks the Hive partition directory, uploads each Parquet file under
+a versioned prefix, and reports the public URL on completion. Credentials
+come from a JSON file at the repo root (``publish.credentials_file``).
+
+.. automodule:: openpois.io.source_coop
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+openpois.io.credentials
+~~~~~~~~~~~~~~~~~~~~~~~
 
-Upload a locally partitioned dataset to a public S3 bucket. Walks the Hive
-partition directory, uploads each Parquet file under a versioned S3 prefix
-with public-read ACL, and reports the public base URL on completion. Requires
-AWS credentials via environment variables or ``~/.aws/credentials``.
+Load Source Cooperative AWS-compatible credentials from a JSON file. Tokens
+are short-lived (~1 hour); the loader logs a clear error pointing at the
+credentials regeneration URL when the file is stale or missing.
 
-.. automodule:: openpois.io.s3
+.. automodule:: openpois.io.credentials
    :members:
    :undoc-members:
    :show-inheritance:

docs/conf.py

@@ -9,7 +9,7 @@
 # -- Project information -------------------------------------------------------
 
 project = "openpois"
-copyright = "2024, Nathaniel Henry"
+copyright = "2026, Nathaniel Henry"
 author = "Nathaniel Henry"
 release = "0.1.0"
 

docs/workflows.md

@@ -1,19 +1,46 @@
 Workflows
 =========
 
-This page describes the four end-to-end pipelines that make up the openpois
-data processing system, in the order they should be executed. Each pipeline
-is implemented as a series of scripts in the ``scripts/`` directory; the
-scripts call library functions documented in the :doc:`api`.
+This page describes the four end-to-end pipelines that produce the openpois
+dataset, in the order they are executed. Each pipeline is implemented as a
+series of scripts in the ``scripts/`` directory; the scripts call library
+functions documented in the :doc:`api`.
 
 All scripts read their configuration from ``config.yaml`` via
 ``config_versioned.Config``. See the individual script docstrings for the
 exact config keys each script uses.
 
----
+
+Prerequisites
+-------------
+
+**Python environment.** Install the conda env from ``environment.yml`` and
+the package itself in editable mode:
+
+.. code-block:: bash
+
+   make build_env       # conda env create -f environment.yml (env name: openpois)
+   conda activate openpois
+   make install_package # pip install -e .
+
+**Geofabrik OAuth (Pipeline 2 only).** Pipeline 2 downloads full-history
+PBFs from Geofabrik's OAuth-protected internal server. Any OSM account grants
+access. Generate a Netscape-format cookie jar by logging in at
+``https://osm-internal.download.geofabrik.de/`` and exporting cookies, or by
+running Geofabrik's ``oauth_cookie_client.py``. Save the cookie jar at the
+path configured in ``config.yaml`` under ``download.osm.history_cookie_file``.
+
+**Source Cooperative credentials (publishing only).** Publishing the data
+back to Source Cooperative requires short-lived AWS-style credentials in a
+JSON file at the path configured under ``publish.credentials_file`` (default:
+``.env.json`` at the repo root). The format is documented in the
+``scripts/publish/upload_to_source_coop.py`` docstring. Replicators who do
+not intend to publish can stop after Pipeline 4 Step 3.
+
+----
 
 Pipeline 1: POI Snapshot Downloads
-------------------------------------
+----------------------------------
 
 These two scripts are independent and can be run in any order (or in
 parallel). Each downloads a current US-wide POI snapshot from one data
@@ -51,27 +78,29 @@ See :mod:`openpois.io.overture`.
 Reads the first 100 rows of each snapshot without loading the full files,
 saving snippet CSVs to the ``testing/`` directory for column inspection.
 
----
+----
 
 Pipeline 2: OSM Historical Change-Rate Model
 --------------------------------------------
 
-This pipeline downloads OpenStreetMap element histories for a Seattle-area
-bounding box and fits a Poisson change-rate model to estimate how quickly
-different POI categories become outdated.
+This pipeline downloads OpenStreetMap full-history PBFs (US + Puerto Rico)
+and fits a Poisson change-rate model to estimate how quickly different POI
+categories become outdated.
 
-**Step 1 — Download OSM element histories**
+**Step 1 — Download full-history PBFs**
 
 .. code-block:: bash
 
-   python scripts/osm_data/download.py
+   python scripts/osm_data/download_history.py
 
-Queries the Overpass API across a configured date range to collect element IDs
-for each tag key, then fetches the full version history of each element via the
-OSM API. Outputs ``osm_elements.csv``, ``osm_versions.csv``,
-``osm_changes.csv``, and ``osm_failed_elements.csv``.
+Requires the Geofabrik OAuth cookie jar described in *Prerequisites* above.
+Downloads the US-mainland and Puerto Rico full-history extracts, filters
+each with ``osmium tags-filter`` (POI tag keys only) and ``osmium
+time-filter`` (the ``download.osm.start_date`` / ``end_date`` window), then
+parses with pyosmium into per-version and per-change Parquet tables.
+Outputs: ``osm_versions.parquet`` and ``osm_changes.parquet``.
 
-See :mod:`openpois.io.osm_history`.
+See :mod:`openpois.io.osm_history_pbf`.
 
 **Step 2 — Reformat into observations**
 
@@ -82,7 +111,7 @@ See :mod:`openpois.io.osm_history`.
 Converts raw version histories into one-row-per-observation records, each
 flagged for whether the configured ``osm_data.tag_key`` changed, then
 assigns a shared taxonomy label and explodes rows for POIs mapping to
-multiple labels. Output: ``osm_observations.csv``.
+multiple labels. Output: ``osm_observations.parquet``.
 
 See :mod:`openpois.osm.format_observations`.
 
@@ -111,10 +140,10 @@ Produces Kaplan-Meier-style survival curve plots saved to
 
 See :mod:`openpois.osm.change_plots`.
 
----
+----
 
 Pipeline 3: Rate the OSM Snapshot
-------------------------------------
+---------------------------------
 
 This pipeline applies the fitted change-rate model (Pipeline 2) to the OSM
 snapshot (Pipeline 1) to assign a confidence score to every POI.
@@ -141,31 +170,30 @@ See :mod:`openpois.models.apply`.
 
    python scripts/osm_snapshot/format_for_upload.py
 
-Adds geohash columns and writes a Hive-style partitioned dataset so that the
-web map can fetch only the tiles it needs. Output:
-``osm_snapshot_partitioned/``.
+Adds geohash columns and writes a Hive-style partitioned dataset so the web
+map can fetch only the tiles it needs. Output: ``osm_snapshot_partitioned/``.
 
 See :mod:`openpois.io.geohash_partition`.
 
-**Step 3 — Upload to S3**
+**Step 3 — Build OSM PMTiles**
 
 .. code-block:: bash
 
-   python scripts/osm_snapshot/upload_to_s3.py
+   python scripts/osm_snapshot/prepare_pmtiles.py
 
-Uploads the partitioned dataset to the configured public S3 bucket with
-public-read ACL. Requires AWS credentials (``AWS_ACCESS_KEY_ID`` /
-``AWS_SECRET_ACCESS_KEY`` env vars or ``~/.aws/credentials``).
+Generates a single-zoom (z14) PMTiles archive from the partitioned dataset
+for use by the web map. Output: ``osm_snapshot.pmtiles``.
 
-See :mod:`openpois.io.s3`.
+See :mod:`openpois.io.pmtiles`.
 
----
+----
 
-Pipeline 4: Conflation and Upload
-------------------------------------
+Pipeline 4: Conflation and Publishing
+-------------------------------------
 
-This pipeline conflates the rated OSM snapshot with the Overture Maps snapshot
-into a single unified POI dataset for the web map.
+This pipeline conflates the rated OSM snapshot with the Overture Maps
+snapshot into a single unified POI dataset and publishes it to Source
+Cooperative.
 
 **Prerequisites:** Pipeline 3 rated OSM snapshot and Pipeline 1 Overture
 snapshot.
@@ -193,23 +221,29 @@ See :mod:`openpois.conflation.match`, :mod:`openpois.conflation.merge`, and
 Produces a summary CSV with match counts and average match scores per
 shared taxonomy label. Output: ``summary_by_label.csv``.
 
-**Step 3 — Partition for upload**
+**Step 3 — Partition and build conflated PMTiles**
 
 .. code-block:: bash
 
    python scripts/conflation/format_for_upload.py
+   python scripts/conflation/prepare_pmtiles.py
 
-Adds geohash columns and writes a Hive-style partitioned dataset.
-Output: ``conflated_partitioned/``.
+Adds geohash columns and writes a Hive-style partitioned dataset, then
+builds a single-zoom (z14) PMTiles archive of the conflated points.
+Outputs: ``conflated_partitioned/`` and ``conflated.pmtiles``.
 
-See :mod:`openpois.io.geohash_partition`.
+See :mod:`openpois.io.geohash_partition` and :mod:`openpois.io.pmtiles`.
 
-**Step 4 — Upload to S3**
+**Step 4 — Publish to Source Cooperative** *(optional)*
 
 .. code-block:: bash
 
-   python scripts/conflation/upload_to_s3.py
+   python scripts/publish/upload_to_source_coop.py
 
-Uploads the partitioned conflated dataset to S3 with public-read ACL.
+Uploads the partitioned conflated dataset, the partitioned OSM dataset,
+both PMTiles archives, and a per-version README to Source Cooperative
+under the ``versions.source_coop`` folder. Requires the credentials file
+described in *Prerequisites*. Skip this step if you only want the data
+locally.
 
-See :mod:`openpois.io.s3`.
+See :mod:`openpois.io.source_coop` and :mod:`openpois.publish.build_readme`.

docs/workflows.rst

@@ -1,4 +1,4 @@
-#!/home/nathenry/miniforge3/envs/openpois/bin/python
+#!/usr/bin/env python
 """
 Conflate rated OSM POIs with Overture Maps POIs into a unified dataset.