Update machine-readable documentation.

Author: njhenry

.claude/CLAUDE.md

@@ -1,82 +1,59 @@
 # CLAUDE.md
 
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+Guidance for Claude Code working in this repository. Deep-dives live in [skills/](skills/) and [docs/](docs/); this file is orientation.
 
-## Environment Setup
+## Environment setup
 
 ```bash
-make build_env       # Create conda environment from environment.yml
-make install_package # Install openpois in editable mode (pip install -e .)
+make build_env       # Create conda env from environment.yml (name: openpois, Python 3.10+)
+make install_package # pip install -e . (editable install)
 ```
 
-The conda environment is named `openpois` and requires Python 3.10+.
+Python executable: `/home/nathenry/miniforge3/envs/openpois/bin/python`.
 
-## Common Commands
+## Common commands
 
 ```bash
 pytest               # Run tests
-make export_env      # Export conda environment to environment.yml after adding dependencies
+make export_env      # Export conda env to environment.yml after adding deps
 ```
 
-Code style is enforced by Black (format on save in VSCode). Linting via flake8 and pylint, both configured in `pyproject.toml`.
+Style: Black (format-on-save in VSCode). Lint: flake8 + pylint, configured in `pyproject.toml`.
 
-## Architecture
+## Architecture at a glance
 
-**openpois** models POI (Point of Interest) stability over time using historical OpenStreetMap data. The workflow is:
+**openpois** models POI stability over time from OpenStreetMap history, and produces unified OSM + Overture + Foursquare snapshots for web consumption. Work splits into four pipelines:
 
-1. **Download OSM history** — two options depending on scope:
-   - **US + Puerto Rico (default, `src/openpois/io/osm_history_pbf.py`)** — downloads Geofabrik full-history PBFs (`us-internal.osh.pbf` + `puerto-rico-internal.osh.pbf`), runs `osmium tags-filter --omit-referenced` then `osmium time-filter`, and streams the result through pyosmium into `osm_versions.parquet` + `osm_changes.parquet`. Requires an OSM-account OAuth cookie jar for Geofabrik's internal server. Entry point: `scripts/osm_data/download_history.py`.
-   - **City-scale fallback (`src/openpois/io/osm_history.py`)** — queries the Overpass API for element IDs in a bounding box, then fetches per-element histories from the OSM API. Seattle-scoped by default; Overpass cannot serve US-wide histories. Entry point: `scripts/osm_data/download.py`.
-2. **Format observations** (`src/openpois/osm/format_observations.py`) — converts raw OSM version histories into observation records (one row per version) with flags for tag changes and deletions
-3. **Model change rates** (`src/openpois/models/`) — fits an empirical Bayes model using PyTorch to estimate per-group POI change rates (λ) as a Poisson process
-4. **Visualize stability** (`src/openpois/osm/change_plots.py`) — plots how long POI tags remain unchanged
+| Pipeline | Skill |
+|---|---|
+| Fit λ from OSM history, rate current snapshots | [skills/model-history-pipeline](skills/model-history-pipeline/SKILL.md) |
+| Iterate model variants on a pinned history run | [skills/iterate-model-types](skills/iterate-model-types/SKILL.md) |
+| Refresh the three POI snapshots (OSM / Overture / FSQ) | [skills/full-data-pull](skills/full-data-pull/SKILL.md) |
+| Conflate OSM + Overture, partition, upload to S3 | [skills/conflate-snapshots](skills/conflate-snapshots/SKILL.md) |
+| Bump the frontend to the new data version | [skills/update-site](skills/update-site/SKILL.md) |
+| Post-run QA on any of the above | [skills/verify-pipeline-run](skills/verify-pipeline-run/SKILL.md) |
 
-The **scripts/** directory contains end-to-end pipelines that call library functions using settings from `config.yaml`. They are not part of the installed package and serve as reference implementations.
+## Where things live
 
-### Key classes and files
+| Path | Purpose |
+|---|---|
+| [src/openpois/io/](../src/openpois/io/) | I/O adapters: OSM history/snapshot, Overture, Foursquare, Census boundary |
+| [src/openpois/osm/](../src/openpois/osm/) | OSM-specific transforms: `format_observations`, `change_plots` |
+| [src/openpois/models/](../src/openpois/models/) | PyTorch empirical Bayes: `EventRate`, `ModelFitter`, model registry |
+| [src/openpois/conflation/](../src/openpois/conflation/) | OSM×Overture matching: `taxonomy`, `match`, `merge` |
+| [scripts/](../scripts/) | End-to-end pipelines using config.yaml — not installed, reference only |
+| [site/](../site/) | Vue 3 + Vite frontend |
 
-- `EventRate` (`models/event_rate.py`) — wraps a constant or time-varying λ; computes change probabilities via integration
-- `ModelFitter` (`models/model_fitter.py`) — fits λ using PyTorch L-BFGS optimizer with optional priors; supports parameter draws for uncertainty
-- `pytorch_setup()` / `prepare_data_for_model()` (`models/setup.py`) — initializes torch (GPU/CPU) and prepares filtered, grouped observation data
-- `download_osm_history()` (`io/osm_history_pbf.py`) — US+PR history pipeline entry: Geofabrik full-history PBFs → osmium tags-filter (`--omit-referenced`) → osmium time-filter → pyosmium stream → `osm_versions.parquet` + `osm_changes.parquet`. Requires `download.osm.history_cookie_file` to point at a Netscape-format cookie jar with valid Geofabrik OAuth cookies.
-- `download_element_histories()` (`io/osm_history.py`) — legacy city-scale entry point (Overpass, `download.osm.history_bbox` config key, Seattle-scoped; Overpass cannot serve US-wide histories)
+## Reference docs
 
-### Configuration
+- [docs/data-sources.md](docs/data-sources.md) — URLs, auth, schema quirks for every source
+- [docs/taxonomy-setup.md](docs/taxonomy-setup.md) — crosswalk CSVs, build_taxonomy.py, frontend sync
+- [docs/data-versioning.md](docs/data-versioning.md) — `versions:` block, path resolution, external references
 
-`config.yaml` holds all shared settings (spatial boundary, date ranges, OSM tag keys, model hyperparameters, output directory paths with versioning). The `config_versioned` package (external dependency) reads this file. Scripts load config at startup; library functions accept parameters directly.
+## Running to-do
 
-- `.get()` raises `ValueError` for null config values — pass `fail_if_none=False` for optional fields like `release_date: null`
+[TODO.md](TODO.md) — curated running list. Not auto-synced to git status.
 
-## POI Snapshot Downloads
+## Config gotcha worth surfacing
 
-Three separate utilities download current snapshots covering the 50 US states + DC + Puerto Rico (separate from the historical OSM workflow):
-
-### Spatial boundary (`src/openpois/io/boundary.py`)
-- Single source of truth for the US+PR extent used by all three snapshot downloaders
-- Downloads the Census 1:20M cartographic state shapefile (`cb_2023_us_state_20m`) on first use; cached under `directories.boundary`
-- `get_us_pr_boundary()` returns `(boundary_gdf, coarse_bboxes)` — a single-row dissolved+buffered polygon (EPSG:4326) plus a list of bboxes for predicate pushdown
-- Buffering is done in `EPSG:6933` (World Equal-Area Cylindrical) so the `coastline_buffer_m` (default 100 m) is accurate across CONUS / AK / HI / PR. Because `.dissolve()` removes internal state borders, the uniform outward buffer effectively only expands coastline; land-border expansion into CA/MX is negligible.
-- `coarse_bboxes` splits the Aleutians at the antimeridian into two bboxes (Near Islands at +172°E vs. rest of AK at negative longitudes)
-
-### OSM (`src/openpois/io/osm_snapshot.py`)
-- `download_pbf` / `filter_pbf` / `parse_pbf_to_geodataframe` / `download_osm_snapshot`
-- Two Geofabrik extracts: `us-latest.osm.pbf` (~11 GB, 50 states incl. AK+HI) + `puerto-rico-latest.osm.pbf` (PR is NOT in the US extract) → osmium tags-filter → pyosmium parse → concat → GeoParquet
-- Geofabrik extracts are pre-cut to admin boundaries, so no polygon post-filter is needed
-- `osmium` is in the conda env bin but NOT on shell PATH; code resolves it via `Path(sys.executable).parent / "osmium"`
-- Run: `python scripts/osm_snapshot/download.py`
-
-### Overture Maps (`src/openpois/io/overture.py`)
-- DuckDB + httpfs + spatial extensions; queries public S3 directly, no auth
-- **Two-stage spatial filter:** DuckDB `WHERE` clause ORs one disjunct per coarse bbox (predicate pushdown on Overture's `bbox` struct column), then a GeoPandas `sjoin(predicate='within')` post-filter against the exact US+PR polygon
-- `taxonomy` field is a named STRUCT: use `taxonomy.hierarchy[1]` (not `taxonomy[1]`)
-- `brand` is a singular struct (not array); geometry is native DuckDB GEOMETRY type requiring `LOAD spatial` and `ST_X()/ST_Y()`
-- L0 category names (Feb 2026+): `food_and_drink`, `shopping`, `arts_and_entertainment`, `sports_and_recreation`, `health_care`
-- Run: `python scripts/overture/download.py`
-
-### Foursquare OS Places (`src/openpois/io/foursquare.py`)
-- PyIceberg `RestCatalog`; requires `warehouse="places"` parameter
-- Catalog: `uri=https://catalog.h3-hub.foursquare.com/iceberg`, namespace=`datasets`, tables=`places_os` / `categories_os`
-- Table is **unpartitioned** (no `dt` column); release date inferred from `last_updated_at` in partition metadata
-- Row filter: `country IN ('US', 'PR') AND date_closed IS NULL` — Foursquare uses ISO alpha-2 codes, so PR must be listed explicitly; PyIceberg has no spatial predicate support, so an exact `sjoin(predicate='within')` post-filter runs after the rows are loaded
-- `fsq_category_ids` arrives as numpy/pyarrow array — use `len(x) == 0` not `if not x:`
-- Token in `FSQ_PORTAL_TOKEN` env var; run: `python scripts/foursquare/download.py`
+`config_versioned.Config.get()` raises `ValueError` on null values. For optional fields (e.g., `release_date: null`), pass `fail_if_none=False`. Prefer `config.get_file_path(section, file_key)` over composing `get_dir_path()` + `get()` manually.

.claude/TODO.md

@@ -0,0 +1,19 @@
+# openpois — running to-do
+
+Short running list of in-progress / upcoming work. Edit freely; trim older completed items when the list gets long. Date items `YYYY-MM-DD` when added.
+
+## In progress
+
+_(no items — add some when work is underway)_
+
+## Upcoming
+
+- [ ] Watch for Overture L0/L1 → flat `basic_category` migration (~June 2026). Crosswalk CSV + `assign_overture_shared_label` will need updating. See [docs/taxonomy-setup.md](docs/taxonomy-setup.md).
+
+## Recently done
+
+_(trim after a few weeks)_
+
+---
+
+**Agent note:** When uncommitted changes are present in the repo, do not assume they belong in "In progress" here — confirm with the user first. This file is curated, not auto-synced to git status.

.claude/docs/data-sources.md

@@ -0,0 +1,81 @@
+# Data sources
+
+Reference for every external data source openpois ingests. For the workflow that orchestrates these, see the skills under [.claude/skills/](../skills/).
+
+## OSM history (Geofabrik full-history PBFs)
+
+**Used by**: the historical modeling pipeline ([skills/model-history-pipeline](../skills/model-history-pipeline/SKILL.md)).
+
+- **URLs**:
+  - `download.osm.history_pbf_url` → `https://osm-internal.download.geofabrik.de/north-america/us-internal.osh.pbf`
+  - `download.osm.pr_history_pbf_url` → `.../us/puerto-rico-internal.osh.pbf`
+- **Auth**: OAuth — any OSM account works. Produce a Netscape-format cookie jar (browser export or Geofabrik's `oauth_cookie_client.py`). Path: `download.osm.history_cookie_file` (default `~/data/openpois/.creds/geofabrik_cookies.txt`).
+- **Pipeline**: `osmium tags-filter --omit-referenced` → `osmium time-filter` → pyosmium streams to `osm_versions.parquet` + `osm_changes.parquet`.
+- **Entry**: [src/openpois/io/osm_history_pbf.py](../../src/openpois/io/osm_history_pbf.py) (`download_osm_history`).
+- **Config**: `download.osm.start_date`, `end_date`, `date_interval_days`, `filter_keys`, `extract_keys`.
+
+## OSM snapshot (Geofabrik standard PBFs)
+
+**Used by**: current-state snapshot (`osm_snapshot.parquet`).
+
+- **URLs**:
+  - US: `https://download.geofabrik.de/north-america/us-latest.osm.pbf` (~11 GB, 50 states incl. AK+HI)
+  - PR: `https://download.geofabrik.de/north-america/us/puerto-rico-latest.osm.pbf` — **PR is not in the US extract**
+- **Auth**: none (public).
+- **Pipeline**: `osmium tags-filter` → pyosmium parse → concat US+PR → GeoParquet.
+- **Entry**: [src/openpois/io/osm_snapshot.py](../../src/openpois/io/osm_snapshot.py).
+- **Quirks**:
+  - `osmium` is in the conda env's `bin/` but **not** on shell PATH. Code resolves via `Path(sys.executable).parent / "osmium"`.
+  - Geofabrik extracts are pre-cut to admin boundaries → no polygon post-filter needed.
+
+## Overture Maps
+
+**Used by**: current-state Overture snapshot (`overture_snapshot.parquet`).
+
+- **URL**: public S3 at `s3://overturemaps-us-west-2/`.
+- **Auth**: none (DuckDB + httpfs queries directly).
+- **Pipeline**: two-stage spatial filter — DuckDB `WHERE` clause ORs one disjunct per coarse bbox (predicate pushdown on Overture's `bbox` struct), then GeoPandas `sjoin(predicate='within')` against the exact US+PR polygon.
+- **Entry**: [src/openpois/io/overture.py](../../src/openpois/io/overture.py).
+- **Schema quirks (as of Feb 2026 schema)**:
+  - `taxonomy` is a named STRUCT `{primary, hierarchy[], alternates[]}` — use `taxonomy.hierarchy[1]` **not** `taxonomy[1]`.
+  - `brand` is a singular struct, **not** a `brands[]` array.
+  - L0 category names: `food_and_drink`, `shopping`, `arts_and_entertainment`, `sports_and_recreation`, `health_care`.
+  - Geometry is native DuckDB GEOMETRY — must `LOAD spatial;` and use `ST_X()` / `ST_Y()`.
+- **Upcoming migration (~June 2026)**: L0/L1 hierarchy → flat `basic_category`. Crosswalk CSV + `assign_overture_shared_label` will need updating.
+
+## Foursquare OS Places
+
+**Used by**: current-state Foursquare snapshot (`foursquare_snapshot.parquet`).
+
+- **Catalog**: `https://catalog.h3-hub.foursquare.com/iceberg` — PyIceberg `RestCatalog`.
+- **Auth**: `FSQ_PORTAL_TOKEN` env var.
+- **Params** (all in config.yaml):
+  - `warehouse="places"` (required)
+  - `namespace="datasets"`
+  - `places_table="places_os"`, `categories_table="categories_os"`
+- **Pipeline**: row filter `country IN ('US', 'PR') AND date_closed IS NULL` → PyIceberg scan → sjoin against exact US+PR polygon (PyIceberg has no spatial predicates).
+- **Entry**: [src/openpois/io/foursquare.py](../../src/openpois/io/foursquare.py).
+- **Quirks**:
+  - Table is **unpartitioned** (no `dt` column); release date inferred from `last_updated_at` in partition metadata.
+  - `fsq_category_ids` arrives as numpy/pyarrow array — use `len(x) == 0`, **not** `if not x:`.
+  - PR uses alpha-2 `'PR'`, not `'US'` — silent drop regression pre-2026-04-16 when filter was `'US'`-only.
+
+## Census boundary
+
+**Used by**: all three snapshot downloaders (spatial clipping).
+
+- **URL**: `download.general.boundary.source_url` → `https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_us_state_20m.zip` (1:20M cartographic, 50 states + DC + PR).
+- **Auth**: none.
+- **Pipeline**: download ZIP → cache under `directories.boundary` (first-use) → dissolve → buffer outward by `coastline_buffer_m` (default 100 m) in EPSG:6933 (equal-area, so buffer accurate across CONUS/AK/HI/PR).
+- **Entry**: [src/openpois/io/boundary.py](../../src/openpois/io/boundary.py) (`get_us_pr_boundary`).
+- **Returns**: `(boundary_gdf, coarse_bboxes)` — single-row dissolved+buffered polygon (EPSG:4326) plus a list of bboxes for predicate pushdown.
+- **Antimeridian**: Aleutians split into two bboxes (Near Islands at +172°E vs. rest of AK at negative longitudes).
+
+## Legacy: Overpass-based OSM history
+
+Still wired up but superseded by the PBF pipeline. Queries Overpass API for element IDs in a bbox, then fetches per-element histories from the OSM API.
+
+- **Config**: `download.osm.history_bbox` (Seattle-scoped; Overpass can't serve US-wide histories).
+- **Entry**: [src/openpois/io/osm_history.py](../../src/openpois/io/osm_history.py) (`download_element_histories`).
+- **Script**: `scripts/osm_data/download.py`.
+- **When to use**: city-scale testing, or if Geofabrik OAuth is unavailable.

.claude/docs/data-versioning.md

@@ -0,0 +1,62 @@
+# Data versioning
+
+Every pipeline output is versioned via a single `versions:` block in [config.yaml](../../config.yaml). The external `config_versioned` package resolves these into filesystem paths.
+
+## Source of truth
+
+```yaml
+versions:
+  osm_data: "20260416"                 # historical PBF pipeline outputs
+  model_output: "20260416_by_leisure"  # fitted model artifacts (suffix indicates variant)
+  snapshot_osm: "20260416"             # OSM current-state snapshot
+  snapshot_overture: "20260417"        # Overture snapshot
+  snapshot_foursquare: "20260416"      # Foursquare snapshot
+  aws: "20260416"                      # S3 prefix for uploaded data
+  conflation: "20260417"               # conflated output
+```
+
+Each key corresponds to a `directories.<key>` entry in `config.yaml` with `versioned: true`.
+
+## Path resolution
+
+External `config_versioned.Config` API:
+
+```python
+config.get_dir_path("osm_data")
+# → ~/data/openpois/osm_data/20260416/
+
+config.get_file_path("osm_data", "osm_versions")
+# → ~/data/openpois/osm_data/20260416/osm_versions.parquet
+```
+
+**Prefer `get_file_path` over composing `get_dir_path()` + `get()` manually.**
+
+`.get()` raises `ValueError` on null values — pass `fail_if_none=False` for optional fields like `download.overture.release_date: null` and `download.foursquare.release_date: null`.
+
+`config.write_self(section)` snapshots the effective config into the output directory — used by model and conflation scripts to record the state of a run.
+
+## Naming conventions
+
+- **Dates**: `YYYYMMDD`, e.g., `20260416`.
+- **Model variants**: `{date}_by_{group_key}` (e.g., `20260416_by_leisure`, `20260416_by_amenity`) or `{date}_constant`. See [skills/iterate-model-types](../skills/iterate-model-types/SKILL.md).
+- **Independent cadences**: snapshot versions can (and should) differ across sources — Overture releases ~monthly, Foursquare separately. Don't force them to match.
+
+## External references (hand-update when bumping)
+
+Version strings appear in these places outside `versions:` — grep before any cross-source version change:
+
+| File | References |
+|---|---|
+| [config.yaml](../../config.yaml) | `upload.latest_url_osm`, `upload.latest_url_conflation` (full URL with date) |
+| [site/src/constants.js](../../site/src/constants.js) | `OSM_S3_BASE`, `FSQ_S3_BASE`, `CONFLATED_S3_BASE` |
+| [site/public/about.html](../../site/public/about.html) | Hardcoded S3 browse links in the data-access section |
+| `osm_data.apply_model.model_stub` (config.yaml) | Which model family [scripts/osm_snapshot/apply_model.py](../../scripts/osm_snapshot/apply_model.py) ingests |
+
+[skills/update-site](../skills/update-site/SKILL.md) covers the frontend side; [skills/conflate-snapshots](../skills/conflate-snapshots/SKILL.md) covers the upload + config side.
+
+## Workflow
+
+1. Bump the relevant `versions.*` keys before running a pipeline.
+2. Run the pipeline — outputs land in the versioned directory.
+3. After upload, update `upload.latest_url_*` and the frontend references.
+4. Old versions stay on disk / S3 — delete manually when confident nothing references them.