Search across the entire US (including Puerto Rico) for data.

Author: njhenry

.claude/CLAUDE.md

@@ -24,47 +24,56 @@ Code style is enforced by Black (format on save in VSCode). Linting via flake8 a
 
 **openpois** models POI (Point of Interest) stability over time using historical OpenStreetMap data. The workflow is:
 
-1. **Download OSM history** (`src/openpois/osm/download.py`) — queries the Overpass API for element histories within a bounding box and date range, producing version/change tables
+1. **Download OSM history** (`src/openpois/io/osm_history.py`) — queries the Overpass API for element histories within a bounding box and date range, producing version/change tables
 2. **Format observations** (`src/openpois/osm/format_observations.py`) — converts raw OSM version histories into observation records (one row per version) with flags for tag changes and deletions
 3. **Model change rates** (`src/openpois/models/`) — fits an empirical Bayes model using PyTorch to estimate per-group POI change rates (λ) as a Poisson process
 4. **Visualize stability** (`src/openpois/osm/change_plots.py`) — plots how long POI tags remain unchanged
 
-The **exploratory/** scripts are end-to-end pipelines that call library functions using settings from `config.yaml`. They are not part of the installed package and serve as reference implementations.
+The **scripts/** directory contains end-to-end pipelines that call library functions using settings from `config.yaml`. They are not part of the installed package and serve as reference implementations.
 
 ### Key classes and files
 
 - `EventRate` (`models/event_rate.py`) — wraps a constant or time-varying λ; computes change probabilities via integration
 - `ModelFitter` (`models/model_fitter.py`) — fits λ using PyTorch L-BFGS optimizer with optional priors; supports parameter draws for uncertainty
 - `pytorch_setup()` / `prepare_data_for_model()` (`models/setup.py`) — initializes torch (GPU/CPU) and prepares filtered, grouped observation data
-- `download_element_histories()` (`osm/download.py`) — main entry point for OSM history acquisition (Overpass, Seattle bbox only — do NOT modify for nationwide use)
+- `download_element_histories()` (`io/osm_history.py`) — main entry point for OSM history acquisition (Overpass, `download.osm.history_bbox` config key, Seattle-scoped — do NOT repurpose for nationwide use; Overpass cannot serve US-wide histories)
 
 ### Configuration
 
-`config.yaml` holds all shared settings (bounding box, date ranges, OSM tag keys, model hyperparameters, output directory paths with versioning). The `config_versioned` package (external dependency) reads this file. Exploratory scripts load config at startup; library functions accept parameters directly.
+`config.yaml` holds all shared settings (spatial boundary, date ranges, OSM tag keys, model hyperparameters, output directory paths with versioning). The `config_versioned` package (external dependency) reads this file. Scripts load config at startup; library functions accept parameters directly.
 
 - `.get()` raises `ValueError` for null config values — pass `fail_if_none=False` for optional fields like `release_date: null`
 
 ## POI Snapshot Downloads
 
-Three separate utilities download current US-wide snapshots (separate from the historical OSM workflow):
+Three separate utilities download current snapshots covering the 50 US states + DC + Puerto Rico (separate from the historical OSM workflow):
 
-### OSM (`src/openpois/osm/snapshot.py`)
+### Spatial boundary (`src/openpois/io/boundary.py`)
+- Single source of truth for the US+PR extent used by all three snapshot downloaders
+- Downloads the Census 1:20M cartographic state shapefile (`cb_2023_us_state_20m`) on first use; cached under `directories.boundary`
+- `get_us_pr_boundary()` returns `(boundary_gdf, coarse_bboxes)` — a single-row dissolved+buffered polygon (EPSG:4326) plus a list of bboxes for predicate pushdown
+- Buffering is done in `EPSG:6933` (World Equal-Area Cylindrical) so the `coastline_buffer_m` (default 100 m) is accurate across CONUS / AK / HI / PR. Because `.dissolve()` removes internal state borders, the uniform outward buffer effectively only expands coastline; land-border expansion into CA/MX is negligible.
+- `coarse_bboxes` splits the Aleutians at the antimeridian into two bboxes (Near Islands at +172°E vs. rest of AK at negative longitudes)
+
+### OSM (`src/openpois/io/osm_snapshot.py`)
 - `download_pbf` / `filter_pbf` / `parse_pbf_to_geodataframe` / `download_osm_snapshot`
-- Geofabrik US extract (~11 GB) → osmium tags-filter → pyosmium parse → GeoParquet
+- Two Geofabrik extracts: `us-latest.osm.pbf` (~11 GB, 50 states incl. AK+HI) + `puerto-rico-latest.osm.pbf` (PR is NOT in the US extract) → osmium tags-filter → pyosmium parse → concat → GeoParquet
+- Geofabrik extracts are pre-cut to admin boundaries, so no polygon post-filter is needed
 - `osmium` is in the conda env bin but NOT on shell PATH; code resolves it via `Path(sys.executable).parent / "osmium"`
-- Run: `python exploratory/osm_snapshot/download.py`
+- Run: `python scripts/osm_snapshot/download.py`
 
-### Overture Maps (`src/openpois/overture/download.py`)
+### Overture Maps (`src/openpois/io/overture.py`)
 - DuckDB + httpfs + spatial extensions; queries public S3 directly, no auth
+- **Two-stage spatial filter:** DuckDB `WHERE` clause ORs one disjunct per coarse bbox (predicate pushdown on Overture's `bbox` struct column), then a GeoPandas `sjoin(predicate='within')` post-filter against the exact US+PR polygon
 - `taxonomy` field is a named STRUCT: use `taxonomy.hierarchy[1]` (not `taxonomy[1]`)
 - `brand` is a singular struct (not array); geometry is native DuckDB GEOMETRY type requiring `LOAD spatial` and `ST_X()/ST_Y()`
 - L0 category names (Feb 2026+): `food_and_drink`, `shopping`, `arts_and_entertainment`, `sports_and_recreation`, `health_care`
-- Run: `python exploratory/overture/download.py`
+- Run: `python scripts/overture/download.py`
 
-### Foursquare OS Places (`src/openpois/foursquare/download.py`)
+### Foursquare OS Places (`src/openpois/io/foursquare.py`)
 - PyIceberg `RestCatalog`; requires `warehouse="places"` parameter
 - Catalog: `uri=https://catalog.h3-hub.foursquare.com/iceberg`, namespace=`datasets`, tables=`places_os` / `categories_os`
 - Table is **unpartitioned** (no `dt` column); release date inferred from `last_updated_at` in partition metadata
-- Row filter: `country = 'US' AND date_closed IS NULL` (no dt filter)
+- Row filter: `country IN ('US', 'PR') AND date_closed IS NULL` — Foursquare uses ISO alpha-2 codes, so PR must be listed explicitly; PyIceberg has no spatial predicate support, so an exact `sjoin(predicate='within')` post-filter runs after the rows are loaded
 - `fsq_category_ids` arrives as numpy/pyarrow array — use `len(x) == 0` not `if not x:`
-- Token in `FSQ_PORTAL_TOKEN` env var; run: `python exploratory/foursquare/download.py`
+- Token in `FSQ_PORTAL_TOKEN` env var; run: `python scripts/foursquare/download.py`

config.yaml

@@ -1,27 +1,38 @@
 # Versioned directories (used with config.get_dir_path())
 versions:
-  osm_data: "20260313"
-  model_output: "20260315_constant"
-  snapshot_osm: "20260313"
-  snapshot_overture: "20260313"
-  snapshot_foursquare: "20260313"
-  aws: "20260318"
-  conflation: "20260318"
+  osm_data: "20260416"
+  model_output: "20260315"
+  snapshot_osm: "20260416"
+  snapshot_overture: "20260416"
+  snapshot_foursquare: "20260416"
+  aws: "20260416"
+  conflation: "20260416"
 
 # Settings for downloading data
 download:
   general:
     timeout: 1_000
-    bbox:
-      xmin: -125.0
-      ymin: 24.5
-      xmax: -66.9
-      ymax: 49.4
+    # Census 1:20M cartographic state boundary file; includes 50 states + DC
+    # + PR. Used by all three snapshot downloads to restrict POIs to the US
+    # plus Puerto Rico. The coastline buffer expands the dissolved polygon
+    # outward by N metres so near-shore POIs are retained; internal state
+    # borders disappear on dissolve so the buffer only affects the coast.
+    boundary:
+      source_url: "https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_us_state_20m.zip"
+      coastline_buffer_m: 100
   osm:
     start_date: 2016-01-01
     end_date: 2025-12-31
     date_interval_days: 7
+    # Seattle-area bbox used only by the Overpass-based historical download.
+    # Overpass cannot serve US-wide histories; keep this scoped to a city.
+    history_bbox:
+      xmin: -122.45
+      ymin: 47.50
+      xmax: -122.25
+      ymax: 47.70
     pbf_url: "https://download.geofabrik.de/north-america/us-latest.osm.pbf"
+    pr_pbf_url: "https://download.geofabrik.de/north-america/us/puerto-rico-latest.osm.pbf"
     overwrite_download: false
     overwrite_filter: false
     source_label: "osm"
@@ -76,7 +87,7 @@ osm_data:
     - last_obs_timestamp
     - last_tag_timestamp
   apply_model:
-    model_stub: '20260315'
+    model_stub: '20260319_discounted'
 
 # Settings for exploratory/models/pytorch_simple.py
 osm_turnover_model:
@@ -119,10 +130,15 @@ directories:
     files:
       raw_pbf: us-latest.osm.pbf
       filtered_pbf: us-pois.osm.pbf
+      raw_pr_pbf: puerto-rico-latest.osm.pbf
+      filtered_pr_pbf: puerto-rico-pois.osm.pbf
       snapshot: osm_snapshot.parquet
       rated_snapshot: osm_snapshot_rated.parquet
       partitioned: osm_snapshot_partitioned
       pmtiles: osm_snapshot.pmtiles
+  boundary:
+    versioned: false
+    path: ~/data/openpois/boundary
   snapshot_overture:
     versioned: true
     path: ~/data/openpois/snapshots/overture

scripts/foursquare/download.py

@@ -1,10 +1,12 @@
 """
-Download the current US Foursquare OS Places snapshot as a GeoParquet file.
+Download the current US+PR Foursquare OS Places snapshot as a GeoParquet file.
 
 Authenticates to the Foursquare Places Portal Apache Iceberg REST catalog
-using a portal token, loads the unpartitioned places_os table filtered to US
-records with no closed date, joins against categories_os to resolve L1
-category names, and saves the result as a GeoParquet file.
+using a portal token, loads the unpartitioned places_os table filtered to
+places whose country is 'US' or 'PR' with no closed date, joins against
+categories_os to resolve L1 category names, applies an exact within-polygon
+filter against the US+PR Census boundary, and saves the result as a
+GeoParquet file.
 
 Authentication:
     Set the FSQ_PORTAL_TOKEN environment variable before running:
@@ -20,13 +22,17 @@
     download.foursquare.categories_table    — categories table name ("categories_os")
     download.foursquare.token_env_var       — env var name for the portal token
     download.foursquare.l1_category_names   — L1 category filter list
+    download.general.boundary.source_url    — Census state-boundary zip URL
+    download.general.boundary.coastline_buffer_m — outward coastline buffer (m)
+    directories.boundary                    — cache directory for boundary file
     directories.snapshot_foursquare         — output directory
 
 Output file:
-    foursquare_snapshot.parquet — GeoParquet with ~8.3M US POIs
+    foursquare_snapshot.parquet — GeoParquet with US+PR POIs
         Columns: fsq_place_id, name, fsq_category_ids, geometry, source
 """
 from config_versioned import Config
+from openpois.io.boundary import get_us_pr_boundary
 from openpois.io.foursquare import download_foursquare_snapshot
 
 # -----------------------------------------------------------------------------
@@ -44,6 +50,11 @@
 CATEGORIES_TABLE = config.get("download", "foursquare", "categories_table")
 TOKEN_ENV_VAR = config.get("download", "foursquare", "token_env_var")
 L1_CATEGORIES = config.get("download", "foursquare", "l1_category_names")
+BOUNDARY_URL = config.get("download", "general", "boundary", "source_url")
+COASTLINE_BUFFER_M = config.get(
+    "download", "general", "boundary", "coastline_buffer_m"
+)
+BOUNDARY_DIR = config.get_dir_path("boundary")
 
 SAVE_DIR = config.get_dir_path("snapshot_foursquare")
 SAVE_DIR.mkdir(parents=True, exist_ok=True)
@@ -55,6 +66,11 @@
 # -----------------------------------------------------------------------------
 
 if __name__ == "__main__":
+    boundary_gdf, _ = get_us_pr_boundary(
+        source_url = BOUNDARY_URL,
+        cache_dir = BOUNDARY_DIR,
+        coastline_buffer_m = COASTLINE_BUFFER_M,
+    )
     gdf = download_foursquare_snapshot(
         output_path=OUTPUT_PATH,
         l1_category_names=L1_CATEGORIES,
@@ -64,6 +80,7 @@
         places_table=PLACES_TABLE,
         categories_table=CATEGORIES_TABLE,
         token_env_var=TOKEN_ENV_VAR,
+        boundary_gdf=boundary_gdf,
         release_date=RELEASE_DATE,
     )
     print(f"Saved {len(gdf):,} Foursquare POIs to {OUTPUT_PATH}")

scripts/osm_data/data_viz.py

@@ -49,7 +49,6 @@
 END_DATE = pd.Timestamp(config.get("download", "osm", "end_date"), tz='UTC')
 MODEL_BASE = config.get_dir_path("model_output").parent
 MODEL_STUB = config.get("osm_data", "apply_model", "model_stub")
-ADJ_FACTOR = 1.0
 
 max_days = 365 * 10
 VIZ_DIR.mkdir(parents=True, exist_ok=True)
@@ -82,7 +81,7 @@ def fig_save(
         **kwargs
     )
 
-def get_preds_dict(model_stub: str | None, adj_factor: float = 1.0) -> dict[str, pd.DataFrame]:
+def get_preds_dict(model_stub: str | None) -> dict[str, pd.DataFrame]:
     """
     Load model predictions from the model output directory.
     """
@@ -98,9 +97,9 @@ def get_preds_df(model_stub: str, subset: str | None = None) -> Path:
             return None
         return pd.read_csv(preds_fp).assign(
             year = pd.col('t2'),
-            conf_mean = (1.0 - pd.col('p_mean')) * adj_factor,
-            conf_lower = (1.0 - pd.col('p_upper')) * adj_factor,
-            conf_upper = (1.0 - pd.col('p_lower')) * adj_factor,
+            conf_mean = (1.0 - pd.col('p_mean')),
+            conf_lower = (1.0 - pd.col('p_upper')),
+            conf_upper = (1.0 - pd.col('p_lower')),
         )
     preds = dict()
     preds["constant"] = get_preds_df(model_stub)
@@ -115,7 +114,7 @@ def get_preds_df(model_stub: str, subset: str | None = None) -> Path:
 
 if __name__ == "__main__":
     # Read model predictions
-    preds = get_preds_dict(MODEL_STUB, adj_factor = ADJ_FACTOR)
+    preds = get_preds_dict(MODEL_STUB)
     # Read observations
     # Drop the first observation for each POI (when the POI was first added) - the last
     #   observation timestamp will be missing for these rows

scripts/osm_data/download.py

@@ -6,7 +6,7 @@
 element via the OSM API.
 
 Config keys used (config.yaml):
-    download.general.bbox            — WGS-84 bbox [xmin, ymin, xmax, ymax]
+    download.osm.history_bbox        — WGS-84 bbox [xmin, ymin, xmax, ymax]
     download.general.timeout         — request timeout in seconds
     download.osm.start_date          — earliest snapshot date (min: 2012-09-13)
     download.osm.end_date            — latest snapshot date
@@ -35,7 +35,7 @@
 config = Config("~/repos/openpois/config.yaml")
 
 TIMEOUT = config.get("download", "general", "timeout")
-BBOX = config.get("download", "general", "bbox")
+BBOX = config.get("download", "osm", "history_bbox")
 # Earliest option is September 13, 2012
 START_DATE = datetime.datetime.combine(
     config.get("download", "osm", "start_date"), datetime.time.min

scripts/osm_snapshot/download.py

@@ -1,20 +1,23 @@
 """
-Download the current US OpenStreetMap POI snapshot as a GeoParquet file.
+Download the current US+PR OpenStreetMap POI snapshot as a GeoParquet file.
 
-Downloads the Geofabrik North America PBF extract (~11 GB), uses osmium
-tags-filter to extract nodes and ways matching the configured tag keys, then
-parses the result with pyosmium into a GeoDataFrame and saves as GeoParquet.
-Incremental: skips the PBF download or filter step if output files already
-exist (controlled by overwrite_download and overwrite_filter config flags).
+Downloads two Geofabrik PBF extracts — the US-mainland extract (~11 GB,
+covers all 50 states incl. AK + HI) and the Puerto Rico extract — uses
+osmium tags-filter to extract nodes and ways matching the configured tag
+keys, parses the result with pyosmium into GeoDataFrames, concatenates the
+US + PR results, and saves as GeoParquet. Incremental: skips any PBF
+download or filter step whose output file already exists (controlled by
+overwrite_download and overwrite_filter config flags).
 
 Note: osmium is resolved from the conda env bin rather than the shell PATH;
 no manual PATH modification is needed.
 
 Config keys used (config.yaml):
-    download.osm.pbf_url             — Geofabrik PBF URL (North America extract)
+    download.osm.pbf_url             — Geofabrik US PBF URL (50 states)
+    download.osm.pr_pbf_url          — Geofabrik Puerto Rico PBF URL
     download.osm.filter_keys         — OSM tag keys to retain (e.g. amenity, shop)
     download.osm.extract_keys        — tag keys to include as output columns
-    download.osm.overwrite_download  — re-download PBF even if it already exists
+    download.osm.overwrite_download  — re-download PBFs even if they already exist
     download.osm.overwrite_filter    — re-run osmium filter even if output exists
     download.osm.source_label        — value written to the "source" column
     download.osm.keep_all_keys       — retain all discovered tag columns in output
@@ -24,7 +27,7 @@
     directories.snapshot_osm         — output directory; also used for temp PBF files
 
 Output file:
-    osm_snapshot.parquet — GeoParquet with ~7.8M US POIs (nodes + area centroids)
+    osm_snapshot.parquet — GeoParquet with US+PR POIs (nodes + area centroids)
         Columns: osm_id, osm_type, name, geometry, last_edited, source,
         plus all extract_keys columns
 """
@@ -38,6 +41,7 @@
 config = Config("~/repos/openpois/config.yaml")
 
 PBF_URL = config.get("download", "osm", "pbf_url")
+PR_PBF_URL = config.get("download", "osm", "pr_pbf_url")
 FILTER_KEYS = config.get("download", "osm", "filter_keys")
 EXTRACT_KEYS = config.get("download", "osm", "extract_keys")
 OVERWRITE_DOWNLOAD = config.get("download", "osm", "overwrite_download")
@@ -54,6 +58,8 @@
 
 RAW_PBF = config.get_file_path("snapshot_osm", "raw_pbf")
 FILTERED_PBF = config.get_file_path("snapshot_osm", "filtered_pbf")
+RAW_PR_PBF = config.get_file_path("snapshot_osm", "raw_pr_pbf")
+FILTERED_PR_PBF = config.get_file_path("snapshot_osm", "filtered_pr_pbf")
 OUTPUT_PATH = config.get_file_path("snapshot_osm", "snapshot")
 
 
@@ -66,6 +72,9 @@
         pbf_url = PBF_URL,
         raw_pbf_path = RAW_PBF,
         filtered_pbf_path = FILTERED_PBF,
+        pr_pbf_url = PR_PBF_URL,
+        raw_pr_pbf_path = RAW_PR_PBF,
+        filtered_pr_pbf_path = FILTERED_PR_PBF,
         output_path = OUTPUT_PATH,
         filter_keys = FILTER_KEYS,
         extract_keys = EXTRACT_KEYS,