Merge pull request #31 from henryspatialanalysis/feature/us-overseas-territories

Expand project scope to US + all overseas territories.

Author: njhenry

.claude/docs/data-sources.md

@@ -6,35 +6,40 @@ Reference for every external data source openpois ingests. For the workflow that
 
 **Used by**: the historical modeling pipeline ([skills/model-history-pipeline](../skills/model-history-pipeline/SKILL.md)).
 
-- **URLs**:
+- **URLs** (passed via `HistoryExtract` specs from `scripts/osm_data/download_history.py`):
   - `download.osm.history_pbf_url` → `https://osm-internal.download.geofabrik.de/north-america/us-internal.osh.pbf`
-  - `download.osm.pr_history_pbf_url` → `.../us/puerto-rico-internal.osh.pbf`
+  - `download.osm.pr_history_pbf_url` → `.../north-america/us/puerto-rico-internal.osh.pbf`
+  - `download.osm.usvi_history_pbf_url` → `.../north-america/us/us-virgin-islands-internal.osh.pbf`
+  - `download.osm.american_oceania_history_pbf_url` → `.../australia-oceania/american-oceania-internal.osh.pbf` (covers Guam, NMI, American Samoa, plus uninhabited US Pacific possessions)
 - **Auth**: OAuth — any OSM account works. Produce a Netscape-format cookie jar (browser export or Geofabrik's `oauth_cookie_client.py`). Path: `download.osm.history_cookie_file` (default `~/data/openpois/.creds/geofabrik_cookies.txt`).
-- **Pipeline**: `osmium tags-filter --omit-referenced` → `osmium time-filter` → pyosmium streams to `osm_versions.parquet` + `osm_changes.parquet`.
+- **Pipeline**: per-extract loop — `osmium tags-filter --omit-referenced` → `osmium time-filter` → pyosmium streams to intermediate `*_versions.parquet` + `*_changes.parquet`. Iterative N-way dedup (`_concat_history`) drops `(type, id)` overlap across extracts before writing the final `osm_versions.parquet` + `osm_changes.parquet`.
+- **Per-extract failure tolerance**: if a territory's history PBF returns HTTP 404 (e.g. Geofabrik stops publishing it), the loader logs a warning and continues without that territory's history; the rater then falls back to the global-mean δ for that territory's `shared_label`s.
 - **Entry**: [src/openpois/io/osm_history_pbf.py](../../src/openpois/io/osm_history_pbf.py) (`download_osm_history`).
 - **Config**: `download.osm.start_date`, `end_date`, `filter_keys`, `extract_keys`.
 
 ## OSM snapshot (Geofabrik standard PBFs)
 
 **Used by**: current-state snapshot (`osm_snapshot.parquet`).
 
-- **URLs**:
+- **URLs** (passed via `SnapshotExtract` specs from `scripts/osm_snapshot/download.py`):
   - US: `https://download.geofabrik.de/north-america/us-latest.osm.pbf` (~11 GB, 50 states incl. AK+HI)
   - PR: `https://download.geofabrik.de/north-america/us/puerto-rico-latest.osm.pbf` — **PR is not in the US extract**
+  - USVI: `https://download.geofabrik.de/north-america/us/us-virgin-islands-latest.osm.pbf`
+  - American Oceania: `https://download.geofabrik.de/australia-oceania/american-oceania-latest.osm.pbf` (covers Guam, NMI, American Samoa, and uninhabited US Pacific possessions; Geofabrik does not publish per-territory PBFs for the inhabited western Pacific territories)
 - **Auth**: none (public).
-- **Pipeline**: `osmium tags-filter` → pyosmium parse → concat US+PR → GeoParquet.
+- **Pipeline**: per-extract loop — `osmium tags-filter` → pyosmium parse → write intermediate parquets → concat all intermediates → GeoParquet.
 - **Entry**: [src/openpois/io/osm_snapshot.py](../../src/openpois/io/osm_snapshot.py).
 - **Quirks**:
   - `osmium` is in the conda env's `bin/` but **not** on shell PATH. Code resolves via `Path(sys.executable).parent / "osmium"`.
-  - Geofabrik extracts are pre-cut to admin boundaries → no polygon post-filter needed.
+  - Geofabrik extracts are pre-cut to admin boundaries → no polygon post-filter needed. `american-oceania-latest.osm.pbf` ships a few non-target uninhabited US Pacific possessions (Wake, Midway, Howland, Baker, Jarvis, Palmyra, Kingman); they contain near-zero POIs and pass through as bonus coverage.
 
 ## Overture Maps
 
 **Used by**: current-state Overture snapshot (`overture_snapshot.parquet`).
 
 - **URL**: public S3 at `s3://overturemaps-us-west-2/`.
 - **Auth**: none (DuckDB + httpfs queries directly).
-- **Pipeline**: per-part resumable download → exact-polygon filter, all inside DuckDB. Each of the 16 `part-*.parquet` files streams through a fresh DuckDB connection into a local parquet intermediate under `.parts/<release>/`; coarse-bbox `WHERE` pushes down on Overture's `bbox` struct. Once every part is present, a final `COPY` applies `ST_Within` against the dissolved US+PR polygon and writes the GeoParquet. No pandas materialization; crashed runs resume by skipping existing intermediates.
+- **Pipeline**: per-part resumable download → exact-polygon filter, all inside DuckDB. Each of the 16 `part-*.parquet` files streams through a fresh DuckDB connection into a local parquet intermediate under `.parts/<release>/`; coarse-bbox `WHERE` pushes down on Overture's `bbox` struct. Once every part is present, a final `COPY` applies `ST_Within` against the dissolved US + territories polygon and writes the GeoParquet. No pandas materialization; crashed runs resume by skipping existing intermediates.
 - **Entry**: [src/openpois/io/overture.py](../../src/openpois/io/overture.py). Returns a `Path`, not a `GeoDataFrame`.
 - **DuckDB version pin**: `environment.yml` pins `duckdb==1.4.1`. 1.4.4+ and every 1.5.x crash mid-scan on WSL2 with "Information loss on integer cast" in `HTTPFileSystem::ReadInternal` — tracked as DuckDB issue #21669, fix merged to main but not in any tagged release as of 2026-04-17. See [memory: project_duckdb_pin.md] for the bump checklist.
 - **Schema quirks (as of Feb 2026 schema)**:
@@ -48,10 +53,10 @@ Reference for every external data source openpois ingests. For the workflow that
 
 **Used by**: both snapshot downloaders (spatial clipping).
 
-- **URL**: `download.general.boundary.source_url` → `https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_us_state_20m.zip` (1:20M cartographic, 50 states + DC + PR).
+- **URL**: `download.general.boundary.source_url` → `https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_us_state_5m.zip` (1:5M cartographic, 50 states + DC + 5 inhabited territories: PR, VI, GU, MP, AS). Note: the 1:20M variant used previously does **not** include territories.
 - **Auth**: none.
-- **Pipeline**: download ZIP → cache under `directories.boundary` (first-use) → dissolve → buffer outward by `coastline_buffer_m` (default 100 m) in EPSG:6933 (equal-area, so buffer accurate across CONUS/AK/HI/PR).
+- **Pipeline**: download ZIP → cache under `directories.boundary` (first-use) → dissolve → buffer outward by `coastline_buffer_m` (default 100 m) in EPSG:6933 (equal-area, so buffer accurate across CONUS / AK / HI / Caribbean territories / western Pacific territories).
 - **Entry**: [src/openpois/io/boundary.py](../../src/openpois/io/boundary.py) (`get_us_pr_boundary`).
 - **Returns**: `(boundary_gdf, coarse_bboxes)` — single-row dissolved+buffered polygon (EPSG:4326) plus a list of bboxes for predicate pushdown.
-- **Antimeridian**: Aleutians split into two bboxes (Near Islands at +172°E vs. rest of AK at negative longitudes).
+- **Antimeridian**: two bboxes returned, split via per-part centroid at lon=0. The negative-longitude bbox covers CONUS, AK mainland, HI, PR, USVI, and American Samoa (~-170°W). The positive-longitude bbox covers the Aleutian Near Islands (~+172°E), Guam (~+144°E), and the Northern Mariana Islands (~+145°E).
 

.claude/skills/full-data-pull/SKILL.md

@@ -5,7 +5,7 @@ description: Use when the user wants to refresh the independent POI snapshots (O
 
 # Full data pull
 
-Downloads the snapshot sources (50 US states + DC + PR) and applies the rating model to OSM so conflation can run.
+Downloads the snapshot sources (50 US states + DC + 5 inhabited territories: PR, VI, GU, MP, AS) and applies the rating model to OSM so conflation can run.
 
 ## Prerequisites
 

README.md

@@ -1,7 +1,9 @@
 # OpenPOIs
 
-A unified, confidence-scored open dataset of U.S. points of interest, built
-from [OpenStreetMap](https://www.openstreetmap.org) and
+A unified, confidence-scored open dataset of U.S. points of interest —
+covering the 50 states, DC, and the 5 inhabited U.S. territories (Puerto
+Rico, US Virgin Islands, Guam, Northern Mariana Islands, American Samoa) —
+built from [OpenStreetMap](https://www.openstreetmap.org) and
 [Overture Maps](https://overturemaps.org).
 
 ![OpenPOIs interactive map](docs/_static/hero.png)
@@ -22,7 +24,8 @@ OpenPOIs conflates points of interest from OpenStreetMap and Overture Maps
 into a single unified dataset, then attaches a per-POI confidence score
 estimating the probability that the place still exists. Confidence comes from
 a Bayesian turnover model fit on OSM tag-edit history. The published dataset
-covers the United States and is refreshed monthly, following the Overture Maps monthly release cycle.
+covers the United States and its 5 inhabited territories, and is refreshed
+monthly, following the Overture Maps monthly release cycle.
 
 This repository contains the Python library used to produce the data, the
 end-to-end pipelines that download and conflate sources, and the Vue

config.yaml

@@ -15,25 +15,42 @@ versions:
 download:
   general:
     timeout: 1_000
-    # Census 1:20M cartographic state boundary file; includes 50 states + DC
-    # + PR. Used by all three snapshot downloads to restrict POIs to the US
-    # plus Puerto Rico. The coastline buffer expands the dissolved polygon
-    # outward by N metres so near-shore POIs are retained; internal state
-    # borders disappear on dissolve so the buffer only affects the coast.
+    # Census 1:5M cartographic state boundary file; includes 50 states + DC
+    # + 5 inhabited US territories (PR, USVI, GU, MP, AS). The 1:20M variant
+    # used previously only covered states + DC + PR. Used by all three
+    # snapshot downloads to restrict POIs to the US footprint. The coastline
+    # buffer expands the dissolved polygon outward by N metres so near-shore
+    # POIs are retained; internal state borders disappear on dissolve so the
+    # buffer only affects the coast.
     boundary:
-      source_url: "https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_us_state_20m.zip"
+      source_url: "https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_us_state_5m.zip"
       coastline_buffer_m: 100
   osm:
     start_date: 2016-01-01
     end_date: 2025-12-31
+    # Snapshot extracts. Geofabrik publishes a dedicated PBF for PR and USVI
+    # under north-america/us/, but the western-Pacific inhabited territories
+    # (Guam, Northern Mariana Islands, American Samoa) have no per-territory
+    # files. They are bundled into a single `american-oceania` extract that
+    # also includes the uninhabited US Pacific possessions (Wake, Midway,
+    # Howland, Baker, Jarvis, Palmyra, Kingman) — those contribute near-zero
+    # POIs and are accepted as bonus coverage.
     pbf_url: "https://download.geofabrik.de/north-america/us-latest.osm.pbf"
     pr_pbf_url: "https://download.geofabrik.de/north-america/us/puerto-rico-latest.osm.pbf"
+    usvi_pbf_url: "https://download.geofabrik.de/north-america/us/us-virgin-islands-latest.osm.pbf"
+    american_oceania_pbf_url: "https://download.geofabrik.de/australia-oceania/american-oceania-latest.osm.pbf"
     # Full-history PBFs live on Geofabrik's OAuth-protected internal server.
     # Any OSM account grants access; generate a Netscape-format cookie jar by
     # logging in at https://osm-internal.download.geofabrik.de/ and exporting
-    # cookies, or by running Geofabrik's oauth_cookie_client.py.
+    # cookies, or by running Geofabrik's oauth_cookie_client.py. The
+    # `usvi_*` and `american_oceania_*` history URLs follow the same path
+    # convention as the snapshot URLs; if any are missing on the server, the
+    # history loader logs a warning and continues without that territory's
+    # history (the rater then falls back to the global-mean delta).
     history_pbf_url: "https://osm-internal.download.geofabrik.de/north-america/us-internal.osh.pbf"
     pr_history_pbf_url: "https://osm-internal.download.geofabrik.de/north-america/us/puerto-rico-internal.osh.pbf"
+    usvi_history_pbf_url: "https://osm-internal.download.geofabrik.de/north-america/us/us-virgin-islands-internal.osh.pbf"
+    american_oceania_history_pbf_url: "https://osm-internal.download.geofabrik.de/australia-oceania/american-oceania-internal.osh.pbf"
     history_cookie_file: "~/data/openpois/.creds/geofabrik_cookies.txt"
     overwrite_download: true
     overwrite_filter: true
@@ -139,7 +156,7 @@ directories:
     versioned: true
     path: ~/data/openpois/osm_data
     files:
-      # US+PR full-history pipeline (PBF-based)
+      # US + territories full-history pipeline (PBF-based)
       osm_changes: osm_changes.parquet
       osm_versions: osm_versions.parquet
       raw_history_pbf: us-internal.osh.pbf
@@ -148,10 +165,20 @@ directories:
       raw_pr_history_pbf: puerto-rico-internal.osh.pbf
       filtered_pr_history_pbf: puerto-rico-pois.osh.pbf
       time_filtered_pr_history_pbf: puerto-rico-pois-timefilt.osh.pbf
+      raw_usvi_history_pbf: us-virgin-islands-internal.osh.pbf
+      filtered_usvi_history_pbf: us-virgin-islands-pois.osh.pbf
+      time_filtered_usvi_history_pbf: us-virgin-islands-pois-timefilt.osh.pbf
+      raw_american_oceania_history_pbf: american-oceania-internal.osh.pbf
+      filtered_american_oceania_history_pbf: american-oceania-pois.osh.pbf
+      time_filtered_american_oceania_history_pbf: american-oceania-pois-timefilt.osh.pbf
       us_versions: us_osm_versions.parquet
       us_changes: us_osm_changes.parquet
       pr_versions: pr_osm_versions.parquet
       pr_changes: pr_osm_changes.parquet
+      usvi_versions: usvi_osm_versions.parquet
+      usvi_changes: usvi_osm_changes.parquet
+      american_oceania_versions: american_oceania_osm_versions.parquet
+      american_oceania_changes: american_oceania_osm_changes.parquet
       # Modelling-ready observations (one row per POI version × shared_label)
       osm_observations: osm_observations.parquet
   model_output:
@@ -171,6 +198,10 @@ directories:
       filtered_pbf: us-pois.osm.pbf
       raw_pr_pbf: puerto-rico-latest.osm.pbf
       filtered_pr_pbf: puerto-rico-pois.osm.pbf
+      raw_usvi_pbf: us-virgin-islands-latest.osm.pbf
+      filtered_usvi_pbf: us-virgin-islands-pois.osm.pbf
+      raw_american_oceania_pbf: american-oceania-latest.osm.pbf
+      filtered_american_oceania_pbf: american-oceania-pois.osm.pbf
       snapshot: osm_snapshot.parquet
       rated_snapshot: osm_snapshot_rated.parquet
       partitioned: osm_snapshot_partitioned

docs/api.rst

@@ -54,11 +54,14 @@ io
 openpois.io.osm_history_pbf
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Download Geofabrik full-history PBFs (US + Puerto Rico), filter to POI tags
-with ``osmium tags-filter``, time-window with ``osmium time-filter``, and parse
-with pyosmium into per-version and per-change Parquet tables suitable for the
-change-rate model. Uses an OAuth cookie jar against Geofabrik's internal
-server.
+Download Geofabrik full-history PBFs (US + inhabited territories: Puerto
+Rico, US Virgin Islands, plus Guam / NMI / American Samoa via the
+``american-oceania`` extract), filter to POI tags with ``osmium tags-filter``,
+time-window with ``osmium time-filter``, and parse with pyosmium into
+per-version and per-change Parquet tables suitable for the change-rate
+model. Uses an OAuth cookie jar against Geofabrik's internal server.
+Per-extract failure tolerance: missing-on-server (HTTP 404) territory PBFs
+are skipped with a warning rather than aborting the run.
 
 .. automodule:: openpois.io.osm_history_pbf
    :members:

docs/workflows.rst

@@ -83,9 +83,10 @@ saving snippet CSVs to the ``testing/`` directory for column inspection.
 Pipeline 2: OSM Historical Change-Rate Model
 --------------------------------------------
 
-This pipeline downloads OpenStreetMap full-history PBFs (US + Puerto Rico)
-and fits a Poisson change-rate model to estimate how quickly different POI
-categories become outdated.
+This pipeline downloads OpenStreetMap full-history PBFs (US + inhabited
+territories: Puerto Rico, US Virgin Islands, plus Guam / NMI / American
+Samoa via the ``american-oceania`` extract) and fits a Poisson change-rate
+model to estimate how quickly different POI categories become outdated.
 
 **Step 1 — Download full-history PBFs**
 
@@ -94,11 +95,15 @@ categories become outdated.
    python scripts/osm_data/download_history.py
 
 Requires the Geofabrik OAuth cookie jar described in *Prerequisites* above.
-Downloads the US-mainland and Puerto Rico full-history extracts, filters
-each with ``osmium tags-filter`` (POI tag keys only) and ``osmium
-time-filter`` (the ``download.osm.start_date`` / ``end_date`` window), then
-parses with pyosmium into per-version and per-change Parquet tables.
-Outputs: ``osm_versions.parquet`` and ``osm_changes.parquet``.
+Downloads each Geofabrik full-history extract in turn, filters each with
+``osmium tags-filter`` (POI tag keys only) and ``osmium time-filter`` (the
+``download.osm.start_date`` / ``end_date`` window), then parses with
+pyosmium into per-extract Parquet tables and concatenates with iterative
+``(type, id)`` dedup. If a territory's history PBF is missing on the server
+(HTTP 404), the loader logs a warning and continues; that territory's
+snapshot/Overture coverage is unaffected and the rater falls back to the
+global-mean delta. Outputs: ``osm_versions.parquet`` and
+``osm_changes.parquet``.
 
 See :mod:`openpois.io.osm_history_pbf`.