Update Overture download to resolve httpfs bug in DuckDB 1.4.1 through 1.5.2.

Author: njhenry

.claude/TODO.md

@@ -4,16 +4,18 @@ Short running list of in-progress / upcoming work. Edit freely; trim older compl
 
 ## In progress
 
-_(no items — add some when work is underway)_
-
 ## Upcoming
 
+- [ ] Watch for a DuckDB release that fixes the WSL2 httpfs "Information loss on integer cast" crash (issue #21669, fix PR #21395). Once a tagged release ships with the fix and a full `scripts/overture/download.py` run on WSL2 completes, we can unpin from `duckdb==1.4.1` and revert the per-part download to a single-query DuckDB scan. Added 2026-04-17.
+- [ ] Auto-check taxonomy changes whenever we switch to a new Overture Maps version (detect new/removed L0/L1/L2 categories vs. `taxonomy_crosswalk_overture_maps.csv` and flag gaps). Added 2026-04-16.
 - [ ] Watch for Overture L0/L1 → flat `basic_category` migration (~June 2026). Crosswalk CSV + `assign_overture_shared_label` will need updating. See [docs/taxonomy-setup.md](docs/taxonomy-setup.md).
 
 ## Recently done
 
 _(trim after a few weeks)_
 
+- [x] Fix: CONUS Overture download crashed DuckDB on httpfs scans — 2026-04-17. Refactored [src/openpois/io/overture.py](../src/openpois/io/overture.py) to per-part resumable download + final filter-in-DuckDB; pinned `duckdb==1.4.1` to dodge bug #21669. Full run produced 13,054,244 POIs.
+
 ---
 
 **Agent note:** When uncommitted changes are present in the repo, do not assume they belong in "In progress" here — confirm with the user first. This file is curated, not auto-synced to git status.

.claude/docs/data-sources.md

@@ -34,8 +34,9 @@ Reference for every external data source openpois ingests. For the workflow that
 
 - **URL**: public S3 at `s3://overturemaps-us-west-2/`.
 - **Auth**: none (DuckDB + httpfs queries directly).
-- **Pipeline**: two-stage spatial filter — DuckDB `WHERE` clause ORs one disjunct per coarse bbox (predicate pushdown on Overture's `bbox` struct), then GeoPandas `sjoin(predicate='within')` against the exact US+PR polygon.
-- **Entry**: [src/openpois/io/overture.py](../../src/openpois/io/overture.py).
+- **Pipeline**: per-part resumable download → exact-polygon filter, all inside DuckDB. Each of the 16 `part-*.parquet` files streams through a fresh DuckDB connection into a local parquet intermediate under `.parts/<release>/`; coarse-bbox `WHERE` pushes down on Overture's `bbox` struct. Once every part is present, a final `COPY` applies `ST_Within` against the dissolved US+PR polygon and writes the GeoParquet. No pandas materialization; crashed runs resume by skipping existing intermediates.
+- **Entry**: [src/openpois/io/overture.py](../../src/openpois/io/overture.py). Returns a `Path`, not a `GeoDataFrame`.
+- **DuckDB version pin**: `environment.yml` pins `duckdb==1.4.1`. 1.4.4+ and every 1.5.x crash mid-scan on WSL2 with "Information loss on integer cast" in `HTTPFileSystem::ReadInternal` — tracked as DuckDB issue #21669, fix merged to main but not in any tagged release as of 2026-04-17. See [memory: project_duckdb_pin.md] for the bump checklist.
 - **Schema quirks (as of Feb 2026 schema)**:
   - `taxonomy` is a named STRUCT `{primary, hierarchy[], alternates[]}` — use `taxonomy.hierarchy[1]` **not** `taxonomy[1]`.
   - `brand` is a singular struct, **not** a `brands[]` array.

.claude/scheduled_tasks.lock

@@ -0,0 +1 @@
+{"sessionId":"6fe31aa6-c607-4ab5-b54d-07e6afb28372","pid":37405,"acquiredAt":1776408925552}

.claude/skills/full-data-pull/SKILL.md

@@ -48,9 +48,9 @@ Downloads the three snapshot sources (50 US states + DC + PR) and applies the ra
 
 ## Verification
 
-Hand off to [skills/verify-pipeline-run](../verify-pipeline-run/SKILL.md). Baseline totals (as of 2026-04-16):
+Hand off to [skills/verify-pipeline-run](../verify-pipeline-run/SKILL.md). Baseline totals (as of 2026-04-17):
 - OSM: ~7.78M POIs
-- Overture: ~7.23M POIs
+- Overture: ~13.05M POIs (jumped from ~7.23M after widening `download.overture.taxonomy_allowlist` to include `services_and_business` + `lifestyle_services` sub-branches)
 - Foursquare: ~8.32M POIs
 
 Flag >5% drops — Foursquare in particular has had silent country-filter regressions (PR alpha-2 code quirk).

.claude/skills/verify-pipeline-run/SKILL.md

@@ -9,9 +9,9 @@ Post-run QA runbook. Pick the subsection that matches what just ran.
 
 ## Snapshots (OSM / Overture / Foursquare)
 
-Baseline row counts (2026-04-16):
+Baseline row counts (2026-04-17):
 - OSM: ~7.78M
-- Overture: ~7.23M
+- Overture: ~13.05M (up from ~7.23M after widening `taxonomy_allowlist`; pre-2026-04-17 runs will be lower)
 - Foursquare: ~8.32M
 
 Check:
@@ -23,7 +23,7 @@ pd.read_parquet(path).shape[0]
 Flag >5% drops. Known regression patterns:
 - **Foursquare**: PR alpha-2 code — filter must be `country IN ('US', 'PR')`, not `'US'` only.
 - **OSM**: PR is a *separate* PBF — confirm both `us-latest.osm.pbf` and `puerto-rico-latest.osm.pbf` got downloaded, filtered, and concat'd.
-- **Overture**: coarse-bbox pushdown + exact `sjoin` — drop means the Aleutian antimeridian split was lost or the Census boundary failed to load.
+- **Overture**: coarse-bbox pushdown + final DuckDB `ST_Within` — drop means the Aleutian antimeridian split was lost or the Census boundary failed to load. If the run crashed with "Information loss on integer cast", the DuckDB pin was bumped off 1.4.1 (see [docs/data-sources.md](../../docs/data-sources.md) → Overture Maps).
 
 ## Model output
 

config.yaml

@@ -65,6 +65,13 @@ download:
     release_date: null  # null = auto-detect latest
     s3_bucket: "overturemaps-us-west-2"
     s3_region: "us-west-2"
+    # DuckDB resource caps for the per-part S3 scans and the final polygon
+    # filter. Peak host RAM ~= workers * memory_limit, peak CPU ~= workers *
+    # threads. Scale per-worker values down if raising workers above 1.
+    duckdb:
+      memory_limit: "4GB"
+      threads: 2
+      workers: 2
     # (L0, L1) allowlist. L1 = null means "all of this L0".
     # Entries intentionally exclude office/B2B-style L1s (corporate offices,
     # media services, etc.), transit/parking/airports (covered elsewhere), and

environment.yml

@@ -196,7 +196,7 @@ dependencies:
       - coverage==7.13.4
       - cryptography==46.0.5
       - dill==0.4.1
-      - duckdb==1.5.0
+      - duckdb==1.4.1
       - et-xmlfile==2.0.0
       - filelock==3.20.0
       - flake8==7.3.0

scripts/overture/download.py

@@ -2,10 +2,12 @@
 Download the current US+PR Overture Maps Places snapshot as a GeoParquet file.
 
 Queries Overture Maps GeoParquet files on public S3 using DuckDB's httpfs and
-spatial extensions, filtering with a two-stage spatial filter (coarse bbox
-prefilter in the DuckDB query, then exact within-polygon filter in Python
-against the US+PR Census boundary) and by L0 taxonomy category. No
-authentication required — Overture Maps data is publicly accessible.
+spatial extensions. Iterates the release's ``part-*.parquet`` files, writing a
+bounded-memory DuckDB COPY per part into a ``.parts/<release>/`` directory.
+Once every part is present, a single DuckDB COPY applies the exact US+PR
+polygon filter and writes the final GeoParquet without materializing rows in
+Python. Interrupted runs resume by skipping parts whose intermediates already
+exist. No authentication required — Overture Maps data is publicly accessible.
 
 Auto-detects the latest available Overture release from S3 unless a specific
 release_date is pinned in config.yaml.
@@ -15,6 +17,9 @@
     download.overture.s3_bucket              — Overture Maps S3 bucket name
     download.overture.s3_region              — AWS region of the Overture bucket
     download.overture.taxonomy_allowlist     — list of [L0, L1] pairs; L1 null = any
+    download.overture.duckdb.memory_limit    — per-connection DuckDB memory cap
+    download.overture.duckdb.threads         — per-connection DuckDB thread count
+    download.overture.duckdb.workers         — parallel part downloads (must be 1)
     download.general.boundary.source_url     — Census state-boundary zip URL
     download.general.boundary.coastline_buffer_m — outward coastline buffer (m)
     directories.boundary                     — cache directory for boundary file
@@ -25,6 +30,7 @@
         Columns: overture_id, overture_name, taxonomy_l0, taxonomy_l1,
         taxonomy_l2, brand_name, confidence, geometry, source
 """
+import pyarrow.parquet as pq
 from config_versioned import Config
 from openpois.io.boundary import get_us_pr_boundary
 from openpois.io.overture import download_overture_snapshot
@@ -40,6 +46,15 @@
 S3_BUCKET = config.get("download", "overture", "s3_bucket")
 S3_REGION = config.get("download", "overture", "s3_region")
 TAXONOMY_ALLOWLIST = config.get("download", "overture", "taxonomy_allowlist")
+DUCKDB_MEMORY_LIMIT = config.get(
+    "download", "overture", "duckdb", "memory_limit", fail_if_none=False
+) or "4GB"
+DUCKDB_THREADS = config.get(
+    "download", "overture", "duckdb", "threads", fail_if_none=False
+) or 2
+DUCKDB_WORKERS = config.get(
+    "download", "overture", "duckdb", "workers", fail_if_none=False
+) or 2
 BOUNDARY_URL = config.get("download", "general", "boundary", "source_url")
 COASTLINE_BUFFER_M = config.get(
     "download", "general", "boundary", "coastline_buffer_m"
@@ -62,13 +77,17 @@
         cache_dir = BOUNDARY_DIR,
         coastline_buffer_m = COASTLINE_BUFFER_M,
     )
-    gdf = download_overture_snapshot(
-        output_path=OUTPUT_PATH,
-        taxonomy_allowlist=TAXONOMY_ALLOWLIST,
-        boundary_gdf=boundary_gdf,
-        coarse_bboxes=coarse_bboxes,
-        bucket=S3_BUCKET,
-        s3_region=S3_REGION,
-        release_date=RELEASE_DATE,
+    output_path = download_overture_snapshot(
+        output_path = OUTPUT_PATH,
+        taxonomy_allowlist = TAXONOMY_ALLOWLIST,
+        boundary_gdf = boundary_gdf,
+        coarse_bboxes = coarse_bboxes,
+        bucket = S3_BUCKET,
+        s3_region = S3_REGION,
+        release_date = RELEASE_DATE,
+        duckdb_memory_limit = DUCKDB_MEMORY_LIMIT,
+        duckdb_threads = DUCKDB_THREADS,
+        workers = DUCKDB_WORKERS,
     )
-    print(f"Saved {len(gdf):,} Overture POIs to {OUTPUT_PATH}")
+    n_rows = pq.read_metadata(output_path).num_rows
+    print(f"Saved {n_rows:,} Overture POIs to {output_path}")

site/src/constants.js

@@ -66,19 +66,22 @@ export const CONFLATED_LABELS = [
   'Alternative Medicine',
   'Arcade',
   'Arts Venue',
+  'ATM',
   'Bakery',
   'Bank',
   'Bar',
   'Bike shop',
   'Bookstore',
   'Bowling Alley',
   'Cafe',
+  'Campground',
   'Car Dealer',
   'Car Rental',
   'Car Repair',
   'Car Wash',
   'Casino',
   'Cell Phone Store',
+  'Cemetery',
   'Charging Station',
   'Clinic',
   'Clothing Store',
@@ -99,10 +102,15 @@ export const CONFLATED_LABELS = [
   'Garden Store',
   'Gas Station',
   'Golf Course',
+  'Government Office',
+  'Hair and Beauty',
   'Hardware',
+  'Home Service',
+  'Hotel',
   'Jewelry Store',
   'Kindergarten',
   'Laundromat',
+  'Legal Service',
   'Library',
   'Liquor Store',
   'Marina',
@@ -119,11 +127,14 @@ export const CONFLATED_LABELS = [
   'Pet Store',
   'Pharmacy',
   'Physical Therapy',
+  'Place of Worship',
   'Playground',
   'Post Office',
+  'Public Restroom',
+  'Public Safety',
+  'Real Estate',
   'Recreation',
   'Restaurant',
-  'Salon and Hair',
   'School',
   'Shoe Store',
   'Shopping Center',
@@ -141,7 +152,9 @@ export const CONFLATED_LABELS = [
   'Wholesale Store',
   // "Other" categories last, unchecked by default
   'Other Amenity',
+  'Other Financial',
   'Other Healthcare',
+  'Other Professional',
   'Other Shop',
 ]