Foursquare is already incorporated into Overture Maps; remove Foursquare code and documentation references.

Author: njhenry

.claude/CLAUDE.md

@@ -22,13 +22,13 @@ Style: Black (format-on-save in VSCode). Lint: flake8 + pylint, configured in `p
 
 ## Architecture at a glance
 
-**openpois** models POI stability over time from OpenStreetMap history, and produces unified OSM + Overture + Foursquare snapshots for web consumption. Work splits into four pipelines:
+**openpois** models POI stability over time from OpenStreetMap history, and produces unified OSM + Overture snapshots for web consumption. Work splits into four pipelines:
 
 | Pipeline | Skill |
 |---|---|
 | Fit λ from OSM history, rate current snapshots | [skills/model-history-pipeline](skills/model-history-pipeline/SKILL.md) |
 | Iterate model variants on a pinned history run | [skills/iterate-model-types](skills/iterate-model-types/SKILL.md) |
-| Refresh the three POI snapshots (OSM / Overture / FSQ) | [skills/full-data-pull](skills/full-data-pull/SKILL.md) |
+| Refresh the POI snapshots (OSM / Overture) | [skills/full-data-pull](skills/full-data-pull/SKILL.md) |
 | Conflate OSM + Overture, partition, upload to S3 | [skills/conflate-snapshots](skills/conflate-snapshots/SKILL.md) |
 | Bump the frontend to the new data version | [skills/update-site](skills/update-site/SKILL.md) |
 | Post-run QA on any of the above | [skills/verify-pipeline-run](skills/verify-pipeline-run/SKILL.md) |
@@ -37,7 +37,7 @@ Style: Black (format-on-save in VSCode). Lint: flake8 + pylint, configured in `p
 
 | Path | Purpose |
 |---|---|
-| [src/openpois/io/](../src/openpois/io/) | I/O adapters: OSM history/snapshot, Overture, Foursquare, Census boundary |
+| [src/openpois/io/](../src/openpois/io/) | I/O adapters: OSM history/snapshot, Overture, Census boundary |
 | [src/openpois/osm/](../src/openpois/osm/) | OSM-specific transforms: `format_observations`, `change_plots` |
 | [src/openpois/models/](../src/openpois/models/) | JAX/BlackJAX empirical Bayes: `ModelFitter`, model registry |
 | [src/openpois/conflation/](../src/openpois/conflation/) | OSM×Overture matching: `taxonomy`, `match`, `merge` |

.claude/docs/data-sources.md

@@ -44,26 +44,9 @@ Reference for every external data source openpois ingests. For the workflow that
   - Geometry is native DuckDB GEOMETRY — must `LOAD spatial;` and use `ST_X()` / `ST_Y()`.
 - **Upcoming migration (~June 2026)**: L0/L1 hierarchy → flat `basic_category`. Crosswalk CSV + `assign_overture_shared_label` will need updating.
 
-## Foursquare OS Places
-
-**Used by**: current-state Foursquare snapshot (`foursquare_snapshot.parquet`).
-
-- **Catalog**: `https://catalog.h3-hub.foursquare.com/iceberg` — PyIceberg `RestCatalog`.
-- **Auth**: `FSQ_PORTAL_TOKEN` env var.
-- **Params** (all in config.yaml):
-  - `warehouse="places"` (required)
-  - `namespace="datasets"`
-  - `places_table="places_os"`, `categories_table="categories_os"`
-- **Pipeline**: row filter `country IN ('US', 'PR') AND date_closed IS NULL` → PyIceberg scan → sjoin against exact US+PR polygon (PyIceberg has no spatial predicates).
-- **Entry**: [src/openpois/io/foursquare.py](../../src/openpois/io/foursquare.py).
-- **Quirks**:
-  - Table is **unpartitioned** (no `dt` column); release date inferred from `last_updated_at` in partition metadata.
-  - `fsq_category_ids` arrives as numpy/pyarrow array — use `len(x) == 0`, **not** `if not x:`.
-  - PR uses alpha-2 `'PR'`, not `'US'` — silent drop regression pre-2026-04-16 when filter was `'US'`-only.
-
 ## Census boundary
 
-**Used by**: all three snapshot downloaders (spatial clipping).
+**Used by**: both snapshot downloaders (spatial clipping).
 
 - **URL**: `download.general.boundary.source_url` → `https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_us_state_20m.zip` (1:20M cartographic, 50 states + DC + PR).
 - **Auth**: none.

.claude/docs/data-versioning.md

@@ -10,7 +10,6 @@ versions:
   model_output: "20260416_by_leisure"  # fitted model artifacts (suffix indicates variant)
   snapshot_osm: "20260416"             # OSM current-state snapshot
   snapshot_overture: "20260417"        # Overture snapshot
-  snapshot_foursquare: "20260416"      # Foursquare snapshot
   aws: "20260416"                      # S3 prefix for uploaded data
   conflation: "20260417"               # conflated output
 ```
@@ -31,15 +30,15 @@ config.get_file_path("osm_data", "osm_versions")
 
 **Prefer `get_file_path` over composing `get_dir_path()` + `get()` manually.**
 
-`.get()` raises `ValueError` on null values — pass `fail_if_none=False` for optional fields like `download.overture.release_date: null` and `download.foursquare.release_date: null`.
+`.get()` raises `ValueError` on null values — pass `fail_if_none=False` for optional fields like `download.overture.release_date: null`.
 
 `config.write_self(section)` snapshots the effective config into the output directory — used by model and conflation scripts to record the state of a run.
 
 ## Naming conventions
 
 - **Dates**: `YYYYMMDD`, e.g., `20260416`.
 - **Model variants**: `{date}_by_{group_key}` (e.g., `20260416_by_leisure`, `20260416_by_amenity`) or `{date}_constant`. See [skills/iterate-model-types](../skills/iterate-model-types/SKILL.md).
-- **Independent cadences**: snapshot versions can (and should) differ across sources — Overture releases ~monthly, Foursquare separately. Don't force them to match.
+- **Independent cadences**: snapshot versions can (and should) differ across sources — Overture releases ~monthly. Don't force them to match.
 
 ## External references (hand-update when bumping)
 
@@ -48,7 +47,7 @@ Version strings appear in these places outside `versions:` — grep before any c
 | File | References |
 |---|---|
 | [config.yaml](../../config.yaml) | `upload.latest_url_osm`, `upload.latest_url_conflation` (full URL with date) |
-| [site/src/constants.js](../../site/src/constants.js) | `OSM_S3_BASE`, `FSQ_S3_BASE`, `CONFLATED_S3_BASE` |
+| [site/src/constants.js](../../site/src/constants.js) | `OSM_S3_BASE`, `CONFLATED_S3_BASE` |
 | [site/public/about.html](../../site/public/about.html) | Hardcoded S3 browse links in the data-access section |
 | `osm_data.apply_model.model_stub` (config.yaml) | Which model family [scripts/osm_snapshot/apply_model.py](../../scripts/osm_snapshot/apply_model.py) ingests |
 

.claude/skills/full-data-pull/SKILL.md

@@ -1,16 +1,15 @@
 ---
 name: full-data-pull
-description: Use when the user wants to refresh the three independent POI snapshots (OSM, Overture, Foursquare) and rate the OSM snapshot for conflation. Triggers: "refresh all snapshots", "do a new data pull", "download new OSM/Overture/Foursquare", "monthly data refresh", "pull the latest POI data". Does NOT include conflation or S3 upload — those live in conflate-snapshots.
+description: Use when the user wants to refresh the independent POI snapshots (OSM, Overture) and rate the OSM snapshot for conflation. Triggers: "refresh all snapshots", "do a new data pull", "download new OSM/Overture", "monthly data refresh", "pull the latest POI data". Does NOT include conflation or S3 upload — those live in conflate-snapshots.
 ---
 
 # Full data pull
 
-Downloads the three snapshot sources (50 US states + DC + PR) and applies the rating model to OSM so conflation can run.
+Downloads the snapshot sources (50 US states + DC + PR) and applies the rating model to OSM so conflation can run.
 
 ## Prerequisites
 
 - conda env `openpois` active.
-- For Foursquare: `FSQ_PORTAL_TOKEN` env var set.
 - For OSM: `osmium` in env bin (resolved automatically via `Path(sys.executable).parent / "osmium"`).
 - Boundary cache at `directories.boundary` (auto-downloads on first use).
 - A fitted model exists for the OSM rating step (see [skills/model-history-pipeline](../model-history-pipeline/SKILL.md)).
@@ -22,16 +21,14 @@ Downloads the three snapshot sources (50 US states + DC + PR) and applies the ra
    versions:
      snapshot_osm: "YYYYMMDD"
      snapshot_overture: "YYYYMMDD"
-     snapshot_foursquare: "YYYYMMDD"
    ```
    See [docs/data-versioning.md](../../docs/data-versioning.md).
 
-2. **Run the three downloads** (independent — order doesn't matter, can run in parallel):
+2. **Run the downloads** (independent — order doesn't matter, can run in parallel):
 
    ```bash
    python scripts/osm_snapshot/download.py     # ~11 GB US PBF + PR PBF → osm_snapshot.parquet
    python scripts/overture/download.py         # DuckDB over S3           → overture_snapshot.parquet
-   python scripts/foursquare/download.py       # PyIceberg catalog        → foursquare_snapshot.parquet
    ```
    Per-source details, auth, and schema quirks are in [docs/data-sources.md](../../docs/data-sources.md).
 
@@ -51,9 +48,8 @@ Downloads the three snapshot sources (50 US states + DC + PR) and applies the ra
 Hand off to [skills/verify-pipeline-run](../verify-pipeline-run/SKILL.md). Baseline totals (as of 2026-04-17):
 - OSM: ~7.78M POIs
 - Overture: ~13.05M POIs (jumped from ~7.23M after widening `download.overture.taxonomy_allowlist` to include `services_and_business` + `lifestyle_services` sub-branches)
-- Foursquare: ~8.32M POIs
 
-Flag >5% drops — Foursquare in particular has had silent country-filter regressions (PR alpha-2 code quirk).
+Flag >5% drops.
 
 ## Next
 

.claude/skills/update-site/SKILL.md

@@ -9,7 +9,7 @@ Vue 3 + Vite frontend lives in [site/](../../../site/). After a data pull + uplo
 
 ## Prerequisites
 
-- New data published to S3 via [skills/conflate-snapshots](../conflate-snapshots/SKILL.md) (or the latest FSQ snapshot uploaded separately).
+- New data published to S3 via [skills/conflate-snapshots](../conflate-snapshots/SKILL.md).
 - Node + npm available (see `site/package.json` for engine requirements).
 
 ## Steps
@@ -18,7 +18,6 @@ Vue 3 + Vite frontend lives in [site/](../../../site/). After a data pull + uplo
 
 2. **Update S3 URLs in [site/src/constants.js](../../../site/src/constants.js)** — each source may have a different date:
    - `OSM_S3_BASE` → `snapshots/osm/YYYYMMDD/osm_snapshot_partitioned`
-   - `FSQ_S3_BASE` → `snapshots/foursquare/YYYYMMDD/foursquare_snapshot_partitioned`
    - `CONFLATED_S3_BASE` → `snapshots/conflated/YYYYMMDD/conflated_partitioned`
    - `OVERTURE_PMTILES_URL` → bump on monthly Overture release
 
@@ -30,7 +29,7 @@ Vue 3 + Vite frontend lives in [site/](../../../site/). After a data pull + uplo
    ```
    Verify:
    - Map loads POIs at zoom 14+ without CORS/404 errors
-   - Source filter dropdown (OSM / Overture / Foursquare / Conflated) toggles data
+   - Source filter dropdown (OSM / Overture / Conflated) toggles data
    - Taxonomy legend renders from `taxonomy.html`
    - POI popups show non-empty name/category/confidence
 

.claude/skills/verify-pipeline-run/SKILL.md

@@ -7,12 +7,11 @@ description: Use when the user wants a QA/sanity check on a recently completed p
 
 Post-run QA runbook. Pick the subsection that matches what just ran.
 
-## Snapshots (OSM / Overture / Foursquare)
+## Snapshots (OSM / Overture)
 
 Baseline row counts (2026-04-17):
 - OSM: ~7.78M
 - Overture: ~13.05M (up from ~7.23M after widening `taxonomy_allowlist`; pre-2026-04-17 runs will be lower)
-- Foursquare: ~8.32M
 
 Check:
 ```python
@@ -21,7 +20,6 @@ pd.read_parquet(path).shape[0]
 ```
 
 Flag >5% drops. Known regression patterns:
-- **Foursquare**: PR alpha-2 code — filter must be `country IN ('US', 'PR')`, not `'US'` only.
 - **OSM**: PR is a *separate* PBF — confirm both `us-latest.osm.pbf` and `puerto-rico-latest.osm.pbf` got downloaded, filtered, and concat'd.
 - **Overture**: coarse-bbox pushdown + final DuckDB `ST_Within` — drop means the Aleutian antimeridian split was lost or the Census boundary failed to load. If the run crashed with "Information loss on integer cast", the DuckDB pin was bumped off 1.4.1 (see [docs/data-sources.md](../../docs/data-sources.md) → Overture Maps).
 
@@ -63,7 +61,7 @@ Confirm `conf_mean`, `conf_lower`, `conf_upper` columns are populated for every
 
 - Open the deployed site (or `npm run dev` locally after a constants.js bump).
 - Browser console: no CORS, no 404s on S3 URLs.
-- Filter dropdown: each source (OSM / Overture / Foursquare / Conflated) loads.
+- Filter dropdown: each source (OSM / Overture / Conflated) loads.
 - Popups non-empty; taxonomy legend rendered; PMTiles overlay visible at zoom 14+.
 
 ## Recording issues

README.md

@@ -11,7 +11,7 @@ make install_package  # Install openpois in editable mode
 
 ## POI Snapshot Downloads
 
-Three exploratory scripts download current US-wide POI snapshots from different sources. All output GeoParquet to `~/data/`.
+Two exploratory scripts download current US-wide POI snapshots from different sources. Both output GeoParquet to `~/data/`.
 
 ### OpenStreetMap
 
@@ -31,18 +31,7 @@ Queries the public Overture Maps S3 bucket directly via DuckDB. No authenticatio
 python exploratory/overture/download.py
 ```
 
-Output: `~/data/openpois/snapshots/overture/<VERSION>/overture_snapshot.parquet` (~7.2M POIs)
-
-### Foursquare OS Places
-
-Queries the Foursquare Places Portal Apache Iceberg catalog. Requires a free API token from [places.foursquare.com](https://places.foursquare.com).
-
-```bash
-export FSQ_PORTAL_TOKEN="<your_token>"
-python exploratory/foursquare/download.py
-```
-
-Output: `~/data/openpois/snapshots/foursquare/<VERSION>/foursquare_snapshot.parquet` (~8.3M POIs)
+Output: `~/data/openpois/snapshots/overture/<VERSION>/overture_snapshot.parquet` (~13M POIs)
 
 ### Configuration
 

config.yaml

@@ -4,7 +4,6 @@ versions:
   model_output: "20260422_by_shared_label"
   snapshot_osm: "20260417"
   snapshot_overture: "20260423"
-  snapshot_foursquare: "20260416"
   aws: "20260423"
   conflation: "20260423"
 
@@ -103,20 +102,6 @@ download:
       - [community_and_government, public_safety_service]
       - [travel_and_transportation, fueling_station]
       - [travel_and_transportation, vehicle_service]
-  foursquare:
-    release_date: null  # null = auto-detect latest
-    catalog_uri: "https://catalog.h3-hub.foursquare.com/iceberg"
-    catalog_warehouse: "places"
-    catalog_namespace: "datasets"
-    places_table: "places_os"
-    categories_table: "categories_os"
-    token_env_var: "FSQ_PORTAL_TOKEN"
-    l1_category_names:
-      - "Dining and Drinking"
-      - "Retail"
-      - "Arts and Entertainment"
-      - "Sports and Recreation"
-      - "Health and Medicine"
 
 # Settings for OSM exploratory data analysis
 osm_data:
@@ -185,12 +170,6 @@ directories:
       predictions: predictions.csv
       diagnostics: diagnostics.csv
       inference_data: inference_data.nc
-  snapshot_foursquare:
-    versioned: true
-    path: ~/data/openpois/snapshots/foursquare
-    files:
-      snapshot: foursquare_snapshot.parquet
-      partitioned: foursquare_snapshot_partitioned
   snapshot_osm:
     versioned: true
     path: ~/data/openpois/snapshots/osm
@@ -226,7 +205,6 @@ directories:
     files:
       osm_snippet: osm_snippet.csv
       overture_snippet: overture_snippet.csv
-      foursquare_snippet: foursquare_snippet.csv
 
 # Settings for POI conflation
 conflation:

docs/api.rst

@@ -92,19 +92,6 @@ from S3 if a specific date is not pinned.
    :undoc-members:
    :show-inheritance:
 
-openpois.io.foursquare
-~~~~~~~~~~~~~~~~~~~~~~
-
-Download a current US-wide Foursquare OS Places snapshot via the Foursquare
-Places Portal Apache Iceberg REST catalog. Authenticates with a portal token,
-loads US open venues filtered by L1 category, and resolves category names from
-the categories table. Requires the ``FSQ_PORTAL_TOKEN`` environment variable.
-
-.. automodule:: openpois.io.foursquare
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 openpois.io.geohash_partition
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/index.rst

@@ -2,8 +2,8 @@ openpois
 ========
 
 A Python library for conflating Points of Interest (POIs) across
-OpenStreetMap, Overture Maps, and Foursquare OS Places, with utilities for
-modeling POI stability over time using historical OSM data.
+OpenStreetMap and Overture Maps, with utilities for modeling POI
+stability over time using historical OSM data.
 
 - **GitHub:** https://github.com/henryspatialanalysis/openpois
 - **Web map:** https://openpois.org/