Publish to Source Cooperative instead of personal S3.

Author: njhenry

.claude/CLAUDE.md

@@ -29,7 +29,7 @@ Style: Black (format-on-save in VSCode). Lint: flake8 + pylint, configured in `p
 | Fit λ from OSM history, rate current snapshots | [skills/model-history-pipeline](skills/model-history-pipeline/SKILL.md) |
 | Iterate model variants on a pinned history run | [skills/iterate-model-types](skills/iterate-model-types/SKILL.md) |
 | Refresh the POI snapshots (OSM / Overture) | [skills/full-data-pull](skills/full-data-pull/SKILL.md) |
-| Conflate OSM + Overture, partition, upload to S3 | [skills/conflate-snapshots](skills/conflate-snapshots/SKILL.md) |
+| Conflate OSM + Overture, partition, publish to Source Cooperative | [skills/conflate-snapshots](skills/conflate-snapshots/SKILL.md) |
 | Bump the frontend to the new data version | [skills/update-site](skills/update-site/SKILL.md) |
 | Post-run QA on any of the above | [skills/verify-pipeline-run](skills/verify-pipeline-run/SKILL.md) |
 

.claude/TODO.md

@@ -6,6 +6,11 @@ Short running list of in-progress / upcoming work. Edit freely; trim older compl
 
 ## Upcoming
 
+- [ ] **Auto-capture the three per-version README fields** so the publish step doesn't need `publish.version_metadata` overrides. Added 2026-04-24. Today `build_version_readme` in [src/openpois/publish/build_readme.py](../src/openpois/publish/build_readme.py) falls back to config overrides or best-effort guesses; aim is for the pipeline to write authoritative values alongside the data it produces, and the publish step to just read them.
+    - *OSM snapshot date* — `scripts/osm_snapshot/download.py` should write a `~/data/openpois/snapshots/osm/<version>/download_metadata.json` containing `{"downloaded_at": "<ISO date>", "pbf_url": "..."}` after the PBF download completes. `_resolve_osm_snapshot_date` then reads that file before falling back to the version string.
+    - *Overture release* — `scripts/overture/download.py` already resolves a concrete release (pinned or auto-detected) inside `download_overture_snapshot`; currently only the `.parts/<release>/` directory records it and `.parts/` is deleted on success. Surface the resolved release by writing `~/data/openpois/snapshots/overture/<version>/download_metadata.json` with `{"release": "2026-04-15.0", ...}` before the cleanup step. `_resolve_overture_release` reads that file ahead of the `.parts/` heuristic.
+    - *Turnover-model commit* — `scripts/models/osm_turnover.py` should capture `git rev-parse HEAD` at training time and either (a) extend `config.write_self("model_output")` to include a `git_commit` entry or (b) drop a `git_commit.txt` next to the model artifacts. `_resolve_model_commit` reads that value instead of the publish-time HEAD, which is the right fingerprint if code has changed between training and publishing.
+    - Publishing behaviour: if any of the three files is missing, keep the current fallback (and print a visible warning) so old pipeline runs still publish cleanly.
 - [ ] Watch for a DuckDB release that fixes the WSL2 httpfs "Information loss on integer cast" crash (issue #21669, fix PR #21395). Once a tagged release ships with the fix and a full `scripts/overture/download.py` run on WSL2 completes, we can unpin from `duckdb==1.4.1` and revert the per-part download to a single-query DuckDB scan. Added 2026-04-17.
 - [ ] Auto-check taxonomy changes whenever we switch to a new Overture Maps version (detect new/removed L0/L1/L2 categories vs. `taxonomy_crosswalk_overture_maps.csv` and flag gaps). Added 2026-04-16.
 - [ ] Watch for Overture L0/L1 → flat `basic_category` migration (~June 2026). Crosswalk CSV + `assign_overture_shared_label` will need updating. See [docs/taxonomy-setup.md](docs/taxonomy-setup.md).

.claude/docs/data-versioning.md

@@ -10,11 +10,11 @@ versions:
   model_output: "20260416_by_leisure"  # fitted model artifacts (suffix indicates variant)
   snapshot_osm: "20260416"             # OSM current-state snapshot
   snapshot_overture: "20260417"        # Overture snapshot
-  aws: "20260416"                      # S3 prefix for uploaded data
   conflation: "20260417"               # conflated output
+  source_coop: "2026-04-17-v0"         # Source Cooperative upload folder (see below)
 ```
 
-Each key corresponds to a `directories.<key>` entry in `config.yaml` with `versioned: true`.
+Each key corresponds to a `directories.<key>` entry in `config.yaml` with `versioned: true`, except `source_coop`, which only names the remote folder.
 
 ## Path resolution
 
@@ -36,8 +36,9 @@ config.get_file_path("osm_data", "osm_versions")
 
 ## Naming conventions
 
-- **Dates**: `YYYYMMDD`, e.g., `20260416`.
+- **Local dates**: `YYYYMMDD`, e.g., `20260416`.
 - **Model variants**: `{date}_by_{group_key}` (e.g., `20260416_by_leisure`, `20260416_by_amenity`) or `{date}_constant`. See [skills/iterate-model-types](../skills/iterate-model-types/SKILL.md).
+- **Source Coop folder**: `YYYY-MM-DD-v<IDX>`. Default `v0` for every fresh publish; only bump `v1`, `v2`, … if republishing under the same calendar date (e.g. a hot-fix). The Source Coop upload script writes the per-version README into this folder, so the suffix must be unique per upload round.
 - **Independent cadences**: snapshot versions can (and should) differ across sources — Overture releases ~monthly. Don't force them to match.
 
 ## External references (hand-update when bumping)
@@ -46,16 +47,15 @@ Version strings appear in these places outside `versions:` — grep before any c
 
 | File | References |
 |---|---|
-| [config.yaml](../../config.yaml) | `upload.latest_url_osm`, `upload.latest_url_conflation` (full URL with date) |
-| [site/src/constants.js](../../site/src/constants.js) | `OSM_S3_BASE`, `CONFLATED_S3_BASE` |
-| [site/public/about.html](../../site/public/about.html) | Hardcoded S3 browse links in the data-access section |
+| [site/src/constants.js](../../site/src/constants.js) | `OSM_PMTILES_URL`, `CONFLATED_PMTILES_URL` (full `data.source.coop` URLs) |
+| [site/public/about.html](../../site/public/about.html) | Hardcoded Source Coop browse links in the data-access section |
 | `osm_data.apply_model.model_stub` (config.yaml) | Which model family [scripts/osm_snapshot/apply_model.py](../../scripts/osm_snapshot/apply_model.py) ingests |
 
-[skills/update-site](../skills/update-site/SKILL.md) covers the frontend side; [skills/conflate-snapshots](../skills/conflate-snapshots/SKILL.md) covers the upload + config side.
+[skills/update-site](../skills/update-site/SKILL.md) covers the frontend side; [skills/conflate-snapshots](../skills/conflate-snapshots/SKILL.md) covers the publish + config side.
 
 ## Workflow
 
-1. Bump the relevant `versions.*` keys before running a pipeline.
+1. Bump the relevant `versions.*` keys before running a pipeline. For a public release, also bump `versions.source_coop` to the new `YYYY-MM-DD-v0`.
 2. Run the pipeline — outputs land in the versioned directory.
-3. After upload, update `upload.latest_url_*` and the frontend references.
-4. Old versions stay on disk / S3 — delete manually when confident nothing references them.
+3. After publishing, update the frontend references in `site/src/constants.js` and `site/public/about.html`.
+4. Old local versions stay on disk — delete manually when confident nothing references them. Old Source Coop folders stay published indefinitely and serve as an immutable archive.

.claude/docs/partitioning-strategy.md

@@ -96,7 +96,7 @@ GROUP BY 1, 2;
 
 ## When NOT to use this layout
 
-The geohash-partitioned layout is a better fit for **small-bbox, many-types-at-once** queries — which is exactly the web-map viewport case we moved away from. If the S3 / map-viewport path comes back, the helpers are still in place: see `add_geohash_columns` and `write_partitioned_dataset` in [src/openpois/io/geohash_partition.py](../../src/openpois/io/geohash_partition.py), and the original S3 upload step in [scripts/conflation/upload_to_s3.py](../../scripts/conflation/upload_to_s3.py). Swap the function calls in the two `format_for_upload.py` scripts back to the geohash variants.
+The geohash-partitioned layout is a better fit for **small-bbox, many-types-at-once** queries — which is exactly the web-map viewport case we moved away from. If the map-viewport path comes back, the helpers are still in place: see `add_geohash_columns` and `write_partitioned_dataset` in [src/openpois/io/geohash_partition.py](../../src/openpois/io/geohash_partition.py), and the Source Cooperative publish step in [scripts/publish/upload_to_source_coop.py](../../scripts/publish/upload_to_source_coop.py). Swap the function calls in the two `format_for_upload.py` scripts back to the geohash variants.
 
 ## Maintenance
 
@@ -107,7 +107,7 @@ python -u scripts/osm_snapshot/format_for_upload.py   2>&1 | tee ~/data/openpois
 python -u scripts/conflation/format_for_upload.py     2>&1 | tee ~/data/openpois/logs/conflated_repartition_<version>.log
 ```
 
-Each script deletes the existing partitioned directory at its versioned path and rewrites it. Geohash precision is controlled by `upload.geohash_precision_sort` in [config.yaml](../../config.yaml) (currently 6 ≈ 0.6 × 1.2 km).
+Each script deletes the existing partitioned directory at its versioned path and rewrites it. Geohash precision is controlled by `publish.geohash_precision_sort` in [config.yaml](../../config.yaml) (currently 6 ≈ 0.6 × 1.2 km).
 
 **Where the code lives:**
 
@@ -116,4 +116,4 @@ Each script deletes the existing partitioned directory at its versioned path and
 - [scripts/osm_snapshot/format_for_upload.py](../../scripts/osm_snapshot/format_for_upload.py) — OSM partitioning entry point.
 - [tests/test_geohash_partition.py](../../tests/test_geohash_partition.py) — unit tests + a DuckDB Hive-decode round-trip.
 
-**S3 upload is currently disabled** — `scripts/conflation/upload_to_s3.py` is not run as part of this flow. The `upload.latest_url_*` / `upload.s3_*` keys in `config.yaml` are stale but harmless; clean them up in a later pass if the frontend integration is formally retired.
+The Source Cooperative publish flow ([scripts/publish/upload_to_source_coop.py](../../scripts/publish/upload_to_source_coop.py)) uploads these same partitioned trees to `<version>/osm-parquet/` and `<version>/conflated-parquet/`. PMTiles generation remains downstream of partitioning.

.claude/skills/conflate-snapshots/SKILL.md

@@ -1,21 +1,33 @@
 ---
 name: conflate-snapshots
-description: Use when the user wants to match rated OSM POIs with Overture POIs into a unified dataset, partition it for web consumption, and push to S3. Triggers: "run conflation", "push new conflated data to S3", "bump conflation version", "reconflate with new parameters", "re-upload the partitioned parquet".
+description: Use when the user wants to match rated OSM POIs with Overture POIs into a unified dataset, partition it for web consumption, and push to Source Cooperative. Triggers: "run conflation", "publish new data", "push new conflated data to Source Cooperative", "bump conflation version", "reconflate with new parameters", "re-upload the partitioned parquet".
 ---
 
-# Conflate snapshots + publish to S3
+# Conflate snapshots + publish to Source Cooperative
 
-Taxonomy-aware matching between rated OSM and Overture, then partition and upload for web consumption.
+Taxonomy-aware matching between rated OSM and Overture, then partition and
+upload for web consumption.
 
 ## Prerequisites
 
 - Rated OSM snapshot (`osm_snapshot_rated.parquet`) at `versions.snapshot_osm` — produced by [skills/full-data-pull](../full-data-pull/SKILL.md) step 3.
 - Overture snapshot (`overture_snapshot.parquet`) at `versions.snapshot_overture`.
-- AWS credentials configured for the `openpois-public` bucket (region `us-west-2`).
+- **Fresh Source Cooperative temp credentials** in `.env.json` at the repo root. Tokens expire in ~1 hour.
+
+> ⚠️ **Credential refresh check.** Source Cooperative uses short-lived AWS
+> credentials (`aws_access_key_id` starting with `ASIA…`). **Before** running
+> step 7, ask the user to regenerate them at
+> <https://source.coop/repositories/henryspatialanalysis/openpois/manage>
+> and overwrite `~/repos/openpois/.env.json`. The upload script will warn if
+> the file looks stale, but it cannot tell whether the token itself has
+> expired until it actually fails.
 
 ## Steps
 
-1. **Bump `versions.conflation` and `versions.aws`** in `config.yaml`. These typically track together since the upload uses the conflation output.
+1. **Bump `versions.conflation` and `versions.source_coop`** in `config.yaml`.
+   `versions.source_coop` is the remote folder name — `YYYY-MM-DD-vN`. Keep
+   `vN` at `v0`; only bump `v1`, `v2`, … if you re-upload under the same
+   calendar date.
 
 2. **Review conflation parameters** (`config.yaml` → `conflation`):
    - `min_match_score` (default 0.50) — raises/lowers match acceptance
@@ -52,33 +64,34 @@ Taxonomy-aware matching between rated OSM and Overture, then partition and uploa
      python -u scripts/conflation/prepare_pmtiles.py \
        2>&1 | tee ~/data/openpois/logs/pmtiles_conflated_<version>.log
      ```
-     Properties and zoom range are configured under `upload.pmtiles` in
+     Properties and zoom range are configured under `publish.pmtiles` in
      `config.yaml`.
 
-7. **Upload to S3** — pushes partitioned parquet AND the matching `.pmtiles`
-   (single file at `…/<version>/<name>.pmtiles`) under `versions.aws`.
-   ```bash
-   python scripts/osm_snapshot/upload_to_s3.py     # OSM parquet + pmtiles
-   python scripts/conflation/upload_to_s3.py       # conflated parquet + pmtiles
-   ```
-   To upload only the PMTiles (e.g., after regenerating tiles without touching
-   the parquet), use:
+7. **Publish to Source Cooperative** — uploads OSM + conflated parquet,
+   both PMTiles, and a freshly-rendered per-version `README.md` under
+   `<repo>/<versions.source_coop>/`. Confirm the credential check above first.
    ```bash
-   python scripts/osm_snapshot/upload_pmtiles_to_s3.py [--s3-version YYYYMMDD]
-   python scripts/conflation/upload_pmtiles_to_s3.py  [--s3-version YYYYMMDD]
-   ```
+   # Preview everything that would be uploaded:
+   python scripts/publish/upload_to_source_coop.py --dry-run
+
+   # Real upload (datasets + version README):
+   python -u scripts/publish/upload_to_source_coop.py \
+     2>&1 | tee ~/data/openpois/logs/publish_<version>.log
 
-8. **Update latest-URL pointers** in `config.yaml`:
-   ```yaml
-   upload:
-     latest_url_osm:       "https://openpois-public.s3.us-west-2.amazonaws.com/snapshots/osm/YYYYMMDD/osm_snapshot_partitioned/"
-     latest_url_conflation: "https://openpois-public.s3.us-west-2.amazonaws.com/snapshots/conflated/YYYYMMDD/conflated_partitioned/"
+   # If the top-level README or LICENSE changed:
+   python scripts/publish/upload_to_source_coop.py --update-top-level
    ```
+   `--skip-osm-parquet`, `--skip-conflated-parquet`, and `--skip-pmtiles`
+   allow partial reuploads (e.g. after regenerating PMTiles alone).
 
 ## Verification
 
 - `summary_by_label.csv` match rates should resemble the prior run; large drifts mean a parameter or crosswalk regression.
 - `match_diagnostics.parquet` for per-pair forensics on surprising matches.
+- Spot-check the version landing page at
+  <https://source.coop/henryspatialanalysis/openpois/> and confirm the
+  per-version `README.md` renders with the expected OSM date, Overture
+  release, and row counts.
 - See [skills/verify-pipeline-run](../verify-pipeline-run/SKILL.md).
 
 ## Next
@@ -90,4 +103,6 @@ Taxonomy-aware matching between rated OSM and Overture, then partition and uploa
 - Matching: [src/openpois/conflation/match.py](../../../src/openpois/conflation/match.py)
 - Merging: [src/openpois/conflation/merge.py](../../../src/openpois/conflation/merge.py)
 - Taxonomy assignment: [src/openpois/conflation/taxonomy.py](../../../src/openpois/conflation/taxonomy.py)
+- Publish orchestration: [scripts/publish/upload_to_source_coop.py](../../../scripts/publish/upload_to_source_coop.py)
+- Source Coop S3 adapter: [src/openpois/io/source_coop.py](../../../src/openpois/io/source_coop.py)
 - Conflation algorithm docs: [scripts/conflation/README.md](../../../scripts/conflation/README.md)

.claude/skills/full-data-pull/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: full-data-pull
-description: Use when the user wants to refresh the independent POI snapshots (OSM, Overture) and rate the OSM snapshot for conflation. Triggers: "refresh all snapshots", "do a new data pull", "download new OSM/Overture", "monthly data refresh", "pull the latest POI data". Does NOT include conflation or S3 upload — those live in conflate-snapshots.
+description: Use when the user wants to refresh the independent POI snapshots (OSM, Overture) and rate the OSM snapshot for conflation. Triggers: "refresh all snapshots", "do a new data pull", "download new OSM/Overture", "monthly data refresh", "pull the latest POI data". Does NOT include conflation or Source Cooperative publishing — those live in conflate-snapshots.
 ---
 
 # Full data pull