From 2bb3c3c3f0b5e973f28a3a37ed7b8b963e8bfca1 Mon Sep 17 00:00:00 2001 From: Brandon Liu Date: Thu, 16 Jul 2026 13:23:57 -0400 Subject: [PATCH 1/2] add skills --- .claude/skills/analyze-tiles/SKILL.md | 123 ++++++++++++++++++ .claude/skills/build-tiles/SKILL.md | 80 ++++++++++++ .claude/skills/download-osm/SKILL.md | 148 ++++++++++++++++++++++ .claude/skills/download-overture/SKILL.md | 98 ++++++++++++++ .claude/skills/render-image/SKILL.md | 43 +++++++ .claude/skills/view-tiles/SKILL.md | 108 ++++++++++++++++ 6 files changed, 600 insertions(+) create mode 100644 .claude/skills/analyze-tiles/SKILL.md create mode 100644 .claude/skills/build-tiles/SKILL.md create mode 100644 .claude/skills/download-osm/SKILL.md create mode 100644 .claude/skills/download-overture/SKILL.md create mode 100644 .claude/skills/render-image/SKILL.md create mode 100644 .claude/skills/view-tiles/SKILL.md diff --git a/.claude/skills/analyze-tiles/SKILL.md b/.claude/skills/analyze-tiles/SKILL.md new file mode 100644 index 00000000..f59ff8d6 --- /dev/null +++ b/.claude/skills/analyze-tiles/SKILL.md @@ -0,0 +1,123 @@ +--- +name: analyze-tiles +description: > + Inspect a built .pmtiles archive — header/metadata, per-layer size breakdown, and the biggest + tiles — using pmtiles show and planetiler's TileSizeStats plus duckdb. Use when asked to + analyze/inspect/profile a tileset, find what's making tiles big, compare two archives' sizes, + or check an archive's metadata. Takes the archive name as its argument. +--- + +# Analyze tiles (what's in the archive, and what's making it big) + +Read-only inspection of `tiles/data/.pmtiles`. Run from the `tiles/` directory at the +repo root. + +**Requires two external tools** that this repo does not vendor: the `pmtiles` binary +([go-pmtiles](https://github.com/protomaps/go-pmtiles) releases) and +[`duckdb`](https://duckdb.org). Check with `command -v pmtiles duckdb` before relying on +them; if either is missing, say so and point at the install rather than improvising a +substitute. Note `tiles/Makefile` invokes `pmtiles` as `~/Downloads/pmtiles`, so a checkout +may have the binary without it being on PATH. + +Two independent tools, for two different questions. Pick by what was asked — don't run both by +reflex: + +- **"What is this archive?"** → `pmtiles show` (instant). +- **"Why is it big / which layer dominates?"** → `TileSizeStats` + duckdb (a minute on a large + archive). + +## 0. Resolve the archive (the argument) + +Required — names `tiles/data/.pmtiles` (e.g. `/analyze-tiles new-york`). List with +`ls -lh data/*.pmtiles`. If it doesn't exist, stop and offer `/build-tiles `. + +## 1. Header and metadata — `pmtiles show` + +``` +pmtiles show data/.pmtiles +``` + +Gives bounds, min/max zoom, center, tile/entry/content counts, compression, clustering, and +the planetiler provenance keys. The useful ones to actually read back: + +- `planetiler:osm:osmosisreplicationtime` — **the OSM snapshot date of the source extract.** + This is how you tell whether an archive predates a data change you're looking for; a missing + feature is often a stale extract, not a code bug. +- `addressed tiles count` vs `tile contents count` — the gap is deduplication (identical tiles + stored once). A big gap is normal and healthy, especially for sparse extracts. +- `min zoom` / `max zoom` — confirms a `--maxzoom` actually took effect. + +`vector_layers` is elided in the default output; add `--metadata` for the full JSON when the +question is about layer/attribute schema. + +Related subcommands worth knowing: `pmtiles verify ` checks archive structure (use when +a build was interrupted and corruption is suspected — the Makefile has a `clean-pmtiles` +target for exactly this), and `pmtiles tile ` dumps one tile. + +## 2. Size breakdown — TileSizeStats + duckdb + +Planetiler's stats tool runs standalone against an already-built archive: + +``` +java -cp target/*-with-deps.jar com.onthegomap.planetiler.util.TileSizeStats \ + --input=data/.pmtiles \ + --output=/tmp/-layerstats.tsv.gz +``` + +It logs the biggest tiles directly (with a demo-map link and the dominant layer per tile), +which is often the whole answer — read the `Biggest tiles (gzipped)` block before writing any +query. + +It also writes a gzipped TSV, one row per tile *per layer*, with columns: + +``` +z x y hilbert archived_tile_bytes layer layer_bytes layer_features +layer_geometries layer_attr_bytes layer_attr_keys layer_attr_values +``` + +Query it with duckdb (reads the `.gz` directly, no decompression step): + +``` +duckdb -c "select layer, sum(layer_bytes)/1024/1024 as mb, sum(layer_features) as feats + from read_csv('/tmp/-layerstats.tsv.gz', delim='\t', header=true) + group by layer order by mb desc;" +``` + +Other cuts that answer real questions: + +- **Which zoom is expensive** — `group by z` instead of `layer`. +- **Worst individual tiles** — `select z,x,y,layer,layer_bytes … order by layer_bytes desc limit 20`. +- **Attribute bloat** — compare `layer_attr_bytes` against `layer_bytes`; a high ratio means + the tags are the payload, not the geometry. Relevant when a change adds per-feature + attributes (shield text, network names) rather than geometry. + +To get layerstats at build time instead, planetiler takes `--output-layerstats`, which writes +`.layerstats.tsv.gz` alongside the tileset — cheaper than a separate pass if you know +in advance you'll want it. + +## Comparing two archives + +The common real task ("did my change bloat the tiles?") is a diff, and the repo keeps prior +builds around (`ny-shields.pmtiles`, `ny-norefs.pmtiles`, …) for this. Run TileSizeStats over +both, then join: + +``` +duckdb -c "select a.layer, sum(a.layer_bytes)/1024/1024 as before_mb, + sum(b.layer_bytes)/1024/1024 as after_mb + from read_csv('/tmp/before.tsv.gz', delim='\t', header=true) a + full join read_csv('/tmp/after.tsv.gz', delim='\t', header=true) b + on a.z=b.z and a.x=b.x and a.y=b.y and a.layer=b.layer + group by a.layer order by after_mb - before_mb desc;" +``` + +Only compare archives built from the **same extract** — a different source area or snapshot +date makes the numbers meaningless, and `pmtiles show` tells you both. + +## Notes + +- TileSizeStats decompresses and re-parses every tile, so it's minutes on a large archive and + seconds on a small one. Consider testing a query shape against a small archive + (`data/output.pmtiles`) before running it on a 500MB one. +- Report layer sizes in MB with the dominant layer named, not raw duckdb tables — the answer + is usually "pois is 40% of the archive", not a grid of numbers. +- These tools read the archive only; nothing here can corrupt a build. diff --git a/.claude/skills/build-tiles/SKILL.md b/.claude/skills/build-tiles/SKILL.md new file mode 100644 index 00000000..0458792a --- /dev/null +++ b/.claude/skills/build-tiles/SKILL.md @@ -0,0 +1,80 @@ +--- +name: build-tiles +description: > + Compile the current working-tree Java source into a fresh planetiler jar and build a + pmtiles tileset for a user-specified source area, streaming live build progress into the + chat. Use when asked to build tiles, (re)build the tileset, compile-and-run planetiler, or + build/generate a pmtiles for an area/extract. Takes the source area as its argument. +--- + +# Build tiles (compile current source, then run + watch) + +Always does three things, in order: **compile the current source**, **build the area**, and +**stream progress**. Run from the `tiles/` directory at the repo root. + +## 0. Resolve the source area (the argument) + +The area is user-defined and required — it's the skill's argument (e.g. `/build-tiles sfbay`). + +- A local extract lives at `data/sources/.osm.pbf`; pass `--area=` and planetiler + uses it directly (no download). List what's available with `ls data/sources/*.osm.pbf`. +- If no area was given, list the local extracts and ask which one. +- If the area isn't present locally but is a known Geofabrik region, add `--download` so + planetiler fetches it (slower). Default the output to `data/.pmtiles`. + +## 1. Compile the current source (always) + +Rebuild the fat jar so it reflects working-tree changes — this is the whole point of the +skill, so never skip it: + +``` +mvn -q package -DskipTests +``` + +Tests are skipped for speed; this only compiles + packages. Confirm +`target/*-with-deps.jar` exists and its mtime is fresh before continuing. If the build fails, +stop and report the compile error — do not run the old jar. + +## 2. Build the area under a progress watch + +`Monitor` is a deferred tool: load it first with `ToolSearch` query `select:Monitor`. + +Launch planetiler *as* the Monitor command so the watch ends when the build exits. Pipe +through a **single unbuffered `awk` stage** that splits planetiler's carriage-return progress +bar into lines, keeps the per-stage milestones, and `fflush()`es each so events stream live +(a `tr | grep` pipeline block-buffers and dumps everything at the end — do not use it): + +``` +Monitor( + description = "build-tiles ", + command = "cd \"$(git rev-parse --show-toplevel)/tiles\" && java -jar target/*-with-deps.jar --area= --output=data/.pmtiles --force 2>&1 | awk 'BEGIN{RS=\"[\\r\\n]\"} /INF \\[|Finished in|Exception|ERROR/{print; fflush()}'", + timeout_ms = , +) +``` + +(`git rev-parse --show-toplevel` resolves the repo root wherever the checkout lives, so the +command works regardless of the session's cwd.) + +You get one milestone line per ~10s tick — a sequence of progress snapshots, not a single +in-place bar (the chat renders each event as its own message and cannot rewrite a prior one). + +## Reading the stream + +- `[ne]` — Natural Earth prep (~13s). +- `[osm_pass1]` — reads nodes/ways/relations (fast). +- `[osm_pass2]` — the long phase; watch `ways: [ … NN% … ]` climb (e.g. 7→39→68→98%). +- `[osm_water]`, `[osm_land]`, `[landcover]`, `[sort]` — quick. +- `[archive]` — encodes/writes tiles z0…z15; `archive NNNMB` is the final size. +- `Finished in …` / `FINISHED!` — done. + +Relay progress by stage and percentage, not raw line dumps. + +## Notes + +- `--force` is required when re-running to an existing `--output`; planetiler otherwise aborts + immediately (`… already exists, use the --force argument …`). That abort exits before any + `INF [stage]` line, which is why the filter also matches `Exception|ERROR` — otherwise a + config crash looks like a silent hang. +- A full NY extract is ~2.5 min; sfbay ~1 min. Set `timeout_ms` generously above that. +- The harness re-invokes you when the build process exits, so you always get a completion + signal even between progress ticks. diff --git a/.claude/skills/download-osm/SKILL.md b/.claude/skills/download-osm/SKILL.md new file mode 100644 index 00000000..002572fe --- /dev/null +++ b/.claude/skills/download-osm/SKILL.md @@ -0,0 +1,148 @@ +--- +name: download-osm +description: > + Cut an OSM extract for a named place using the SliceOSM API and save it into + tiles/data/sources/ as a .osm.pbf ready for /build-tiles. Use when asked to download/fetch/cut + an OSM extract, get the data for a place, or prepare sources before a build. Takes the place + name as its argument. +--- + +# Download OSM (SliceOSM extract for a named place) + +Turns a **place name** into a `.osm.pbf` in `tiles/data/sources/`. Does not build anything — +`/build-tiles ` picks it up from there. + +Do **not** use planetiler's `--download` or Geofabrik for this. SliceOSM cuts an arbitrary +bbox from a live OSM Express database, so it isn't limited to Geofabrik's fixed regions and +the snapshot is current. + +API: **`https://slice.openstreetmap.us/api/`** (docs: https://github.com/SliceOSM/sliceosm-api) + +## 0. Place name → bbox (do this yourself, no geocoder) + +The place is the skill's argument (e.g. `/download-osm minneapolis`). **Derive the bounding +box from your own world knowledge — do not call a geocoding service.** + +Aim for a box that covers the place's built-up extent plus a little margin. Rough is fine and +expected; a slightly generous box costs extract time, a too-tight one silently clips the map +at the edges, so err generous. + +State the bbox you chose and roughly what it covers before submitting — it's a guess, and the +user is the one who knows if it's wrong. Ask first only when the name is genuinely ambiguous +(Portland OR vs ME; Cambridge UK vs MA) or the box is large enough to risk the node limit. + +## 1. Submit the task + +`POST /` with a JSON body. **The bbox order is `[min_lat, min_lon, max_lat, max_lon]` — latitude +first.** This is the single biggest trap here: it matches neither GeoJSON (lon,lat) nor +planetiler's `--bounds=w,s,e,n` (lon first). Swapping them either errors or silently cuts a +region on the other side of the world. + +``` +curl -s -X POST https://slice.openstreetmap.us/api/ \ + -d '{"Name":"","RegionType":"bbox","RegionData":[,,,]}' +``` + +The response is a bare **UUID** on success, or an error message — it is not JSON, so check it +looks like a UUID before polling. `Name` is just a human label; use the slug. + +For a non-rectangular cut, `"RegionType":"geojson"` takes a Polygon/MultiPolygon geometry in +`RegionData` (normal GeoJSON lon,lat order, unlike the bbox form). There's an unspecified +vertex limit. + +## 2. Poll until complete + +`GET /{uuid}` returns progress; the fields are strings: + +``` +{"Timestamp":"","CellsProg":"","CellsTotal":"","NodesProg":"","NodesTotal":"", + "ElemsProg":"","ElemsTotal":"","SizeBytes":"","Elapsed":"","Complete":""} +``` + +Poll with `Monitor` (a deferred tool: load it with `ToolSearch` query `select:Monitor`) so +progress streams and the watch ends when the job does: + +``` +Monitor( + description = "sliceosm ", + command = "for i in $(seq 1 120); do r=$(curl -s https://slice.openstreetmap.us/api/ || true); echo \"$r\" | grep -q '\"Complete\":\"\\?true' && { echo \"COMPLETE $r\"; break; }; echo \"$r\"; sleep 5; done", + timeout_ms = 600000, +) +``` + +Emit the raw progress line each tick and break on `Complete` — a loop that only prints on +success is silent through a stuck job, which is indistinguishable from a slow one. The task +URL is valid for 24 hours. + +Relay progress as nodes/elements percentages, not raw JSON dumps. + +## 3. Save with the naming convention + +Download to `data/sources/` — **never** to the repo root or `tiles/`: + +``` +curl -s -o data/sources/-.osm.pbf https://slice.openstreetmap.us/api/.osm.pbf +``` + +**Naming convention: `-.osm.pbf`**, where: + +- `` is the place, lowercased and hyphenated (`new-york`, `sf-bay`). +- `` is the **OSM snapshot date** from the task's `Timestamp` field (or the API root's + `Timestamp`) — *not* today's date, and not the download date. + +This matches the existing `new-york-260503.osm.pbf` (a 2026-05-03 snapshot). The date is the +point: extracts of the same place from different days are different data, and `pmtiles show` +on a built archive reports `planetiler:osm:osmosisreplicationtime`, so a dated filename is +what lets you line an archive up with its source. Undated names like `nyc.osm.pbf` are older +files that predate this convention — don't add more. + +The slug becomes the `/build-tiles` argument, since planetiler resolves `--area=` to +`data/sources/.osm.pbf`. So this saves as `new-york-260503.osm.pbf` and builds with +`/build-tiles new-york-260503`. + +Verify before reporting success: + +``` +ls -lh data/sources/-.osm.pbf +``` + +A 0-byte or few-hundred-byte file means the download raced ahead of completion or returned an +error body — check it's really a pbf, delete it if not. A corrupt pbf fails confusingly later, +deep inside a build. `GET /{uuid}.osm.pbf` 404s until the task completes, so an early grab +saves an error page under a `.osm.pbf` name. + +Two cheap confirmations: + +``` +file data/sources/-.osm.pbf # "OpenStreetMap Protocolbuffer Binary Format" +osmium fileinfo -e data/sources/-.osm.pbf # bbox, counts, replication timestamp +``` + +`file` needs nothing extra. `osmium` is a separate install (`osmium-tool`; `brew install +osmium-tool` on macOS) — `render-tests/generate_pmtiles.sh` depends on it too, so a full +checkout usually has it, but skip this check rather than failing the skill if it's absent. + +`osmium fileinfo -e` reports `osmosis_replication_timestamp` — the authoritative source for +the `` in the filename, and the same value `pmtiles show` will later report on any +archive built from this extract. + +**The extract's actual bbox is wider than the one you asked for.** SliceOSM keeps ways and +relations whole when they cross the boundary, so long features (highways, boundary relations) +drag geometry well past the box — a San Diego request of `-117.35,-116.85` came back spanning +to `-116.11`, over half a degree beyond. This is normal and not a bug. If you need a hard +edge, clip at *build* time with planetiler's `--bounds=` (lon-first there, unlike +here); don't try to fix it by shrinking the request. + +## Notes + +- **Node limit is 100,000,000** (`NodesLimit` at `GET /`). A bbox over it is rejected. For + scale: the New York state extract is ~56M nodes, so metro-area boxes are comfortable and a + whole-country box is not. If rejected, cut a smaller box rather than retrying. +- `GET /` also reports `QueueSize` — if a submit seems to hang, check whether the server is + backed up rather than assuming the job failed. +- `GET /{uuid}_region.json` returns the submitted geometry immediately, which is a quick way + to confirm the bbox landed where you meant before waiting on the extract. +- Check `ls data/sources/*.osm.pbf` first — if a recent extract for the place is already + there, offer it instead of re-cutting. +- SliceOSM has no fixed-region concept, so there's no `us/` path prefix and none of the + slash-in-the-name problems the Geofabrik path had. diff --git a/.claude/skills/download-overture/SKILL.md b/.claude/skills/download-overture/SKILL.md new file mode 100644 index 00000000..35037ee1 --- /dev/null +++ b/.claude/skills/download-overture/SKILL.md @@ -0,0 +1,98 @@ +--- +name: download-overture +description: > + Download Overture Maps GeoParquet data for a bounding box into tiles/data/overture/ using the + overturemaps CLI, for building tiles from the pm:overture source instead of OSM. Use when + asked to download/fetch Overture data, get Overture parquet, or build tiles from Overture. + Takes the Overture theme type as its argument. +--- + +# Download Overture (GeoParquet, for the `pm:overture` source) + +Downloads Overture GeoParquet and stops. Run from the `tiles/` directory at the repo root. + +Overture is a **parallel source to OSM, not an addition to it**. `Basemap.java` rejects +`--area` and `--overture` together — a build is one or the other. Each layer registers a +separate `pm:overture` handler (`buildings::processOverture`, `roads::processOverture`, …), so +Overture data flows through different code than the OSM path. + +## 0. Resolve the type and bbox (the arguments) + +The **type** is required — it's the skill's argument (e.g. `/download-overture building`). +Valid values, from `overturemaps download --help`: + +``` +address, bathymetry, building, building_part, division, division_area, +division_boundary, place, segment, connector, infrastructure, land, +land_cover, land_use, water +``` + +Note these are Overture's names, not this repo's layer names — roads come from `segment` +(plus `connector`), not a type called `roads`. + +A **bbox** is effectively required too: without `--bbox` the CLI pulls the type globally, +which is enormous. Ask for one if the user didn't give a region, and pass it as +`--bbox=west,south,east,north`. + +## 1. Download + +The CLI is the `overturemaps` Python package. Don't assume it's on PATH — a checkout may keep +it in a repo-local venv at `tiles/venv/`, so check both: + +``` +command -v overturemaps || ls ./venv/bin/overturemaps +``` + +Use whichever resolves (`./venv/bin/overturemaps` below; drop the prefix if it's on PATH): + +``` +./venv/bin/overturemaps download \ + --bbox= \ + -f geoparquet \ + -t \ + -o data/overture/.parquet +``` + +`-f geoparquet` is what the build consumes. The CLI also offers `geojson`/`geojsonseq`, which +are fine for eyeballing data but **cannot** be fed to `--overture`. + +Add `-r ` to pin a release; it defaults to the latest, so an unpinned download is not +reproducible over time. Pin it if the user cares about matching a previous run. + +Downloads are network-bound and can be slow — stream with `Monitor` (a deferred tool: load via +`ToolSearch` query `select:Monitor`) rather than blocking, and confirm with +`ls -lh data/overture/`. + +## 2. Build from it (what the download is for) + +Single file, or a hive-partitioned directory: + +``` +java -jar target/*-with-deps.jar --overture=data/overture/.parquet --output=data/.pmtiles --force +``` + +`Basemap.java` branches on the path's extension: + +- **Ends in `.parquet`** — that exact file is the only input, no hive partitioning. +- **Anything else** — treated as a base directory, globbed as `**/*.parquet` with hive + partitioning **on**. This is the multi-type case: download several types under + `data/overture/` and point `--overture` at the directory to build them together. + +Bounds are read automatically from the GeoParquet metadata when `--bounds` isn't given, so a +bbox-limited download builds only its own region without you restating the extent. If bounds +come out wrong, check that metadata before suspecting the build. + +## Notes + +- If neither `overturemaps` nor `./venv/bin/overturemaps` resolves, it isn't installed: + `python3 -m venv venv && ./venv/bin/pip install overturemaps` from `tiles/`. Prefer the venv + over a global install. +- Overture releases are versioned `YYYY-MM-DD.N`. Record which release was used when + reporting results; "latest" drifts and makes two runs incomparable for no visible reason. +- Overture parquet is large per unit area compared to an OSM extract. Start with a small bbox + when validating that a `processOverture` path works at all, then widen. +- Sanity-check a download without a full build, if [duckdb](https://duckdb.org) is available: + `duckdb -c "select count(*) from 'data/overture/.parquet';"` +- The `pm:overture` source layer is keyed on the parquet `type` field, and feature ids on + `id` (see the `addParquetSource` call in `Basemap.java`). A type whose rows lack those + fields won't map onto the layer handlers. diff --git a/.claude/skills/render-image/SKILL.md b/.claude/skills/render-image/SKILL.md new file mode 100644 index 00000000..1fac9ae6 --- /dev/null +++ b/.claude/skills/render-image/SKILL.md @@ -0,0 +1,43 @@ +--- +name: render-image +description: > + Render a static map image (PNG) from a local .pmtiles archive at a given location and zoom. + Use when asked to render an image, screenshot the map, or produce a static map picture. NOT + YET IMPLEMENTED — a native renderer is planned; this skill is a stub. +--- + +# Render image (STUB — not yet implemented) + +**This skill is a placeholder.** The intended implementation is a **native renderer**, not +in-browser screenshotting. Until that lands, do not improvise a substitute — say it's not +implemented yet and offer `/view-tiles `, which opens the same archive in a live map. + +## Intended interface (when implemented) + +``` +/render-image [--center=] [--zoom=] [--size=x] [--flavor=] +``` + +Reading `tiles/data/.pmtiles` and writing a PNG. + +## What already exists (context for whoever implements this) + +- **`render-tests/`** — a fixture-comparison harness (puppeteer + maplibre-gl + pixelmatch), + not a general renderer. It drives per-test `style.json` fixtures with a `metadata.test` + block (`width`, `height`, `flavor`, `lang`) plus `center`/`zoom`, serves pmtiles over + express on port 2900, and diffs against `expected.png`. Useful as a reference for how a + style + archive + camera turn into pixels; wrong shape to reuse as a one-shot CLI. +- **`app/`** — the live MapView. `vite.config.ts` serves `../tiles` for any `.pmtiles` path; + `MapView.tsx` takes `tiles=`, `flavorName=`, `lang=`, `local_sprites=` from the location + hash and passes `hash: "map"` to MapLibre, so `#map=//` aims the camera. See + `/view-tiles` for the details. +- **`styles/`** — the `@protomaps/basemaps` style package (`namedFlavor`, `layers`) that both + of the above build their style from. A native renderer needs the same style JSON. + +## Notes + +- Flavors are `light`, `dark`, `white`, `grayscale`, `black`. +- Sprites: published CDN sprites are the default; `sprites/dist` holds locally-generated ones + and is currently stale. See `/view-tiles`. +- When implementing, prefer reading the archive directly over standing up an HTTP server — + the whole point of the native path is skipping the browser. diff --git a/.claude/skills/view-tiles/SKILL.md b/.claude/skills/view-tiles/SKILL.md new file mode 100644 index 00000000..e2cc29ef --- /dev/null +++ b/.claude/skills/view-tiles/SKILL.md @@ -0,0 +1,108 @@ +--- +name: view-tiles +description: > + Run the local MapView web app in app/ and open a named .pmtiles archive from the tiles data + folder in it. Use when asked to view/inspect/look at a tileset, open a pmtiles in the map, + preview a build, or check how a change renders. Takes the archive name as its argument. +--- + +# View tiles (run the app, then open an archive in it) + +Does two things: **start the `app/` dev server** and **hand back a URL that loads the named +archive**. Run from the `app/` directory at the repo root. + +This skill does not build anything. If the archive doesn't exist yet, or the user wants it to +reflect current working-tree Java changes, that's `/build-tiles ` first. + +## 0. Resolve the archive (the argument) + +The archive is user-defined and required — it's the skill's argument (e.g. `/view-tiles +new-york`), naming `tiles/data/.pmtiles`. + +- List what's available with `ls ../tiles/data/*.pmtiles`. +- If no archive was given, list them and ask which one. +- If the name doesn't match exactly but one archive is an obvious near-match, say which one + you picked and why rather than silently substituting. +- If nothing matches, stop and offer `/build-tiles ` — don't start the server on a + path that will 404. + +## 1. Start the dev server + +The dev server is long-running and must stay up while the user looks at the map, so start it +**detached** — `Bash` with `run_in_background: true`. A foreground `npm run dev` blocks until +timeout and hangs the session. + +``` +npm run dev +``` + +Vite serves on **http://localhost:5173** by default; it silently picks 5174, 5175, … if the +port is taken, so read the actual `Local:` line out of the startup output rather than +assuming. If a server is already up on 5173 serving this app, reuse it instead of starting a +second one. + +## 2. Build the URL + +`vite.config.ts` mounts `../tiles` for any request path ending in `.pmtiles`, so +`tiles/data/.pmtiles` is served at `/data/.pmtiles`. `MapView.tsx` reads its +config from the location hash, and `tiles=` accepts that server-relative path: + +``` +http://localhost:5173/#tiles=/data/.pmtiles +``` + +Verify the archive actually serves *before* opening the browser — a range request is what the +map will make: + +``` +curl -s -o /dev/null -w "%{http_code}\n" -r 0-99 "http://localhost:5173/data/.pmtiles" +``` + +`206` is success. `404` means the path is wrong (check the name against `ls`); `200` with a +full body means the middleware didn't match. + +## 3. Open it + +Once the range request returns `206`, open the URL in the default browser — this is the +point of the skill, so don't ask first: + +``` +open "http://localhost:5173/#tiles=/data/.pmtiles" +``` + +Quote the URL: the `#` starts a shell comment unquoted, and everything after it — the whole +`tiles=` fragment — is silently dropped, so the app loads the hosted demo bucket instead of +the local build and the map looks plausible but wrong. + +Print the URL in the reply too, so the user can reopen or hand-edit the hash. + +## Useful hash parameters + +`MapView.tsx` reads these from the hash; combine with `&`. Values must not contain `=` — the +parser in `src/utils.ts` splits naively on it. + +- `tiles=/data/.pmtiles` — the archive to load. Omitted, it falls back to the hosted + `DEFAULT_TILES` demo bucket, which is *not* the local build. +- `flavorName=light|dark|white|grayscale|black` — style flavor. +- `local_sprites=true` — serve sprites from `sprites/dist` instead of the published CDN + sprites. **Off by default — do not add it unless the user asks.** `sprites/dist` is + generated and currently stale, so turning it on silently swaps in months-old icons. Revisit + when the sprite pipeline is being worked on. +- `lang=en` — label language. +- `show_boxes=true` — draw label collision boxes. +- `npm_version=` — compare against a published style version. +- `#map=//` — MapLibre's own hash; append to jump somewhere specific. + +## Notes + +- Default flavor sprites come from the published CDN, so icons reflect what's released, not + the working tree. That's the intended default for now. If someone does opt into + `local_sprites=true`, `sprites/dist` is generated and needs `cd sprites && make` (after + `cargo build --release` builds `spritegen`) to pick up a changed SVG — suspect a stale + `dist` before suspecting the style. +- The app also serves a drag-and-drop path ("Drag .pmtiles here to view") for archives outside + `tiles/`; the `tiles=` hash is the scriptable equivalent and the one to prefer. +- The dev server keeps running after the skill finishes — that's intended. Mention that it's + still up, and stop it with `pkill -f vite` when the user is done. +- Vite hot-reloads style/app edits, but **not** the .pmtiles — a rebuilt archive needs a + browser reload (hard-reload, since range responses cache). From b6407f01c847eb8955010c1b7bd546e986a84b7e Mon Sep 17 00:00:00 2001 From: Brandon Liu Date: Wed, 22 Jul 2026 14:30:46 -0400 Subject: [PATCH 2/2] update skill to use DuckDB and download-overture script [#624] --- .claude/skills/download-overture/SKILL.md | 86 ++++++++++------------- 1 file changed, 39 insertions(+), 47 deletions(-) diff --git a/.claude/skills/download-overture/SKILL.md b/.claude/skills/download-overture/SKILL.md index 35037ee1..2e59d117 100644 --- a/.claude/skills/download-overture/SKILL.md +++ b/.claude/skills/download-overture/SKILL.md @@ -1,10 +1,10 @@ --- name: download-overture description: > - Download Overture Maps GeoParquet data for a bounding box into tiles/data/overture/ using the - overturemaps CLI, for building tiles from the pm:overture source instead of OSM. Use when - asked to download/fetch Overture data, get Overture parquet, or build tiles from Overture. - Takes the Overture theme type as its argument. + Download Overture Maps GeoParquet data for a bounding box using tiles/download-overture.sh, + which queries the public S3 release with DuckDB, for building tiles from the pm:overture + source instead of OSM. Use when asked to download/fetch Overture data, get Overture parquet, + or build tiles from Overture. Takes the area (place name or bbox) as its argument. --- # Download Overture (GeoParquet, for the `pm:overture` source) @@ -16,67 +16,62 @@ Overture is a **parallel source to OSM, not an addition to it**. `Basemap.java` separate `pm:overture` handler (`buildings::processOverture`, `roads::processOverture`, …), so Overture data flows through different code than the OSM path. -## 0. Resolve the type and bbox (the arguments) +## 0. Resolve the bbox (the argument) -The **type** is required — it's the skill's argument (e.g. `/download-overture building`). -Valid values, from `overturemaps download --help`: +The skill's argument is the **area**. `download-overture.sh` takes a bbox as +`xmin,ymin,xmax,ymax` in WGS84 — if the user named a place instead, work out the bbox for it +and state the numbers you used before downloading. -``` -address, bathymetry, building, building_part, division, division_area, -division_boundary, place, segment, connector, infrastructure, land, -land_cover, land_use, water -``` - -Note these are Overture's names, not this repo's layer names — roads come from `segment` -(plus `connector`), not a type called `roads`. - -A **bbox** is effectively required too: without `--bbox` the CLI pulls the type globally, -which is enormous. Ask for one if the user didn't give a region, and pass it as -`--bbox=west,south,east,north`. +There is no per-type argument: the script pulls the `transportation`, `places`, `base`, +`buildings` and `divisions` themes in one query and writes one Parquet file. Bbox is what +bounds the download, so a loose bbox is the expensive mistake here. ## 1. Download -The CLI is the `overturemaps` Python package. Don't assume it's on PATH — a checkout may keep -it in a repo-local venv at `tiles/venv/`, so check both: - ``` -command -v overturemaps || ls ./venv/bin/overturemaps +./download-overture.sh [RELEASE] [BBOX] [OUTPUT] ``` -Use whichever resolves (`./venv/bin/overturemaps` below; drop the prefix if it's on PATH): +All three are positional and optional, but in practice pass at least the first two, since +`BBOX` can't be given without `RELEASE`: + +- `RELEASE` — Overture release, e.g. `2026-06-17.0`. Defaults to the latest, resolved by + DuckDB from `https://stac.overturemaps.org/catalog.json`. Pass one explicitly when the run + needs to be reproducible. +- `BBOX` — `xmin,ymin,xmax,ymax`. Defaults to Monaco. +- `OUTPUT` — output file path. Defaults to `data/sources/monaco.parquet`. ``` -./venv/bin/overturemaps download \ - --bbox= \ - -f geoparquet \ - -t \ - -o data/overture/.parquet +./download-overture.sh 2026-06-17.0 -122.52,37.70,-122.35,37.83 data/sources/sf.parquet ``` -`-f geoparquet` is what the build consumes. The CLI also offers `geojson`/`geojsonseq`, which -are fine for eyeballing data but **cannot** be fed to `--overture`. +`./download-overture.sh --help` prints the same usage and the current defaults. -Add `-r ` to pin a release; it defaults to the latest, so an unpinned download is not -reproducible over time. Pin it if the user cares about matching a previous run. +The script shells out to `duckdb`, which reads the release directly from +`s3://overturemaps-us-west-2/release//` with hive partitioning and filters rows on the +`bbox` struct. Requirements: + +- `duckdb` on PATH (`brew install duckdb`). No Python, no venv, no `overturemaps` package. +- Network access to both the STAC catalog (for release resolution) and the S3 bucket. If the + catalog is unreachable the script exits telling you to pass a release explicitly — that + failure is about release resolution, not the download itself. Downloads are network-bound and can be slow — stream with `Monitor` (a deferred tool: load via `ToolSearch` query `select:Monitor`) rather than blocking, and confirm with -`ls -lh data/overture/`. +`ls -lh data/sources/`. ## 2. Build from it (what the download is for) -Single file, or a hive-partitioned directory: - ``` -java -jar target/*-with-deps.jar --overture=data/overture/.parquet --output=data/.pmtiles --force +java -jar target/*-with-deps.jar --overture=data/sources/.parquet --output=data/.pmtiles --force ``` `Basemap.java` branches on the path's extension: -- **Ends in `.parquet`** — that exact file is the only input, no hive partitioning. +- **Ends in `.parquet`** — that exact file is the only input, no hive partitioning. This is + what the script produces. - **Anything else** — treated as a base directory, globbed as `**/*.parquet` with hive - partitioning **on**. This is the multi-type case: download several types under - `data/overture/` and point `--overture` at the directory to build them together. + partitioning **on**, for a manually assembled multi-file layout. Bounds are read automatically from the GeoParquet metadata when `--bounds` isn't given, so a bbox-limited download builds only its own region without you restating the extent. If bounds @@ -84,15 +79,12 @@ come out wrong, check that metadata before suspecting the build. ## Notes -- If neither `overturemaps` nor `./venv/bin/overturemaps` resolves, it isn't installed: - `python3 -m venv venv && ./venv/bin/pip install overturemaps` from `tiles/`. Prefer the venv - over a global install. - Overture releases are versioned `YYYY-MM-DD.N`. Record which release was used when reporting results; "latest" drifts and makes two runs incomparable for no visible reason. - Overture parquet is large per unit area compared to an OSM extract. Start with a small bbox when validating that a `processOverture` path works at all, then widen. -- Sanity-check a download without a full build, if [duckdb](https://duckdb.org) is available: - `duckdb -c "select count(*) from 'data/overture/.parquet';"` +- Sanity-check a download without a full build: + `duckdb -c "select theme, count(*) from 'data/sources/.parquet' group by 1;"` - The `pm:overture` source layer is keyed on the parquet `type` field, and feature ids on - `id` (see the `addParquetSource` call in `Basemap.java`). A type whose rows lack those - fields won't map onto the layer handlers. + `id` (see the `addParquetSource` call in `Basemap.java`). Rows lacking those fields won't + map onto the layer handlers.