Promote 0.4.0 beta: VizX notebook rendering, pmtiles_gbx reader, sample downloaders, XYZ rescale, Helios series#48
Merged
Merged
Conversation
added 30 commits
June 27, 2026 20:20
Approved brainstorming output for the 0.4.0 "twofer": three sequenced sub-projects — an Overture STAC-driven distributed data-source module (gbx.sample.overture), net-new VizX PMTiles/COG viewers + a reusable PMTiles inspector, and a config_nb-spined 3-notebook series (SF AOI, solar site-selection narrative) demonstrating MVT/XYZ/COG -> PMTiles. Co-authored-by: Isaac
Add a metadata Delta table output target to OvertureClient.download: one row per asset with a source/path column holding the Volumes path, UPSERTed via MERGE keyed by (theme, type, source). Matches the GeoBrix reader source/path convention so the table can drive distributed reads and a repair()-style re-download, mirroring the StacClient/eo-series pattern. Co-authored-by: Isaac
Serverless is the target, not Classic. Distribution now models the h3-rasterize fan-out: distributed Spark read of Overture parquet over the cloud path with bbox predicate pushdown writing the AOI subset in parallel (performant default), asset-level repartition(N,col) HTTP download as fallback, finer-than-file row-group/byte-range fan-out to avoid the file-count bottleneck, and the hard Serverless constraints (repartition by column only; no spark.conf/cache/persist/.rdd; TEMP TABLE = Serverless/18.1+). Co-authored-by: Isaac
The plot_pmtiles vector static fallback and plot_cog render over a contextily basemap, consistent with the existing plot_static. contextily is already a [vizx] dependency, so no new dep and the static path stays visually consistent with the rest of VizX. Co-authored-by: Isaac
Standing practice for all tiling work: every light/Serverless gain is assessed for applicability to similar light-tier and same+similar heavy-tier functions, and captured as a reusable pattern in two homes — a version-controlled docs/superpowers/performance/ corpus and a thin pointer agent memory that links to it — kept distinct from the user-facing performance.mdx. Co-authored-by: Isaac
Per decision, no separate precursor: the docs/superpowers/performance/ corpus README + pattern files are created as each tiling gain is validated, with paired pointer memories added alongside; each SP plan carries a "capture validated gains" step. Co-authored-by: Isaac
Three sequenced TDD implementation plans from the approved Helios spec. SP1 (12 tasks): OvertureClient STAC-driven distributed download + metadata Delta table. SP2 (10 tasks): plot_pmtiles/plot_cog + pmtiles_info. SP3 (8 tasks): config_nb-spined 3-notebook SF series. Corrected SP2 fallback semantics to interactive-by-default with auto-degrade when the base64 embed exceeds max_embed_mb (max_embed_mb=0 forces static); the draft had inverted it (and was internally inconsistent with its own interactive-default test). Co-authored-by: Isaac
Add INTERACTIVE_PLOTS=False (GitHub-renderable static default) to config_nb, routing all notebook plot calls through toggle-aware helpers (static plot_pmtiles max_embed_mb=0 / plot_static / plot_cog; interactive plot_pmtiles / plot_interactive). Inject Databricks-native ST/H3 where it fits the solar narrative — NB01 roof area + H3 density, NB03 H3 slope aggregation — as the product on-ramp; NB02 left raster-only. Native names + geometry encoding flagged for confirmation and capability-gated. Co-authored-by: Isaac
Keep the two-step install (force-reinstall --no-deps for fresh wheel bytes, then extras so deps resolve) with an inline rationale so it is not collapsed to one line; matches the eo-series / h3-rasterize pattern and is resilient to same-session wheel rebuilds. Add the [overture] extra (pystac, introduced by SP1) so OvertureClient imports cleanly. Co-authored-by: Isaac
Add the target-environment run: a databricks-sdk runner + gbx:run:helios- series command that submits the series to Serverless as sequential notebook job tasks (nb01->nb02->nb03), each %run-ing config_nb (job tasks don't share Python state) with depends_on threading the Volume/Delta artifacts; emits the run_page_url. Complements Task 7's Docker cell-by-cell validation. Co-authored-by: Isaac
Foundation for the Overture discovery module: axis-aligned bbox intersection, bbox normalization, and the canonical theme->type map + expansion (None => all themes). Pure driver-side helpers, no Spark/network, unit-tested in isolation. Co-authored-by: Isaac
traverse_catalog walks a pystac.Catalog-shaped root (children -> items), filters items by AOI bbox intersection, restricts to requested (theme,type) pairs, and emits one row per GeoParquet asset (theme/type/href/asset_bbox). Tested offline with an importable fake catalog (no network). Co-authored-by: Isaac
resolve_release picks the latest release from catalog metadata (sorted overture:releases extra field, or catalog id) or passes an explicit pin through unchanged; raises ValueError when no release is discoverable. cli_discover shells out to the optional overturemaps CLI per (theme, type) pair and returns None when absent, signalling the caller to fall back to static STAC traversal. Runner is injectable for offline testing. Test file imports consolidated to top (isort gate). Co-authored-by: Isaac
OvertureClient mirrors StacClient's shape and injection seams. discover() resolves the release, expands themes (None=>all), tries the CLI fast-path (production only, treats None or empty result as fallback) then static traversal, and returns a typed DataFrame (theme/type/href/asset_bbox/release). Re-exported from sample/__init__. test_sample_bundle updated to reflect new __all__. Co-authored-by: Isaac
_open_catalog is only reached when self._catalog_opener is None (routing happens in _opener()), so the inner guard is unreachable and misleading. Simplify to always open the real catalog. Co-authored-by: Isaac
Performant default download path: distributed-read each asset's GeoParquet with a bbox-struct predicate pushdown (AOI rows only), write the subset per (theme,type) to the Volume, emit metadata. Repartition by column (never number-only) so the plan stays distributed on Serverless; verified via getNumPartitions > 1. Adds _meta_schema/_meta_dataframe/_META_COLS module helpers reused by the Task 6 fallback path. Co-authored-by: Isaac
Empty _meta_dataframe() already flows through the same withColumn/select chain as the non-empty path, so _META_COLS matches; this test locks that contract so Tasks 7/8 union logic can rely on schema consistency. Co-authored-by: Isaac
Asset-level HTTP-href download for when no cloud read is available: fan out by href (column-hash repartition, Serverless-safe), temp-file then sequential copy (Volume-safe), retry + parquet-open validate + idempotent skip. _get_fn injectable; tested offline with a fake fetcher serving real parquet bytes. Co-authored-by: Isaac
Public download() routes cloud-readable assets to the distributed read+AOI-rewrite path and http hrefs to the whole-file fallback, returns the pinned metadata schema (source aliased as path), and is idempotent on re-run. Partitions default to asset count. Co-authored-by: Isaac
The existing test_download_routes_cloud_to_distributed uses an absolute tmp_path href (starts with /), which the cloud predicate treats as a FUSE Volume path and routes to _download_distributed -- so download()'s routing to the HTTP fallback branch was untested, and the test comment claiming otherwise was misleading. Add test_download_routes_http_to_fallback: an http:// href fails the cloud predicate, so download() routes through _download_fallback (offline, via the injected fake _get_fn). Asserts the _META_COLS 10-column schema in order and is_out_file_valid. Correct the misleading comment and assert against _META_COLS instead of an inline list. Fix download()'s docstring: it claimed table= UPSERTs to a Delta table, which does not exist yet (Task 8) -- a caller passing table= today gets a silent no-op. Document table= as reserved/no-op and spell out the cloud predicate (/-prefix Volume + s3a:// schemes, http(s):// -> fallback). Co-authored-by: Isaac
table=<name> persists/UPSERTs the per-asset metadata to a Delta table, idempotent MERGE keyed by (theme,type,source) (mirrors StacClient.repair). First run creates the table; re-runs update volatile cols so the catalog stays queryable and re-runnable without accumulating duplicate rows. Co-authored-by: Isaac
read() loads downloaded GeoParquet back into Spark from a Volume directory, a metadata Delta table name, or a metadata DataFrame (source/path column), with an optional bbox-struct AOI filter. A path-guard prevents spark.catalog.tableExists() ParseException on filesystem paths. The metadata table branch can directly drive distributed reads. Co-authored-by: Isaac
Fix three review findings on OvertureClient.read(): - Add test_read_from_table_name (Delta-gated, skips offline) to cover the table-name branch that had no test; verifies real row retrieval. - Add test_read_empty_metadata_raises (offline) to cover the IndexError regression in _read_paths([]); guard now raises a clear ValueError. - Hoist `from pyspark.sql import functions as F` from inside test_read_from_metadata_dataframe to the module-level import block (isort/E402 compliance). Co-authored-by: Isaac
Module-level convenience that discovers an AOI's Overture assets and downloads them in one call (themes=None => all), passing the AOI bbox pushdown and optional metadata Delta table through. Re-exported from sample/__init__. Co-authored-by: Isaac
Add pystac>=1.9,<2 to requirements-pyrx-ci.in (explicit pin; already present transitively via rio-tiler, now marked as a direct dep so supply-chain intent is clear) and requirements-dev-container.in (same reason, alongside pystac-client). Regenerate both hash-pinned .txt locks so the via annotation correctly shows -r <in-file>. Add the [overture] optional-dependencies extra to pyproject.toml (pystac>=1.9,<2), mirroring the [stac] extra pattern. Register test/sample in both required locations: 1. _LIGHT_TEST_DIRS in test/conftest.py (so the heavy CI phase skips it and avoids ModuleNotFoundError at collection time). 2. pytest dir list in .github/actions/pyrx_build/action.yml (so the light CI phase actually runs it). Co-authored-by: Isaac
Three pre-existing flake8 failures (gating CI lint): - test/ds/test_scratch_isolation.py: remove unused pytest import (F401) - test/ds/test_vector_reader_contract.py: drop unused _ShapefileReader import (F401) - src/databricks/labs/gbx/ds/vector.py: suppress C901 on _write_local_osgeo_gdb (complexity 26 > max 20; genuine refactor is out of scope for this wiring task, noqa is the least-surprise fix) SP1 accumulated minor fixes: - sample/overture.py _download_fallback: drop dead _validate/_max_tries local aliases; UDF closure captures validate/max_tries from the enclosing function scope directly - test/sample/test_overture.py: hoist module-body `import databricks.labs.gbx.sample.overture as ov` to module level with # noqa: E402 (consistent with existing importorskip pattern) - test/sample/test_sample_bundle.py: add download_overture_aoi to the expected __all__ set (was missing; sample/__init__.py exports it) Docker black drift (black 26.3.1 in container): - test/ds/test_vector_filename.py, test_vector_writer.py: reformatted Co-authored-by: Isaac
Start the performance engineering corpus at docs/superpowers/performance/. First entry: serverless-aoi-ingestion-strategy — the pattern validated while building the Overture data source: distributed read-in-place with bbox-struct predicate pushdown + column-hash repartition, with a whole-file HTTP-href fallback. Captures the applicability matrix (light AOI downloaders: yes; heavy OGR/GDAL readers: N/A) and the honest evidence status (pattern/correctness confirmed; cluster-scale speedup not yet measured, deferred to the optional cluster smoke). Distinct from the existing serverless-fanout-repartition-by-column memory (which covers repartition mechanics); this entry covers the full AOI ingestion strategy and generalizes to future sample/ downloaders. Co-authored-by: Isaac
Whole-branch review flagged the spec's download() signature omitted the bbox= AOI param the impl requires for distributed-read pushdown, and did not state the per-asset unique-subdirectory write contract that prevents sharded-type clobber (the C1 fix). Spec now matches the implemented surface. Co-authored-by: Isaac
_download_distributed was writing all assets sharing the same (theme,
type) to the same `out_dir/<theme>/<type>` path. The 2nd asset's
mode("overwrite") deleted the 1st asset's output, so only the last
shard survived; metadata source values collapsed (same path), hiding
the loss in the MERGE key.
Fix: each asset writes to `out_dir/<theme>/<type>/<token>` where
`token = sha1(href.split("?")[0])[:12]` — stable, deterministic (same
asset → same subdir across re-runs → idempotent MERGE key), unique per
asset. mode("overwrite") is now safe and idempotent per asset. read()
uses recursiveFileLookup and was already correct — no change needed.
Adds test_download_distributed_multi_asset_no_clobber: two assets
sharing (theme, type) → RED on old code (sources collapse to 1),
GREEN after fix (both shards present, 3 AOI rows, 2 distinct sources,
idempotency confirmed).
Co-authored-by: Isaac
Spark-side PMTiles read is unsupported, so a local-driver header reader is needed by the VizX viewers for vector/raster type detection and the static fallback. Lazy re-export keeps heavy-tier imports pandas-free. Note: the brief's fixture used identical PNG bytes for z=0 and z=1, causing the PMTiles writer to deduplicate them (max_zoom collapses to 0). Fixed by using distinct per-zoom payloads in the raster fixture. Co-authored-by: Isaac
added 16 commits
June 30, 2026 22:25
Two-task TDD plan mirroring NaipDownloader: discover/read + module scaffold (Task 1), download gsd-selection + download_dem_aoi (Task 2). Offline mock tests; NB-03 wiring + docs + live Serverless smoke are follow-ups. Co-authored-by: Isaac
…ter STAC Adds DemDownloader + download_dem_aoi to gbx.sample, mirroring NaipDownloader. Selection axis is gsd (resolution): finest picks minimum gsd (10 m over 30 m), int picks that exact gsd, graceful no-op when source has no gsd property. Serverless-safe, _stac_client injection seam for offline tests. 11/11 offline mock tests passing. Co-authored-by: Isaac
Expose opacity (raster-opacity / fill-opacity) and color (vector fill-color) on pmtiles_layer; _pmtiles already consumes layer.opacity/color, they just weren't surfaced. Lets a PMTiles layer be a muted underlay or styled context in a multi-layer overlay (e.g. a dimmed hillshade under a solar-score heatmap). Co-authored-by: Isaac
…/grid The interactive renderer only ever drew a flat fill — column/cmap were ignored, so an H3 heatmap came out one color. _vector_or_grid now maps layer.column through layer.cmap to a per-feature color (fill-color = ['get','_gbx_color']); an explicit color still wins. build_html renders a colormap legend (gradient bar + value range) for each data-driven layer, stripping the _gbx_legend sidecar from the serialized sources. Co-authored-by: Isaac
… bounds Two gaps behind flat grid heatmaps + globe-zoom: - _gdf_for(grid) called cells_as_gdf WITHOUT extra_cols, dropping the value column, so the data-driven fill/legend never triggered (grids rendered one flat color, no legend). Now passes extra_cols=[column]. - autofit downzoom recomputed header bounds from the kept low-zoom tiles (a z0-9 overview -> a z2 quarter-of-world), so fitBounds opened on the globe. Now preserves the source archive's data bounds (dropping high zooms doesn't change the extent). Co-authored-by: Isaac
…le+alpha A single-band raster (hillshade/DEM relief) rendered as an LA (grayscale+alpha) PNG, which MapLibre's raster layer cannot paint -> the hillshade layer showed blank. render_tile now replicates the lone band to R=G=B (mask -> alpha) so the tile is RGB(A) — a neutral grayscale relief that web renderers display. Multi-band (RGB/RGBA, e.g. NAIP) untouched. Co-authored-by: Isaac
… grayscale+alpha" This reverts commit 7570447.
…lank) build_html set the opening view via MapLibre's constructor bounds/canvas, which measure the container at construction. Inside a notebook iframe the map is built before layout, so the container is 0-sized: a constructor bounds fit collapses to minZoom (a globe when min zoom is low, e.g. a downzoomed overview), and the initial 0x0 canvas loads no tiles (a blank map when an explicit center/zoom is given). Apply resize() + fitBounds/jumpTo on load AND via a ResizeObserver, once the container has real dimensions — Databricks often sizes the output iframe after the map is constructed. Co-authored-by: Isaac
grid_layer/vector_layer gain scale= ('linear' default | 'quantile'). Linear normalization
crushes a skewed distribution into one end of the ramp (e.g. H3 roof counts where most
cells hold 1..25 of a 1..128 range all render as indistinguishable pale yellow). Quantile
colors by percentile rank, spreading the dense low range across the full ramp; the legend
gains a median tick so the non-linear mapping stays honest.
Co-authored-by: Isaac
The auto-view (no caller center/zoom) framed via MapLibre's container-measured fitBounds. In a notebook iframe the container is 0-sized at construction and unreliable even on load, so the fit collapsed to minZoom (a globe for a low-min-zoom overview) or left the map blank in Databricks displayHTML — while maps opened with an explicit center+zoom render reliably there. Derive the opening zoom from the archive extent container-independently (_fit_zoom_for_bounds) and open via jumpTo, matching the path that works. Co-authored-by: Isaac
Add the '3DEP Downloader (DEM)' sample-data page (mirrors the NAIP downloader) and
register it in the sidebar. Document the new grid_layer/vector_layer scale= option
('linear' vs 'quantile') on the multi-layer compositor page, with the skewed-distribution
rationale and an H3 heatmap example.
Co-authored-by: Isaac
The vizx/pmtiles/helios pages cited a 64 MB embed budget for notebook rendering — wrong. The default is ~3 MB (~6 MB when vizx raises the cell-output cap); Databricks caps cell output at ~10 MB (20 MB max) and displayHTML inflates the payload ~2-3x, so an archive over ~4-5 MB falls back to static or must stream from a URL. Correct every mention and point large-archive guidance at interactive_fit='downzoom' / URL mode. Also add a 'Ready-made downloaders' section to the STAC page linking the three sample downloaders (Overture, NAIP, 3DEP) that wrap StacClient / the same STAC pattern. Co-authored-by: Isaac
Correct stale claims across the Helios docs page and folder README: NB02/NB03 now use NaipDownloader/DemDownloader (not raw StacClient / 'AWS Open Data'); NB03 is online-only (remove the non-existent SRTM offline fallback); NB04 produces three shards, not four (the SW tile is open water) and has no offline fallback; the ETL tree is data/helios (not data/sf); and the plot_pmtiles embed ceiling is ~4-5 MB (not 64 MB). Link the NAIP/3DEP downloader pages. Co-authored-by: Isaac
NB02 stage -> NaipDownloader; NB03 stage -> DemDownloader + gbx_rst_boundingbox catalog accessor (was StacClient); NB04 -> three shard archives with the SW z11 parent shown as open water (no archive), matching the corrected notebook/README narrative. Regenerated helios-02/03/04 PNGs from the updated generator. Co-authored-by: Isaac
Remove the top-right red 'coming soon' banner on the intro page. Add three v0.4.0 release-note items that were shipped but unlisted: the AOI sample downloaders (Overture/NAIP/3DEP under gbx.sample), the inline PMTiles/COG viewers (plot_pmtiles/plot_cog/pmtiles_info), and the Helios distributed-tiling notebook series. Co-authored-by: Isaac
Auto-format imports/code (isort+black) and drop unused imports / a duplicate StringType import flagged by flake8, so the Python lint CI gate passes. Includes an in-container black pass on test_vector_raster_bridge.py (host black differed from the CI/Docker black). Co-authored-by: Isaac
mjohns-databricks
had a problem deploying
to
runtime
GitHub Actions
Failure
Light CI caught three failures from the pushed commits: - _pmtiles_register_js gained a required uid (multi-map isolation) but the dynamic-widget caller passes none -> TypeError. Make uid optional: key by uid_sid when given, else by sid alone (matches the bare pmtiles://<sid> URL the widget keeps). Fixes 8 test_dynamic. - test_sample_package_all asserted the pre-Dem/Naip __all__; add the new sample exports. - test_cell_output_cap assumed no IPython; block IPython.display so plot_interactive hits its no-display-channel return-html path (IPython is now in the light CI env). Co-authored-by: Isaac
Commit the executed Helios notebooks in their shippable state: toggles at defaults (FORCE_REBUILD=False, INTERACTIVE_PLOTS=False) with static PNG map outputs baked in for GitHub rendering. Covers the session's work — NB01 §3b quantile roof-density heatmap + §6 buildings PMTiles; NB02 NaipDownloader + pmtiles_gbx; NB03 DemDownloader + gtiff_gbx COG + raster_gbx catalog + solar score + hillshade; NB04 shard build/FORCE_REBUILD guard + mosaic + static shard views; config_nb markdown->print cleanup. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the
beta/0.4.0work accumulated since #47 tomain(210 files, +34.6k / −1.5k). Highlights by theme:VizX — reliable interactive rendering in Databricks notebooks
resize()+jumpTo/fitBoundson load + aResizeObserver) — no more "globe" or blank map from MapLibre's constructor fit computing against a zero-sized-at-construction container. Resolves theplot_pmtiles/ multi-layer blank-and-globe class of bugs.%set_cell_max_output_size_in_mb, raised automatically) anddisplayHTML's ~2–3× base64 inflation — so an archive over ~4–5 MB falls back cleanly to static, or drops its densest zooms (interactive_fit="downzoom").displayHTMLis resolved viadbruntimeon Serverless kernels.grid_layer/vector_layercolor per feature bycolumnthroughcmap, with ascale="quantile"option (percentile-rank) so a skewed distribution (e.g. H3 roof counts) spreads across the full ramp instead of collapsing into one end; a colormap legend (with a median tick underquantile) renders inline.pmtiles_layergainsopacity/color;emphasis="data"|"blend".plot_pmtiles(raster/vector auto-detected, base64FileSource, no tile server),plot_cog(COG over a contextily basemap),pmtiles_info. Plus multi-map JS isolation, correct rastertileSize, a populated-zoom static fallback, anddebug_mode.pmtiles_gbxreader (lightweight DataSource)source=raster— a scalable per-tile mosaic pyramid;source=archive— read tiles back from an existing.pmtiles. RGBA-from-RGB imagery fix, archive-path normalization, and the_xyz_mosaicper-tile compositing core.AOI sample downloaders (
databricks.labs.gbx.sample)NaipDownloader(NAIP) andDemDownloader(USGS 3DEP), both wrappingStacClienton Planetary Computer with window-on-read;OvertureClientCLI download hardened (--stacretries →--no-stacfallback, real stderr surfaced). Docs pages for all three, linked from the STAC page.STAC + reader AOI windowing
StacClient.download(bbox=, bbox_crs=)windowed fetch withmax_mppdecimation (bounds Serverless UDF memory);bbox/bboxCrswindow-on-read onraster_gbx/gtiff_gbx; parallel download via range-scan fan-out (Serverless AQE-coalesce-proof).Data-aware XYZ rescale
rescaleonRST_TileXYZ/RST_XYZPyramid(both tiers) for data-aware 8-bit encoding of non-uint8 rasters, resolved once per pyramid; cross-tier value-distribution parity gate.Helios notebook series + docs
mosaic.json), running on the lightweight tier / Serverless. Overview page + folder README + regenerated diagrams; release-notes end-story additions (sample downloaders, PMTiles/COG viewers, Helios);plot_pmtilesembed-ceiling doc corrections (was mis-stated as 64 MB); intro "v0.4.0 (coming soon)" box removed.CI: Python lint (isort / black / flake8) is green, and
requirements-pyrx-ciplus the light test-dir lists already cover the new modules (ds,stac,vizx,sample).This pull request and its description were written by Isaac.