diff --git a/dev-docs/specs/2026-05-27-nldas-icechunk-example-design.md b/dev-docs/specs/2026-05-27-nldas-icechunk-example-design.md new file mode 100644 index 00000000..3bba6773 --- /dev/null +++ b/dev-docs/specs/2026-05-27-nldas-icechunk-example-design.md @@ -0,0 +1,200 @@ +# NLDAS icechunk example — Design + +- **Date:** 2026-05-27 +- **Issues:** [#569](https://github.com/developmentseed/deck.gl-raster/issues/569) +- **Status:** Proposed +- **Related:** [`2026-04-17-ecmwf-zarr-animation-design.md`](2026-04-17-ecmwf-zarr-animation-design.md) — the closest precedent example; this design deliberately strips its UI down to a static frame + +## Problem + +We want an example proving that an [icechunk](https://icechunk.io) repository can +be read in the browser and rendered with `deck.gl-raster`, using +[`icechunk-js`](https://github.com/EarthyScience/icechunk-js) as a +zarrita-compatible store. icechunk is increasingly used to publish analysis-ready +and *virtual* Zarr (chunks that reference byte ranges inside other cloud objects), +and we currently have no example demonstrating that path. + +The target dataset, suggested in the issue, is +[NLDAS-3](https://github.com/virtual-zarr/nldas-icechunk) — NASA's North American +Land Data Assimilation System v3 daily forcing data, virtualized into an icechunk +repo: + +- **Repo:** `https://nasa-waterinsight.s3.us-west-2.amazonaws.com/virtual-zarr-store/NLDAS-3-icechunk` +- **Access:** public / anonymous. +- **Virtual chunks:** reference the original NLDAS-3 files under + `s3://nasa-waterinsight/NLDAS3/forcing/daily/` — **the same bucket**. + +Feasibility facts verified during design: + +- The entire `nasa-waterinsight` bucket returns `Access-Control-Allow-Origin: *` + for `GET`/`HEAD`, so both the icechunk repo metadata and the virtual source + objects are reachable from a browser origin. +- `icechunk-js@0.4.0` declares `zarrita ^0.5 || ^0.6 || ^0.7` as a peer + dependency, matching this repo's `zarrita@0.7.3`. +- The repo's `config.yaml` declares exactly one virtual chunk container: + ```yaml + virtual_chunk_containers: + s3://nasa-waterinsight/NLDAS3/forcing/daily/: + url_prefix: s3://nasa-waterinsight/NLDAS3/forcing/daily/ + store: !s3 { region: us-west-2, anonymous: false, ... } + ``` + The container's underlying objects are nonetheless publicly readable over + HTTPS (verified), so the browser can fetch them unsigned despite + `anonymous: false` in the stored config. + +The working Python recipe (provided by @kylebarron) confirms what the browser +code must replicate: + +```python +storage = icechunk.s3_storage(bucket='nasa-waterinsight', + prefix="virtual-zarr-store/NLDAS-3-icechunk", region="us-west-2", anonymous=True) +virtual_credentials = icechunk.containers_credentials({ + "s3://nasa-waterinsight/NLDAS3/forcing/daily/": icechunk.s3_anonymous_credentials()}) +repo = icechunk.Repository.open(storage=storage, + authorize_virtual_chunk_access=virtual_credentials) +session = repo.readonly_session('main') +ds = xr.open_zarr(session.store, consolidated=False, zarr_version=3, chunks={}) +``` + +Two requirements fall out of this: **region** `us-west-2` (in the browser this is +just encoded in the HTTPS host — `icechunk-js` has no region param), and +**explicit authorization of the virtual chunk container** before chunk reads +work. + +## Goals + +- A new `examples/nldas-icechunk` that renders a single Tair (air temperature) + timestep over North America with a temperature colormap. +- Exercise the real integration seam: `IcechunkStore` → `zarrita.open` → + `ZarrLayer` (the existing `@developmentseed/deck.gl-zarr` layer). +- Reuse this repo's idiomatic GPU colormap pipeline (rescale + colormap on the + GPU via `deck.gl-raster`'s gpu-modules), as in the ECMWF example. +- `pnpm typecheck` passes and `pnpm dev` shows the rendered frame. + +## Non-goals + +- **No animation and no UI controls.** This is a minimal "plumbing" demo — one + pinned timestep, a fixed colormap, and a fixed rescale range. A time slider and + colormap/rescale controls are obvious follow-ups but explicitly out of scope. +- **No GeoZarr support work.** The NLDAS virtual store is not GeoZarr-compliant; + we hard-code synthetic spatial attrs (the established ECMWF approach) rather + than teaching anything to parse NLDAS's native layout. +- **No icechunk version-control UI** (snapshots/tags/branches). The example + checks out the default `main` branch only. + +## Design + +### Directory layout + +Mirrors `examples/dynamical-zarr-ecmwf`, minus the control-panel UI: + +``` +examples/nldas-icechunk/ + index.html + package.json + tsconfig.json + vite.config.ts + README.md + src/ + main.tsx React entry + App.tsx open store -> open Tair -> build ZarrLayer -> map + overlay + nldas/ + metadata.ts REPO_URL, VARIABLE, TIME_INDEX, rescale range, + colormap choice, hard-coded NLDAS_GEOZARR_ATTRS + get-tile-data.ts zarr.get(arr, sliceSpec) -> Float32 tile {data,width,height,byteLength} + render-tile.ts GPU rescale + colormap pipeline (trimmed ECMWF render-tile) +``` + +No Chakra UI — there are no controls. Dependencies: `icechunk-js@^0.4.0`, +`zarrita`, the workspace deck.gl-raster / deck.gl-zarr packages, the deck.gl +peer packages, `maplibre-gl`, `react-map-gl`, and the shared +`deck.gl-raster-examples-shared` (`DeckGlOverlay`). + +### Data flow + +1. **Open (once, on mount).** Build the store with the virtual chunk container + authorized. The `virtualChunkContainers` option lives on `ReadSession.open` + (not on `IcechunkStore.open(url, …)`), so we construct the session explicitly + and wrap it: + ```ts + const storage = new HttpStorage(REPO_URL); // region encoded in the HTTPS host + // VCC name (from config.yaml) -> public HTTPS prefix for the source objects + const virtualChunkContainers = new Map([[ + "s3://nasa-waterinsight/NLDAS3/forcing/daily/", + "https://nasa-waterinsight.s3.us-west-2.amazonaws.com/NLDAS3/forcing/daily/", + ]]); + // exact entry point (Repository.checkoutBranch vs ReadSession.open with an + // explicit snapshot id) is pinned at the smoke-test step below + const session = await /* main-branch read session */; + const store = await IcechunkStore.open(session); // withRangeCoalescing is fn-typed; omit + const arr = await zarr.open(store.resolve("/Tair"), { kind: "array" }); + ``` + Use `zarr.open.v3` if auto-detection misfires — icechunk is always Zarr v3. + Assert the dtype is float; throw with a clear message otherwise (ECMWF + precedent). No custom `FetchClient` is needed: the source objects are public, + so the default client's unsigned `fetch` succeeds. +2. **Colormap.** Fetch the shipped `colormaps.png`, `decodeColormapSprite` to + `ImageData`, and `createColormapTexture` once the luma `Device` arrives via + the overlay's `onDeviceInitialized` callback. Identical to ECMWF. +3. **Layer.** Construct + `ZarrLayer({ node: arr, metadata: NLDAS_GEOZARR_ATTRS, selection: { : TIME_INDEX }, getTileData, renderTile, maxRequests })`. + The layer tiles the single-level array; `getTileData` pulls one chunk per tile + via `zarr.get(arr, options.sliceSpec)`; `renderTile` applies the fixed rescale + + colormap on the GPU. +4. **Map.** A `maplibre-gl` basemap centered on North America (≈ lon −98, lat 39, + zoom ≈ 3.5) with the shared `DeckGlOverlay` (interleaved). + +### Spatial metadata (the one non-obvious piece) + +NLDAS-3 virtual-zarr is not GeoZarr-compliant, so — exactly like ECMWF's +`ECMWF_GEOZARR_ATTRS` — we hard-code a synthetic attrs object and pass it as +`ZarrLayer`'s `metadata` prop: + +```ts +{ + "spatial:dimensions": [, ], // e.g. ["lat", "lon"] + "spatial:transform": [a, b, c, d, e, f], // @developmentseed/affine convention + "spatial:shape": [height, width], + "proj:code": "EPSG:4326", +} +``` + +The **exact grid values** (spatial dim names, the non-spatial time dim name, +shape, origin, pixel size, and crucially the latitude **row direction**) are not +guessed — they are read from the store once during implementation by logging +`arr.shape` and reading the 1-D lat/lon coordinate arrays, then frozen into +`metadata.ts` with a comment recording where they came from. Tair's units +(likely Kelvin) are confirmed the same way and drive a fixed rescale range plus a +temperature colormap choice. + +### Error handling + +- Async open effect uses a `cancelled` flag (ECMWF precedent) to avoid setting + state after unmount. +- Non-float dtype throws with a descriptive message. +- Layer is only constructed once both the opened array and the colormap texture + are ready. + +## Risks / smoke-test before building UI + +- **Virtual chunk resolution — the load-bearing risk.** Mechanism is understood + (the `virtualChunkContainers` map above), but two unknowns remain until run: the + exact session entry point that accepts `virtualChunkContainers` + (`Repository.checkoutBranch` vs `ReadSession.open` with an explicit snapshot id), + and whether the manifest stores chunk locations such that the container's + `url_prefix` matches and rewrites cleanly to the HTTPS prefix. **First + implementation step is a throwaway script/console call** that opens the store + and `zarr.get`s a single Tair chunk, confirming bytes return, *before* any + layer/UI work. If it fails, revisit the approach here rather than pressing on. +- **Zarr version.** icechunk is Zarr v3 with no consolidated metadata; the store + serves metadata directly. Prefer `zarr.open.v3` if plain `open` mis-detects. + +## Testing + +Examples in this repo are demos without unit tests; verification is: + +- `pnpm typecheck` in the example. +- Manual `pnpm dev` confirming Tair renders over North America with the colormap. + +A fresh worktree first needs submodule init + `pnpm install` + `pnpm build` so the +workspace packages resolve. diff --git a/examples/nldas-icechunk/README.md b/examples/nldas-icechunk/README.md new file mode 100644 index 00000000..1acbbc04 --- /dev/null +++ b/examples/nldas-icechunk/README.md @@ -0,0 +1,30 @@ +# NLDAS-3 icechunk Example + +Renders a single timestep of NLDAS-3 near-surface air temperature from a public +[icechunk](https://icechunk.io) repository, read in the browser via +[`icechunk-js`](https://github.com/EarthyScience/icechunk-js) + zarrita and +displayed with `@developmentseed/deck.gl-zarr`'s `ZarrLayer`. + +The store is a *virtual* Zarr: its chunks reference NLDAS-3 source objects in +the same public `nasa-waterinsight` S3 bucket, authorized through a +`virtualChunkContainers` map. + +## Setup + +1. Install dependencies from the repository root: + ```bash + pnpm install + ``` +2. Build the packages: + ```bash + pnpm build + ``` +3. Run the development server: + ```bash + cd examples/nldas-icechunk + pnpm dev + ``` +4. Open your browser to http://localhost:3000 + +`src/nldas/metadata.ts` hard-codes the grid (origin, pixel size, shape, units, +fill) because the virtual store is not GeoZarr-compliant. diff --git a/examples/nldas-icechunk/index.html b/examples/nldas-icechunk/index.html new file mode 100644 index 00000000..91e90dfb --- /dev/null +++ b/examples/nldas-icechunk/index.html @@ -0,0 +1,22 @@ + + + + + + NLDAS icechunk Example + + + +
+ + + diff --git a/examples/nldas-icechunk/package.json b/examples/nldas-icechunk/package.json new file mode 100644 index 00000000..9f5b2898 --- /dev/null +++ b/examples/nldas-icechunk/package.json @@ -0,0 +1,38 @@ +{ + "name": "deck.gl-nldas-icechunk", + "private": true, + "type": "module", + "scripts": { + "dev": "vite", + "build": "vite build", + "preview": "vite preview", + "typecheck": "tsc --noEmit", + "publish": "pnpm build && gh-pages -d dist -b gh-pages -e examples/nldas-icechunk" + }, + "dependencies": { + "@chakra-ui/react": "^3.34.0", + "@deck.gl/core": "^9.3.2", + "@deck.gl/geo-layers": "^9.3.2", + "@deck.gl/layers": "^9.3.2", + "@deck.gl/mapbox": "^9.3.2", + "@developmentseed/deck.gl-raster": "workspace:^", + "@developmentseed/deck.gl-zarr": "workspace:^", + "@emotion/react": "^11.14.0", + "@luma.gl/core": "^9.3.2", + "@luma.gl/shadertools": "^9.3.2", + "deck.gl-raster-examples-shared": "workspace:*", + "icechunk-js": "^0.4.0", + "maplibre-gl": "^5.24.0", + "react": "^19.2.5", + "react-dom": "^19.2.5", + "react-map-gl": "^8.1.1", + "zarrita": "^0.7.3" + }, + "devDependencies": { + "@types/react": "^19.2.14", + "@types/react-dom": "^19.2.3", + "@vitejs/plugin-react": "^6.0.1", + "gh-pages": "^6.3.0", + "vite": "^8.0.0" + } +} diff --git a/examples/nldas-icechunk/src/App.tsx b/examples/nldas-icechunk/src/App.tsx new file mode 100644 index 00000000..b6994f0f --- /dev/null +++ b/examples/nldas-icechunk/src/App.tsx @@ -0,0 +1,242 @@ +import { NativeSelect, Text } from "@chakra-ui/react"; +import { + COLORMAP_INDEX, + createColormapTexture, + decodeColormapSprite, +} from "@developmentseed/deck.gl-raster/gpu-modules"; +import colormapsPngUrl from "@developmentseed/deck.gl-raster/gpu-modules/colormaps.png"; +import { ZarrLayer } from "@developmentseed/deck.gl-zarr"; +import type { Device, Texture } from "@luma.gl/core"; +import { + ColormapPreview, + ControlPanel, + DeckGlOverlay, + ExternalLink, + Field, + RangeSlider, +} from "deck.gl-raster-examples-shared"; +import "maplibre-gl/dist/maplibre-gl.css"; +import { useEffect, useMemo, useRef, useState } from "react"; +import type { MapRef } from "react-map-gl/maplibre"; +import { Map as MaplibreMap } from "react-map-gl/maplibre"; +import type * as zarr from "zarrita"; +import type { ColormapId } from "./nldas/colormap-choices.js"; +import { + COLORMAP_CHOICES, + DEFAULT_COLORMAP_ID, +} from "./nldas/colormap-choices.js"; +import type { NldasTileData } from "./nldas/get-tile-data.js"; +import { getTileData } from "./nldas/get-tile-data.js"; +import { + NLDAS_GEOZARR_ATTRS, + RESCALE_MAX, + RESCALE_MIN, + RESCALE_SLIDER_MAX, + RESCALE_SLIDER_MIN, + RESCALE_SLIDER_STEP, + TIME_DIM, + TIME_INDEX, +} from "./nldas/metadata.js"; +import { makeRenderTile } from "./nldas/render-tile.js"; +import type { SurfaceTempSource } from "./nldas/store.js"; +import { openSurfaceTemp } from "./nldas/store.js"; + +// Keyless CARTO basemap; light background reads well under a data overlay. +const BASEMAP_STYLE = + "https://basemaps.cartocdn.com/gl/positron-gl-style/style.json"; + +// [[Min longitude, min latitude], [max longitude, max latitude]] +const DATA_BOUNDS: [[number, number], [number, number]] = [ + [-180, -20], + [-20, 80], +]; + +/** Total number of rows in the shipped colormap sprite. */ +const COLORMAP_ROW_COUNT = Object.keys(COLORMAP_INDEX).length; + +/** + * Convert a Kelvin value to an integer °C for display. The slider operates in + * Kelvin (the data's native unit, which the rescale shader expects); only the + * label is shown in the friendlier °C, with Kelvin in parentheses. + */ +const kelvinToCelsius = (k: number) => Math.round(k - 273.15); + +export default function App() { + const mapRef = useRef(null); + const [source, setSource] = useState(null); + const [device, setDevice] = useState(null); + const [colormapImage, setColormapImage] = useState(null); + const [colormapTexture, setColormapTexture] = useState(null); + const [colormapId, setColormapId] = useState(DEFAULT_COLORMAP_ID); + const [rescaleMin, setRescaleMin] = useState(RESCALE_MIN); + const [rescaleMax, setRescaleMax] = useState(RESCALE_MAX); + + const colormapChoice = useMemo( + () => + COLORMAP_CHOICES.find((c) => c.id === colormapId) ?? COLORMAP_CHOICES[0], + [colormapId], + ); + + // Open the icechunk store + surface temperature array once. + useEffect(() => { + let cancelled = false; + (async () => { + const opened = await openSurfaceTemp(); + if (!cancelled) { + setSource(opened); + } + })(); + return () => { + cancelled = true; + }; + }, []); + + // Decode the shipped colormap sprite once (no GPU device needed). + useEffect(() => { + let cancelled = false; + (async () => { + const resp = await fetch(colormapsPngUrl); + const bytes = await resp.arrayBuffer(); + const image = await decodeColormapSprite(bytes); + if (!cancelled) { + setColormapImage(image); + } + })(); + return () => { + cancelled = true; + }; + }, []); + + // Upload the colormap sprite once both the Device and the decoded image exist. + useEffect(() => { + if (!device || !colormapImage) { + return; + } + setColormapTexture(createColormapTexture(device, colormapImage)); + }, [device, colormapImage]); + + const layers = + source && colormapTexture + ? [ + new ZarrLayer({ + id: "nldas-surface-temp", + node: source.array, + metadata: NLDAS_GEOZARR_ATTRS, + selection: { [TIME_DIM]: TIME_INDEX }, + getTileData, + renderTile: makeRenderTile({ + colormapTexture, + colormapIndex: colormapChoice.colormapIndex, + colormapReversed: colormapChoice.reversed, + noDataValue: source.noDataValue, + rescaleMin, + rescaleMax, + }), + // Re-run renderTile on cached tiles when the colormap or rescale + // range changes. + updateTriggers: { + renderTile: [colormapId, rescaleMin, rescaleMax], + }, + }), + ] + : []; + + return ( +
+ + + + + + Reads{" "} + + NLDAS-3 + {" "} + temperature data directly from a public{" "} + Icechunk{" "} + repository. + + + Zarr data is rendered by a{" "} + + ZarrLayer + + , using + + icechunk-js + {" "} + and{" "} + + zarrita + {" "} + to fetch data directly in the browser, without a server in between. + The store is{" "} + + virtualized + + : the underlying raw data is technically read from NetCDF files. + + + + + setColormapId(e.target.value as ColormapId)} + > + {COLORMAP_CHOICES.map((c) => ( + + ))} + + + + + + + + Rescale range: {kelvinToCelsius(rescaleMin)}°C ({rescaleMin} K) –{" "} + {kelvinToCelsius(rescaleMax)}°C ({rescaleMax} K) + + } + > + { + if (nextMin !== rescaleMin) { + setRescaleMin(nextMin); + } + if (nextMax !== rescaleMax) { + setRescaleMax(nextMax); + } + }} + thumbLabels={["Rescale min (K)", "Rescale max (K)"]} + /> + + +
+ ); +} diff --git a/examples/nldas-icechunk/src/main.tsx b/examples/nldas-icechunk/src/main.tsx new file mode 100644 index 00000000..171b36eb --- /dev/null +++ b/examples/nldas-icechunk/src/main.tsx @@ -0,0 +1,12 @@ +import { ExampleProvider } from "deck.gl-raster-examples-shared"; +import { StrictMode } from "react"; +import { createRoot } from "react-dom/client"; +import App from "./App.js"; + +createRoot(document.getElementById("root")!).render( + + + + + , +); diff --git a/examples/nldas-icechunk/src/nldas/colormap-choices.ts b/examples/nldas-icechunk/src/nldas/colormap-choices.ts new file mode 100644 index 00000000..d72b42ae --- /dev/null +++ b/examples/nldas-icechunk/src/nldas/colormap-choices.ts @@ -0,0 +1,59 @@ +import { COLORMAP_INDEX } from "@developmentseed/deck.gl-raster/gpu-modules"; + +/** Shape constraint for a single colormap choice. */ +type ColormapChoiceShape = { + /** Stable identifier used as the select-option value. */ + id: string; + /** Human-readable label shown in the dropdown. */ + label: string; + /** Layer index into the colormap sprite. */ + colormapIndex: number; + /** Whether to sample the colormap in reverse. */ + reversed: boolean; +}; + +/** + * Shortlist of colormap options appropriate for near-surface air temperature. + * Order drives the dropdown order in the control panel. + */ +export const COLORMAP_CHOICES = [ + { + id: "thermal", + label: "thermal (cmocean sequential)", + colormapIndex: COLORMAP_INDEX.thermal, + reversed: false, + }, + { + id: "coolwarm", + label: "coolwarm (diverging)", + colormapIndex: COLORMAP_INDEX.coolwarm, + reversed: false, + }, + { + id: "rdbu_r", + label: "RdBu reversed (blue→red)", + colormapIndex: COLORMAP_INDEX.rdbu, + reversed: true, + }, + { + id: "balance", + label: "balance (cmocean diverging)", + colormapIndex: COLORMAP_INDEX.balance, + reversed: false, + }, + { + id: "turbo", + label: "turbo", + colormapIndex: COLORMAP_INDEX.turbo, + reversed: false, + }, +] as const satisfies readonly ColormapChoiceShape[]; + +/** Union of valid colormap choice ids (e.g. `"coolwarm" | "rdbu_r" | ...`). */ +export type ColormapId = (typeof COLORMAP_CHOICES)[number]["id"]; + +/** An entry from {@link COLORMAP_CHOICES}. */ +export type ColormapChoice = (typeof COLORMAP_CHOICES)[number]; + +/** Default colormap on first load. */ +export const DEFAULT_COLORMAP_ID: ColormapId = COLORMAP_CHOICES[0].id; diff --git a/examples/nldas-icechunk/src/nldas/get-tile-data.ts b/examples/nldas-icechunk/src/nldas/get-tile-data.ts new file mode 100644 index 00000000..1313c4a3 --- /dev/null +++ b/examples/nldas-icechunk/src/nldas/get-tile-data.ts @@ -0,0 +1,58 @@ +import type { MinimalTileData } from "@developmentseed/deck.gl-raster"; +import type { GetTileDataOptions } from "@developmentseed/deck.gl-zarr"; +import type { Texture } from "@luma.gl/core"; +import * as zarr from "zarrita"; + +/** Per-tile data: one spatial chunk uploaded as an r32float texture. */ +export type NldasTileData = MinimalTileData & { + /** r32float 2D texture holding the tile's temperature values. */ + texture: Texture; +}; + +/** + * Slice one spatial chunk of the near-surface air temperature array (time + * pinned by the layer's selection) and upload it as a single-channel float + * texture. Fill pixels keep their sentinel value (the store's `missing_value`) + * and are discarded on the GPU by `FilterNoDataVal`. + */ +export async function getTileData( + arr: zarr.Array<"float32", zarr.Readable>, + options: GetTileDataOptions, +): Promise { + const { device, sliceSpec, width, height, signal } = options; + + const chunk = await zarr.get(arr, sliceSpec, { signal }); + if (chunk.shape.length !== 2) { + throw new Error( + `Expected 2D sliced chunk (y, x), got shape [${chunk.shape.join(", ")}]`, + ); + } + if (chunk.shape[0] !== height || chunk.shape[1] !== width) { + throw new Error( + `Tile shape mismatch: expected [${height}, ${width}], got ` + + `[${chunk.shape.join(", ")}]`, + ); + } + + const data = chunk.data as Float32Array; + + const texture = device.createTexture({ + format: "r32float", + width, + height, + data, + sampler: { + minFilter: "nearest", + magFilter: "nearest", + addressModeU: "clamp-to-edge", + addressModeV: "clamp-to-edge", + }, + }); + + return { + texture, + width, + height, + byteLength: data.byteLength, + }; +} diff --git a/examples/nldas-icechunk/src/nldas/metadata.ts b/examples/nldas-icechunk/src/nldas/metadata.ts new file mode 100644 index 00000000..46f61bbe --- /dev/null +++ b/examples/nldas-icechunk/src/nldas/metadata.ts @@ -0,0 +1,57 @@ +/** Public NLDAS-3 icechunk repo (anonymous, CORS-enabled). */ +export const REPO_URL = + "https://nasa-waterinsight.s3.us-west-2.amazonaws.com/virtual-zarr-store/NLDAS-3-icechunk"; + +/** Branch to read. */ +export const BRANCH = "main"; + +/** + * Path to the near-surface air temperature array within the store. `Tair` is + * the array's name in the NLDAS-3 store, so the literal path can't change. + */ +export const SURFACE_TEMP_PATH = "/Tair"; + +/** + * Virtual chunk container map: the container name declared in the repo's + * `config.yaml` → the public HTTPS prefix the browser can fetch. The + * temperature chunks are virtual references into the original NLDAS-3 source + * objects in the same `nasa-waterinsight` bucket. + */ +export const VIRTUAL_CHUNK_CONTAINERS = new Map([ + [ + "s3://nasa-waterinsight/NLDAS3/forcing/daily/", + "https://nasa-waterinsight.s3.us-west-2.amazonaws.com/NLDAS3/forcing/daily/", + ], +]); + +/** Name of the non-spatial dimension (array dims are ["time", "lat", "lon"]). */ +export const TIME_DIM = "time"; + +/** + * Which timestep to render (single static frame). Time is "days since + * 2001-01-01" with one-day increments, so index 3482 = 2010-07-16 — a summer + * day with good thermal contrast over North America. + */ +export const TIME_INDEX = 3482; + +/** + * Initial rescale range in Kelvin (near-surface air temperature `vmin`/`vmax` + * ≈ 228–304 K). Adjustable at runtime via the rescale slider. + */ +export const RESCALE_MIN = 260; +export const RESCALE_MAX = 305; + +/** Bounds + step for the rescale slider (Kelvin). */ +export const RESCALE_SLIDER_MIN = 240; +export const RESCALE_SLIDER_MAX = 320; +export const RESCALE_SLIDER_STEP = 1; + +/** + * Synthetic GeoZarr-compliant attrs (the virtual store is not GeoZarr). + */ +export const NLDAS_GEOZARR_ATTRS = { + "spatial:dimensions": ["lat", "lon"], + "spatial:transform": [0.01, 0, -169, 0, 0.01, 7], + "spatial:shape": [6500, 11700], + "proj:code": "EPSG:4326", +} as const; diff --git a/examples/nldas-icechunk/src/nldas/render-tile.ts b/examples/nldas-icechunk/src/nldas/render-tile.ts new file mode 100644 index 00000000..036fffcd --- /dev/null +++ b/examples/nldas-icechunk/src/nldas/render-tile.ts @@ -0,0 +1,55 @@ +import type { RenderTileResult } from "@developmentseed/deck.gl-raster"; +import { + Colormap, + CreateTexture, + FilterNoDataVal, + LinearRescale, +} from "@developmentseed/deck.gl-raster/gpu-modules"; +import type { Texture } from "@luma.gl/core"; +import type { NldasTileData } from "./get-tile-data.js"; + +/** Arguments for {@link makeRenderTile}. */ +export type MakeRenderTileArgs = { + /** Colormap sprite texture (2d-array; shared across tiles). */ + colormapTexture: Texture; + /** Which layer of the sprite to sample. */ + colormapIndex: number; + /** Whether to reverse the colormap. */ + colormapReversed: boolean; + /** Sentinel value to discard (set by getTileData on fill/NaN pixels). */ + noDataValue: number; + /** Minimum value for rescale (variable units). */ + rescaleMin: number; + /** Maximum value for rescale. */ + rescaleMax: number; +}; + +/** + * Build a renderTile callback that samples the tile's float texture, discards + * nodata, rescales to [0, 1], and applies a colormap — all on the GPU. + */ +export function makeRenderTile(args: MakeRenderTileArgs) { + const { + colormapTexture, + colormapIndex, + colormapReversed, + noDataValue, + rescaleMin, + rescaleMax, + } = args; + return function renderTile(data: NldasTileData): RenderTileResult { + return { + renderPipeline: [ + // r32float sample → color = vec4(value, 0, 0, 1) + { module: CreateTexture, props: { textureName: data.texture } }, + // Discard fills on the raw value before rescale clamps it. + { module: FilterNoDataVal, props: { value: noDataValue } }, + { module: LinearRescale, props: { rescaleMin, rescaleMax } }, + { + module: Colormap, + props: { colormapTexture, colormapIndex, reversed: colormapReversed }, + }, + ], + }; + }; +} diff --git a/examples/nldas-icechunk/src/nldas/store.ts b/examples/nldas-icechunk/src/nldas/store.ts new file mode 100644 index 00000000..351440e0 --- /dev/null +++ b/examples/nldas-icechunk/src/nldas/store.ts @@ -0,0 +1,63 @@ +import { + HttpStorage, + IcechunkStore, + ReadSession, + Repository, +} from "icechunk-js"; +import * as zarr from "zarrita"; +import { + BRANCH, + REPO_URL, + SURFACE_TEMP_PATH, + VIRTUAL_CHUNK_CONTAINERS, +} from "./metadata.js"; + +/** The opened temperature array plus the fill sentinel read from its attrs. */ +export interface SurfaceTempSource { + /** The near-surface air temperature array. */ + array: zarr.Array<"float32", zarr.Readable>; + /** Fill value, read from the array's `missing_value` attribute. */ + noDataValue: number; +} + +/** + * Open the NLDAS-3 near-surface air temperature array from the public icechunk + * repo, with the virtual chunk container authorized so chunk reads resolve to + * public HTTPS objects. + * + * The container map is only accepted by `ReadSession.open`, so we resolve the + * branch snapshot id first, then open a session that carries it. + */ +export async function openSurfaceTemp(): Promise { + const storage = new HttpStorage(REPO_URL); + // NLDAS-3 is a v1 icechunk repo + const repo = await Repository.open({ storage, formatVersion: "v1" }); + + const branchSession = await repo.checkoutBranch(BRANCH); + const snapshotId = branchSession.getSnapshotId(); + + const session = await ReadSession.open(storage, snapshotId, { + virtualChunkContainers: VIRTUAL_CHUNK_CONTAINERS, + }); + const store = await IcechunkStore.open(session); + + const node = await zarr.open(store.resolve(SURFACE_TEMP_PATH), { + kind: "array", + }); + if (!node.is("float32")) { + throw new Error( + `Expected ${SURFACE_TEMP_PATH} to be float32, got ${node.dtype}`, + ); + } + + // Read the fill sentinel from the array's attrs rather than hard-coding it. + const missingValue = node.attrs.missing_value; + if (typeof missingValue !== "number") { + throw new Error( + `Expected ${SURFACE_TEMP_PATH} to have a numeric "missing_value" attr, ` + + `got ${typeof missingValue}`, + ); + } + + return { array: node, noDataValue: missingValue }; +} diff --git a/examples/nldas-icechunk/src/vite-env.d.ts b/examples/nldas-icechunk/src/vite-env.d.ts new file mode 100644 index 00000000..11f02fe2 --- /dev/null +++ b/examples/nldas-icechunk/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/examples/nldas-icechunk/tsconfig.json b/examples/nldas-icechunk/tsconfig.json new file mode 100644 index 00000000..4bd6962d --- /dev/null +++ b/examples/nldas-icechunk/tsconfig.json @@ -0,0 +1,4 @@ +{ + "extends": "../tsconfig.base.json", + "include": ["src"] +} diff --git a/examples/nldas-icechunk/vite.config.ts b/examples/nldas-icechunk/vite.config.ts new file mode 100644 index 00000000..27711d63 --- /dev/null +++ b/examples/nldas-icechunk/vite.config.ts @@ -0,0 +1,11 @@ +import react from "@vitejs/plugin-react"; +import { defineConfig } from "vite"; + +export default defineConfig({ + plugins: [react()], + base: "/deck.gl-raster/examples/nldas-icechunk/", + worker: { format: "es" }, + server: { + port: 3000, + }, +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 69f9e273..f55447e8 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -544,6 +544,76 @@ importers: specifier: ^8.0.0 version: 8.0.10(@types/node@25.6.0)(esbuild@0.27.2)(jiti@1.21.7)(terser@5.46.2)(tsx@4.21.0)(yaml@2.8.3) + examples/nldas-icechunk: + dependencies: + '@chakra-ui/react': + specifier: ^3.34.0 + version: 3.35.0(@emotion/react@11.14.0(@types/react@19.2.14)(react@19.2.5))(react-dom@19.2.5(react@19.2.5))(react@19.2.5) + '@deck.gl/core': + specifier: ^9.3.1 + version: 9.3.2 + '@deck.gl/geo-layers': + specifier: ^9.3.1 + version: 9.3.2(@deck.gl/core@9.3.2)(@deck.gl/extensions@9.3.2(@deck.gl/core@9.3.2)(@luma.gl/core@9.3.3)(@luma.gl/engine@9.3.3(@luma.gl/core@9.3.3)(@luma.gl/shadertools@9.3.3(@luma.gl/core@9.3.3))))(@deck.gl/layers@9.3.2(@deck.gl/core@9.3.2)(@loaders.gl/core@4.4.1)(@luma.gl/core@9.3.3)(@luma.gl/engine@9.3.3(@luma.gl/core@9.3.3)(@luma.gl/shadertools@9.3.3(@luma.gl/core@9.3.3))))(@deck.gl/mesh-layers@9.3.2(@deck.gl/core@9.3.2)(@loaders.gl/core@4.4.1)(@luma.gl/core@9.3.3)(@luma.gl/engine@9.3.3(@luma.gl/core@9.3.3)(@luma.gl/shadertools@9.3.3(@luma.gl/core@9.3.3)))(@luma.gl/gltf@9.3.3(@luma.gl/core@9.3.3)(@luma.gl/engine@9.3.3(@luma.gl/core@9.3.3)(@luma.gl/shadertools@9.3.3(@luma.gl/core@9.3.3)))(@luma.gl/shadertools@9.3.3(@luma.gl/core@9.3.3)))(@luma.gl/shadertools@9.3.3(@luma.gl/core@9.3.3)))(@loaders.gl/core@4.4.1)(@luma.gl/core@9.3.3)(@luma.gl/engine@9.3.3(@luma.gl/core@9.3.3)(@luma.gl/shadertools@9.3.3(@luma.gl/core@9.3.3))) + '@deck.gl/layers': + specifier: ^9.3.1 + version: 9.3.2(@deck.gl/core@9.3.2)(@loaders.gl/core@4.4.1)(@luma.gl/core@9.3.3)(@luma.gl/engine@9.3.3(@luma.gl/core@9.3.3)(@luma.gl/shadertools@9.3.3(@luma.gl/core@9.3.3))) + '@deck.gl/mapbox': + specifier: ^9.3.1 + version: 9.3.2(@deck.gl/core@9.3.2)(@luma.gl/core@9.3.3)(@math.gl/web-mercator@4.1.0) + '@developmentseed/deck.gl-raster': + specifier: workspace:^ + version: link:../../packages/deck.gl-raster + '@developmentseed/deck.gl-zarr': + specifier: workspace:^ + version: link:../../packages/deck.gl-zarr + '@emotion/react': + specifier: ^11.14.0 + version: 11.14.0(@types/react@19.2.14)(react@19.2.5) + '@luma.gl/core': + specifier: ^9.3.3 + version: 9.3.3 + '@luma.gl/shadertools': + specifier: ^9.3.3 + version: 9.3.3(@luma.gl/core@9.3.3) + deck.gl-raster-examples-shared: + specifier: workspace:* + version: link:../_shared + icechunk-js: + specifier: ^0.4.0 + version: 0.4.0(zarrita@0.7.3) + maplibre-gl: + specifier: ^5.24.0 + version: 5.24.0 + react: + specifier: ^19.2.5 + version: 19.2.5 + react-dom: + specifier: ^19.2.5 + version: 19.2.5(react@19.2.5) + react-map-gl: + specifier: ^8.1.1 + version: 8.1.1(maplibre-gl@5.24.0)(react-dom@19.2.5(react@19.2.5))(react@19.2.5) + zarrita: + specifier: ^0.7.3 + version: 0.7.3 + devDependencies: + '@types/react': + specifier: ^19.2.14 + version: 19.2.14 + '@types/react-dom': + specifier: ^19.2.3 + version: 19.2.3(@types/react@19.2.14) + '@vitejs/plugin-react': + specifier: ^6.0.1 + version: 6.0.1(vite@8.0.10(@types/node@25.6.0)(esbuild@0.27.2)(jiti@1.21.7)(terser@5.46.2)(tsx@4.21.0)(yaml@2.8.3)) + gh-pages: + specifier: ^6.3.0 + version: 6.3.0 + vite: + specifier: ^8.0.0 + version: 8.0.10(@types/node@25.6.0)(esbuild@0.27.2)(jiti@1.21.7)(terser@5.46.2)(tsx@4.21.0)(yaml@2.8.3) + examples/sentinel-2: dependencies: '@chakra-ui/react': @@ -3274,6 +3344,10 @@ packages: '@module-federation/webpack-bundler-runtime@0.22.0': resolution: {integrity: sha512-aM8gCqXu+/4wBmJtVeMeeMN5guw3chf+2i6HajKtQv7SJfxV/f4IyNQJUeUQu9HfiAZHjqtMV5Lvq/Lvh8LdyA==} + '@msgpack/msgpack@3.1.3': + resolution: {integrity: sha512-47XIizs9XZXvuJgoaJUIE2lFoID8ugvc0jzSHP+Ptfk8nTbnR8g788wv48N03Kx0UkAv559HWRQ3yzOgzlRNUA==} + engines: {node: '>= 18'} + '@napi-rs/wasm-runtime@0.2.12': resolution: {integrity: sha512-ZVWUcfwY4E/yPitQJl481FjFo3K22D6qF0DuFH6Y/nbnE11GY5uguDxZMGXPQ8WQ0128MXQD7TnfHyK4oWoIJQ==} @@ -6268,6 +6342,14 @@ packages: resolution: {integrity: sha512-Y93lCzHYgGWdrJ66yIktxiaGULYc6oGiABxhcO5AufBeOyoIdZF7bIfLaOrbM0iGIOXQQgxxRrFEnb+Y6w1n4A==} engines: {node: '>=10.18'} + icechunk-js@0.4.0: + resolution: {integrity: sha512-au8ALQqpd60LAVHgeQzf0c1V7xSfvH89JFPCyjdZoy5qcQ9Uutu3h1SQBKXIa5iccf6VPjfFOYxlavoFFS8Rfg==} + peerDependencies: + zarrita: ^0.5.0 || ^0.6.0 || ^0.7.0 + peerDependenciesMeta: + zarrita: + optional: true + iconv-lite@0.4.24: resolution: {integrity: sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==} engines: {node: '>=0.10.0'} @@ -12747,6 +12829,8 @@ snapshots: '@module-federation/runtime': 0.22.0 '@module-federation/sdk': 0.22.0 + '@msgpack/msgpack@3.1.3': {} + '@napi-rs/wasm-runtime@0.2.12': dependencies: '@emnapi/core': 1.10.0 @@ -16226,6 +16310,14 @@ snapshots: hyperdyperid@1.2.0: {} + icechunk-js@0.4.0(zarrita@0.7.3): + dependencies: + '@msgpack/msgpack': 3.1.3 + flatbuffers: 25.9.23 + fzstd: 0.1.1 + optionalDependencies: + zarrita: 0.7.3 + iconv-lite@0.4.24: dependencies: safer-buffer: 2.1.2