Skip to content

Commit 2cf36ad

Browse files
kylebarronclaude
andauthored
feat: NLDAS-3 icechunk example (#577)
* docs: design spec for NLDAS icechunk example (#569) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: pin icechunk-js virtual chunk container config in NLDAS spec (#569) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: fix withRangeCoalescing usage in NLDAS spec (#569) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): scaffold example package shell Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(nldas-icechunk): add virtual-chunk smoke spike Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): hard-code grid metadata and store opener Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): fetch tile data as r32float texture Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): GPU colormap render pipeline Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): render Tair via ZarrLayer over icechunk store Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(nldas-icechunk): add README Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): add top-left intro panel Adds a ControlPanel (intro-only, no controls) describing the example, matching the other examples' info box. Re-adds Chakra deps for Text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): constrain map to the NLDAS-3 data extent Set maxBounds to lon [-169, -52], lat [7, 72] so the map can't pan or zoom out beyond where data exists. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix map bounds * chore(nldas-icechunk): drop one-off smoke spike script The grid constants are baked into metadata.ts; the spike was scaffolding. Removes scripts/smoke.ts, its README section, the tsx devDep, and the scripts tsconfig include. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(nldas-icechunk): address review feedback - Drop the CPU nodata-replacement loop: the source uses a finite -9999 sentinel (no NaN), already discarded on the GPU by FilterNoDataVal. - Remove redundant mipLevels: 1 (luma's default is already 1). - Rename Tair -> descriptive surface-temperature naming in identifiers and docstrings; keep the literal "/Tair" store path (the array's real name). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): add colormap picker and rescale slider Adds runtime controls to the intro panel (reusing the shared Field, NativeSelect, ColormapPreview, and RangeSlider), matching the other examples. A new colormap-choices list drives the picker; rescale min/max become state with updateTriggers so cached tiles re-render on change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(nldas-icechunk): label rescale range in °C (with K in parens) Slider still operates in Kelvin (the shader's unit); only the label converts, e.g. "10°C (283 K) – 40°C (313 K)". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(nldas-icechunk): use exact 0.01° grid transform The store's lat/lon coords are float32; deriving the affine by subtracting them injected precision noise (dLon ~0.00999451), drifting the east edge to -52.064 (~7 km). Use the exact intended grid: origin (-169, 7), step 0.01°. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * source is http/1.1, not http/2 * fix(nldas-icechunk): skip v2 /repo probe (declare v1 format) NLDAS-3 is a v1 icechunk repo. Without an explicit formatVersion, Repository.open auto-detects by fetching the v2 "/repo" marker, which 404s (then falls back to v1). Passing formatVersion: "v1" skips that probe. Verified: /repo requests drop 1 -> 0 per open (2 -> 0 with StrictMode's dev double-mount) while the array still opens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * concise * refactor(nldas-icechunk): read fill value from the array's missing_value attr Instead of hard-coding NODATA_VALUE = -9999, openSurfaceTemp now reads (and validates) the array's `missing_value` attribute and returns it alongside the array. The render pipeline gets the fill sentinel from the store rather than a constant. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update info box * narrower rescale range --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent a56f239 commit 2cf36ad

15 files changed

Lines changed: 944 additions & 0 deletions
Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
# NLDAS icechunk example — Design
2+
3+
- **Date:** 2026-05-27
4+
- **Issues:** [#569](https://github.com/developmentseed/deck.gl-raster/issues/569)
5+
- **Status:** Proposed
6+
- **Related:** [`2026-04-17-ecmwf-zarr-animation-design.md`](2026-04-17-ecmwf-zarr-animation-design.md) — the closest precedent example; this design deliberately strips its UI down to a static frame
7+
8+
## Problem
9+
10+
We want an example proving that an [icechunk](https://icechunk.io) repository can
11+
be read in the browser and rendered with `deck.gl-raster`, using
12+
[`icechunk-js`](https://github.com/EarthyScience/icechunk-js) as a
13+
zarrita-compatible store. icechunk is increasingly used to publish analysis-ready
14+
and *virtual* Zarr (chunks that reference byte ranges inside other cloud objects),
15+
and we currently have no example demonstrating that path.
16+
17+
The target dataset, suggested in the issue, is
18+
[NLDAS-3](https://github.com/virtual-zarr/nldas-icechunk) — NASA's North American
19+
Land Data Assimilation System v3 daily forcing data, virtualized into an icechunk
20+
repo:
21+
22+
- **Repo:** `https://nasa-waterinsight.s3.us-west-2.amazonaws.com/virtual-zarr-store/NLDAS-3-icechunk`
23+
- **Access:** public / anonymous.
24+
- **Virtual chunks:** reference the original NLDAS-3 files under
25+
`s3://nasa-waterinsight/NLDAS3/forcing/daily/`**the same bucket**.
26+
27+
Feasibility facts verified during design:
28+
29+
- The entire `nasa-waterinsight` bucket returns `Access-Control-Allow-Origin: *`
30+
for `GET`/`HEAD`, so both the icechunk repo metadata and the virtual source
31+
objects are reachable from a browser origin.
32+
- `icechunk-js@0.4.0` declares `zarrita ^0.5 || ^0.6 || ^0.7` as a peer
33+
dependency, matching this repo's `zarrita@0.7.3`.
34+
- The repo's `config.yaml` declares exactly one virtual chunk container:
35+
```yaml
36+
virtual_chunk_containers:
37+
s3://nasa-waterinsight/NLDAS3/forcing/daily/:
38+
url_prefix: s3://nasa-waterinsight/NLDAS3/forcing/daily/
39+
store: !s3 { region: us-west-2, anonymous: false, ... }
40+
```
41+
The container's underlying objects are nonetheless publicly readable over
42+
HTTPS (verified), so the browser can fetch them unsigned despite
43+
`anonymous: false` in the stored config.
44+
45+
The working Python recipe (provided by @kylebarron) confirms what the browser
46+
code must replicate:
47+
48+
```python
49+
storage = icechunk.s3_storage(bucket='nasa-waterinsight',
50+
prefix="virtual-zarr-store/NLDAS-3-icechunk", region="us-west-2", anonymous=True)
51+
virtual_credentials = icechunk.containers_credentials({
52+
"s3://nasa-waterinsight/NLDAS3/forcing/daily/": icechunk.s3_anonymous_credentials()})
53+
repo = icechunk.Repository.open(storage=storage,
54+
authorize_virtual_chunk_access=virtual_credentials)
55+
session = repo.readonly_session('main')
56+
ds = xr.open_zarr(session.store, consolidated=False, zarr_version=3, chunks={})
57+
```
58+
59+
Two requirements fall out of this: **region** `us-west-2` (in the browser this is
60+
just encoded in the HTTPS host — `icechunk-js` has no region param), and
61+
**explicit authorization of the virtual chunk container** before chunk reads
62+
work.
63+
64+
## Goals
65+
66+
- A new `examples/nldas-icechunk` that renders a single Tair (air temperature)
67+
timestep over North America with a temperature colormap.
68+
- Exercise the real integration seam: `IcechunkStore` → `zarrita.open` →
69+
`ZarrLayer` (the existing `@developmentseed/deck.gl-zarr` layer).
70+
- Reuse this repo's idiomatic GPU colormap pipeline (rescale + colormap on the
71+
GPU via `deck.gl-raster`'s gpu-modules), as in the ECMWF example.
72+
- `pnpm typecheck` passes and `pnpm dev` shows the rendered frame.
73+
74+
## Non-goals
75+
76+
- **No animation and no UI controls.** This is a minimal "plumbing" demo — one
77+
pinned timestep, a fixed colormap, and a fixed rescale range. A time slider and
78+
colormap/rescale controls are obvious follow-ups but explicitly out of scope.
79+
- **No GeoZarr support work.** The NLDAS virtual store is not GeoZarr-compliant;
80+
we hard-code synthetic spatial attrs (the established ECMWF approach) rather
81+
than teaching anything to parse NLDAS's native layout.
82+
- **No icechunk version-control UI** (snapshots/tags/branches). The example
83+
checks out the default `main` branch only.
84+
85+
## Design
86+
87+
### Directory layout
88+
89+
Mirrors `examples/dynamical-zarr-ecmwf`, minus the control-panel UI:
90+
91+
```
92+
examples/nldas-icechunk/
93+
index.html
94+
package.json
95+
tsconfig.json
96+
vite.config.ts
97+
README.md
98+
src/
99+
main.tsx React entry
100+
App.tsx open store -> open Tair -> build ZarrLayer -> map + overlay
101+
nldas/
102+
metadata.ts REPO_URL, VARIABLE, TIME_INDEX, rescale range,
103+
colormap choice, hard-coded NLDAS_GEOZARR_ATTRS
104+
get-tile-data.ts zarr.get(arr, sliceSpec) -> Float32 tile {data,width,height,byteLength}
105+
render-tile.ts GPU rescale + colormap pipeline (trimmed ECMWF render-tile)
106+
```
107+
108+
No Chakra UI — there are no controls. Dependencies: `icechunk-js@^0.4.0`,
109+
`zarrita`, the workspace deck.gl-raster / deck.gl-zarr packages, the deck.gl
110+
peer packages, `maplibre-gl`, `react-map-gl`, and the shared
111+
`deck.gl-raster-examples-shared` (`DeckGlOverlay`).
112+
113+
### Data flow
114+
115+
1. **Open (once, on mount).** Build the store with the virtual chunk container
116+
authorized. The `virtualChunkContainers` option lives on `ReadSession.open`
117+
(not on `IcechunkStore.open(url, …)`), so we construct the session explicitly
118+
and wrap it:
119+
```ts
120+
const storage = new HttpStorage(REPO_URL); // region encoded in the HTTPS host
121+
// VCC name (from config.yaml) -> public HTTPS prefix for the source objects
122+
const virtualChunkContainers = new Map([[
123+
"s3://nasa-waterinsight/NLDAS3/forcing/daily/",
124+
"https://nasa-waterinsight.s3.us-west-2.amazonaws.com/NLDAS3/forcing/daily/",
125+
]]);
126+
// exact entry point (Repository.checkoutBranch vs ReadSession.open with an
127+
// explicit snapshot id) is pinned at the smoke-test step below
128+
const session = await /* main-branch read session */;
129+
const store = await IcechunkStore.open(session); // withRangeCoalescing is fn-typed; omit
130+
const arr = await zarr.open(store.resolve("/Tair"), { kind: "array" });
131+
```
132+
Use `zarr.open.v3` if auto-detection misfires — icechunk is always Zarr v3.
133+
Assert the dtype is float; throw with a clear message otherwise (ECMWF
134+
precedent). No custom `FetchClient` is needed: the source objects are public,
135+
so the default client's unsigned `fetch` succeeds.
136+
2. **Colormap.** Fetch the shipped `colormaps.png`, `decodeColormapSprite` to
137+
`ImageData`, and `createColormapTexture` once the luma `Device` arrives via
138+
the overlay's `onDeviceInitialized` callback. Identical to ECMWF.
139+
3. **Layer.** Construct
140+
`ZarrLayer({ node: arr, metadata: NLDAS_GEOZARR_ATTRS, selection: { <timeDim>: TIME_INDEX }, getTileData, renderTile, maxRequests })`.
141+
The layer tiles the single-level array; `getTileData` pulls one chunk per tile
142+
via `zarr.get(arr, options.sliceSpec)`; `renderTile` applies the fixed rescale
143+
+ colormap on the GPU.
144+
4. **Map.** A `maplibre-gl` basemap centered on North America (≈ lon −98, lat 39,
145+
zoom ≈ 3.5) with the shared `DeckGlOverlay` (interleaved).
146+
147+
### Spatial metadata (the one non-obvious piece)
148+
149+
NLDAS-3 virtual-zarr is not GeoZarr-compliant, so — exactly like ECMWF's
150+
`ECMWF_GEOZARR_ATTRS` — we hard-code a synthetic attrs object and pass it as
151+
`ZarrLayer`'s `metadata` prop:
152+
153+
```ts
154+
{
155+
"spatial:dimensions": [<yDim>, <xDim>], // e.g. ["lat", "lon"]
156+
"spatial:transform": [a, b, c, d, e, f], // @developmentseed/affine convention
157+
"spatial:shape": [height, width],
158+
"proj:code": "EPSG:4326",
159+
}
160+
```
161+
162+
The **exact grid values** (spatial dim names, the non-spatial time dim name,
163+
shape, origin, pixel size, and crucially the latitude **row direction**) are not
164+
guessed — they are read from the store once during implementation by logging
165+
`arr.shape` and reading the 1-D lat/lon coordinate arrays, then frozen into
166+
`metadata.ts` with a comment recording where they came from. Tair's units
167+
(likely Kelvin) are confirmed the same way and drive a fixed rescale range plus a
168+
temperature colormap choice.
169+
170+
### Error handling
171+
172+
- Async open effect uses a `cancelled` flag (ECMWF precedent) to avoid setting
173+
state after unmount.
174+
- Non-float dtype throws with a descriptive message.
175+
- Layer is only constructed once both the opened array and the colormap texture
176+
are ready.
177+
178+
## Risks / smoke-test before building UI
179+
180+
- **Virtual chunk resolution — the load-bearing risk.** Mechanism is understood
181+
(the `virtualChunkContainers` map above), but two unknowns remain until run: the
182+
exact session entry point that accepts `virtualChunkContainers`
183+
(`Repository.checkoutBranch` vs `ReadSession.open` with an explicit snapshot id),
184+
and whether the manifest stores chunk locations such that the container's
185+
`url_prefix` matches and rewrites cleanly to the HTTPS prefix. **First
186+
implementation step is a throwaway script/console call** that opens the store
187+
and `zarr.get`s a single Tair chunk, confirming bytes return, *before* any
188+
layer/UI work. If it fails, revisit the approach here rather than pressing on.
189+
- **Zarr version.** icechunk is Zarr v3 with no consolidated metadata; the store
190+
serves metadata directly. Prefer `zarr.open.v3` if plain `open` mis-detects.
191+
192+
## Testing
193+
194+
Examples in this repo are demos without unit tests; verification is:
195+
196+
- `pnpm typecheck` in the example.
197+
- Manual `pnpm dev` confirming Tair renders over North America with the colormap.
198+
199+
A fresh worktree first needs submodule init + `pnpm install` + `pnpm build` so the
200+
workspace packages resolve.

examples/nldas-icechunk/README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# NLDAS-3 icechunk Example
2+
3+
Renders a single timestep of NLDAS-3 near-surface air temperature from a public
4+
[icechunk](https://icechunk.io) repository, read in the browser via
5+
[`icechunk-js`](https://github.com/EarthyScience/icechunk-js) + zarrita and
6+
displayed with `@developmentseed/deck.gl-zarr`'s `ZarrLayer`.
7+
8+
The store is a *virtual* Zarr: its chunks reference NLDAS-3 source objects in
9+
the same public `nasa-waterinsight` S3 bucket, authorized through a
10+
`virtualChunkContainers` map.
11+
12+
## Setup
13+
14+
1. Install dependencies from the repository root:
15+
```bash
16+
pnpm install
17+
```
18+
2. Build the packages:
19+
```bash
20+
pnpm build
21+
```
22+
3. Run the development server:
23+
```bash
24+
cd examples/nldas-icechunk
25+
pnpm dev
26+
```
27+
4. Open your browser to http://localhost:3000
28+
29+
`src/nldas/metadata.ts` hard-codes the grid (origin, pixel size, shape, units,
30+
fill) because the virtual store is not GeoZarr-compliant.

examples/nldas-icechunk/index.html

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8" />
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
6+
<title>NLDAS icechunk Example</title>
7+
<style>
8+
body {
9+
margin: 0;
10+
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
11+
}
12+
#root {
13+
width: 100vw;
14+
height: 100vh;
15+
}
16+
</style>
17+
</head>
18+
<body>
19+
<div id="root"></div>
20+
<script type="module" src="/src/main.tsx"></script>
21+
</body>
22+
</html>
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
{
2+
"name": "deck.gl-nldas-icechunk",
3+
"private": true,
4+
"type": "module",
5+
"scripts": {
6+
"dev": "vite",
7+
"build": "vite build",
8+
"preview": "vite preview",
9+
"typecheck": "tsc --noEmit",
10+
"publish": "pnpm build && gh-pages -d dist -b gh-pages -e examples/nldas-icechunk"
11+
},
12+
"dependencies": {
13+
"@chakra-ui/react": "^3.34.0",
14+
"@deck.gl/core": "^9.3.2",
15+
"@deck.gl/geo-layers": "^9.3.2",
16+
"@deck.gl/layers": "^9.3.2",
17+
"@deck.gl/mapbox": "^9.3.2",
18+
"@developmentseed/deck.gl-raster": "workspace:^",
19+
"@developmentseed/deck.gl-zarr": "workspace:^",
20+
"@emotion/react": "^11.14.0",
21+
"@luma.gl/core": "^9.3.2",
22+
"@luma.gl/shadertools": "^9.3.2",
23+
"deck.gl-raster-examples-shared": "workspace:*",
24+
"icechunk-js": "^0.4.0",
25+
"maplibre-gl": "^5.24.0",
26+
"react": "^19.2.5",
27+
"react-dom": "^19.2.5",
28+
"react-map-gl": "^8.1.1",
29+
"zarrita": "^0.7.3"
30+
},
31+
"devDependencies": {
32+
"@types/react": "^19.2.14",
33+
"@types/react-dom": "^19.2.3",
34+
"@vitejs/plugin-react": "^6.0.1",
35+
"gh-pages": "^6.3.0",
36+
"vite": "^8.0.0"
37+
}
38+
}

0 commit comments

Comments
 (0)