Skip to content

Commit 6d878a2

Browse files
authored
docs: Add Zarr example to docs (#468)
* New Zarr example * Add zarr example * exclude examples from pyright * try other exclude * remove pyright
1 parent b419d64 commit 6d878a2

14 files changed

Lines changed: 1498 additions & 8 deletions

File tree

.github/workflows/test-python.yml

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -77,8 +77,3 @@ jobs:
7777

7878
- name: Add venv to PATH (for pyright action)
7979
run: echo "$PWD/.venv/bin" >> $GITHUB_PATH
80-
81-
- name: Run pyright
82-
uses: jakebailey/pyright-action@v2
83-
with:
84-
pylance-version: latest-release

docs/assets/zarr-example.png

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../examples/zarr/zarr-example.png

docs/examples/fastapi.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ It's easy to integrate obstore with FastAPI routes, where you want to download a
66

77
FastAPI has a [`StreamingResponse`](https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse), which neatly integrates with [`BytesStream`][obstore.BytesStream] to stream the response to the user.
88

9+
## Example
10+
911
!!! note
1012

1113
This example is also [available on Github](https://github.com/developmentseed/obstore/blob/main/examples/fastapi/README.md) if you'd like to test it out locally.

docs/examples/minio.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22

33
[MinIO](https://github.com/minio/minio) is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license. It's often used for testing or self-hosting S3-compatible storage.
44

5+
## Example
6+
7+
!!! note
8+
9+
This example is also [available on Github](https://github.com/developmentseed/obstore/blob/main/examples/minio/README.md) if you'd like to test it out locally.
10+
511
We can run minio locally using docker:
612

713
```shell

docs/examples/zarr.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Zarr
2+
3+
[Zarr-Python](https://zarr.readthedocs.io/en/stable/index.html) is a Python library for reading and writing the [Zarr file format](https://zarr.dev/) for N-dimensional arrays. Zarr-Python is often used in conjunction with [Xarray](https://xarray.dev/).
4+
5+
Zarr datasets are often very large and thus stored in object storage for cost effectiveness. As of Zarr-Python version 3.0.7 and later, you can [use Obstore as a backend](https://zarr.readthedocs.io/en/stable/user-guide/storage.html#object-store) for Zarr-Python. For large queries this [can be significantly faster](https://github.com/maxrjones/zarr-obstore-performance) than the default fsspec-based backend.
6+
7+
## Example
8+
9+
!!! note
10+
11+
This example is also [available on Github](https://github.com/developmentseed/obstore/blob/main/examples/zarr/README.md) if you'd like to test it out locally.
12+
13+
```py
14+
import matplotlib.pyplot as plt
15+
import pystac_client
16+
import xarray as xr
17+
from zarr.storage import ObjectStore
18+
19+
from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider
20+
from obstore.store import AzureStore
21+
22+
# These first lines are specific to Zarr stored in the Microsoft Planetary
23+
# Computer. We use pystac-client to find the metadata for this specific Zarr
24+
# store.
25+
catalog = pystac_client.Client.open(
26+
"https://planetarycomputer.microsoft.com/api/stac/v1/",
27+
)
28+
collection = catalog.get_collection("daymet-daily-hi")
29+
asset = collection.assets["zarr-abfs"]
30+
31+
# We construct an AzureStore because this Zarr dataset is stored in Azure
32+
# storage
33+
azure_store = AzureStore(
34+
credential_provider=PlanetaryComputerCredentialProvider.from_asset(asset),
35+
)
36+
37+
# Next we use the Zarr ObjectStorage adapter and pass it to xarray.
38+
zarr_store = ObjectStore(azure_store, read_only=True)
39+
ds = xr.open_dataset(zarr_store, consolidated=True, engine="zarr")
40+
41+
# And plot with matplotlib
42+
fig, ax = plt.subplots(figsize=(12, 12))
43+
ds.sel(time="2009")["tmax"].mean(dim="time").plot.imshow(ax=ax, cmap="inferno")
44+
fig.savefig("zarr-example.png")
45+
```
46+
47+
This plots:
48+
49+
![](../assets/zarr-example.png)

docs/integrations/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,6 @@ Various integrations with external libraries exist:
44

55
- [`dagster`](https://dagster.io/): Refer to [`dagster-obstore`](https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-obstore).
66
- [`fsspec`](https://github.com/fsspec/filesystem_spec): Use the [`obstore.fsspec`][obstore.fsspec] module.
7-
- [`zarr-python`](https://zarr.readthedocs.io/en/stable/): [In progress](https://github.com/zarr-developers/zarr-python/pull/1661).
7+
- [`zarr-python`](https://zarr.readthedocs.io/en/stable/): Use [`zarr.storage.ObjectStore`](https://zarr.readthedocs.io/en/stable/user-guide/storage.html#object-store), included as of Zarr version `3.0.7` and later. See also the [Obstore-Zarr example](../examples/zarr.md).
88

99
Know of an integration that doesn't exist here? [Edit this document](https://github.com/developmentseed/obstore/edit/main/docs/integrations.md).

examples/zarr/.python-version

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
3.12

examples/zarr/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Obstore Zarr example
2+
3+
Example using Zarr with the Obstore backend.
4+
5+
```
6+
uv run main.py
7+
```
8+
9+
![](./zarr-example.png)
10+
11+
This is a port of the Zarr example in the [Planetary Computer documentation](https://planetarycomputer.microsoft.com/docs/quickstarts/reading-zarr-data/).

examples/zarr/main.py

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
"""Example using Zarr with the Obstore backend."""
2+
3+
import matplotlib.pyplot as plt
4+
import pystac_client
5+
import xarray as xr
6+
from zarr.storage import ObjectStore
7+
8+
from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider
9+
from obstore.store import AzureStore
10+
11+
# These first lines are specific to Zarr stored in the Microsoft Planetary
12+
# Computer. We use pystac-client to find the metadata for this specific Zarr
13+
# store.
14+
catalog = pystac_client.Client.open(
15+
"https://planetarycomputer.microsoft.com/api/stac/v1/",
16+
)
17+
collection = catalog.get_collection("daymet-daily-hi")
18+
asset = collection.assets["zarr-abfs"]
19+
20+
# We construct an AzureStore because this Zarr dataset is stored in Azure
21+
# storage
22+
azure_store = AzureStore(
23+
credential_provider=PlanetaryComputerCredentialProvider.from_asset(asset),
24+
)
25+
26+
# Next we use the Zarr ObjectStorage adapter and pass it to xarray.
27+
zarr_store = ObjectStore(azure_store, read_only=True)
28+
ds = xr.open_dataset(zarr_store, consolidated=True, engine="zarr")
29+
30+
# And plot with matplotlib
31+
fig, ax = plt.subplots(figsize=(12, 12))
32+
ds.sel(time="2009")["tmax"].mean(dim="time").plot.imshow(ax=ax, cmap="inferno")
33+
fig.savefig("zarr-example.png")

examples/zarr/pyproject.toml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
[project]
2+
name = "zarr-example"
3+
version = "0.1.0"
4+
description = "Add your description here"
5+
readme = "README.md"
6+
requires-python = ">=3.11"
7+
dependencies = [
8+
"matplotlib>=3.10.1",
9+
"obstore>=0.6.0",
10+
"pystac-client>=0.8.6",
11+
"xarray>=2025.3.0",
12+
"zarr>=3.0.8",
13+
]
14+
15+
[dependency-groups]
16+
dev = ["ipykernel>=6.29.5"]

0 commit comments

Comments
 (0)