Skip to content

Commit b873705

Browse files
committed
Executable storage guide
1 parent 901f04e commit b873705

2 files changed

Lines changed: 31 additions & 35 deletions

File tree

docs/user-guide/storage.md

Lines changed: 30 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -14,25 +14,29 @@ Zarr-Python 3, stores must implement the abstract store API from
1414
In most cases, it is not required to create a `Store` object explicitly. Passing a string
1515
to Zarr's top level API will result in the store being created automatically:
1616

17-
```python
17+
```python exec="true" session="storage" source="above" result="ansi"
1818
import zarr
1919

2020
# Implicitly create a writable LocalStore
21-
zarr.create_group(store='data/foo/bar')
22-
# <Group file://data/foo/bar>
21+
group = zarr.create_group(store='data/foo/bar')
22+
print(group)
23+
```
2324

25+
```python exec="true" session="storage" source="above" result="ansi"
2426
# Implicitly create a read-only FsspecStore
25-
zarr.open_group(
27+
group = zarr.open_group(
2628
store='s3://noaa-nwm-retro-v2-zarr-pds',
2729
mode='r',
2830
storage_options={'anon': True}
2931
)
30-
# <Group <FsspecStore(S3FileSystem, noaa-nwm-retro-v2-zarr-pds)>>
32+
print(group)
33+
```
3134

35+
```python exec="true" session="storage" source="above" result="ansi"
3236
# Implicitly creates a MemoryStore
3337
data = {}
34-
zarr.create_group(store=data)
35-
# <Group memory://...>
38+
group = zarr.create_group(store=data)
39+
print(group)
3640
```
3741

3842
## Explicit Store Creation
@@ -46,21 +50,21 @@ built-in store: [`zarr.storage.LocalStore`][], [`zarr.storage.FsspecStore`][],
4650
The [`zarr.storage.LocalStore`][] stores data in a nested set of directories on a local
4751
filesystem:
4852

49-
```python
53+
```python exec="true" session="storage" source="above" result="ansi"
5054
store = zarr.storage.LocalStore('data/foo/bar', read_only=True)
51-
zarr.open_group(store=store, mode='r')
52-
# <Group file://data/foo/bar>
55+
group = zarr.open_group(store=store, mode='r')
56+
print(group)
5357
```
5458

5559
### Zip Store
5660

5761
The [`zarr.storage.ZipStore`][] stores the contents of a Zarr hierarchy in a single
5862
Zip file. The [Zip Store specification](https://github.com/zarr-developers/zarr-specs/pull/311) is currently in draft form:
5963

60-
```python
64+
```python exec="true" session="storage" source="above" result="ansi"
6165
store = zarr.storage.ZipStore('data.zip', mode='w')
62-
zarr.create_array(store=store, shape=(2,), dtype='float64')
63-
# <Array zip://data.zip shape=(2,) dtype=float64>
66+
array = zarr.create_array(store=store, shape=(2,), dtype='float64')
67+
print(array)
6468
```
6569

6670
### Remote Store
@@ -72,26 +76,27 @@ such as cloud object storage (e.g. AWS S3, Google Cloud Storage, Azure Blob Stor
7276
that implements the [AbstractFileSystem](https://filesystem-spec.readthedocs.io/en/stable/api.html#fsspec.spec.AbstractFileSystem)
7377
API. `storage_options` can be used to configure the fsspec backend:
7478

75-
```python
79+
```python exec="true" session="storage" source="above" result="ansi"
7680
store = zarr.storage.FsspecStore.from_url(
7781
's3://noaa-nwm-retro-v2-zarr-pds',
7882
read_only=True,
7983
storage_options={'anon': True}
8084
)
81-
zarr.open_group(store=store, mode='r')
82-
# <Group <FsspecStore(S3FileSystem, noaa-nwm-retro-v2-zarr-pds)>>
85+
group = zarr.open_group(store=store, mode='r')
86+
print(group)
8387
```
8488

8589
The type of filesystem (e.g. S3, https, etc..) is inferred from the scheme of the url (e.g. s3 for "**s3**://noaa-nwm-retro-v2-zarr-pds").
8690
In case a specific filesystem is needed, one can explicitly create it. For example to create a S3 filesystem:
8791

88-
```python
92+
```python exec="true" session="storage" source="above" result="ansi"
8993
import fsspec
9094
fs = fsspec.filesystem(
9195
's3', anon=True, asynchronous=True,
9296
client_kwargs={'endpoint_url': "https://noaa-nwm-retro-v2-zarr-pds.s3.amazonaws.com"}
9397
)
9498
store = zarr.storage.FsspecStore(fs)
99+
print(store)
95100
```
96101

97102

@@ -100,12 +105,11 @@ store = zarr.storage.FsspecStore(fs)
100105
The [`zarr.storage.MemoryStore`][] a in-memory store that allows for serialization of
101106
Zarr data (metadata and chunks) to a dictionary:
102107

103-
```python
108+
```python exec="true" session="storage" source="above" result="ansi"
104109
data = {}
105110
store = zarr.storage.MemoryStore(data)
106-
# TODO: replace with create_array after #2463
107-
zarr.create_array(store=store, shape=(2,), dtype='float64')
108-
# <Array memory://... shape=(2,) dtype=float64>
111+
array = zarr.create_array(store=store, shape=(2,), dtype='float64')
112+
print(array)
109113
```
110114

111115
### Object Store
@@ -114,33 +118,25 @@ zarr.create_array(store=store, shape=(2,), dtype='float64')
114118
[storage implementation](https://developmentseed.org/obstore/latest/api/store/), including AWS S3 ([`obstore.store.S3Store`][]), Google Cloud Storage ([`obstore.store.GCSStore`][]), and Azure Blob Storage ([`obstore.store.AzureStore`][]). This store is backed by [obstore](https://developmentseed.org/obstore/latest/), which
115119
builds on the production quality Rust library [object_store](https://docs.rs/object_store/latest/object_store/).
116120

117-
```python
121+
```python exec="true" session="storage" source="above" result="ansi"
118122
from zarr.storage import ObjectStore
119123
from obstore.store import MemoryStore
120124

121125
store = ObjectStore(MemoryStore())
122-
zarr.create_array(store=store, shape=(2,), dtype='float64')
123-
# <Array object_store://... shape=(2,) dtype=float64>
126+
array = zarr.create_array(store=store, shape=(2,), dtype='float64')
127+
print(array)
124128
```
125129

126130
Here's an example of using ObjectStore for accessing remote data:
127131

128-
```python
132+
```python exec="true" session="storage" source="above" result="ansi"
129133
from zarr.storage import ObjectStore
130134
from obstore.store import S3Store
131135

132136
s3_store = S3Store('noaa-nwm-retro-v2-zarr-pds', skip_signature=True, region="us-west-2")
133137
store = zarr.storage.ObjectStore(store=s3_store, read_only=True)
134138
group = zarr.open_group(store=store, mode='r')
135-
group.info
136-
# Name :
137-
# Type : Group
138-
# Zarr format : 2
139-
# Read-only : True
140-
# Store type : ObjectStore
141-
# No. members : 12
142-
# No. arrays : 12
143-
# No. groups : 0
139+
print(group.info)
144140
```
145141

146142
!!! warning

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,7 @@ fix = "rm -r data/; pytest docs/user-guide --doctest-glob='*.rst' --accept"
252252
list-env = "pip list"
253253

254254
[tool.hatch.envs.docs]
255-
features = ['docs']
255+
features = ['docs', 'remote']
256256

257257
[tool.hatch.envs.docs.scripts]
258258
serve = "mkdocs serve"

0 commit comments

Comments
 (0)