Working with arrays

Creating an array

Zarr has several functions for creating arrays. For example:

import shutil
shutil.rmtree('data', ignore_errors=True)
import numpy as np

np.random.seed(0)

import zarr
store = zarr.storage.MemoryStore()
z = zarr.create_array(store=store, shape=(10000, 10000), chunks=(1000, 1000), dtype='int32')
print(z)

The code above creates a 2-dimensional array of 32-bit integers with 10000 rows and 10000 columns, divided into chunks where each chunk has 1000 rows and 1000 columns (and so there will be 100 chunks in total). The data is written to a [zarr.storage.MemoryStore][] (e.g. an in-memory dict). See Persistent arrays for details on storing arrays in other stores, and see Data types for an in-depth look at the data types supported by Zarr.

See the creation API documentation for more detailed information about creating arrays.

Reading and writing data

Zarr arrays support a similar interface to NumPy arrays for reading and writing data. For example, the entire array can be filled with a scalar value:

z[:] = 42

Regions of the array can also be written to, e.g.:

import numpy as np

z[0, :] = np.arange(10000)
z[:, 0] = np.arange(10000)

The contents of the array can be retrieved by slicing, which will load the requested region into memory as a NumPy array, e.g.:

print(z[0, 0])

print(z[-1, -1])

print(z[0, :])

print(z[:, 0])

print(z[:])

More information about NumPy-style indexing can be found in the NumPy documentation.

Persistent arrays

In the examples above, compressed data for each chunk of the array was stored in main memory. Zarr arrays can also be stored on a file system, enabling persistence of data between sessions. To do this, we can change the store argument to point to a filesystem path:

z1 = zarr.create_array(store='data/example-1.zarr', shape=(10000, 10000), chunks=(1000, 1000), dtype='int32')

The array above will store its configuration metadata and all compressed chunk data in a directory called 'data/example-1.zarr' relative to the current working directory. The [zarr.create_array][] function provides a convenient way to create a new persistent array or continue working with an existing array. Note, there is no need to close an array: data are automatically flushed to disk, and files are automatically closed whenever an array is modified.

Persistent arrays support the same interface for reading and writing data, e.g.:

z1[:] = 42
z1[0, :] = np.arange(10000)
z1[:, 0] = np.arange(10000)

Check that the data have been written and can be read again:

z2 = zarr.open_array('data/example-1.zarr', mode='r')
print(np.all(z1[:] == z2[:]))

If you are just looking for a fast and convenient way to save NumPy arrays to disk then load back into memory later, the functions [zarr.save][] and [zarr.load][] may be useful. E.g.:

a = np.arange(10)
zarr.save('data/example-2.zarr', a)
print(zarr.load('data/example-2.zarr'))

Please note that there are a number of other options for persistent array storage, see the Storage Guide for more details.

Resizing and appending

A Zarr array can be resized, which means that any of its dimensions can be increased or decreased in length. For example:

z = zarr.create_array(store='data/example-3.zarr', shape=(10000, 10000), dtype='int32',chunks=(1000, 1000))
z[:] = 42
print(f"Original shape: {z.shape}")
z.resize((20000, 10000))
print(f"New shape: {z.shape}")

Note that when an array is resized, the underlying data are not rearranged in any way. If one or more dimensions are shrunk, any chunks falling outside the new array shape will be deleted from the underlying store.

[zarr.Array.append][] is provided as a convenience function, which can be used to append data to any axis. E.g.:

a = np.arange(10000000, dtype='int32').reshape(10000, 1000)
z = zarr.create_array(store='data/example-4.zarr', shape=a.shape, dtype=a.dtype, chunks=(1000, 100))
z[:] = a
print(f"Original shape: {z.shape}")
z.append(a)
print(f"Shape after first append: {z.shape}")
z.append(np.vstack([a, a]), axis=1)
print(f"Shape after second append: {z.shape}")

Runtime configuration

Zarr arrays are parametrized with a configuration that determines certain aspects of array behavior.

We currently support two configuration options for arrays: write_empty_chunks and order.

field	type	default	description
`write_empty_chunks`	`bool`	`False`	Controls whether empty chunks are written to storage. See Empty chunks.
`order`	`Literal["C", "F"]`	`"C"`	The memory layout of arrays returned when reading data from the store.

You can specify the configuration when you create an array with the config keyword argument. config can be passed as either a dict or an ArrayConfig object.

arr = zarr.create_array({}, shape=(10,), dtype='int8', config={"write_empty_chunks": True})
print(arr.config)

To get an array view with a different config, use the with_config method.

arr_f = arr.with_config({"order": "F"})
print(arr_f.config)

Compressors

A number of different compressors can be used with Zarr. Zarr includes Blosc, Zstandard and Gzip compressors. Additional compressors are available through a separate package called NumCodecs which provides various compressor libraries including LZ4, Zlib, BZ2 and LZMA. Different compressors can be provided via the compressors keyword argument accepted by all array creation functions. For example:

compressors = zarr.codecs.BloscCodec(cname='zstd', clevel=3, shuffle=zarr.codecs.BloscShuffle.bitshuffle)
data = np.arange(100000000, dtype='int32').reshape(10000, 10000)
z = zarr.create_array(store='data/example-5.zarr', shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=compressors)
z[:] = data
print(z.compressors)

This array above will use Blosc as the primary compressor, using the Zstandard algorithm (compression level 3) internally within Blosc, and with the bit-shuffle filter applied.

When using a compressor, it can be useful to get some diagnostics on the compression ratio. Zarr arrays provide the [zarr.Array.info][] property which can be used to print useful diagnostics, e.g.:

print(z.info)

The [zarr.Array.info_complete][] method inspects the underlying store and prints additional diagnostics, e.g.:

print(z.info_complete())

!!! note [zarr.Array.info_complete][] will inspect the underlying store and may be slow for large arrays. Use [zarr.Array.info][] if detailed storage statistics are not needed.

If you don't specify a compressor, by default Zarr uses the Zstandard compressor.

To create an array without any compression, set compressors=None:

z_no_compress = zarr.create_array(store='data/example-uncompressed.zarr', shape=(10000, 10000), chunks=(1000, 1000), dtype='int32', compressors=None)
print(f"Compressors: {z_no_compress.compressors}")

In addition to Blosc and Zstandard, other compression libraries can also be used. For example, here is an array using Gzip compression, level 1:

data = np.arange(100000000, dtype='int32').reshape(10000, 10000)
z = zarr.create_array(store='data/example-6.zarr', shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=zarr.codecs.GzipCodec(level=1))
z[:] = data
print(f"Compressors: {z.compressors}")

Here is an example using LZMA from NumCodecs with a custom filter pipeline including LZMA's built-in delta filter:

import lzma
from zarr.codecs.numcodecs import LZMA

lzma_filters = [dict(id=lzma.FILTER_DELTA, dist=4), dict(id=lzma.FILTER_LZMA2, preset=1)]
compressors = LZMA(filters=lzma_filters)
data = np.arange(100000000, dtype='int32').reshape(10000, 10000)
z = zarr.create_array(store='data/example-7.zarr', shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=compressors)
print(f"Compressors: {z.compressors}")

To disable compression, set compressors=None when creating an array, e.g.:

z = zarr.create_array(
    store='data/example-8.zarr',
    shape=(100000000,),
    chunks=(1000000,),
    dtype='int32',
    compressors=None
)
print(f"Compressors: {z.compressors}")

Filters

In some cases, compression can be improved by transforming the data in some way. For example, if nearby values tend to be correlated, then shuffling the bytes within each numerical value or storing the difference between adjacent values may increase compression ratio. Some compressors provide built-in filters that apply transformations to the data prior to compression. For example, the Blosc compressor has built-in implementations of byte- and bit-shuffle filters, and the LZMA compressor has a built-in implementation of a delta filter. However, to provide additional flexibility for implementing and using filters in combination with different compressors, Zarr also provides a mechanism for configuring filters outside of the primary compressor.

Here is an example using a delta filter with the Blosc compressor:

from zarr.codecs.numcodecs import Delta

filters = [Delta(dtype='int32')]
compressors = zarr.codecs.BloscCodec(cname='zstd', clevel=1, shuffle=zarr.codecs.BloscShuffle.shuffle)
data = np.arange(100000000, dtype='int32').reshape(10000, 10000)
z = zarr.create_array(store='data/example-9.zarr', shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), filters=filters, compressors=compressors)
print(z.info_complete())

For more information about available filter codecs, see the Numcodecs documentation.

Advanced indexing

Zarr arrays support several methods for advanced or "fancy" indexing, which enable a subset of data items to be extracted or updated in an array without loading the entire array into memory.

Note that although this functionality is similar to some of the advanced indexing capabilities available on NumPy arrays and on h5py datasets, the Zarr API for advanced indexing is different from both NumPy and h5py, so please read this section carefully. For a complete description of the indexing API, see the documentation for the [zarr.Array][] class.

Indexing with coordinate arrays

Items from a Zarr array can be extracted by providing an integer array of coordinates. E.g.:

data = np.arange(10) ** 2
z = zarr.create_array(store='data/example-10.zarr', shape=data.shape, dtype=data.dtype)
z[:] = data
print(z[:])
print(z.get_coordinate_selection([2, 5]))

Coordinate arrays can also be used to update data, e.g.:

z.set_coordinate_selection([2, 5], [-1, -2])
print(z[:])

For multidimensional arrays, coordinates must be provided for each dimension, e.g.:

data = np.arange(15).reshape(3, 5)
z = zarr.create_array(store='data/example-11.zarr', shape=data.shape, dtype=data.dtype)
z[:] = data
print(z[:])

print(z.get_coordinate_selection(([0, 2], [1, 3])))

z.set_coordinate_selection(([0, 2], [1, 3]), [-1, -2])
print(z[:])

For convenience, coordinate indexing is also available via the vindex property, as well as the square bracket operator, e.g.:

print(z.vindex[[0, 2], [1, 3]])
z.vindex[[0, 2], [1, 3]] = [-3, -4]

print(z[:])

print(z[[0, 2], [1, 3]])

When the indexing arrays have different shapes, they are broadcast together. That is, the following two calls are equivalent:

print(z[1, [1, 3]])
print(z[[1, 1], [1, 3]])

Indexing with a mask array

Items can also be extracted by providing a Boolean mask. E.g.:

data = np.arange(10) ** 2
z = zarr.create_array(store='data/example-12.zarr', shape=data.shape, dtype=data.dtype)
z[:] = data
print(z[:])

sel = np.zeros_like(z, dtype=bool)
sel[2] = True
sel[5] = True
print(z.get_mask_selection(sel))

z.set_mask_selection(sel, [-1, -2])
print(z[:])

Here's a multidimensional example:

data = np.arange(15).reshape(3, 5)
z = zarr.create_array(store='data/example-13.zarr', shape=data.shape, dtype=data.dtype)
z[:] = data
print(z[:])

sel = np.zeros_like(z, dtype=bool)
sel[0, 1] = True
sel[2, 3] = True
print(z.get_mask_selection(sel))

z.set_mask_selection(sel, [-1, -2])
print(z[:])

For convenience, mask indexing is also available via the vindex property, e.g.:

print(z.vindex[sel])

z.vindex[sel] = [-3, -4]
print(z[:])

Mask indexing is conceptually the same as coordinate indexing, and is implemented internally via the same machinery. Both styles of indexing allow selecting arbitrary items from an array, also known as point selection.

Orthogonal indexing

Zarr arrays also support methods for orthogonal indexing, which allows selections to be made along each dimension of an array independently. For example, this allows selecting a subset of rows and/or columns from a 2-dimensional array. E.g.:

data = np.arange(15).reshape(3, 5)
z = zarr.create_array(store='data/example-14.zarr', shape=data.shape, dtype=data.dtype)
z[:] = data
print(z[:])

print(z.get_orthogonal_selection(([0, 2], slice(None))))  # select first and third rows

print(z.get_orthogonal_selection((slice(None), [1, 3])))  # select second and fourth columns)

print(z.get_orthogonal_selection(([0, 2], [1, 3])))  # select rows [0, 2] and columns [1, 4]

Data can also be modified, e.g.:

z.set_orthogonal_selection(([0, 2], [1, 3]), [[-1, -2], [-3, -4]])

For convenience, the orthogonal indexing functionality is also available via the oindex property, e.g.:

data = np.arange(15).reshape(3, 5)
z = zarr.create_array(store='data/example-15.zarr', shape=data.shape, dtype=data.dtype)
z[:] = data
print(z.oindex[[0, 2], :])  # select first and third rows

print(z.oindex[:, [1, 3]])  # select second and fourth columns

print(z.oindex[[0, 2], [1, 3]])  # select rows [0, 2] and columns [1, 4]

z.oindex[[0, 2], [1, 3]] = [[-1, -2], [-3, -4]]
print(z[:])

Any combination of integer, slice, 1D integer array and/or 1D Boolean array can be used for orthogonal indexing.

If the index contains at most one iterable, and otherwise contains only slices and integers, orthogonal indexing is also available directly on the array:

data = np.arange(15).reshape(3, 5)
z = zarr.create_array(store='data/example-16.zarr', shape=data.shape, dtype=data.dtype)
z[:] = data
print(np.all(z.oindex[[0, 2], :] == z[[0, 2], :]))

Block Indexing

Zarr also support block indexing, which allows selections of whole chunks based on their logical indices along each dimension of an array. For example, this allows selecting a subset of chunk aligned rows and/or columns from a 2-dimensional array. E.g.:

data = np.arange(100).reshape(10, 10)
z = zarr.create_array(store='data/example-17.zarr', shape=data.shape, dtype=data.dtype, chunks=(3, 3))
z[:] = data

Retrieve items by specifying their block coordinates:

print(z.get_block_selection(1))

Equivalent slicing:

print(z[3:6])

For convenience, the block selection functionality is also available via the blocks property, e.g.:

print(z.blocks[1])

Block index arrays may be multidimensional to index multidimensional arrays. For example:

print(z.blocks[0, 1:3])

Data can also be modified. Let's start by a simple 2D array:

z = zarr.create_array(store='data/example-18.zarr', shape=(6, 6), dtype=int, chunks=(2, 2))

Set data for a selection of items:

z.set_block_selection((1, 0), 1)
print(z[...])

For convenience, this functionality is also available via the blocks property. E.g.:

z.blocks[:, 2] = 7
print(z[...])

Any combination of integer and slice can be used for block indexing:

print(z.blocks[2, 1:3])

root = zarr.create_group('data/example-19.zarr')
foo = root.create_array(name='foo', shape=(1000, 100), chunks=(10, 10), dtype='float32')
bar = root.create_array(name='bar', shape=(100,), dtype='int32')
foo[:, :] = np.random.random((1000, 100))
bar[:] = np.arange(100)
print(root.tree())

Sharding

Using small chunk shapes in very large arrays can lead to a very large number of chunks. This can become a performance issue for file systems and object storage. With Zarr format 3, a new sharding feature has been added to address this issue.

With sharding, multiple chunks can be stored in a single storage object (e.g. a file). Within a shard, chunks are compressed and serialized separately. This allows individual chunks to be read independently. However, when writing data, a full shard must be written in one go for optimal performance and to avoid concurrency issues. That means that shards are the units of writing and chunks are the units of reading. Users need to configure the chunk and shard shapes accordingly.

Sharded arrays can be created by providing the shards parameter to [zarr.create_array][].

a = zarr.create_array('data/example-20.zarr', shape=(10000, 10000), shards=(1000, 1000), chunks=(100, 100), dtype='uint8')
a[:] = (np.arange(10000 * 10000) % 256).astype('uint8').reshape(10000, 10000)
print(a.info_complete())

In this example a shard shape of (1000, 1000) and a chunk shape of (100, 100) is used. This means that 10*10 chunks are stored in each shard, and there are 10*10 shards in total. Without the shards argument, there would be 10,000 chunks stored as individual files.

Rectilinear (variable) chunk grids

!!! warning "Experimental" Rectilinear chunk grids are an experimental feature and may change in future releases. This feature is expected to stabilize in Zarr version 3.3.

Because the feature is still stabilizing, it is disabled by default and
must be explicitly enabled:

```python
import zarr
zarr.config.set({"array.rectilinear_chunks": True})
```

Or via the environment variable `ZARR_ARRAY__RECTILINEAR_CHUNKS=True`.

The examples below assume this config has been set.

By default, Zarr arrays use a regular chunk grid where every chunk along a given dimension has the same size (except possibly the final boundary chunk). Rectilinear chunk grids allow each chunk along a dimension to have a different size. This is useful when the natural partitioning of the data is not uniform — for example, satellite swaths of varying width, time series with irregular intervals, or spatial tiles of different extents.

Creating arrays with rectilinear chunks

To create an array with rectilinear chunks, pass a nested list to the chunks parameter where each inner list gives the chunk sizes along one dimension:

zarr.config.set({"array.rectilinear_chunks": True})
z = zarr.create_array(
    store=zarr.storage.MemoryStore(),
    shape=(60, 100),
    chunks=[[10, 20, 30], [50, 50]],
    dtype='int32',
)
print(z.info)

In this example the first dimension is split into three chunks of sizes 10, 20, and 30, while the second dimension is split into two equal chunks of size 50.

Reading and writing data

Rectilinear arrays support the same indexing interface as regular arrays. Reads and writes that cross chunk boundaries of different sizes are handled automatically:

import numpy as np
data = np.arange(60 * 100, dtype='int32').reshape(60, 100)
z[:] = data
# Read a slice that spans the first two chunks (sizes 10 and 20) along axis 0
print(z[5:25, 0:5])

Inspecting chunk sizes

The .write_chunk_sizes property returns the actual data size of each storage chunk along every dimension. It works for both regular and rectilinear arrays and returns a tuple of tuples (matching the dask Array.chunks convention). When sharding is used, .read_chunk_sizes returns the inner chunk sizes instead:

print(z.write_chunk_sizes)

For regular arrays, this includes the boundary chunk:

z_regular = zarr.create_array(
    store=zarr.storage.MemoryStore(),
    shape=(100, 80),
    chunks=(30, 40),
    dtype='int32',
)
print(z_regular.write_chunk_sizes)

Note that the .chunks property is only available for regular chunk grids. For rectilinear arrays, use .write_chunk_sizes (or .read_chunk_sizes) instead.

Resizing and appending

Rectilinear arrays can be resized. When growing past the current edge sum, a new chunk is appended covering the additional extent. When shrinking, the chunk edges are preserved and the extent is re-bound (chunks beyond the new extent simply become inactive):

z = zarr.create_array(
    store=zarr.storage.MemoryStore(),
    shape=(30,),
    chunks=[[10, 20]],
    dtype='float64',
)
z[:] = np.arange(30, dtype='float64')
print(f"Before resize: chunk_sizes={z.write_chunk_sizes}")
z.resize((50,))
print(f"After resize:  chunk_sizes={z.write_chunk_sizes}")

The append method also works with rectilinear arrays:

z.append(np.arange(10, dtype='float64'))
print(f"After append:  shape={z.shape}, chunk_sizes={z.write_chunk_sizes}")

Compressors and filters

Rectilinear arrays work with all codecs — compressors, filters, and checksums. Since each chunk may have a different size, the codec pipeline processes each chunk independently:

z = zarr.create_array(
    store=zarr.storage.MemoryStore(),
    shape=(60, 100),
    chunks=[[10, 20, 30], [50, 50]],
    dtype='float64',
    filters=[zarr.codecs.TransposeCodec(order=(1, 0))],
    compressors=[zarr.codecs.BloscCodec(cname='zstd', clevel=3)],
)
z[:] = np.arange(60 * 100, dtype='float64').reshape(60, 100)
np.testing.assert_array_equal(z[:], np.arange(60 * 100, dtype='float64').reshape(60, 100))
print("Roundtrip OK")

Rectilinear shard boundaries

Rectilinear chunk grids can also be used for shard boundaries when combined with sharding. In this case, the outer grid (shards) is rectilinear while the inner chunks remain regular. Each shard dimension must be divisible by the corresponding inner chunk size:

z = zarr.create_array(
    store=zarr.storage.MemoryStore(),
    shape=(120, 100),
    chunks=(10, 10),
    shards=[[60, 40, 20], [50, 50]],
    dtype='int32',
)
z[:] = np.arange(120 * 100, dtype='int32').reshape(120, 100)
print(z[50:70, 40:60])

Note that rectilinear inner chunks with sharding are not supported — only the shard boundaries can be rectilinear.

Metadata format

Rectilinear chunk grid metadata uses run-length encoding (RLE) for compact serialization. When reading metadata, both bare integers and [value, count] pairs are accepted:

[10, 20, 30] — three chunks with explicit sizes
[[10, 3]] — three chunks of size 10 (RLE shorthand)
[[10, 3], 5] — three chunks of size 10, then one chunk of size 5

When writing, Zarr automatically compresses repeated values into RLE format.

Missing features in 3.0

The following features have not been ported to 3.0 yet.

Copying and migrating data

See the Zarr-Python 2 documentation on Copying and migrating data for more details.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Working with arrays

Creating an array

Reading and writing data

Persistent arrays

Resizing and appending

Runtime configuration

Compressors

Filters

Advanced indexing

Indexing with coordinate arrays

Indexing with a mask array

Orthogonal indexing

Block Indexing

Sharding

Rectilinear (variable) chunk grids

Creating arrays with rectilinear chunks

Reading and writing data

Inspecting chunk sizes

Resizing and appending

Compressors and filters

Rectilinear shard boundaries

Metadata format

Missing features in 3.0

Copying and migrating data

FilesExpand file tree

arrays.md

Latest commit

History

arrays.md

File metadata and controls

Working with arrays

Creating an array

Reading and writing data

Persistent arrays

Resizing and appending

Runtime configuration

Compressors

Filters

Advanced indexing

Indexing with coordinate arrays

Indexing with a mask array

Orthogonal indexing

Block Indexing

Sharding

Rectilinear (variable) chunk grids

Creating arrays with rectilinear chunks

Reading and writing data

Inspecting chunk sizes

Resizing and appending

Compressors and filters

Rectilinear shard boundaries

Metadata format

Missing features in 3.0

Copying and migrating data