Merge branch 'main' of github.com:zarr-developers/zarr-python into feat/v3-scale-offset-cast

Author: d-v-b

docs/user-guide/glossary.md

@@ -0,0 +1,110 @@
+# Glossary
+
+This page defines key terms used throughout the zarr-python documentation and API.
+
+## Array Structure
+
+### Array
+
+An N-dimensional typed array stored in a Zarr [store](#store). An array's
+[metadata](#metadata) defines its shape, data type, chunk layout, and codecs.
+
+### Chunk
+
+The fundamental unit of data in a Zarr array. An array is divided into chunks
+along each dimension according to the [chunk grid](#chunk-grid), which is currently
+part of Zarr's private API. Each chunk is independently compressed and encoded
+through the array's [codec](#codec) pipeline.
+
+When [sharding](#shard) is used, "chunk" refers to the inner chunks within each
+shard, because those are the compressible units. The chunks are the smallest units
+that can be read independently.
+
+!!! warning "Convention specific to zarr-python"
+    The use of "chunk" to mean the inner sub-chunk within a shard is a convention
+    adopted by zarr-python's `Array` API. In the Zarr V3 specification and in other
+    Zarr implementations, "chunk" may refer to the top-level grid cells (which
+    zarr-python calls "shards" when the sharding codec is used). Be aware of this
+    distinction when working across libraries.
+
+**API**: [`Array.chunks`][zarr.Array.chunks] returns the chunk shape. When
+sharding is used, this is the inner chunk shape.
+
+### Chunk Grid
+
+The partitioning of an array's elements into [chunks](#chunk). In Zarr V3, the
+chunk grid is defined in the array [metadata](#metadata) and determines the
+boundaries of each storage object.
+
+When sharding is used, the chunk grid defines the [shard](#shard) boundaries,
+not the inner chunk boundaries. The inner chunk shape is defined within the
+[sharding codec](#shard).
+
+**API**: The `chunk_grid` field in array metadata contains the storage-level
+grid.
+
+### Shard
+
+A storage object that contains one or more [chunks](#chunk). Sharding reduces the
+number of objects in a [store](#store) by grouping chunks together, which
+improves performance on file systems and object storage.
+
+Within each shard, chunks are compressed independently and can be read
+individually. However, writing requires updating the full shard for consistency,
+making shards the unit of writing and chunks the unit of reading.
+
+Sharding is implemented as a [codec](#codec) (the sharding indexed codec).
+When sharding is used:
+
+- The [chunk grid](#chunk-grid) in metadata defines the shard boundaries
+- The sharding codec's `chunk_shape` defines the inner chunk size
+- Each shard contains `shard_shape / chunk_shape` chunks per dimension
+
+**API**: [`Array.shards`][zarr.Array.shards] returns the shard shape, or `None`
+if sharding is not used. [`Array.chunks`][zarr.Array.chunks] returns the inner
+chunk shape.
+
+## Storage
+
+### Store
+
+A key-value storage backend that holds Zarr data and metadata. Stores implement
+the [`zarr.abc.store.Store`][] interface. Examples include local file systems,
+cloud object storage (S3, GCS, Azure), zip files, and in-memory dictionaries.
+
+Each [chunk](#chunk) or [shard](#shard) is stored as a single value (object or
+file) in the store, addressed by a key derived from its grid coordinates.
+
+### Metadata
+
+The JSON document (`zarr.json`) that describes an [array](#array) or group. For
+arrays, metadata includes the shape, data type, [chunk grid](#chunk-grid), fill
+value, and [codec](#codec) pipeline. Metadata is stored alongside the data in
+the [store](#store). Zarr-Python does not yet expose its internal metadata
+representation as part of its public API.
+
+## Codecs
+
+### Codec
+
+A transformation applied to array data during reading and writing. Codecs are
+chained into a pipeline and come in three types:
+
+- **Array-to-array**: Transforms like transpose that rearrange array elements
+- **Array-to-bytes**: Serialization that converts an array to a byte sequence
+  (exactly one required)
+- **Bytes-to-bytes**: Compression or checksums applied to the serialized bytes
+
+The [sharding indexed codec](#shard) is a special array-to-bytes codec that
+groups multiple [chunks](#chunk) into a single storage object.
+
+## API Properties
+
+The following properties are available on [`zarr.Array`][]:
+
+| Property | Description |
+|----------|-------------|
+| `.chunks` | Chunk shape — the inner chunk shape when sharding is used |
+| `.shards` | Shard shape, or `None` if no sharding |
+| `.nchunks` | Total number of independently compressible units across the array |
+| `.cdata_shape` | Number of independently compressible units per dimension |

docs/user-guide/index.md

@@ -35,6 +35,10 @@ Take your skills to the next level:
 - **[Extending](extending.md)** - Extend functionality with custom code
 - **[Consolidated Metadata](consolidated_metadata.md)** - Advanced metadata management
 
+## Reference
+
+- **[Glossary](glossary.md)** - Definitions of key terms (chunks, shards, codecs, etc.)
+
 ## Need Help?
 
 - Browse the [API Reference](../api/zarr/index.md) for detailed function documentation

mkdocs.yml

@@ -27,6 +27,7 @@ nav:
       - user-guide/gpu.md
       - user-guide/consolidated_metadata.md
       - user-guide/experimental.md
+      - user-guide/glossary.md
   - Examples:
       - user-guide/examples/custom_dtype.md
   - API Reference:

src/zarr/core/chunk_grids.py

@@ -126,11 +126,14 @@ def normalize_chunks(chunks: Any, shape: tuple[int, ...], typesize: int) -> tupl
         chunks = tuple(int(chunks) for _ in shape)
 
     # handle dask-style chunks (iterable of iterables)
-    if all(isinstance(c, (tuple | list)) for c in chunks):
-        # take first chunk size for each dimension
-        chunks = tuple(
-            c[0] for c in chunks
-        )  # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
+    if all(isinstance(c, (tuple, list)) for c in chunks):
+        for i, c in enumerate(chunks):
+            if any(x != y for x, y in itertools.pairwise(c[:-1])) or (len(c) > 1 and c[-1] > c[0]):
+                raise ValueError(
+                    f"Irregular chunk sizes in dimension {i}: {tuple(c)}. "
+                    "Only uniform chunks (with an optional smaller final chunk) are supported."
+                )
+        chunks = tuple(c[0] for c in chunks)
 
     # handle bad dimensionality
     if len(chunks) > len(shape):

tests/test_chunk_grids.py

@@ -35,6 +35,10 @@ def test_guess_chunks(shape: tuple[int, ...], itemsize: int) -> None:
         ((30, None, None), (100, 20, 10), 1, (30, 20, 10)),
         ((30, 20, None), (100, 20, 10), 1, (30, 20, 10)),
         ((30, 20, 10), (100, 20, 10), 1, (30, 20, 10)),
+        # dask-style chunks (uniform with optional smaller final chunk)
+        (((100, 100, 100), (50, 50)), (300, 100), 1, (100, 50)),
+        (((100, 100, 50),), (250,), 1, (100,)),
+        (((100,),), (100,), 1, (100,)),
         # auto chunking
         (None, (100,), 1, (100,)),
         (-1, (100,), 1, (100,)),
@@ -52,3 +56,8 @@ def test_normalize_chunks_errors() -> None:
         normalize_chunks("foo", (100,), 1)
     with pytest.raises(ValueError):
         normalize_chunks((100, 10), (100,), 1)
+    # dask-style irregular chunks should raise
+    with pytest.raises(ValueError, match="Irregular chunk sizes"):
+        normalize_chunks(((10, 20, 30),), (60,), 1)
+    with pytest.raises(ValueError, match="Irregular chunk sizes"):
+        normalize_chunks(((100, 100), (10, 20)), (200, 30), 1)