Merge branch 'main' into ian/simple-tree

Author: ianhi

docs/overrides/stylesheets/extra.css

@@ -56,11 +56,6 @@
 .md-header .md-search__input {
   background-color: rgba(255, 255, 255, 0.15);
   border: 1px solid rgba(255, 255, 255, 0.2);
-  color: white;
-}
-
-.md-header .md-search__input::placeholder {
-  color: rgba(255, 255, 255, 0.7);
 }
 
 /* Navigation tabs */

docs/user-guide/glossary.md

@@ -0,0 +1,110 @@
+# Glossary
+
+This page defines key terms used throughout the zarr-python documentation and API.
+
+## Array Structure
+
+### Array
+
+An N-dimensional typed array stored in a Zarr [store](#store). An array's
+[metadata](#metadata) defines its shape, data type, chunk layout, and codecs.
+
+### Chunk
+
+The fundamental unit of data in a Zarr array. An array is divided into chunks
+along each dimension according to the [chunk grid](#chunk-grid), which is currently
+part of Zarr's private API. Each chunk is independently compressed and encoded
+through the array's [codec](#codec) pipeline.
+
+When [sharding](#shard) is used, "chunk" refers to the inner chunks within each
+shard, because those are the compressible units. The chunks are the smallest units
+that can be read independently.
+
+!!! warning "Convention specific to zarr-python"
+    The use of "chunk" to mean the inner sub-chunk within a shard is a convention
+    adopted by zarr-python's `Array` API. In the Zarr V3 specification and in other
+    Zarr implementations, "chunk" may refer to the top-level grid cells (which
+    zarr-python calls "shards" when the sharding codec is used). Be aware of this
+    distinction when working across libraries.
+
+**API**: [`Array.chunks`][zarr.Array.chunks] returns the chunk shape. When
+sharding is used, this is the inner chunk shape.
+
+### Chunk Grid
+
+The partitioning of an array's elements into [chunks](#chunk). In Zarr V3, the
+chunk grid is defined in the array [metadata](#metadata) and determines the
+boundaries of each storage object.
+
+When sharding is used, the chunk grid defines the [shard](#shard) boundaries,
+not the inner chunk boundaries. The inner chunk shape is defined within the
+[sharding codec](#shard).
+
+**API**: The `chunk_grid` field in array metadata contains the storage-level
+grid.
+
+### Shard
+
+A storage object that contains one or more [chunks](#chunk). Sharding reduces the
+number of objects in a [store](#store) by grouping chunks together, which
+improves performance on file systems and object storage.
+
+Within each shard, chunks are compressed independently and can be read
+individually. However, writing requires updating the full shard for consistency,
+making shards the unit of writing and chunks the unit of reading.
+
+Sharding is implemented as a [codec](#codec) (the sharding indexed codec).
+When sharding is used:
+
+- The [chunk grid](#chunk-grid) in metadata defines the shard boundaries
+- The sharding codec's `chunk_shape` defines the inner chunk size
+- Each shard contains `shard_shape / chunk_shape` chunks per dimension
+
+**API**: [`Array.shards`][zarr.Array.shards] returns the shard shape, or `None`
+if sharding is not used. [`Array.chunks`][zarr.Array.chunks] returns the inner
+chunk shape.
+
+## Storage
+
+### Store
+
+A key-value storage backend that holds Zarr data and metadata. Stores implement
+the [`zarr.abc.store.Store`][] interface. Examples include local file systems,
+cloud object storage (S3, GCS, Azure), zip files, and in-memory dictionaries.
+
+Each [chunk](#chunk) or [shard](#shard) is stored as a single value (object or
+file) in the store, addressed by a key derived from its grid coordinates.
+
+### Metadata
+
+The JSON document (`zarr.json`) that describes an [array](#array) or group. For
+arrays, metadata includes the shape, data type, [chunk grid](#chunk-grid), fill
+value, and [codec](#codec) pipeline. Metadata is stored alongside the data in
+the [store](#store). Zarr-Python does not yet expose its internal metadata
+representation as part of its public API.
+
+## Codecs
+
+### Codec
+
+A transformation applied to array data during reading and writing. Codecs are
+chained into a pipeline and come in three types:
+
+- **Array-to-array**: Transforms like transpose that rearrange array elements
+- **Array-to-bytes**: Serialization that converts an array to a byte sequence
+  (exactly one required)
+- **Bytes-to-bytes**: Compression or checksums applied to the serialized bytes
+
+The [sharding indexed codec](#shard) is a special array-to-bytes codec that
+groups multiple [chunks](#chunk) into a single storage object.
+
+## API Properties
+
+The following properties are available on [`zarr.Array`][]:
+
+| Property | Description |
+|----------|-------------|
+| `.chunks` | Chunk shape — the inner chunk shape when sharding is used |
+| `.shards` | Shard shape, or `None` if no sharding |
+| `.nchunks` | Total number of independently compressible units across the array |
+| `.cdata_shape` | Number of independently compressible units per dimension |

docs/user-guide/index.md

@@ -35,6 +35,10 @@ Take your skills to the next level:
 - **[Extending](extending.md)** - Extend functionality with custom code
 - **[Consolidated Metadata](consolidated_metadata.md)** - Advanced metadata management
 
+## Reference
+
+- **[Glossary](glossary.md)** - Definitions of key terms (chunks, shards, codecs, etc.)
+
 ## Need Help?
 
 - Browse the [API Reference](../api/zarr/index.md) for detailed function documentation

mkdocs.yml

@@ -27,6 +27,7 @@ nav:
       - user-guide/gpu.md
       - user-guide/consolidated_metadata.md
       - user-guide/experimental.md
+      - user-guide/glossary.md
   - Examples:
       - user-guide/examples/custom_dtype.md
   - API Reference:

src/zarr/abc/store.py

@@ -16,7 +16,16 @@
 
     from zarr.core.buffer import Buffer, BufferPrototype
 
-__all__ = ["ByteGetter", "ByteSetter", "Store", "set_or_delete"]
+__all__ = [
+    "ByteGetter",
+    "ByteSetter",
+    "Store",
+    "SupportsDeleteSync",
+    "SupportsGetSync",
+    "SupportsSetSync",
+    "SupportsSyncStore",
+    "set_or_delete",
+]
 
 
 @dataclass(frozen=True, slots=True)
@@ -700,6 +709,31 @@ async def delete(self) -> None: ...
     async def set_if_not_exists(self, default: Buffer) -> None: ...
 
 
+@runtime_checkable
+class SupportsGetSync(Protocol):
+    def get_sync(
+        self,
+        key: str,
+        *,
+        prototype: BufferPrototype | None = None,
+        byte_range: ByteRequest | None = None,
+    ) -> Buffer | None: ...
+
+
+@runtime_checkable
+class SupportsSetSync(Protocol):
+    def set_sync(self, key: str, value: Buffer) -> None: ...
+
+
+@runtime_checkable
+class SupportsDeleteSync(Protocol):
+    def delete_sync(self, key: str) -> None: ...
+
+
+@runtime_checkable
+class SupportsSyncStore(SupportsGetSync, SupportsSetSync, SupportsDeleteSync, Protocol): ...
+
+
 async def set_or_delete(byte_setter: ByteSetter, value: Buffer | None) -> None:
     """Set or delete a value in a byte setter
 

src/zarr/core/chunk_grids.py

@@ -126,11 +126,14 @@ def normalize_chunks(chunks: Any, shape: tuple[int, ...], typesize: int) -> tupl
         chunks = tuple(int(chunks) for _ in shape)
 
     # handle dask-style chunks (iterable of iterables)
-    if all(isinstance(c, (tuple | list)) for c in chunks):
-        # take first chunk size for each dimension
-        chunks = tuple(
-            c[0] for c in chunks
-        )  # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
+    if all(isinstance(c, (tuple, list)) for c in chunks):
+        for i, c in enumerate(chunks):
+            if any(x != y for x, y in itertools.pairwise(c[:-1])) or (len(c) > 1 and c[-1] > c[0]):
+                raise ValueError(
+                    f"Irregular chunk sizes in dimension {i}: {tuple(c)}. "
+                    "Only uniform chunks (with an optional smaller final chunk) are supported."
+                )
+        chunks = tuple(c[0] for c in chunks)
 
     # handle bad dimensionality
     if len(chunks) > len(shape):

src/zarr/storage/_common.py

@@ -5,7 +5,13 @@
 from pathlib import Path
 from typing import TYPE_CHECKING, Any, Literal, Self, TypeAlias
 
-from zarr.abc.store import ByteRequest, Store
+from zarr.abc.store import (
+    ByteRequest,
+    Store,
+    SupportsDeleteSync,
+    SupportsGetSync,
+    SupportsSetSync,
+)
 from zarr.core.buffer import Buffer, default_buffer_prototype
 from zarr.core.common import (
     ANY_ACCESS_MODE,
@@ -228,6 +234,37 @@ async def is_empty(self) -> bool:
         """
         return await self.store.is_empty(self.path)
 
+    # -------------------------------------------------------------------
+    # Synchronous IO delegation
+    # -------------------------------------------------------------------
+
+    def get_sync(
+        self,
+        *,
+        prototype: BufferPrototype | None = None,
+        byte_range: ByteRequest | None = None,
+    ) -> Buffer | None:
+        """Synchronous read — delegates to ``self.store.get_sync(self.path, ...)``."""
+        if not isinstance(self.store, SupportsGetSync):
+            raise TypeError(f"Store {type(self.store).__name__} does not support synchronous get.")
+        if prototype is None:
+            prototype = default_buffer_prototype()
+        return self.store.get_sync(self.path, prototype=prototype, byte_range=byte_range)
+
+    def set_sync(self, value: Buffer) -> None:
+        """Synchronous write — delegates to ``self.store.set_sync(self.path, value)``."""
+        if not isinstance(self.store, SupportsSetSync):
+            raise TypeError(f"Store {type(self.store).__name__} does not support synchronous set.")
+        self.store.set_sync(self.path, value)
+
+    def delete_sync(self) -> None:
+        """Synchronous delete — delegates to ``self.store.delete_sync(self.path)``."""
+        if not isinstance(self.store, SupportsDeleteSync):
+            raise TypeError(
+                f"Store {type(self.store).__name__} does not support synchronous delete."
+            )
+        self.store.delete_sync(self.path)
+
     def __truediv__(self, other: str) -> StorePath:
         """Combine this store path with another path"""
         return self.__class__(self.store, _dereference_path(self.path, other))

src/zarr/storage/_local.py

@@ -187,6 +187,56 @@ def __repr__(self) -> str:
     def __eq__(self, other: object) -> bool:
         return isinstance(other, type(self)) and self.root == other.root
 
+    # -------------------------------------------------------------------
+    # Synchronous store methods
+    # -------------------------------------------------------------------
+
+    def _ensure_open_sync(self) -> None:
+        if not self._is_open:
+            if not self.read_only:
+                self.root.mkdir(parents=True, exist_ok=True)
+            if not self.root.exists():
+                raise FileNotFoundError(f"{self.root} does not exist")
+            self._is_open = True
+
+    def get_sync(
+        self,
+        key: str,
+        *,
+        prototype: BufferPrototype | None = None,
+        byte_range: ByteRequest | None = None,
+    ) -> Buffer | None:
+        if prototype is None:
+            prototype = default_buffer_prototype()
+        self._ensure_open_sync()
+        assert isinstance(key, str)
+        path = self.root / key
+        try:
+            return _get(path, prototype, byte_range)
+        except (FileNotFoundError, IsADirectoryError, NotADirectoryError):
+            return None
+
+    def set_sync(self, key: str, value: Buffer) -> None:
+        self._ensure_open_sync()
+        self._check_writable()
+        assert isinstance(key, str)
+        if not isinstance(value, Buffer):
+            raise TypeError(
+                f"LocalStore.set(): `value` must be a Buffer instance. "
+                f"Got an instance of {type(value)} instead."
+            )
+        path = self.root / key
+        _put(path, value)
+
+    def delete_sync(self, key: str) -> None:
+        self._ensure_open_sync()
+        self._check_writable()
+        path = self.root / key
+        if path.is_dir():
+            shutil.rmtree(path)
+        else:
+            path.unlink(missing_ok=True)
+
     async def get(
         self,
         key: str,

src/zarr/storage/_memory.py

@@ -77,6 +77,49 @@ def __eq__(self, other: object) -> bool:
             and self.read_only == other.read_only
         )
 
+    # -------------------------------------------------------------------
+    # Synchronous store methods
+    # -------------------------------------------------------------------
+
+    def get_sync(
+        self,
+        key: str,
+        *,
+        prototype: BufferPrototype | None = None,
+        byte_range: ByteRequest | None = None,
+    ) -> Buffer | None:
+        if prototype is None:
+            prototype = default_buffer_prototype()
+        if not self._is_open:
+            self._is_open = True
+        assert isinstance(key, str)
+        try:
+            value = self._store_dict[key]
+            start, stop = _normalize_byte_range_index(value, byte_range)
+            return prototype.buffer.from_buffer(value[start:stop])
+        except KeyError:
+            return None
+
+    def set_sync(self, key: str, value: Buffer) -> None:
+        self._check_writable()
+        if not self._is_open:
+            self._is_open = True
+        assert isinstance(key, str)
+        if not isinstance(value, Buffer):
+            raise TypeError(
+                f"MemoryStore.set(): `value` must be a Buffer instance. Got an instance of {type(value)} instead."
+            )
+        self._store_dict[key] = value
+
+    def delete_sync(self, key: str) -> None:
+        self._check_writable()
+        if not self._is_open:
+            self._is_open = True
+        try:
+            del self._store_dict[key]
+        except KeyError:
+            logger.debug("Key %s does not exist.", key)
+
     async def get(
         self,
         key: str,
@@ -122,7 +165,6 @@ async def set(self, key: str, value: Buffer, byte_range: tuple[int, int] | None
             raise TypeError(
                 f"MemoryStore.set(): `value` must be a Buffer instance. Got an instance of {type(value)} instead."
             )
-
         if byte_range is not None:
             buf = self._store_dict[key]
             buf[byte_range[0] : byte_range[1]] = value