feat: add experimental support for rectilinear (variable-sized) chunks (#3802)

Introduces a unified `ChunkGridMetadata` model that handles both regular and
rectilinear chunk layouts through a common `RegularDimension`/`VaryingDimension`
abstraction. Rectilinear chunks are gated behind a feature flag
(`zarr.config.set({'array.rectilinear_chunks': True})`).

Key changes:
- New `ChunkGridMetadata` replaces `RegularChunkGrid` as the internal
  representation, supporting both regular and rectilinear dimensions
- Rectilinear chunk grids can be created via nested sequences passed to
  `chunks` (e.g., `[[10, 20, 30], [50, 50]]`)
- Rectilinear sharding: shard boundaries can be rectilinear while inner
  chunks remain regular
- Existing arrays with regular chunk grids are read/written identically

Breaking change:
- `BaseCodec.validate()` and `CodecPipeline.validate()` now receive
  `ChunkGridMetadata` instead of `ChunkGrid` for the `chunk_grid` parameter

---------

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>
Co-authored-by: Ilan Gold <ilanbassgold@gmail.com>
Co-authored-by: Sam Levang <39069044+slevang@users.noreply.github.com>

Author: 6 people

changes/3802.feature.md

@@ -0,0 +1,16 @@
+Add support for rectilinear (variable-sized) chunk grids. This feature is experimental and
+must be explicitly enabled via ``zarr.config.set({'array.rectilinear_chunks': True})``.
+
+Rectilinear chunks can be used through:
+
+- **Creating arrays**: Pass nested sequences (e.g., ``[[10, 20, 30], [50, 50]]``) to ``chunks``
+  in ``zarr.create_array``, ``zarr.from_array``, ``zarr.zeros``, ``zarr.ones``, ``zarr.full``,
+  ``zarr.open``, and related functions, or to ``chunk_shape`` in ``zarr.create``.
+- **Opening existing arrays**: Arrays stored with the ``rectilinear`` chunk grid are read
+  transparently via ``zarr.open`` and ``zarr.open_array``.
+- **Rectilinear sharding**: Shard boundaries can be rectilinear while inner chunks remain regular.
+
+**Breaking change**: The ``validate`` method on ``BaseCodec`` and ``CodecPipeline`` now receives
+a ``ChunkGridMetadata`` instance instead of a ``ChunkGrid`` instance for the ``chunk_grid``
+parameter. Third-party codecs that override ``validate`` and inspect the chunk grid will need to
+update their type annotations. No known downstream packages were using this parameter.

design/chunk-grid.md

@@ -611,6 +611,171 @@ In this example a shard shape of (1000, 1000) and a chunk shape of (100, 100) is
 This means that `10*10` chunks are stored in each shard, and there are `10*10` shards in total.
 Without the `shards` argument, there would be 10,000 chunks stored as individual files.
 
+## Rectilinear (variable) chunk grids
+
+!!! warning "Experimental"
+    Rectilinear chunk grids are an experimental feature and may change in
+    future releases. This feature is expected to stabilize in Zarr version 3.3.
+
+    Because the feature is still stabilizing, it is disabled by default and
+    must be explicitly enabled:
+
+    ```python
+    import zarr
+    zarr.config.set({"array.rectilinear_chunks": True})
+    ```
+
+    Or via the environment variable `ZARR_ARRAY__RECTILINEAR_CHUNKS=True`.
+
+    The examples below assume this config has been set.
+
+By default, Zarr arrays use a regular chunk grid where every chunk along a
+given dimension has the same size (except possibly the final boundary chunk).
+Rectilinear chunk grids allow each chunk along a dimension to have a different
+size. This is useful when the natural partitioning of the data is not uniform —
+for example, satellite swaths of varying width, time series with irregular
+intervals, or spatial tiles of different extents.
+
+### Creating arrays with rectilinear chunks
+
+To create an array with rectilinear chunks, pass a nested list to the `chunks`
+parameter where each inner list gives the chunk sizes along one dimension:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+zarr.config.set({"array.rectilinear_chunks": True})
+z = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(60, 100),
+    chunks=[[10, 20, 30], [50, 50]],
+    dtype='int32',
+)
+print(z.info)
+```
+
+In this example the first dimension is split into three chunks of sizes 10, 20,
+and 30, while the second dimension is split into two equal chunks of size 50.
+
+### Reading and writing data
+
+Rectilinear arrays support the same indexing interface as regular arrays.
+Reads and writes that cross chunk boundaries of different sizes are handled
+automatically:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+import numpy as np
+data = np.arange(60 * 100, dtype='int32').reshape(60, 100)
+z[:] = data
+# Read a slice that spans the first two chunks (sizes 10 and 20) along axis 0
+print(z[5:25, 0:5])
+```
+
+### Inspecting chunk sizes
+
+The `.write_chunk_sizes` property returns the actual data size of each storage
+chunk along every dimension. It works for both regular and rectilinear arrays
+and returns a tuple of tuples (matching the dask `Array.chunks` convention).
+When sharding is used, `.read_chunk_sizes` returns the inner chunk sizes instead:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+print(z.write_chunk_sizes)
+```
+
+For regular arrays, this includes the boundary chunk:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z_regular = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(100, 80),
+    chunks=(30, 40),
+    dtype='int32',
+)
+print(z_regular.write_chunk_sizes)
+```
+
+Note that the `.chunks` property is only available for regular chunk grids. For
+rectilinear arrays, use `.write_chunk_sizes` (or `.read_chunk_sizes`) instead.
+
+### Resizing and appending
+
+Rectilinear arrays can be resized. When growing past the current edge sum, a
+new chunk is appended covering the additional extent. When shrinking, the chunk
+edges are preserved and the extent is re-bound (chunks beyond the new extent
+simply become inactive):
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(30,),
+    chunks=[[10, 20]],
+    dtype='float64',
+)
+z[:] = np.arange(30, dtype='float64')
+print(f"Before resize: chunk_sizes={z.write_chunk_sizes}")
+z.resize((50,))
+print(f"After resize:  chunk_sizes={z.write_chunk_sizes}")
+```
+
+The `append` method also works with rectilinear arrays:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z.append(np.arange(10, dtype='float64'))
+print(f"After append:  shape={z.shape}, chunk_sizes={z.write_chunk_sizes}")
+```
+
+### Compressors and filters
+
+Rectilinear arrays work with all codecs — compressors, filters, and checksums.
+Since each chunk may have a different size, the codec pipeline processes each
+chunk independently:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(60, 100),
+    chunks=[[10, 20, 30], [50, 50]],
+    dtype='float64',
+    filters=[zarr.codecs.TransposeCodec(order=(1, 0))],
+    compressors=[zarr.codecs.BloscCodec(cname='zstd', clevel=3)],
+)
+z[:] = np.arange(60 * 100, dtype='float64').reshape(60, 100)
+np.testing.assert_array_equal(z[:], np.arange(60 * 100, dtype='float64').reshape(60, 100))
+print("Roundtrip OK")
+```
+
+### Rectilinear shard boundaries
+
+Rectilinear chunk grids can also be used for shard boundaries when combined
+with sharding. In this case, the outer grid (shards) is rectilinear while the
+inner chunks remain regular. Each shard dimension must be divisible by the
+corresponding inner chunk size:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(120, 100),
+    chunks=(10, 10),
+    shards=[[60, 40, 20], [50, 50]],
+    dtype='int32',
+)
+z[:] = np.arange(120 * 100, dtype='int32').reshape(120, 100)
+print(z[50:70, 40:60])
+```
+
+Note that rectilinear inner chunks with sharding are not supported — only the
+shard boundaries can be rectilinear.
+
+### Metadata format
+
+Rectilinear chunk grid metadata uses run-length encoding (RLE) for compact
+serialization. When reading metadata, both bare integers and `[value, count]`
+pairs are accepted:
+
+- `[10, 20, 30]` — three chunks with explicit sizes
+- `[[10, 3]]` — three chunks of size 10 (RLE shorthand)
+- `[[10, 3], 5]` — three chunks of size 10, then one chunk of size 5
+
+When writing, Zarr automatically compresses repeated values into RLE format.
+
 ## Missing features in 3.0
 
 The following features have not been ported to 3.0 yet.

docs/user-guide/arrays.md

@@ -30,6 +30,7 @@ Configuration options include the following:
 - Default Zarr format `default_zarr_version`
 - Default array order in memory `array.order`
 - Whether empty chunks are written to storage `array.write_empty_chunks`
+- Enable experimental rectilinear chunks `array.rectilinear_chunks`
 - Whether missing chunks are filled with the array's fill value on read `array.read_missing_chunks` (default `True`). Set to `False` to raise a [`ChunkNotFoundError`][zarr.errors.ChunkNotFoundError] instead.
 - Async and threading options, e.g. `async.concurrency` and `threading.max_workers`
 - Selections of implementations of codecs, codec pipelines and buffers