feat: rectilinear chunks in Zarr backend by maxrjones · Pull Request #11279 · pydata/xarray

maxrjones · 2026-04-02T16:49:05Z

Description

This PR accompanies zarr-developers/zarr-python#3802, adding support for rectilinear zarr chunks in Xarray.

The user-facing difference between this PR and zarr-developers/zarr-python#3369 / #10880 is that rectilinear chunks are gated behind zarr.config.set({'array.rectilinear_chunks': True}) (or ZARR_ARRAY__RECTILINEAR_CHUNKS=True), disabled by default. This gives zarr-python developers an opportunity to gracefully finalize the API, which is especially valuable given that rectilinear chunks are the largest feature addition in zarr-python since Zarr V3/sharding.

What changed

_determine_zarr_chunks now passes through variable (non-uniform) chunk sizes when writing to Zarr V3 with the unified ChunkGrid API, instead of raising an error.
Reading correctly reconstructs chunk information from both RegularChunkGrid and RectilinearChunkGrid metadata.
safe_chunks and align_chunks validation is skipped for rectilinear (tuple-of-tuples) chunks, since those checks assume uniform chunk sizes.
Error messages for chunk validation failures now distinguish between Zarr V2 and V3 and point users toward the rectilinear chunks extension.

To-do

expand test coverage for error messages when using V2 or config flag is off, and a multi-dimensional test case
decide whether to continue silently bypassing safe_chunks/align_chunks or add validations
remove upstream version pin

Checklist

Closes #xxxx
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst
New functions/methods are listed in api.rst

AI Disclosure

This PR contains AI-generated content.
- I have tested any AI-generated content in my PR.
- I take responsibility for any AI-generated content in my PR. Tools: Claude Code

headtr1ck · 2026-04-03T19:39:37Z

Is this a duplicate of #10880?

maxrjones · 2026-04-03T19:49:05Z

Is this a duplicate of #10880?

This would supersede #10880. It implements the same feature, but using a different upstream implementation (zarr-developers/zarr-python#3802), which will likely be merged into Zarr-Python in the coming days. zarr-developers/zarr-python#3802 supersedes zarr-developers/zarr-python#3369, which #10880 was built on top.

keewis

I'll need to look into this a bit more, but for now:

is skipped for rectilinear (tuple-of-tuples) chunks since those checks assume uniform chunk sizes.

That's what the current checks do, but their purpose is to support safely appending data without write conflicts between execution workers (dask / cubed / etc). Do we maybe need different checks that verify that zarr chunks do not overlap with multiple execution chunks?

xarray/backends/zarr.py

Co-authored-by: Justus Magin <keewis@users.noreply.github.com>

dcherian · 2026-04-08T21:29:12Z

pixi.toml

 dask = { git = "https://github.com/dask/dask" }
 distributed = { git = "https://github.com/dask/distributed" }
-zarr = { git = "https://github.com/zarr-developers/zarr-python" }
+zarr = { git = "https://github.com/maxrjones/zarr-python", branch = "poc/unified-chunk-grid" }


Suggested change

zarr = { git = "https://github.com/maxrjones/zarr-python", branch = "poc/unified-chunk-grid" }

Now that it's on main, we can apply the run-upstream label (which i will do now)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

* add `zizmor` to the hooks * set the default permissions to minimum * don't persist credentials * pin `actions/checkout` * pin `xarray-contrib/ci-trigger` * pin `actions/upload-artifact` * pin `actions/download-artifact` * pin `pypa/gh-action-pypi-publish` * pin `actions/setup-python` * pin `prefix-dev/setup-pixi` * pin `codecov/codecov-action` * pin `scientific-python/issue-from-pytest-log-action` * pin `mamba-org/setup-micromamba` * pin `WyriHaximus/github-action-get-previous-tag` * pin `EnricoMi/publish-unit-test-result-action` * pin `actions/labeler` * pin `actions/cache` * actions cooldown for dependabot * avoid potential template injections * broken condition * ignore the `pull_request_target` warning (because `actions/labeler` actually needs it) * ignore zizmor's dangerous-triggers warning for publish-test-results * fetch the `codecov` token from a github environment * correct the pin for `setup-pixi` * split the nightly wheels ci into build and publish jobs * remove the codecov env and ignore the zizmor warning instead * back to the codecov env, but disable deployments * correct the pin for `actions/setup-python` Co-authored-by: Nick Hodgskin <36369090+VeckoTheGecko@users.noreply.github.com> --------- Co-authored-by: Nick Hodgskin <36369090+VeckoTheGecko@users.noreply.github.com>

This PR modifies the few places we relied on a generic `np.timedelta64` dtype to explicitly specify the time resolution: - It removes `NAT_TYPES` and relies instead on checking the `dtype.kind` in `computation.nanops._maybe_null_out`. - It infers the time `unit` using `np.datetime_data` from the input `dtype` to determine the `unit` on the returned `fill_value` in `core.dtypes.maybe_promote`. - It explicitly constructs a zero-valued `np.timedelta64` or `np.datetime64` object for use downstream in `plot.utils._determine_cmap_params`.

Bumps the actions group with 2 updates: [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) and [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish). Updates `prefix-dev/setup-pixi` from 0.9.4 to 0.9.5 - [Release notes](https://github.com/prefix-dev/setup-pixi/releases) - [Commits](prefix-dev/setup-pixi@a0af7a2...1b2de7f) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@ed0c539...cef2210) --- updated-dependencies: - dependency-name: prefix-dev/setup-pixi dependency-version: 0.9.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…a#11282) * remove the accidentally copied pypi publish step * separate test execution from issue creation * upload the artifact only if the tests failed * correct source of the log-file path * debug: print github output [skip-rtd] * typo [skip-rtd] * correct the path to the log file Co-authored-by: Nick Hodgskin <36369090+VeckoTheGecko@users.noreply.github.com> --------- Co-authored-by: Nick Hodgskin <36369090+VeckoTheGecko@users.noreply.github.com>

* use var._root._h5py to get h5py module in h5netcdf backend instead of importing it * fix ros3 tests to use DANDI endpoint and include hdf5 version switch * fix phony_dims for ros3 test * fix import check for ros3 availability * try using property to get around import issue * add whats-new.rst entry

maxrjones added 8 commits March 9, 2026 17:56

Proof of concept: Unified chunk grid in zarr

d2b10d2

Improve preferred chunks

3e27417

Fastforward

cc78540

Fastfoward

2510d31

Merge branch 'main' into poc/unified-zarr-chunk-grid

8a108a8

Support writing variable chunks

9eb62f2

Only test when rectilinear chunks are available

362b5f1

Merge branch 'pydata:main' into poc/unified-zarr-chunk-grid

163abce

github-actions bot added topic-backends topic-zarr Related to zarr storage library io labels Apr 2, 2026

keewis reviewed Apr 3, 2026

View reviewed changes

xarray/backends/zarr.py Outdated Show resolved Hide resolved

xarray/backends/zarr.py Outdated Show resolved Hide resolved

Apply suggestions from code review

bf8d22f

Co-authored-by: Justus Magin <keewis@users.noreply.github.com>

dcherian reviewed Apr 8, 2026

View reviewed changes

dcherian added the run-upstream Run upstream CI label Apr 8, 2026

eendebakpt and others added 7 commits April 8, 2026 17:31

Improve performance of dataset indexing (pydata#11276)

faabee2

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Remove zarr upstream pin

d224c13

github-actions bot added topic-indexing topic-plotting Automation Github bots, testing workflows, release automation labels Apr 8, 2026

Merge branch 'main' into poc/unified-zarr-chunk-grid

f8e4051

maxrjones marked this pull request as draft April 9, 2026 13:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: rectilinear chunks in Zarr backend#11279

feat: rectilinear chunks in Zarr backend#11279
maxrjones wants to merge 17 commits intopydata:mainfrom
maxrjones:poc/unified-zarr-chunk-grid

maxrjones commented Apr 2, 2026

Uh oh!

headtr1ck commented Apr 3, 2026

Uh oh!

maxrjones commented Apr 3, 2026

Uh oh!

keewis left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

dcherian Apr 8, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

maxrjones commented Apr 2, 2026

Description

What changed

To-do

Checklist

AI Disclosure

Uh oh!

headtr1ck commented Apr 3, 2026

Uh oh!

maxrjones commented Apr 3, 2026

Uh oh!

keewis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dcherian Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

keewis left a comment •

edited

Loading

dcherian Apr 8, 2026 •

edited

Loading