Merge branch 'main' into fix-obstore-listdir

Author: antoinegaston

.github/workflows/lint.yml

@@ -0,0 +1,25 @@
+name: Lint
+
+on:
+  push:
+    branches: [main, 3.1.x]
+  pull_request:
+    branches: [main, 3.1.x]
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - name: Install prek
+        run: uv tool install prek
+      - name: Run prek
+        run: prek run --all-files

.pre-commit-config.yaml

@@ -1,8 +1,9 @@
 ci:
   autoupdate_commit_msg: "chore: update pre-commit hooks"
   autoupdate_schedule: "monthly"
-  autofix_commit_msg: "style: pre-commit fixes"
   autofix_prs: false
+  skip: []  # pre-commit.ci only checks for updates, prek runs hooks locally
+
 default_stages: [pre-commit, pre-push]
 
 default_language_version:

changes/3658.misc.md

@@ -0,0 +1 @@
+Switch from `pre-commit` to [`prek`](https://github.com/j178/prek) for pre-commit checks.

docs/contributing.md

@@ -109,26 +109,50 @@ All tests are automatically run via GitHub Actions for every pull request and mu
 
 > **Note:** Previous versions of Zarr-Python made extensive use of doctests. These tests were not maintained during the 3.0 refactor but may be brought back in the future. See issue #2614 for more details.
 
-### Code standards - using pre-commit
+### Code standards - using prek
 
 All code must conform to the PEP8 standard. Regarding line length, lines up to 100 characters are allowed, although please try to keep under 90 wherever possible.
 
-`Zarr` uses a set of `pre-commit` hooks and the `pre-commit` bot to format, type-check, and prettify the codebase. `pre-commit` can be installed locally by running:
+`Zarr` uses a set of git hooks managed by [`prek`](https://github.com/j178/prek), a fast, Rust-based pre-commit hook manager that is fully compatible with `.pre-commit-config.yaml` files. `prek` can be installed locally by running:
 
 ```bash
-python -m pip install pre-commit
+uv tool install prek
+```
+
+or:
+
+```bash
+pip install prek
 ```
 
 The hooks can be installed locally by running:
 
 ```bash
-pre-commit install
+prek install
+```
+
+This would run the checks every time a commit is created locally. The checks will by default only run on the files modified by a commit, but the checks can be triggered for all the files by running:
+
+```bash
+prek run --all-files
+```
+
+You can also run hooks only for files in a specific directory:
+
+```bash
+prek run --directory src/zarr
+```
+
+Or run hooks for files changed in the last commit:
+
+```bash
+prek run --last-commit
 ```
 
-This would run the checks every time a commit is created locally. These checks will also run on every commit pushed to an open PR, resulting in some automatic styling fixes by the `pre-commit` bot. The checks will by default only run on the files modified by a commit, but the checks can be triggered for all the files by running:
+To list all available hooks:
 
 ```bash
-pre-commit run --all-files
+prek list
 ```
 
 If you would like to skip the failing checks and push the code for further discussion, use the `--no-verify` option with `git commit`.

pyproject.toml

@@ -30,7 +30,7 @@ maintainers = [
     { name = "Deepak Cherian" }
 ]
 requires-python = ">=3.11"
-# If you add a new dependency here, please also add it to .pre-commit-config.yml
+# If you add a new dependency here, please also add it to .pre-commit-config.yaml
 dependencies = [
     'packaging>=22.0',
     'numpy>=2.0',
@@ -428,6 +428,7 @@ markers = [
 ignore = [
 	"PC111",  # fix Python code in documentation - enable later
 	"PC180",  # for JavaScript - not interested
+	"PC902",  # pre-commit.ci custom autofix message - not using autofix
 ]
 
 [tool.numpydoc_validation]

src/zarr/abc/store.py

@@ -1,11 +1,14 @@
 from __future__ import annotations
 
+import asyncio
+import json
 from abc import ABC, abstractmethod
-from asyncio import gather
 from dataclasses import dataclass
 from itertools import starmap
 from typing import TYPE_CHECKING, Literal, Protocol, runtime_checkable
 
+from zarr.core.sync import sync
+
 if TYPE_CHECKING:
     from collections.abc import AsyncGenerator, AsyncIterator, Iterable
     from types import TracebackType
@@ -206,6 +209,211 @@ async def get(
         """
         ...
 
+    async def _get_bytes(
+        self, key: str, *, prototype: BufferPrototype, byte_range: ByteRequest | None = None
+    ) -> bytes:
+        """
+        Retrieve raw bytes from the store asynchronously.
+
+        This is a convenience method that wraps ``get()`` and converts the result
+        to bytes. Use this when you need the raw byte content of a stored value.
+
+        Parameters
+        ----------
+        key : str
+            The key identifying the data to retrieve.
+        prototype : BufferPrototype
+            The buffer prototype to use for reading the data.
+        byte_range : ByteRequest, optional
+            If specified, only retrieve a portion of the stored data.
+            Can be a ``RangeByteRequest``, ``OffsetByteRequest``, or ``SuffixByteRequest``.
+
+        Returns
+        -------
+        bytes
+            The raw bytes stored at the given key.
+
+        Raises
+        ------
+        FileNotFoundError
+            If the key does not exist in the store.
+
+        See Also
+        --------
+        get : Lower-level method that returns a Buffer object.
+        get_bytes : Synchronous version of this method.
+        get_json : Asynchronous method for retrieving and parsing JSON data.
+
+        Examples
+        --------
+        >>> store = await MemoryStore.open()
+        >>> await store.set("data", Buffer.from_bytes(b"hello world"))
+        >>> data = await store.get_bytes("data", prototype=default_buffer_prototype())
+        >>> print(data)
+        b'hello world'
+        """
+        buffer = await self.get(key, prototype, byte_range)
+        if buffer is None:
+            raise FileNotFoundError(key)
+        return buffer.to_bytes()
+
+    def _get_bytes_sync(
+        self, key: str = "", *, prototype: BufferPrototype, byte_range: ByteRequest | None = None
+    ) -> bytes:
+        """
+        Retrieve raw bytes from the store synchronously.
+
+        This is a synchronous wrapper around ``get_bytes()``. It should only
+        be called from non-async code. For async contexts, use ``get_bytes()``
+        instead.
+
+        Parameters
+        ----------
+        key : str, optional
+            The key identifying the data to retrieve. Defaults to an empty string.
+        prototype : BufferPrototype
+            The buffer prototype to use for reading the data.
+        byte_range : ByteRequest, optional
+            If specified, only retrieve a portion of the stored data.
+            Can be a ``RangeByteRequest``, ``OffsetByteRequest``, or ``SuffixByteRequest``.
+
+        Returns
+        -------
+        bytes
+            The raw bytes stored at the given key.
+
+        Raises
+        ------
+        FileNotFoundError
+            If the key does not exist in the store.
+
+        Warnings
+        --------
+        Do not call this method from async functions. Use ``get_bytes()`` instead
+        to avoid blocking the event loop.
+
+        See Also
+        --------
+        get_bytes : Asynchronous version of this method.
+        get_json_sync : Synchronous method for retrieving and parsing JSON data.
+
+        Examples
+        --------
+        >>> store = MemoryStore()
+        >>> await store.set("data", Buffer.from_bytes(b"hello world"))
+        >>> data = store.get_bytes_sync("data", prototype=default_buffer_prototype())
+        >>> print(data)
+        b'hello world'
+        """
+
+        return sync(self._get_bytes(key, prototype=prototype, byte_range=byte_range))
+
+    async def _get_json(
+        self, key: str, *, prototype: BufferPrototype, byte_range: ByteRequest | None = None
+    ) -> Any:
+        """
+        Retrieve and parse JSON data from the store asynchronously.
+
+        This is a convenience method that retrieves bytes from the store and
+        parses them as JSON.
+
+        Parameters
+        ----------
+        key : str
+            The key identifying the JSON data to retrieve.
+        prototype : BufferPrototype
+            The buffer prototype to use for reading the data.
+        byte_range : ByteRequest, optional
+            If specified, only retrieve a portion of the stored data.
+            Can be a ``RangeByteRequest``, ``OffsetByteRequest``, or ``SuffixByteRequest``.
+            Note: Using byte ranges with JSON may result in invalid JSON.
+
+        Returns
+        -------
+        Any
+            The parsed JSON data. This follows the behavior of ``json.loads()`` and
+            can be any JSON-serializable type: dict, list, str, int, float, bool, or None.
+
+        Raises
+        ------
+        FileNotFoundError
+            If the key does not exist in the store.
+        json.JSONDecodeError
+            If the stored data is not valid JSON.
+
+        See Also
+        --------
+        get_bytes : Method for retrieving raw bytes.
+        get_json_sync : Synchronous version of this method.
+
+        Examples
+        --------
+        >>> store = await MemoryStore.open()
+        >>> metadata = {"zarr_format": 3, "node_type": "array"}
+        >>> await store.set("zarr.json", Buffer.from_bytes(json.dumps(metadata).encode()))
+        >>> data = await store.get_json("zarr.json", prototype=default_buffer_prototype())
+        >>> print(data)
+        {'zarr_format': 3, 'node_type': 'array'}
+        """
+
+        return json.loads(await self._get_bytes(key, prototype=prototype, byte_range=byte_range))
+
+    def _get_json_sync(
+        self, key: str = "", *, prototype: BufferPrototype, byte_range: ByteRequest | None = None
+    ) -> Any:
+        """
+        Retrieve and parse JSON data from the store synchronously.
+
+        This is a synchronous wrapper around ``get_json()``. It should only
+        be called from non-async code. For async contexts, use ``get_json()``
+        instead.
+
+        Parameters
+        ----------
+        key : str, optional
+            The key identifying the JSON data to retrieve. Defaults to an empty string.
+        prototype : BufferPrototype
+            The buffer prototype to use for reading the data.
+        byte_range : ByteRequest, optional
+            If specified, only retrieve a portion of the stored data.
+            Can be a ``RangeByteRequest``, ``OffsetByteRequest``, or ``SuffixByteRequest``.
+            Note: Using byte ranges with JSON may result in invalid JSON.
+
+        Returns
+        -------
+        Any
+            The parsed JSON data. This follows the behavior of ``json.loads()`` and
+            can be any JSON-serializable type: dict, list, str, int, float, bool, or None.
+
+        Raises
+        ------
+        FileNotFoundError
+            If the key does not exist in the store.
+        json.JSONDecodeError
+            If the stored data is not valid JSON.
+
+        Warnings
+        --------
+        Do not call this method from async functions. Use ``get_json()`` instead
+        to avoid blocking the event loop.
+
+        See Also
+        --------
+        get_json : Asynchronous version of this method.
+        get_bytes_sync : Synchronous method for retrieving raw bytes without parsing.
+
+        Examples
+        --------
+        >>> store = MemoryStore()
+        >>> metadata = {"zarr_format": 3, "node_type": "array"}
+        >>> store.set("zarr.json", Buffer.from_bytes(json.dumps(metadata).encode()))
+        >>> data = store.get_json_sync("zarr.json", prototype=default_buffer_prototype())
+        >>> print(data)
+        {'zarr_format': 3, 'node_type': 'array'}
+        """
+
+        return sync(self._get_json(key, prototype=prototype, byte_range=byte_range))
+
     @abstractmethod
     async def get_partial_values(
         self,
@@ -278,7 +486,7 @@ async def _set_many(self, values: Iterable[tuple[str, Buffer]]) -> None:
         """
         Insert multiple (key, value) pairs into storage.
         """
-        await gather(*starmap(self.set, values))
+        await asyncio.gather(*starmap(self.set, values))
 
     @property
     def supports_consolidated_metadata(self) -> bool: