Feature/zarr

[KirmTwinty]

Zarr support

Zarr is a modern hierarchical file format, heavily inspired by HDF5, but specifically designed for:

storage using distributed systems like cloud object stores, and to enable efficient I/O for parallel computing applications.

A key advantage of Zarr is that its API closely resembles that of HDF5, which simplifies the process of porting H5Grove to support Zarr.

What are your thoughts on this?
If your interested in this feature, I can work on this PR :)

Features

• Zarr file format support through zarr python library
• Extend TypeVar to include other generic types, defined in custom_types.py
• New FileTypeEnum enumeration to distinguish Zarr and HDF5 formats
• New ZarrTypeWrapper class

[loichuder]

Hello @KirmTwinty and thanks for the contribution !

We discussed among ourselves and thought that we would prefer h5grove to only deal with HDF5.

Let me explain: when we designed the project, the idea was to do a simple project to write utilities to serve HDF5 that deal with all the common problems that other HDF5 servers came across: serialization, link resolution and so on. We provided an API as an example of the use of these utilities.

But the API is already an abstraction: an alternative to your PR (that implements the zarr serving) could be to have a new project that provides the same API but serves zarr files instead. As you mentioned, HDF5 and Zarr share some concepts, so this new project could reuse some utilities of h5grove so that it would not be completely decoupled.

It really depends on your usecase and we are really interested to hear it and help you out. So, we would be happy if you could drop us a email at h5web@esrf.fr so that we can discuss the Zarr question further together !