Splitting large data chunks?

[jbusecke]

From reading the man page I get the impression that rechunking will always only combine chunks to increase chunksize? Or is there a way to split large chunks?

In my opinion the community has recently somewhat converged on a chunk size of 1-10MB for cloud optimized access. If this tool could be used to provide a more consistent chunksize (not only bounded below, but also above) this could be very valuable for efforts to virtualize the data ( I am anticipating to work on this with Carbonplan).

Happy to dig into the code and add features, but just wanted to have a discussion first here.

Thanks for all the work.

[davidhassell]

Hi @jbusecke,

Currently there is no option to split large chunks up, based on the assumption that if someone (in particular a CMIP modelling group) already has larger chunks, then they won't want this tool to meddle with them. I.e. if the original chunk size is greater than the -d size chunk size (default 4MiB), then no data rechunking is done.

Do you have use cases of needing to break up "massive" chunks?

However, I think that adding that as an optional command line option would be fine. Something like -s for split, which would allow rechunking of larger chunks down to the -d size chunk size, but would do nothing when the data chunks are originally smaller than the -d size (in which we're growing the chunks, not chopping them up).

We could also think about specify a range that ensured that the data chunks where inside, either coming up from below, or down from above. Again, optional.

I think these feature have to be optional at this particular version of the tool at this time, because the scope of the tool has been discussed in the CMIP7 community, and it has been signed-off by the CMIP governance for a very clear and specific default functionality, which is not possible to change this late in the process.

However, that doesn't stop us putting in more "advanced" features and letting people know that they can use them if they like, but they don't have to (which is good, because some users will not want to to have to think about the implications of chunking strategies, so the default behaviour has to be acceptable).

Looking forward to talking about this further,
David

[jbusecke]

Our usecase is to build virtual zarr stores (implemented by icechunk repositories) that point directly to the ESGF https links in order to:

• Expose datacubes instead of individual files (improved user experience, particularly for users outside of large institutions)
• Integrate easier into cloud-native workflows (the virtual store is readable just like another zarr store).

Different applications require different chunksizes obviously but for our purposes we will likely use icechunk and the ideal chunksize is stated as "3-15 MB". This blogpost by Earthmover shows declining throughput at chunksizes >50MB.