From reading the man page here:
All rechunked variables are de-interlaced with the HDF5 shuffle filter
(which significantly improves compression) before being compressed with
zlib (see the -z option), and also have the Fletcher32 HDF5 checksum
algorithm activated.
Does this mean if a file with large chunks is provided (lets say 200MB), nothing will be done to the compression? Or am I misunderstanding things here?
I was just talking to folks in the QC/QA working group, and mentioned that consistent compression across dataset files is a hard requirement for our efforts to produce virtual zarr references to the data and it seems to me if the application of this tool is required for publication it would present an excellent opportunity to harmonize compression across all files.
From reading the man page here:
Does this mean if a file with large chunks is provided (lets say 200MB), nothing will be done to the compression? Or am I misunderstanding things here?
I was just talking to folks in the QC/QA working group, and mentioned that consistent compression across dataset files is a hard requirement for our efforts to produce virtual zarr references to the data and it seems to me if the application of this tool is required for publication it would present an excellent opportunity to harmonize compression across all files.