Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ Depends on some details of your data.
VirtualiZarr works by mapping your data to the zarr data model from whatever data model is used by the format it was saved in.
This means that if your data contains anything that cannot be represented within the zarr data model, it cannot be virtualized.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This means that if your data contains anything that cannot be represented within the zarr data model, it cannot be virtualized.
This means that if your data contains anything that cannot be represented within the zarr data model, it cannot be virtualized. The following restrictions influence whether you can virtualize a data file.

I think a pre-ample to the list would help


- **Format chunks span contiguous byte ranges** - It's only possible to efficiently access individual chunks of data inside blobs in object storage if each chunk can be fetched via a single HTTP range request, which requires each chunk to occupy a contiguous localized series of bytes within the file layout. Well-designed formats such as netCDF and GRIB have this property, but other formats such as CSV do not. Note also this means that any additional processing which scrambles the byte locations will prevent virtualization - a single netCDF file is virtualizable, but a zipped or gzipped netCDF file is not!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **Format chunks span contiguous byte ranges** - It's only possible to efficiently access individual chunks of data inside blobs in object storage if each chunk can be fetched via a single HTTP range request, which requires each chunk to occupy a contiguous localized series of bytes within the file layout. Well-designed formats such as netCDF and GRIB have this property, but other formats such as CSV do not. Note also this means that any additional processing which scrambles the byte locations will prevent virtualization - a single netCDF file is virtualizable, but a zipped or gzipped netCDF file is not!
- **File must contain chunks of data, where each chunk spans a contiguous segment of the file** - For virtualization to work, each chunk must occupy a contiguous localized series of bytes within the file layout, so that the chunks can be fetched via a single HTTP range request. Well-designed formats such as netCDF and GRIB have this property, but other formats such as CSV do not. Note also this means that any additional processing which scrambles the byte locations will prevent virtualization - a single netCDF file is virtualizable, but a zipped or gzipped netCDF file is not!


When virtualizing multi-file datasets, it is sometimes the case that it is possible to virtualize one file, but not possible to virtualize all the files together as part of one datacube, because of inconsistencies _between_ the files. The following restrictions apply across every file in the datacube you wish to create!

- **Arrays** - The zarr data model is one of a set of arrays, so your data must be decodable as a set of arrays, each of which will map to single zarr array (via the `ManifestArray` class). If your data cannot be directly mapped to an array, for example because it has inconsistent lengths along a common dimension (known as "ragged data"), then it cannot be virtualized.
Expand Down
Loading