Skip to content

Add experimental support for Zarr format 3 #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
tomwhite wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add experimental support for Zarr format 3 #42

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 14 additions & 10 deletions vcf_zarr_spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
This document is a technical specification for VCF Zarr, a means of encoding VCF data in chunked-columnar form using the Zarr format.

This specification depends on definitions and terminology from [The Variant Call Format Specification, VCFv4.3 and BCFv2.2](https://samtools.github.io/hts-specs/VCFv4.3.pdf),
and [Zarr storage specification version 2](https://zarr.readthedocs.io/en/stable/spec/v2.html).
and [Zarr storage specification version 2](https://zarr.readthedocs.io/en/stable/spec/v2.html) or [Zarr core specification version 3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html) [experimental].

@tomwhite tomwhite Sep 25, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marked as "experimental" until we've got experience using implementations of this. (Also, I'm not planning on merging this soon.)


## Compatibility with VCF and BCF

Expand Down Expand Up @@ -42,17 +42,19 @@ Each VCF field is stored in a separate Zarr array. This specification only manda

This document uses a shorthand notation to refer to Zarr data types (dtypes). The following table shows the mapping to VCF types.

| Shorthand | Zarr dtypes | VCF Type |
|-----------|----------------------------------------------------------|-----------|
| `bool` | `\|b1` | Flag |
| `int` | `<i1`, `<i2`, `<i4`, `<i8` or `>i1`, `>i2`, `>i4`, `>i8` | Integer |
| `float` | `<f4`, `<f8` or `>f4`, `>f8` | Float |
| `char` | `\<U1` or `\>U1` | Character |
| `str` | `\|O` | String |
| Shorthand | Zarr 2 data types | Zarr 3 data types | VCF Type |
|-----------|----------------------------------------------------------|-----------------------------------|-----------|
| `bool` | `\|b1` | `bool` | Flag |
| `int` | `<i1`, `<i2`, `<i4`, `<i8` or `>i1`, `>i2`, `>i4`, `>i8` | `int8`, `int16`, `int32`, `int64` | Integer |
| `float` | `<f4`, `<f8` or `>f4`, `>f8` | `float32`, `float64` | Float |
| `char` | `\<U1` or `\>U1` | `string` | Character |

@tomwhite tomwhite Sep 25, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zarr format v3 doesn't use NumPy dtypes any more, and it doesn't have a "char" type, so I've represented VCF Character as a Zarr string. This would work, but would mean that converting VCF Zarr back to VCF would be lossy in the sense that a VCF Character field would be converted back to a VCF String. This may not matter, but it could potentially be fixed by either i) adding a "char" type/fixed-length string type to Zarr, or ii) adding an attribute to VCF Zarr to record the fact that it is logically a single character.

| `str` | `\|O` | `string` | String |

This specification does not mandate a byte order for numeric types: little-endian (e.g. `<i4`) or big-endian (`>i4`) are both permitted.

The `str` dtype is used to represent [variable-length strings](https://zarr.readthedocs.io/en/stable/tutorial.html#string-arrays). In this case a Zarr array filter with and `id` of `vlen-utf8` must be specified for the array.
*[Zarr 2 only]* The `str` dtype is used to represent [variable-length strings](https://zarr.readthedocs.io/en/stable/tutorial.html#string-arrays). In this case a Zarr array filter with and `id` of `vlen-utf8` must be specified for the array.

*[Zarr 3 only]* The `str` dtype is used to represent [variable-length strings](https://github.com/zarr-developers/zarr-extensions/tree/main/data-types/string). In this case a Zarr array [`vlen-utf8`](https://github.com/zarr-developers/zarr-extensions/blob/main/codecs/vlen-utf8/README.md) codec must be specified for the array.

### Missing and fill values

Expand All @@ -73,7 +75,9 @@ An array called `<name>` may have an accompanying array called `<name>_mask` wit

### Array dimension names

Following [Xarray conventions](http://xarray.pydata.org/en/stable/internals/zarr-encoding-spec.html), each Zarr array has an attribute `_ARRAY_DIMENSIONS`, which is a list of strings naming the dimensions.
*[Zarr 2 only]* Following [Xarray conventions](http://xarray.pydata.org/en/stable/internals/zarr-encoding-spec.html), each Zarr array has an attribute `_ARRAY_DIMENSIONS`, which is a list of strings naming the dimensions.

*[Zarr 3 only]* The Zarr array metadata must include `dimension_names`, which is a list of strings naming the dimensions.

The reserved dimension names and their sizes are listed in the following table, along with the corresponding VCF Number value, if applicable.

Expand Down