From ab8d4a180c8079db757e302ba64efe71f78f4d86 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Fri, 18 Aug 2023 04:32:18 +0530 Subject: [PATCH 1/4] Add conventions for ZEP0004 review Co-authored-by: Ryan Abernathey --- .gitignore | 2 ++ docs/conventions/conventions.rst | 21 +++++++++++++++++++++ docs/specs.rst | 6 ++++++ 3 files changed, 29 insertions(+) create mode 100644 docs/conventions/conventions.rst diff --git a/.gitignore b/.gitignore index 7992fe03..d0690f8d 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,5 @@ docs/_build # pycharm .idea + +.DS_Store diff --git a/docs/conventions/conventions.rst b/docs/conventions/conventions.rst new file mode 100644 index 00000000..b374be0f --- /dev/null +++ b/docs/conventions/conventions.rst @@ -0,0 +1,21 @@ +=========== +Conventions +=========== + +This page lists the Zarr conventions. The proposal to formalise the conventions is introduced in `ZEP0004 `_. + +Some of the widely used conventions are: + +- `GDAL `_ +- `OME-NGFF `_ +- `NCZarr `_ +- `Xarray `_ + +Any new conventions accepted by the `ZEP `_ process will be listed here. + +.. toctree:: + :glob: + :maxdepth: 1 + :titlesonly: + :caption: Contents: + diff --git a/docs/specs.rst b/docs/specs.rst index 00e7f1ed..81b6c47d 100644 --- a/docs/specs.rst +++ b/docs/specs.rst @@ -12,6 +12,12 @@ Specifications v3/stores v3/array-storage-transformers +.. toctree:: + :maxdepth: 1 + :caption: Conventions + + Conventions + .. toctree:: :maxdepth: 1 :caption: v2 From 86acb1ba2ab16d2f442e6ca90ee229e00858abcb Mon Sep 17 00:00:00 2001 From: Ryan Abernathey Date: Sun, 4 Feb 2024 15:04:32 -0500 Subject: [PATCH 2/4] added some text to the main conventions page --- docs/conventions/conventions.rst | 120 ++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/docs/conventions/conventions.rst b/docs/conventions/conventions.rst index b374be0f..bc557ac9 100644 --- a/docs/conventions/conventions.rst +++ b/docs/conventions/conventions.rst @@ -2,7 +2,125 @@ Conventions =========== -This page lists the Zarr conventions. The proposal to formalise the conventions is introduced in `ZEP0004 `_. +Why Conventions? +~~~~~~~~~~~~~~~~ + +Zarr Conventions provide a mechanism to standardize metadata and layout of Zarr data +in order to meet domain-specific application needs without changes to the +core Zarr data model and specification, and without specification extensions. + +Conventions must fit completely within the Zarr data / metadata model of groups, arrays, and attributes thereof, requiring +no changes or extension to the specification. +A Zarr implementation itself should not even be aware of the existence of the convention. +The line between a convention and an extension may be blurry in some cases. +The key distinction lies in the implementation: the responsibility for interpreting a *convention* relies completely with downstream, +domain-specific software, while an *extension* must be handled by the Zarr implementation itself. +A good rule of thumb is that a user should be able to safely ignore the convention and still be able to interact with the data via the core Zarr library, +even if some domain-specific context or functionality is missing. +If the data are completely meaningless or unintelligible without the convention, then it should be an extension instead. + +Conventions can also help users switch between different storage libraries more flexibly. +Since Zarr and HDF5 implement nearly identical data models, a single convention can be applied to both formats. +This allows downstream software to maintain better separation of concerns between storage and domain-specific logic. + +Conventions are modular and composable. A single group or array can conform to multiple conventions. + + +Describing Conventions +~~~~~~~~~~~~~~~~~~~~~~ + +Conventions Document +-------------------- + +Conventions are described by a *convention document*. +TODO: say more about the structure and format of this document + +Explicit Conventions +-------------------- + +The preferred way of identifying the presence of a convention in a Zarr group or array is via the attribute `zarr_conventions`. +This attribute must be an array of strings; each string is an identifier for the convention. +Multiple conventions may be present. + +For example, a group metadata JSON document with conventions present might look like this + +.. code-block:: json + + { + "zarr_format": 3, + "node_type": "group", + "attributes": { + "zarr_conventions": ["units-v1", "foo], + } + } + +where `units-v1` and `bar` are the convention identifiers. + + +Legacy Conventions +------------------ + +A legacy convention is a convention already in use that predates this ZEP. +Data conforming to legacy conventions will not have the `zarr_conventions` attribute. +The conventions document must therefore specify how software can identify the presence of the convention through a series of rules or tests. + +For those comfortable with the terminology, legacy conventions can be thought of as a "conformance class" and a corresponding "conformance test". + +Namespacing +----------- + +Conventions may choose to store their attributes on a specific namespace. +This ZEP does not specify how namespacing works; that is up to the convention. +For example, the namespace may be specified as a prefix on attributes, e.g. + +.. code-block:: json + + { + "attributes": {"units-v1:units": "m^2"} + } + + +or via a nested JSON object, e.g. + +.. code-block:: json + + { + "attributes": {"units-v1": {"units: "m^2"}} + } + +The use of namespacing is optional and is up to the convention to decide. + + +Proposing Conventions +~~~~~~~~~~~~~~~~~~~~~ + +New conventions are proposed via a pull-request to the `zarr-specs` repo which adds a new conventions document. +If the convention is already documented elsewhere, the convention document can just contain a reference to the external documentation. +The author of the PR is expected to convene the relevant domain community to review and discuss the ZEP. +This includes posting a link to the PR on relevant forums, mailing lists, and social-media platforms. + +The goal of the discussion is to reach a _consensus_ among the domain community regarding the convention. +The Zarr steering council, together with the PR author, will determine if a consensus has been reached, at which point the PR +can be merged and the convention published on the website. +If a consensus cannot be reached, the steering council may still decide to publish the convention, accompanied by a +disclaimer that it is not a consensus, and noting any objections that were raised during the discussion. + +It is also possible that multiple, competing conventions exist in the same domain. While not ideal, it's not up to +the Zarr community to resolve such domain-specific debates. +These conventions should still be documented in a central location, which hopefully helps move towards alignment. + +Conventions should be versioned using incremental integers, starting from 1. +Or, if the community already has an existing versioning system for their convention, that can be used instead (e.g. CF conventions). +The community is free to update their convention via a pull request using the same consensus process described above. +The conventions document should include a changelog. +Details of how to manage changes and backwards compatibility are left to the domain community. + + +Existing Conventions +~~~~~~~~~~~~~~~~~~~~ + + +This page lists the Zarr conventions. The proposal to formalize the conventions is introduced in `ZEP0004 `_. Some of the widely used conventions are: From e7738bc12d6ef6212432eda02b53347199d5745c Mon Sep 17 00:00:00 2001 From: Ryan Abernathey Date: Sun, 4 Feb 2024 16:03:52 -0500 Subject: [PATCH 3/4] add xarray convention and reorganize --- .../{conventions.rst => index.rst} | 2 + docs/conventions/xarray.rst | 99 +++++++++++++++++++ docs/index.rst | 2 +- docs/specs.rst | 2 +- 4 files changed, 103 insertions(+), 2 deletions(-) rename docs/conventions/{conventions.rst => index.rst} (99%) create mode 100644 docs/conventions/xarray.rst diff --git a/docs/conventions/conventions.rst b/docs/conventions/index.rst similarity index 99% rename from docs/conventions/conventions.rst rename to docs/conventions/index.rst index bc557ac9..7161f638 100644 --- a/docs/conventions/conventions.rst +++ b/docs/conventions/index.rst @@ -137,3 +137,5 @@ Any new conventions accepted by the `ZEP `_ +in the Xarray docs. + + +Description +----------- + +`Xarray`_ is a Python library for working with labeled multi-dimensional arrays. +Xarray was originally designed to read only `NetCDF`_ files, but has since added support for +other formats. +In implementing support for the `Zarr `_ storage format, Xarray developers +made some *ad hoc* choices about how to store NetCDF-style data in Zarr. +These choices have become a de facto convention for mapping the Zarr data model to the +`NetCDF data model `_ + +First, Xarray can only read and write Zarr groups. There is currently no support +for reading / writing individual Zarr arrays. Zarr groups are mapped to +Xarray ``Dataset`` objects, which correspond to NetCDF-4 / HDF5 groups. + +Second, from Xarray's point of view, the key difference between +NetCDF and Zarr is that all NetCDF arrays have *dimension names* while Zarr +arrays do not. Therefore, in order to store NetCDF data in Zarr, Xarray must +somehow encode and decode the name of each array's dimensions. + +To accomplish this, Xarray developers decided to define a special Zarr array +attribute: ``_ARRAY_DIMENSIONS``. The value of this attribute is a list of +dimension names (strings), for example ``["time", "lon", "lat"]``. When writing +data to Zarr, Xarray sets this attribute on all variables based on the variable +dimensions. When reading a Zarr group, Xarray looks for this attribute on all +arrays, raising an error if it can't be found. The attribute is used to define +the variable dimension names and then removed from the attributes dictionary +returned to the user. + +Because of these choices, Xarray cannot read arbitrary array data, but only +Zarr data with valid ``_ARRAY_DIMENSIONS`` attributes on each array. + +After decoding the ``_ARRAY_DIMENSIONS`` attribute and assigning the variable +dimensions, Xarray proceeds to [optionally] decode each variable using its +standard `CF Conventions`_ decoding machinery used for NetCDF data. + +Finally, it's worth noting that Xarray writes (and attempts to read) +"consolidated metadata" by default (the ``.zmetadata`` file), which is another +non-standard Zarr extension, albeit one implemented upstream in Zarr-Python. + +.. _Xarray: http://xarray.dev +.. _NetCDF: https://www.unidata.ucar.edu/software/netcdf +.. _CF Conventions: http://cfconventions.org + + +Identifying the Presence of this Convention +------------------------------------------- + +In implementing this convention, Xarray developers made the unfortunate choice of not +including any explicit identifier in the Zarr metadata. Therefore, the only way to +determine whether the convention is being used is to attempt to examine contents of the +Zarr dataset and look for the following properties: + +* A single flat group containing one or more arrays +* The presence of the ``_ARRAY_DIMENSIONS`` attribute on each array, whose contents are + a list of dimension names (strings) +* If the dimension name corresponds to another array name within the group, that array is + assumed to be a dimension coordinate. Dimension coordinates arrays must be 1D + and have the same length as the corresponding dimension. + + +CF Conventions +-------------- + +It is common for data stored in Zarr using the Xarray convention to also follow +the `Climate and Forecast (CF) Metadata Conventions `_. + +A high-level description of these conventions, quoted from the CF Documentation is as follows: + + The NetCDF library [NetCDF] is designed to read and write data that has been structured + according to well-defined rules and is easily ported across various computer platforms. + The netCDF interface enables but does not require the creation of self-describing datasets. + The purpose of the CF conventions is to require conforming datasets to contain sufficient + metadata that they are self-describing in the sense that each variable in the file has an + associated description of what it represents, including physical units if appropriate, + and that each value can be located in space (relative to earth-based coordinates) and time. + +The CF Conventions are massive and cover a wide range of topics. Readers should consult the +`CF Conventions`_ documentation for more information. \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 50182692..3ed6a964 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -7,7 +7,7 @@ A good starting point is the :ref:`zarr-core-specification-v3.0`. .. toctree:: Home - specs + conventions ZEPs Implementations diff --git a/docs/specs.rst b/docs/specs.rst index 81b6c47d..9cdf538f 100644 --- a/docs/specs.rst +++ b/docs/specs.rst @@ -16,7 +16,7 @@ Specifications :maxdepth: 1 :caption: Conventions - Conventions + Conventions .. toctree:: :maxdepth: 1 From 464ce14df8840a73942c7414b0c6008d759e4fc0 Mon Sep 17 00:00:00 2001 From: Sanket Verma Date: Tue, 26 Mar 2024 21:07:13 +0100 Subject: [PATCH 4/4] Update docs/conventions/index.rst Co-authored-by: Yaroslav Halchenko --- docs/conventions/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conventions/index.rst b/docs/conventions/index.rst index 7161f638..6149fc1b 100644 --- a/docs/conventions/index.rst +++ b/docs/conventions/index.rst @@ -50,7 +50,7 @@ For example, a group metadata JSON document with conventions present might look "zarr_format": 3, "node_type": "group", "attributes": { - "zarr_conventions": ["units-v1", "foo], + "zarr_conventions": ["units-v1", "foo"], } }