Example metadata

[TomAugspurger]

Similar to #36 but perhaps simpler: are there any example JSON documents available showing the required and optional metadata fields? I had a few questions, which an example would answer pretty quickly:

1. Do the required metadata fields go directly in the top level of the attributes? Or is it nested under another key?
2. What should grid_mapping look like?
3. Are there any requirements on the metadata for an Auxiliary Variable?

I have a start here (using Zarr V3 terminology) but I'm not sure if it's correct. It's for a Dataset with a single array (air_temperature) with dimensions (time, lon, lat).

Dataset Root

{
    "zarr_format": 3,
    "node_type": "group"
}

Time

{
    "zarr_format": 3,
    "node_type": "array",
    "shape": [
        365
    ],
    "data_type": "int64",
    "fill_value": 0,
    "chunk_grid": {
        "name": "regular",
        "configuration": {
            "chunk_shape": [
                365
            ]
        }
    },
    "chunk_key_encoding": {
        "name": "default",
        "configuration": {
            "separator": "/"
        }
    },
    "codecs": [
        {
            "name": "bytes",
            "configuration": {
                "endian": "little"
            }
        }
    ],
    "dimension_names": ["time"],
    "attributes": {
        "standard_name": "time"
    }
}

lon

{
    "zarr_format": 3,
    "node_type": "array",
    "shape": [
        360
    ],
    "data_type": "float32",
    "fill_value": 0.0,
    "chunk_grid": {
        "name": "regular",
        "configuration": {
            "chunk_shape": [
                360
            ]
        }
    },
    "chunk_key_encoding": {
        "name": "default",
        "configuration": {
            "separator": "/"
        }
    },
    "codecs": [
        {
            "name": "bytes",
            "configuration": {
                "endian": "little"
            }
        }
    ],
    "dimension_names": ["lon"],
    "attributes": {
        "standard_name": "grid_longitude"
    }
}

lat

{
    "zarr_format": 3,
    "node_type": "array",
    "shape": [
        180
    ],
    "data_type": "float32",
    "fill_value": 0.0,
    "chunk_grid": {
        "name": "regular",
        "configuration": {
            "chunk_shape": [
                180
            ]
        }
    },
    "chunk_key_encoding": {
        "name": "default",
        "configuration": {
            "separator": "/"
        }
    },
    "codecs": [
        {
            "name": "bytes",
            "configuration": {
                "endian": "little"
            }
        }
    ],
    "dimension_names": ["lat"],
    "attributes": {
        "standard_name": "grid_latitude"
    }
}

data

{
    "zarr_format": 3,
    "node_type": "array",
    "shape": [
        365, 360, 180
    ],
    "data_type": "int64",
    "fill_value": 0,
    "chunk_grid": {
        "name": "regular",
        "configuration": {
            "chunk_shape": [
                183, 180, 90
            ]
        }
    },
    "chunk_key_encoding": {
        "name": "default",
        "configuration": {
            "separator": "/"
        }
    },
    "codecs": [
        {
            "name": "bytes",
            "configuration": {
                "endian": "little"
            }
        }
    ],
    "dimension_names": ["time", "lon", "lat"],
    "attributes": {
        "standard_name": "air_temperature"
    }
}

[maxrjones]

Do the required metadata fields go directly in the top level of the attributes? Or is it nested under another key?

Currently, the required metadata fields go directly in the top level of the attributes. But I think they should be nested under another key (e.g., cf for CF attributes, ogc for OGC GeoTIFF/COG attributes, or other keys that could be registered via extensions in the proposal in #67). I think this would simplify implementations because it would be possible to interpret CF metadata without first needing to sort through all arbitrary attributes to identify which are relevant for CF. Using a top-level key also aligns with the trend observed in other Zarr discipline-specific conventions (e.g., OME-Zarr switched to using a top-level key in v0.5.0).

What should grid_mapping look like?
Are there any requirements on the metadata for an Auxiliary Variable?

IIUC correctly these are up in the air. I could answer based on v0.4, which was a partial representation of CF conventions with the addition of a gdal-like string based GeoTransform option and was demonstrated in this notebook. But I think the answer would change if either #65 or #67 are adopted since they both would allow a feature complete version of CF conventions in GeoZarr.

[TomAugspurger]

But I think they should be nested under another key (e.g., cf for CF attributes, ogc for OGC GeoTIFF/COG attributes, or other keys that could be registered via extensions in the proposal in #67).

I think that's worth considering. And if we do that then I agree is shouldn't all be under a geozarr key. GeoZarr isn't the only one who will want to do things with CF, and it'd be a shame to duplicate things.

Thanks for the links to #65 and #67.

[christophenoel]

Closing as stale. No recent activity and likely no longer relevant. Can be reopened if needed.

[TomAugspurger]

Do you have a link to an example metadata document?

[christophenoel]

This is part of the work to be done: finalise a first spec version with examples.

[TomAugspurger]

https://github.com/zarr-developers/geozarr-spec/blob/main/standard/template/sections/clause_9_zarr_encoding_overviews.adoc#metadata-encoding seems to have some metadata for defining multiscales. I'm not sure about others.

[christophenoel]

Apologies for the brief reply – I do not have time to elaborate today, but I will provide more details later.

The first version of GeoZarr does not aim (currently) to define schemas over Zarr. Instead, it will simply establish conventions to unambiguously map NetCDF/GDAL/... data to Zarr (for example, handling compound types, variable-length strings, dimension names, shared dimensions, etc.). As a result, any valid Zarr will also be a valid GeoZarr, and the conventions are not related to the metadata.

However, we plan to introduce an extension (with schemas) to support multiscale data and an extension for affine transformations. A JSON schema will be provided for each required attribute, specifying whether the attribute applies at the group level, array level, or elsewhere.