Multiscale hierarchy structure needs clarification

[emmanuelmathot]

Summary

The specification describes multiscale encoding but doesn't clearly define the exact hierarchical structure and relationship between parent groups and zoom level children.

Current Problem

• Section 9.7.1 mentions "hierarchical layout" but lacks precise structure definition
• Unclear relationship between multiscale root group and zoom level groups
• Missing guidance on metadata inheritance between levels
• Ambiguous about where multiscales metadata should be located

Proposed Solution

1. Define precise hierarchy structure with clear examples
2. Clarify metadata placement and inheritance rules
3. Specify group naming conventions for zoom levels
4. Document parent-child relationships explicitly

Implementation Evidence

The EOPF implementation uses this hierarchy:

/measurements/r10m/          # Parent group with multiscales metadata
├── 0/                       # Native resolution (zoom level 0)
│   ├── band1
│   ├── band2
│   └── spatial_ref
├── 1/                       # First overview level
│   ├── band1
│   ├── band2
│   └── spatial_ref
└── 2/                       # Second overview level
    ├── band1
    ├── band2
    └── spatial_ref

With multiscales metadata at the parent level:

zarr_json_attributes["multiscales"] = {
    "tile_matrix_set": tile_matrix_set,
    "resampling_method": "average",
    "tile_matrix_limits": tile_matrix_limits,
}

Specification Sections to Update

• Section 9.7.1 (Hierarchical Layout)
• Section 9.7.2 (Metadata Encoding)
• Add clear structural diagrams and examples

cc @vincentsarago, @maxrjones, @d-v-b, @briannapagan

[geospatial-jeff]

It should clarify decimation requirements too, the OGC TMS spec isn't clear here. Decimation of 2 is pretty standard but that isn't always the case with OGC TMS - developmentseed/morecantile#147

[christophenoel]

I agree that the current model of overviews described in clause 7 – unified data model and clause 9 – Zarr encoding overviews is not yet fully clear or complete.

In particular:

• A Dataset Group, which contains one or multiple data variables, may include or exclude overviews. I believe the Dataset Group itself would hold the native resolution directly (a dedicated 0/ zoom-level group would not exist) as overviews can be added later, and even not be used/supported by the client.
• As illustrated in your example, a zoom-level group may also contain multiple variables, not just a single one.
• Variables are not necessarily restricted to 2D. While COG only supports 2D, it is not obvious whether the TMS model can always apply to variables with dimensions such as X, Y, Band (i.e. more than two spatial dimensions). This needs clarification.

[d-v-b]

• A Dataset Group, which contains one or multiple data variables, may include or exclude overviews. I believe the Dataset Group itself would hold the native resolution directly (a dedicated 0/ zoom-level group would not exist) as overviews can be added later, and even not be used/supported by the client.

So I understand correctly, are you suggesting a layout like this would be permitted?:

/measurements/r10m/          # Parent group with multiscales metadata
├── band1
├── band2
├── spatial_ref
├── 1/                       # First overview level
│   ├── band1
│   ├── band2
│   └── spatial_ref
└── 2/                       # Second overview level
    ├── band1
    ├── band2
    └── spatial_ref

[emmanuelmathot]

@d-v-b @christophenoel, xarray do not allow this layout since it enforce data alignement

The data in different datatree nodes are not totally independent. In particular dimensions (and indexes) in child nodes must be exactly aligned with those in their parent nodes. Exact alignment means that shared dimensions must be the same length, and indexes along those dimensions must be equal.

from https://docs.xarray.dev/en/latest/user-guide/hierarchical-data.html#data-alignment

I dont know if this is a contraint in the Unified Data Model too...

[christophenoel]

Indeed. I have illustrated in my review: PR #86

[christophenoel]

@d-v-b @christophenoel, xarray do not allow this layout since it enforce data alignement

The note you quoted applies inside a single Dataset (or DataTree node): all data variables within that Dataset must share consistent dimensions and coordinate indexes. This is the same principle as in the Unified Data Model, where a Dataset is defined as a coherent collection of variables with aligned dimensions

It does not mean that a parent Dataset (native resolution) and a child Dataset (overview resolution) must have aligned spatial dimensions. Each zoom-level group is its own Dataset, and xarray will enforce alignment within each of those groups, not across them.

With both approch (native data in parent group or in a children group), each group is a Dataset and thus the variables nodes all share the same dimensions (this is part of the definition of a dataset in our model spec).

Note: xarray indeed can open a single dataset at a time (with potentially multiple variables within that dataset)

[emmanuelmathot]

The data in different datatree nodes are not totally independent

Why then the xarray documentation specifically says "datatree" and not "DataSet" knowing the terminology ?

On the other hand, I am pretty sure to have ran into the xarray error when trying to open a dataset with such a layout. I will try to reproduce it.

[christophenoel]

You're right for the datatree definition, but from what I read Datatree is the new abstraction that xarray introduced to represent a hierarchy of datasets.

In xarray, you can:

• Open a single NetCDF/Zarr group (root or any subgroup) directly as a flat xarray.Dataset.
• Or, if you want to reflect the entire group hierarchy in one object, you use xarray.DataTree.

UDM does not enforce datatree constraint: of course, many NetCDF includes children groups which doesn't share the dimensions with parent groups.

[christophenoel]

Why then the xarray documentation specifically says "datatree" and not "DataSet" knowing the terminology ?

On the other hand, I am pretty sure to have ran into the xarray error when trying to open a dataset with such a layout. I will try to reproduce it.

As clarified by David yesterday:

In xarray, DataTree provides a tree-structured container of multiple Dataset objects arranged in a hierarchy and with dimensions exactly aligned with those in their parent nodes

Dataset itself remains the standard container and is still used for reading groups with variables and coordinates that does not follow these constraints.

Note that the following simplified code create multiscales in chilren group:

import xarray as xr
# Select the 'abs' variable (preserve band dimension)
abs_data = biomass["abs"]
# Define scales for downsampling
scales = [2, 4, 8, 16, 32]  # Example scale factors
for i, scale in enumerate(scales, start=1):
    # Coarsen ONLY on x and y (preserve band dimension)
    downscaled = abs_data.coarsen(y=scale, x=scale, boundary="trim").mean()
    # Write to Zarr store under the subgroup path
    downscaled.to_dataset(name="abs").to_zarr( biomass_zarr_path, pathgroup=f"measurements/{i}",
        mode="w")

The following code open and get information about the downscales:

import zarr
store = zarr.open(biomass_zarr_path, mode="r")
measurements_group = store["measurements"]
# List child groups
child_groups = measurements_group.group_keys()
# Collect info about each group
group_info = []
for group in child_groups:
    grp = measurements_group[group]
    abs_array = grp["abs"]  # access the 'abs' array
    info = {
        "Group": group,
        "Shape": abs_array.shape,
        "Chunks": abs_array.chunks,
        "DType": abs_array.dtype,
        "Attributes": len(grp.attrs),
    }
    group_info.append(info)

The following code shows zoom level 0 and zoom level 5 comparison:

import xarray as xr
import matplotlib.pyplot as plt
# Load root dataset (zoom level 0)
biomass_root = xr.open_zarr(measurement_path, consolidated=False)
rgb_root = biomass_root["abs"].sel(band=[1, 2, 3])
# Load zoom level 5 dataset
zoom_level = 5
zoom_path = f"{measurement_path}/{zoom_level}"
biomass_zoom = xr.open_zarr(zoom_path, consolidated=False)
rgb_zoom = biomass_zoom["abs"].sel(band=[1, 2, 3])
# Plot side by side
fig, axes = plt.subplots(1, 2, figsize=(12, 6))
# Plot root
rgb_root.plot.imshow(ax=axes[0], rgb="band", robust=True)
axes[0].set_title("Zoom Level 0 (Full Resolution)")
# Plot zoom level 3
rgb_zoom.plot.imshow(ax=axes[1], rgb="band", robust=True)
axes[1].set_title(f"Zoom Level {zoom_level}")
plt.tight_layout()
plt.show()

The notebook will be released soon, I will keep you informed.

[christophenoel]

I would be curious to hear more about how option A would break existing readers. Is the name of the dataset of particular important to readers, such that opening r10m/0 is a breaking change compared to opening r10m?

To clarify the context for client libraries:

• There are already many missions (including those from the European Space Agency, which initiated GeoZarr) that expose petabytes of data encoded in Zarr and indexed in catalogues. These products are typically encoded with native data at the dataset level (metadata + child data variables/coordinates). Processing these archives into Zarr has been estimated to cost a lot of money. There is no realistic chance that data providers would adopt GeoZarr if adding overviews required rewriting existing products in this way.
• In addition, there are many client applications developed around these existing products. These applications expect to continue working unchanged if overviews are introduced. Overviews is expected be defined as a backward-compatible extension, not something that alters the dataset’s core structure.

[d-v-b]

In addition, there are many client applications developed around these existing products. These applications expect to continue working unchanged if overviews are introduced. Overviews is expected be defined as a backward-compatible extension, not something that alters the dataset’s core structure.

Thanks, this is very helpful context. If I understand this constraint correctly, it seems to imply that this spec should not introduce any features (e.g., overviews) that would require changes to existing applications. For example, the proposal to nest all geozarr attributes under a "geozarr" key seems to be discouraged for this reason.

If the goal of the spec is not to develop a new standard but rather to document specific existing practices, then I think this scope should be declared early on the spec, and the client applications that function as a hard constraint on spec development should be named as such.

[christophenoel]

Let me nuance my previous comment. We are working in a democratic group, and while I defend the interests of my client (ESA), who initiated the GeoZarr effort, my voice carries no more weight than anyone.

Regarding the proposal to have a geozarr attribute: this would not affect existing applications, since they would simply ignore an unknown attribute and therefore not benefit from the additional functionality.

That said, I believe it is fairly well established that the specification should avoid changes that require restructuring existing data or create critical backward compatibility issues (such as reformatting existing dataset layouts). What was agree is that the existing CF/CDM ecosystem should be extended with GIS capabilities (and STAC, and ...), rather than reinventing an entirely new data model or specification. The scope should rather focus on extensions through additional metadata or new data, without breaking existing products.

I also agree that if this principle is approved, it should be made explicit in the specification, which still largely reflects its initial draft.