Appending consolidated metadata zarr datasets with consolidated metadata slow; how to speed it up?

[C-Loftus]

Thank you very much for the great work on xarray. I had a question on optimizing zarr writes.

---

I have a large government environmental dataset that generates h5 files every day. I want to convert it to zarr and append it to my store every day. However I am finding that ds.to_zarr(consolidated=True) is quite slow to write due to zarr-developers/zarr-python#3313

I've also just tried running zarr.consolidate_metadata(my_zarr_store) once at the end but it seemed to mess with the dimensions and collapse the time down to a small subset of the time values. Perhaps due to #11101 ?

I was curious if there are any ways to improve my logic to make it faster at scale. The first time a user runs my workflow it has to download 4000 h5 files and iteratively add them to the store (they are too big to batch all together on disk) so it is quite slow to run.

My full source code is here if it helps: https://github.com/cgs-earth/ArizonaWaterObservatory/tree/main/SMAP_ETL/src

Here is an abbreviated version

def append_hd5_to_s3_zarr(
    hd5_file_path: Path,
    s3_fs,
    bucket: str,
    store_name: str,
    test_mode: bool = False,
):
    """
    Open hd5 file with xarray, reproject cell_lat/cell_lon to EASE2 meters,
    and append to S3 Zarr store using fsspec mapper.
    """
    root = xr.open_dataset(
        hd5_file_path,
        engine="h5netcdf",
        phony_dims="sort",
    )

    geo = xr.open_dataset(
        hd5_file_path,
        engine="h5netcdf",
        group="Geophysical_Data",
        phony_dims="sort",
    )

    ds = xr.merge([geo, root[["cell_lat", "cell_lon"]]])
    ...

    # --- Reproject cell_lat/cell_lon to EASE2 meters and assign as x/y coords ---
    transformer = Transformer.from_crs(
        "EPSG:4326", "EPSG:6933", always_xy=True
    )

    lons = ds["cell_lon"].values
    lats = ds["cell_lat"].values

    x_meters, y_meters = transformer.transform(lons, lats)

    # EASE2 is a regular grid, so collapse to 1D by taking the first row/col
    x_1d = x_meters[0, :]  # x varies along columns
    y_1d = y_meters[:, 0]  # y varies along rows

    # Get the phony dim names that correspond to spatial axes
    phony_y, phony_x = ds[
        "cell_lat"
    ].dims  # e.g. ("phony_dim_0", "phony_dim_1")

    ds = ds.assign_coords(
        {
            phony_x: x_1d,
            phony_y: y_1d,
        }
    ).rename({phony_x: "x", phony_y: "y"})

    # Extract time from filename; we will use this as the time dimension
    # and serialize it with pandas so it can be used in queries
    time_value = pd.Timestamp(parse_time_from_path(hd5_file_path))

    # Force explicit time dimension
    ds = ds.expand_dims(time=[time_value])
    ds = ds.sortby("time")

    # Create fsspec mapper for Zarr store
    zarr_mapper = s3_fs.get_mapper(f"{bucket}/{store_name}")

    try:
        _ = zarr.open_consolidated(zarr_mapper, mode="r")
        store_exists = True
    except Exception:
        store_exists = False

    if store_exists:
        ds.to_zarr(
            store=zarr_mapper,
            mode="a",
            append_dim="time",
            consolidated=True,
            zarr_format=2,
        )
    else:
        ds.to_zarr(
            store=zarr_mapper,
            mode="w",
            consolidated=True,
            zarr_format=2,
            encoding=time_encoding,
        )

These are my dependencies

# Copyright 2026 Lincoln Institute of Land Policy
# SPDX-License-Identifier: Apache-2.0

[project]
name = "smap-etl"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "earthaccess>=0.17.0",
    "h5netcdf>=1.8.1",
    "h5py>=3.16.0",
    "minio>=7.2.20",
    "pandas>=3.0.2",
    "pyproj>=3.7.2",
    "s3fs>=2026.3.0",
    "xarray>=2026.2.0",
    "zarr>=3.1.6",
]

[dependency-groups]
dev = [
    "pytest>=9.0.2",
    "ruff>=0.15.10",
]

I have seen #6888 and it seems like it might help but I am unclear how to apply it in my current example.

Thank you
Regards
Colton

[khlaifiabilel]

The expensive part here is doing consolidation after every one-file append. Since this is explicitly a Zarr v2 store, write unconsolidated during the ingestion batch and consolidate once after the loop:

store_path = f"{bucket}/{store_name}"
zarr_mapper = s3_fs.get_mapper(store_path)

# Test for the Zarr v2 root metadata, not for consolidated metadata.
store_exists = s3_fs.exists(f"{store_path}/.zgroup")

if store_exists:
    ds.to_zarr(
        store=zarr_mapper,
        mode="a",
        append_dim="time",
        consolidated=False,
        zarr_format=2,
    )
else:
    ds.to_zarr(
        store=zarr_mapper,
        mode="w",
        consolidated=False,
        zarr_format=2,
        encoding=time_encoding,
    )

Then, after all files in this run have been appended:

zarr_mapper = s3_fs.get_mapper(f"{bucket}/{store_name}")
zarr.consolidate_metadata(zarr_mapper, zarr_format=2)

This changes roughly 4,000 consolidation passes into one. During the batch, any code that must read the in-progress store should use xr.open_zarr(..., consolidated=False). After the final call, consumers that require .zmetadata can use consolidated reads again. For later daily batches, repeat the same pattern and refresh .zmetadata once at the end of each batch; until then an old .zmetadata is stale.

The current existence check is unsafe for this pattern. zarr.open_consolidated(...) answers “does usable consolidated metadata exist?”, not “does the store exist?”. Once writes are unconsolidated, it would report a real store as absent and the mode="w" branch could overwrite it. Catching every Exception also turns transient S3/authentication errors into “store absent”. Checking the format-2 root key (.zgroup) avoids both problems.

zarr.consolidate_metadata only gathers existing metadata; it should not itself shorten the time coordinate. The behavior in xarray#11101 is a separate append issue: mode="a" can overwrite non-append coordinate variables with values from the incoming dataset. In this case x and y are intended to be invariant, so validate that they equal the existing store before appending rather than relying on the append operation to align them. Also verify that incoming times are new, since append_dim="time" appends by position and does not de-duplicate or sort the existing store.