Sentinel-1 GRD γ0T RTC GeoZarr V3 — Implementation Tracker

[emmanuelmathot]

Overview

Implementation of Sentinel-1 GRD γ0T RTC time-series GeoZarr V3 stores in the data-model codebase. S1Tiling (CNES/OTB) produces γ0T RTC GeoTIFFs on the S2 MGRS grid; this work converts those GeoTIFFs into GeoZarr V3 stores following the EOPF hierarchy.

Design document: .github/prompts/s1-grd-rtc-implementation-plan-v2.md (on s1-tiling branch)
Working branch: s1-tiling

Implementation Phases

• Phase 0 — Design and prototyping (commit 1ae1b99, faad006)
• Phase 1 — Pydantic-zarr V3 models (PR feat: Pydantic-zarr V3 model for Sentinel-1 GRD γ0T RTC stores #138, commit 4bb1367)
• Phase 2 — GeoTIFF ingestion (PR feat: phase 2 — S1 GRD RTC GeoTIFF ingestion pipeline #146)
• Phase 3 — Conditions ingestion and discovery (PR feat: Phase 3 — S1Tiling conditions ingestion and discovery #153)
• Phase 4 — CLI and S3 (PR feat: Phase 4 — CLI and S3 (#139) #155)
• Phase 5 — STAC and Argo

Devlog

2026-03-23 — Phase 1 complete

Delivered: Pydantic-zarr V3 schema for S1 GRD RTC stores, aligned with the S2 model pattern.

Files:

• src/eopf_geozarr/data_api/s1_rtc.py — 316 lines, model hierarchy (Root → OrbitGroup → Resolution → Arrays)
• tests/_test_data/s1_rtc_examples/s1-grd-rtc-31TCH.json — realistic fixture (MGRS 31TCH, 3 timesteps, sharding codecs)
• tests/test_data_api/test_s1_rtc.py — 11 tests (round-trip, structure, 5 negative cases)

Key decisions:

• Uses pyz.v3 (Zarr V3) instead of pyz.v2 (Zarr V2) used by S2
• extra="allow" on attrs models for forward compatibility
• Conditions group uses dict[str, ArraySpec] (dynamic keys per orbit)
• Both ascending/descending optional with at-least-one validator

PR #138 opened as draft for review by pydantic-zarr schema maintainers.

Open questions for reviewers:

1. Should attrs models inherit from a shared base with S2?
2. Is extra="allow" the right choice, or lock down all fields?
3. Should multiscales use the typed zarr_cm model instead of dict[str, Any]?
4. Should conditions enforce a regex pattern beyond gamma_area_ prefix?

2026-03-22 — Phase 0 complete

Delivered: End-to-end prototype from S1Tiling Docker → GeoTIFF → GeoZarr V3.

• S1Tiling 1.4.0 ran on 3 acquisitions over MGRS 31TCH (orbits 008, 037, 110)
• Discovered EODAG 4.0.0 breaking changes (5 issues, monkey-patch created)
• Real-data conversion validated: 10980×10980 sharded arrays, border_mask, gamma_area conditions
• Inner chunk divisibility constraint: 10980 % 512 ≠ 0, solution: best_chunk_size() → 366
• Store size: ~1.8 GB for 3 timesteps across all overview levels
• xarray readback verified for all levels

[d-v-b]

Should multiscales use the typed zarr_cm model instead of dict[str, Any]?

IMO yes, this will reduce the likelihood of small errors in the multiscales convention declaration

Is extra="allow" the right choice, or lock down all fields?

This depends on how dynamic the source attributes are, and whether we see the geozarr conversion as purely additive. If the goal is to guarantee a strict set of attributes in the output, which means removing attributes from the input, then extra="allow" is probably not what we want. but this depends on the needs / expectations of the consumers.

[emmanuelmathot]

2025-03-24 — Phase 2 complete: GeoTIFF → Zarr V3 ingestion pipeline

Delivered files

File	Description
src/eopf_geozarr/conversion/s1_ingest.py (~500 lines)	Production ingestion pipeline
tests/test_s1_rtc_ingest.py (27 tests)	Full test coverage

Public API

• ingest_s1tiling_acquisition(vv, vh, mask, store, orbit) — create-or-append one acquisition
• consolidate_s1_store(store, orbit) — post-batch consolidation
• discover_s1tiling_acquisitions(input_dir) — file discovery with grouping
• extract_geotiff_metadata(path) — rasterio-based metadata extraction

Test results: 27/27 passed ✅

Key design decisions

• Convention metadata built from zarr_cm CMO dicts (UUID, schema_url, spec_url)
• Private _downsample_2d helper for overview generation (mean downsampling)
• structlog for structured logging throughout
• CRS + shape validation on append (rejects mismatched acquisitions)
• 1D spatial coordinate arrays (x, y) at every resolution level via np.linspace
• Lowercase dimensions ["y", "x"] / ["time", "y", "x"] required by titiler-eopf

Known issues

• Pre-existing Phase 1 test failure: test_s1_rtc_orbit_attrs — Multiscales model not subscriptable (Phase 1 model needs TypedDict x/y members)

Branch: feat/s1-rtc-geotiff-ingestion (based on feat/s1-rtc-pydantic-model)
PR ready: feat/s1-rtc-pydantic-model...feat/s1-rtc-geotiff-ingestion

[emmanuelmathot]

Devlog update — zcm.Multiscales adoption

Context: Reviewer @d-v-b suggested replacing dict[str, Any] with the typed zcm.Multiscales Pydantic model.

Changes (commit 8cc7df1 on PR #138):

• S1RtcOrbitGroupAttrs.multiscales field type changed from dict[str, Any] to Multiscales (imported from eopf_geozarr.data_api.geozarr.multiscales.zcm)
• Removed all inline replacement models (MultiscalesTransform, MultiscalesScaleLevel, Multiscales) that were a workaround for Python 3.11 incompatibility
• spatial:dimensions validator updated to lowercase ['y', 'x'] to align with Python 3.12 conventions
• Test assertions updated to use Pydantic attribute access instead of dict access

Blocker resolved: zcm.py was incompatible with Python 3.11 due to typing.TypedDict vs typing_extensions.TypedDict in Pydantic 2.12. PR #141 bumped minimum Python to 3.12, unblocking the import.

Status: PR #138 rebased on main, force-pushed, 11/11 tests passing.