Skip to content

Commit 23db304

Browse files
emmanuelmathotbenbovy
emmanuelmathot
and
benbovy
authored
Add integration tests and enhance Sentinel-1 GRD to GeoZarr conversion (#30)
* Add integration tests for Sentinel-1 GRD to GeoZarr conversion - Implemented a comprehensive integration test for converting Sentinel-1 GRD datasets to GeoZarr format. - Created a sample EOPF DataTree structure mimicking the Sentinel-1 GRD organization. - Verified the preservation of GRD measurement data, GCP-based georeferencing, and compliance with GeoZarr specifications. - Included checks for basic structure, GeoZarr compliance, and GCP implementation. * Enhance GeoZarr dataset conversion to support GCPs and update integration tests for Sentinel-1 * tests: update sentinel 1 sample (mock) datatree * silence test warnings * clean-up, misc. fixes and formatting * refactor create_geozarr_dataset to include sentinel-1 Deal with current VH and VV top-level groups in sentinel-1 data products. * fix typing * fix zarr variable path * tests: fix temp path * fix write auxiliary coords to zarr Avoid chunking 1-d auxiliary coordinates "line" and "pixel", which raises an error when attempting to write them to zarr with the rest of the single variable dataset and its coordinates. * streamline sentinel-1 special case `iterative_copy` iterates over the whole datatree, so a simpler approach is to just make the cross-product of the groups to convert to geozarr with the two VV/VH polarization groups and pick one of the two `conditions/gcp` groups (both should be equal). * add grid mapping variable with GCPs + tests Native resolution only for now. * add GCP-based multiscale support for Sentinel1 GRD No GCP decimation yet (original GCPs copied at each zoom level). * re-organize tests structure So we can later reuse testing logic that is common to Sentinel 1 and 2, if need. * add tests for sentinel 1 multiscale * add GCP group cli argument * fix tests * check sentinel-1 data but no GCP group provided * move tests So they are installed with the package. * compute overview GCPs from native resolution GCPs * fix sentinel1 grd overview dimensions * ci: update test directory * fix: improve chunk size calculation logic in calculate_aligned_chunk_size function * strip group name slash prefix Co-authored-by: Emmanuel Mathot <emmanuel.mathot@gmail.com> * bands -> variables Co-authored-by: Emmanuel Mathot <emmanuel.mathot@gmail.com> * band -> variable Co-authored-by: Emmanuel Mathot <emmanuel.mathot@gmail.com> * feat: add Sentinel-1 reprojection utilities and update GeoZarr conversion logic * fix: update group name formatting in create_geozarr_dataset function * fix: correct standard names for target coordinates in _create_target_coordinates function * feat: add validation script for Sentinel-1 reprojection compatibility with titiler-eopf * feat: enhance reprojection and downsampling functions with nodata handling * fix: update grid_mapping handling in overview dataset creation * fix: suppress FutureWarning for timedelta decoding in CLI * fix: refine warning suppression for FutureWarning and UserWarning in CLI * fix: refine warning suppression for RuntimeWarning in CLI * feat: add validation script for Sentinel-1 reprojection compatibility with titiler-eopf * Implement code changes to enhance functionality and improve performance * fix: refine warning suppression and improve code formatting across multiple files * fix: update test command to use correct source directory for pytest * fix: update test command to use correct test directory for pytest --------- Co-authored-by: Benoit Bovy <benbovy@gmail.com>
1 parent b59afed commit 23db304

15 files changed

Lines changed: 2111 additions & 738 deletions

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ jobs:
5050

5151
- name: Run tests
5252
run: |
53-
uv run pytest tests/ -v --tb=short -m "not network" --cov=eopf_geozarr --cov-report=xml --cov-report=term-missing
53+
uv run pytest src/eopf_geozarr/tests/ -v --tb=short -m "not network" --cov=eopf_geozarr --cov-report=xml --cov-report=term-missing
5454
5555
- name: Upload coverage to Codecov
5656
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
@@ -105,7 +105,7 @@ jobs:
105105

106106
- name: Run network tests
107107
run: |
108-
uv run pytest tests/ -v --tb=short -m "network"
108+
uv run pytest src/eopf_geozarr/tests/ -v --tb=short -m "network"
109109
110110
security:
111111
runs-on: ubuntu-latest

‎.vscode/launch.json‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,80 @@
123123
"AWS_ENDPOINT_URL": "https://s3.de.io.cloud.ovh.net/"
124124
},
125125

126+
},
127+
{
128+
"name": "Convert to GeoZarr Sentinel-1 GRD (Local)",
129+
"type": "debugpy",
130+
"request": "launch",
131+
"module": "eopf_geozarr",
132+
"args": [
133+
"convert",
134+
"https://objects.eodc.eu/e05ab01a9d56408d82ac32d69a5aae2a:202509-s01siwgrh/12/products/cpm_v256/S1C_IW_GRDH_1SDV_20250912T053648_20250912T053713_004087_0081FD_5AA4.zarr",
135+
"./tests-output/eopf_geozarr/s1_test.zarr",
136+
"--groups", "/measurements",
137+
"--gcp-group", "/conditions/gcp",
138+
"--spatial-chunk", "512",
139+
"--min-dimension", "256",
140+
"--tile-width", "256",
141+
"--max-retries", "2",
142+
"--dask-cluster",
143+
"--verbose"
144+
],
145+
"cwd": "${workspaceFolder}",
146+
"justMyCode": false,
147+
"console": "integratedTerminal",
148+
"env": {
149+
"PYTHONPATH": "${workspaceFolder}/eopf_geozarr/.venv/bin"
150+
},
151+
},
152+
{
153+
"name": "Convert to GeoZarr Sentinel-1 GRD (S3)",
154+
"type": "debugpy",
155+
"request": "launch",
156+
"module": "eopf_geozarr",
157+
"args": [
158+
"convert",
159+
"https://objects.eodc.eu/e05ab01a9d56408d82ac32d69a5aae2a:202509-s01siwgrh/12/products/cpm_v256/S1C_IW_GRDH_1SDV_20250912T053648_20250912T053713_004087_0081FD_5AA4.zarr",
160+
"s3://esa-zarr-sentinel-explorer-fra/tests-output/eopf_geozarr/S1C_IW_GRDH_1SDV_20250912T053648_20250912T053713_004087_0081FD_5AA4.zarr",
161+
"--groups", "/measurements",
162+
"--gcp-group", "/conditions/gcp",
163+
// "--crs-groups", "/conditions/geometry",
164+
"--spatial-chunk", "512",
165+
"--min-dimension", "256",
166+
"--tile-width", "256",
167+
"--max-retries", "2",
168+
"--dask-cluster",
169+
"--verbose"
170+
],
171+
"cwd": "${workspaceFolder}",
172+
"justMyCode": false,
173+
"console": "integratedTerminal",
174+
"env": {
175+
"PYTHONPATH": "${workspaceFolder}/.venv/bin",
176+
"AWS_PROFILE": "eopf-explorer-vs",
177+
"AWS_DEFAULT_REGION": "de",
178+
"AWS_ENDPOINT_URL": "https://s3.de.io.cloud.ovh.net/"
179+
},
180+
181+
},
182+
{
183+
"name": "Info EOPF Zarr (Sample Service)",
184+
"type": "debugpy",
185+
"request": "launch",
186+
"module": "eopf_geozarr",
187+
"args": [
188+
"info",
189+
"https://objects.eodc.eu/e05ab01a9d56408d82ac32d69a5aae2a:202508-s02msil2a/11/products/cpm_v256/S2C_MSIL2A_20250811T112131_N0511_R037_T29TPF_20250811T152216.zarr",
190+
"--verbose",
191+
"--html-output", "eopf_dataset_info.html"
192+
],
193+
"cwd": "${workspaceFolder}",
194+
"justMyCode": false,
195+
"console": "integratedTerminal",
196+
"env": {
197+
"PYTHONPATH": "${workspaceFolder}/.venv/bin",
198+
},
199+
126200
},
127201
{
128202
"name": "Info GeoZarr local",

‎src/eopf_geozarr/cli.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77

88
import argparse
99
import sys
10+
import warnings
1011
from pathlib import Path
1112
from typing import Any, Optional
1213

@@ -20,6 +21,13 @@
2021
validate_s3_access,
2122
)
2223

24+
# Suppress xarray FutureWarning about timedelta decoding
25+
warnings.filterwarnings("ignore", message=".*", category=FutureWarning)
26+
27+
warnings.filterwarnings("ignore", message=".*", category=UserWarning)
28+
29+
warnings.filterwarnings("ignore", message=".*", category=RuntimeWarning)
30+
2331

2432
def setup_dask_cluster(enable_dask: bool, verbose: bool = False) -> Optional[Any]:
2533
"""
@@ -166,6 +174,7 @@ def convert_command(args: argparse.Namespace) -> None:
166174
tile_width=args.tile_width,
167175
max_retries=args.max_retries,
168176
crs_groups=args.crs_groups,
177+
gcp_group=args.gcp_group,
169178
)
170179

171180
print("✅ Successfully converted EOPF dataset to GeoZarr format")
@@ -1087,6 +1096,11 @@ def create_parser() -> argparse.ArgumentParser:
10871096
nargs="*",
10881097
help="Groups that need CRS information added on best-effort basis (e.g., /conditions/geometry)",
10891098
)
1099+
convert_parser.add_argument(
1100+
"--gcp-group",
1101+
type=str,
1102+
help="Groups where Ground Control Points (GCPs) are located (e.g., /conditions/gcp) (Sentinel-1)",
1103+
)
10901104
convert_parser.add_argument(
10911105
"--verbose", action="store_true", help="Enable verbose output"
10921106
)

0 commit comments

Comments
 (0)