Discussion: output chunk-layout policy for optimized GeoZarr (single-chunk policy, sharding, inner-tile size)

[lhoupert]

Follow-up (co-written with Claude ) to the review discussion on #220 (now split: source reads stay in #220, output chunk clamp moved to #221). This issue is to agree on an output chunk-layout policy for optimized GeoZarr, instead of settling it inside a bugfix PR.

Where this comes from

Some source products store quality/atmosphere aot/wvp (and ~25 other L2A quality/mask arrays) as one whole-array chunk (10980×10980 at r10m, ~482 MB raw at f4). Pre-#221, create_original_encoding propagated that layout verbatim into converted outputs — unusable for tiled serving and expensive to write. #221 clamps oversized chunks to 1830×1830 inner tiles inside a whole-array shard (single storage object per array), and is deliberately a draft until this discussion converges.

Open questions

1. Single-chunk policy provenance. In fix(conversion): concurrent-safe source opening (native-chunk-aligned reads + atomic CacheStore) for data-pipeline#339 #220 review, @d-v-b recalled a policy that "we emit Zarr arrays with a single chunk". Where is that written down? It isn't in ADR-101, and ADR-101 doesn't mention sharding at all. If the policy is "one storage object per array", fix(s2): clamp oversized source chunks to sharded 1830px tiles #221's whole-array shard satisfies it; if it is literally "one chunk", fix(s2): clamp oversized source chunks to sharded 1830px tiles #221 changes policy and we should decide that consciously.
2. Inner-tile size. fix(s2): clamp oversized source chunks to sharded 1830px tiles #221 uses 1830×1830 because it's the native S2 tiling the EOPF PSFD documents (§6.7.3, "6×6 chunks") and matches pre-existing cpm_v262-era products. Is that the right serving granularity for titiler (vs e.g. 512/1024), and what is the write-cost impact?
3. Where should the policy live? Amend ADR-101, or a new ADR covering sharding + chunk sizing for optimized outputs (reflectance vs quality/mask arrays)?

Source-side provenance (CPM) — what we know

• CPM's own mapping config (eopf/store/mapping/S02MSIL2A_PSD14/15_*.json, checked at main and tag 3.0.0rc4) declares native 1830×1830 tiling for aot/wvp at every resolution — identical to the reflectance bands — and shows no sign of having changed across tags 2.6.x → 3.0.0rc4.
• The cpm_v262/cpm_v270 labels are just EODC path segments naming the CPM release that produced the source; the whole-array layout has been observed on samples from both, so it does not track CPM version.
• But the Convert fails with blosc decompression: -1 on cpm_v270 scenes: simplecache concurrent same-key race (not a zarr sample service timeout) data-pipeline#339 root-cause work established that the source products at EODC really do carry whole-array 10980×10980 aot/wvp chunks (that is what triggers ~9 concurrent same-key reads under chunks="auto"). Taken together, that implies a discrepancy between CPM's declared mapping and what the delivered products actually contain (e.g. a rechunk step not applied for some production batches).
• Cheap decisive check, proposed as part of this issue: curl the source zarr.json for .../cpm_v270/<item>.zarr/quality/atmosphere/r10m/aot/ directly at EODC and record its chunk_grid, for one v262-era and one v270 sample. If the sources are whole-array while CPM's mapping says 1830, the question of why belongs upstream with the CPM team; either way our converter should not inherit pathological source layouts, which is what fix(s2): clamp oversized source chunks to sharded 1830px tiles #221 enforces.

Proposed outcome

Agreement on (a) the storage-object/chunk policy wording, (b) the inner-tile size for clamped arrays, (c) which document records it — then #221 exits draft (adjusted if needed).

[d-v-b]

@d-v-b recalled a policy that "we emit Zarr arrays with a single chunk"

sorry if I was imprecise -- I meant a single file. This is compatible with sharding ofc

[lhoupert]

sorry to put you under the spotlight :-) Single chunk or single file, I didnt find any reference to it in our ADR, so I think it could be good to update them at some point when we have the "final" version of our data model

[d-v-b]

agreed, it would be good to get this clarified, and then have a single source of truth for the chunking policy