How to handle zone data retrieval returns with collections that are mixed with dimensions.

[tik65536]

Originally posted by @tik65536 in #61 (comment)

related ogc pilot issue : https://gitlab.ogc.org/ogc/AI-DGGS/-/issues/59#note_39233

@fmigneault, I am handling the FIXME here. However, I have a question on the handling of mixed collections with and without the datetime dimension. Should we only assume that the zone ID is the only valid dimension? Otherwise, the dimension of the Zarr won't match.

In general, for other return types, such as DGGS-JSON, how is it handled?

Replied by @fmigneault

This is still an open question (see the sub-comments in the pilot issue https://gitlab.ogc.org/ogc/AI-DGGS/-/issues/59#note_39233).

In the meantime, I propose to handle it this way:

• If the multi-collection request combines with/without datetime dimenions, the collections missing the datetime dimension would pad it with the repeated value for each temporal entry corresponding to the temporal instances of the superset of all datetime dimenions that are available from other collections being considered. Repeating the value for cases of missing datetime dimension makes sense IMO, since they normally return a "zone match" regardless of datetime (ie: the value "always applies").
• If multiple collections have datetime dimensions, but they don't align (eg: offsets in the hours), temporal instances that mismatch between the requested time range would have to be null-padded, with all temporal instances being a superset of all available datetimes.

Eventually (future feature), we could integrate the resolution (see https://gitlab.ogc.org/ogc/AI-DGGS/-/issues/63) or similar feature to help aggregate/align collections temporally with resampling. However, I think the ongoing discussions need more time to make sure the strategy is robust to move forward with that feature.

Seems to be a corresponding strategy used by other participants for experimental endpoints as well: https://gitlab.ogc.org/ogc/AI-DGGS/-/issues/59#note_39362

[tik65536]

According to the discussion above, the returns of data retrieval from the endpoint /dggs/{dggrsId}/zones/{zoneId}/data with collections with mixed dimensions can be structured with three scenarios with respect to dimensions :

We limited ourselves here to two dimensions: zoneID and temporal.

Mixed collections with zoneID as dimension (1D) only

• The current implementation is to create a big dataframe (master_df) to hold the results from different collections. Just like concatenating each collection horizontally, with the column named in <collection_id>.<property_name>.

Mixed collections with zoneID and temporal dimension (2D)

• The master_df can be set with a multi-index, so that other 2D collections can be merged together. As pointed out by @fmigneault, the temporal resolution should also be considered in the future.

Mixed collections with both 2D and 1D

• There is no implementation yet. As suggested by @fmigneault, one way would be to expand the dimension of 1D collections to 2D with temporal data, then pad the temporal dimension with repeated values.

For the suggested expanded dimension handling of mixed dimensions, including 2D and 1D, it may create an illusion for clients/users that the 1D collections are temporal. The return must supply a flag for the particular property to indicate that the temporal dimension is artificially created, so that users/clients can distinguish those from the return.

On the other hand, theoretically, all collections are 2D, considering the time when the data is captured. If we disregard this property and merge it with other temporal collections from other times, it also causes confusion and misalignment in the temporal dimension.

If we adopt the expanding dimension way, then we must have 2 enhancements :

• an indicator at the property level to show whether it is artificially created.
• An extra metadata field for collections to specify the data capturing time, so that the API can refer to it as the pseudo-temporal dimension. In this way, when we join it with other temporal collections, it can resolve the misalignment issue.

The side effect is that all collections will become temporal ones if any one of them is temporal.

The simplest way is to separate collections into temporal and non-temporal ones. The zone data retrieval endpoint defaults to handle non-temporal collections, and it switches to temporal collections when the datetime is specified in the query parameter or in the CQL2 filter.

@fmigneault, @allixender, what do you think?

[tik65536]

To clarify, for the return of two temporal collections on zone data retrieval, should the dimension list only show the merged datetime, or should it include two individual datetime dimensions from each collection?

From the example of the standard section 16.2.1, it seems like the temporal (datetime) is the merged version from collections.

An example with a separated datetime dimension for individual collection.

  "dimensions": [
    {
      "name": "rcm_zarr.datetime",
      "grid": {
        "cellsCount": 4,
        "coordinates": [
          "2022-04-20T00:00:00.000000000",
          "2022-04-30T00:00:00.000000000",
          "2022-05-05T00:00:00.000000000",
          "2022-05-10T00:00:00.000000000"
        ]
      },
      "interval": [
        "2022-04-20T00:00:00.000000000",
        "2022-05-10T00:00:00.000000000"
      ]
    },
    {
      "name": "rcm_parquet.datetime",
      "grid": {
        "cellsCount": 4,
        "coordinates": [
          "2022-04-20T00:00:00.000000000Z",
          "2022-04-30T00:00:00.000000000Z",
          "2022-05-05T00:00:00.000000000Z",
          "2022-05-10T00:00:00.000000000Z"
        ]
      },
      "interval": [
        "2022-04-20T00:00:00.000000000Z",
        "2022-05-10T00:00:00.000000000Z"
      ]
    }
  ],

In addition, for zone data retrieval with zone-depth query parameter (e.g. zone-depth=0-1), the temporal dimension at each zone-depth may be different?

[fmigneault]

A special consideration with 1D is whether the collection has a generic datetime property, and therefore indicates precisely the time at which it applies (like a datetime dimension of a single value), or whether the collection is a purely spatial, in which case it tells the user "time does not matter", and therefore repeating the value should not matter as well.

The first case with a single datetime property could be inserted only at that specific time (and all others null-padded) when mixing it with other single daetime property collections (but at different instants) or collections with multiple datetime values as dimension.

Another way to consider whether temporal mixing should happen or not is whether the ?datetime query parameter is provided or not. It could be fair that the default behaviour is set to return only the latest temporal instant (rather than all available ones by default) for all collections when ?datetime is omitted, and therefore the response is purely spatial, though maybe not technically "temporally aligned" across collections. If the ?datetime is explicitly requested, then the user can expect that some temporal stitching/padding/mixing/realigning must occur, and it is performed as needed across collections.

[fmigneault]

To clarify, for the return of two temporal collections on zone data retrieval, should the dimension list only show the merged datetime, or should it include two individual datetime dimensions from each collection?

It should be a single merged one. This will allow combining the corresponding zoneID/datetime properties where matches happen, which is the interesting part of the collection aggregation.

For the time being, this zoneID/datetime correspondence would happen "by chance" if 2 collections provide exactly equal temporal instants, but this could be improved later on with the resolution parameter to perform minor "realignments / resamplings" of the temporal instants based on a provided sampling period (eg: resolution=P1D to obtain datetimes in the above JSON example even if the original datatime are more specific than them).

[tik65536]

@fmigneault, after fighting with the datetime for the whole day. I finally fixed the zarr return.

There are several major changes to the /model/ogc_dggs/data_retrieval.py, and I would like to list them out as a summary here.

Mark the commit here in case need to revert the changes: 391a504

parquet collection provider :

1. I changed to use the include_properties to inject the datetime column into the SQL statement, instead of adding it at end of the SQL string processing.

data_retrieval.py:

1. For each collection_result dataframe created from get_data, the dataframe is set with a multi-index (zoneID and datetime) if it consists of datetime, otherwise, set only the zoneID as the dataframe index.

2. Changed to use merge with outer join when combining collection_result from different collections.

3. Replaced the data_col_dims with zone_level_dims. zone_level_dims is a dictionary with zone-level as key with a list of Dimension. The list is created from the pandas multi-index.

4. In the case of several zone depths, the dimension list in the DGGS-JSON return will have the corresponding datetime dimension at each zone depth. It is because I am not sure if the datetime dimension will be different for each zone-depth.

5. Changed to use xarray datatree to prepare the zarr+zip return.

Example of point 4 with 2 zone-depth.

  "dimensions": [
    {
      "name": "datetime_at_zone-depth-0",
      "grid": {
        "cellsCount": 4,
        "coordinates": [
          "2022-04-20T00:00:00.000000000",
          "2022-04-30T00:00:00.000000000",
          "2022-05-05T00:00:00.000000000",
          "2022-05-10T00:00:00.000000000"
        ]
      },
      "interval": [
        "2022-04-20T00:00:00.000000000",
        "2022-05-10T00:00:00.000000000"
      ]
    },
    {
      "name": "datetime_at_zone-depth-1",
      "grid": {
        "cellsCount": 4,
        "coordinates": [
          "2022-04-20T00:00:00.000000000",
          "2022-04-30T00:00:00.000000000",
          "2022-05-05T00:00:00.000000000",
          "2022-05-10T00:00:00.000000000"
        ]
      },
      "interval": [
        "2022-04-20T00:00:00.000000000",
        "2022-05-10T00:00:00.000000000"
      ]
    }
  ]

The correspoding value return at each zone-depth

"rcm_igeo7_parquet.band_1": [
      {
        "depth": 0,
        "shape": {
          "count": 4,
          "subZones": 1,
          "dimensions": {
            "datetime_at_zone-depth-0": 4
          }
        },
        "data": [
          null,
          null,
          534.1685791015625,
          665.5203857421875
        ]
      },
      {
        "depth": 1,
        "shape": {
          "count": 60,
          "subZones": 13,
          "dimensions": {
            "datetime_at_zone-depth-1": 4
          }
        },

[fmigneault]

I think the temporal dimension should be consistent across depths.

If the deepest depth yields a match, the higher ones above should (typically) also be available at corresponding datetimes. If not available (i.e.: missing from the data source), I believe it is still preferable to have the missing zones/datetime indicated explicitly as null to avoid having to perform the inverse resolution of matching datetimes entries across different variants of the per-depth dimension.

The advantage of the 1D array is that the offsets are consistent with the amount of zones at each depth. If it is also ensured to be constant across the extra dimensions (datetime) across depths, it is very easy to pinpoint a specific zone/datetime value since only the zone count varies at deeper levels, and that zone-depth count is well-known by a DGGS-aware client/library. The client receiving this response does not have to "double-check" the datetime entries at each level. It can directly use the expected ARRAY[zone-index X datetime-count + datetime-index] to find matching zone-depth values at the equivalent temporal instant. In other words, not making the dense arrays consistent across dimensions looses their advantage of being returned this way.

All other changes seem good. Thanks for the updates and work on it!

[fmigneault]

@tik65536
I noticed this issue on https://github.com/LandscapeGeoinformatics/pydggsapi/compare/alternative_dggrs_representations

I think the following is invalid according to https://typing.python.org/en/latest/spec/literal.html#illegal-parameters-for-literal-at-type-check-time

# pydggsapi/schemas/api/dggrs_providers.py
ZoneIdRepresentationType = ['textual', 'int', 'hexstring']

# pydggsapi/schemas/api/collections.py
dggrs_zoneid_repr: Literal[tuple(ZoneIdRepresentationType)] = "textual"

Instead, you would need to define the type the other way around:

# pydggsapi/schemas/api/dggrs_providers.py
ZoneIdRepresentationType = Literal['textual', 'int', 'hexstring']

# pydggsapi/schemas/api/collections.py
dggrs_zoneid_repr: ZoneIdRepresentationType = "textual"

# wherever the list is needed:
from typing import get_args
get_args(ZoneIdRepresentationType)

This the strategy I have reused across https://github.com/allixender/dggrid4py/blob/master/dggrid4py/dggrid_runner.py

[tik65536]

Thx for checking. It is fixed in 70d4d25