Alternative DGGRS representations for a given collection

[fmigneault]

Currently, the dggrs_zoneid_repr (added in #52, relates to #11) is tightly coupled with the collections[*].collection_provider.

Using the following configuration as example:

dggrs:
  "1":
    IGEO7:
      classname: igeo7_dggrs_provider.IGEO7Provider
  "2":
    ISEA7H-Z7:
      classname: dggal_dggrs_provider.DGGALProvider
      parameters: {"grid": "ISEA7H_Z7"}  # no-op compared to IGEO7Provider, just naming alias for convenience
  "3":
    ISEA7H:
      classname: dggal_dggrs_provider.DGGALProvider
      parameters: {"grid": "ISEA7H_Z7"}  # ideally "ISEA7H", but not currently available in the provider implementation
  "4":
    ISEA7H-HEX:
      classname: igeo7_dggrs_provider.IGEO7Provider  # ??
      parameters: { "dggrs_zoneid_repr": "hex" }  # new property?  not the datasource encoding, but *desired* representation

collections:
  "2":
    manitoba-rcm-ard:
      # ... other metadata ...
      collection_provider:
        providerId: "igeo7-parquet-manitoba-rcm-ard"
        dggrsId: "IGEO7"
        dggrs_zoneid_repr: "textual"  # default
        datasource_id: "igeo7-parquet-manitoba-rcm-ard"

collection_providers:
  "1":
    igeo7-parquet-manitoba-rcm-ard:
      datasources: { "igeo7-parquet-manitoba-rcm-ard": {...} }

Providing the following equivalent endpoints is not possible (unless the DGGRS providers implement a convert() for it):

• /collections/manitoba-rcm-ard/dggs/IGEO7/...
• /collections/manitoba-rcm-ard/dggs/ISEA7H/...
• /collections/manitoba-rcm-ard/dggs/ISEA7H-Z7/...
• /collections/manitoba-rcm-ard/dggs/ISEA7H-HEX/...

Note

Above are equivalent since only the ZIRS changes, but the underlying DGGH is the same.

Currently, the above would require creating a "duplicate" manitoba-rcm-ard-IGEO7, manitoba-rcm-ard-ISEA7H, etc. collections pointing at their respective datasources with pre-converted zone IDs, since the collections[*].collection_provider.dggrs_zoneid_repr is a tight link. This can potentially consist of a lot of data duplication between the datasources for otherwise easily obtainable ZIRS correspondence. It also makes the "equivalent data" access less obvious to users since different collection IDs would represent the same results, rather than offering the same "collection of data" with different DGGS encodings.

It would make sense to implement dggal_dggrs_provider.DGGALProvider.convert for this case. In general, the int/str representation could always be converted for any DGGRS (via DGGAL), and hex from the int representation. Furthermore, the igeo7_dggrs_provider.IGEO7Provider.convert could handle the specific IGEO7/ISEA7H/ISEA7H-Z7 combination that has an additional case of different str representations (root rhombus vs Z7).

---

Another limitation of the current configuration structure seems to relate to the 1->1 mapping of collections[*].collection_provider.datasource_id.

If certain DGGRS cannot be easily converted between each other (eg: a hexagonal vs a square DGGRS), but happen to have been pre-processed into distinct collection_providers[*].datasources for corresponding "data", they still cannot be conceptually combined into a single collection ID.

It would be helpful if the global mapping of collections[*].collection_provider allowed multiple entries, for which "conceptual" collection of data can be provided by distinct collection_provider implementations, essentially "abstracting" the multi-provider configuration from the user's perspective accessing this "data concept".

[fmigneault]

@tik65536 I would be interested to get your thoughts regarding these considerations.

[tik65536]

I will need some time to think through this issue.

To start off the thinking, I would like to list out the assumptions:

• 1. The API should always adopt the textual representation of a zone ID for a DGGR if possible (i.e., the human-readable format).

And clarify the types of conversions we are facing:

• 1.) non-equivalent grids (e.g. H3 to IGEO7), it should be handled by the covert function defined in the dggrs provider.
• 2.) equivalent grids
  • a.) same indexing schema but different representation (e.g. IGEO7 Z7 in DIGITI_STRING and HEXSTRING representation)
    - with the above assumption (1), the API implemented only on textual conversion (i.e. textual -> other, other-> textual)
    - It is handled by the zone_id_from_textual and zone_id_to_textual
    - If the dggrs_zone_id_repr of the datasource is not equal to textual, then perform the necessary conversion.
    - b.) different indexing schemas (e.g. IGEO7 CPI indexing, ISEA7H rhombuses indexing)
    • There are two causes:
      • the indexing schemas are supported by a single DGGRS library (e.g. ISEAH7H (rhombuses) , ISEA7H_Z7 with dggal)
      • the indexing schemas are supported by different DGGRS libraries (e.g. ISEAH7H (rhombuses) with dggal, IGEO7 with dggrid)

For the case of 2a, the question is whether the API accepts non-textual zone ID representations for the DGGRS, like :

• /dggs/igeo7/zones/0023456
• /dggs/igeo7/zones/<hex string of 0023456>
  As stated in the assumption, the textual representation should always be considered first. However, suppose we want the API to support this feature. In that case, I suggest adding a query parameter (or including it in the body as JSON as a customisation for the API to avoid changes to the endpoint signature) to indicate the zone ID type from the client, for example :

The query /dggs/igeo7/zones/<hex string of 0023456> with body :

{ 
   "zone_id_repr": "hexstring"
}

The allowable values of this parameter are the same as dggrs_zone_id_repr defined in collection_provider ['int', 'hexstring', 'textual'], defaults to 'textual'.

Then, the API will perform an additional conversion zone_id_to_textual for the zone ID at the beginning, before it goes on to the normal process.
Another factor to consider is the zone ID representation within the return. Should it keep in the textual format or convert it back to the requested format?

For the case of 2b, I would prefer that each indexing schema has its own dggrs defined. For example :

• isea7h (with dggrid)
• isea7h_rhombuses (with dggal)

Otherwise, clients/users are easily confused about what index they are expecting. Then, for the dggrs that support alternative indexing schema, it can simply override the purpose of the convert function to provide index transformation. Then, collections with igeo7 as native dggrs will automatically support the alternative indexing schema, so users don't need to define the collections-dggrs pairs.

However, this mixture of purposes should be avoided. Or the API can add a new attribute to the dggrs, alternative_schema, to differentiate it from the non-equivalent grids conversion. The flow and logic of the new attribute are the same as those of the non-equivalent grids' convert function; it simply creates an explicit attribute to separate it.

But one point to note is that there may be different libraries for the samedggrs id, which may be slightly different, a good example is the indexing schema. The dggrs id used and defined in this API implementation is tightly coupled with the library that supports it. like the examples given at the start of the 2b case. But does this usage of dggrs id violate the API standard on the definition of dggrs id ?

@fmigneault , what do you think?

[fmigneault]

For the case of 2a, the question is whether the API accepts non-textual zone ID representations for the DGGRS, like :

* /dggs/igeo7/zones/0023456

* /dggs/igeo7/zones/<hex string of `0023456`>
  As stated in the assumption, the textual representation should always be considered first.

I don't think there should be a limitation for textual/non-textual access API-wise (at the very least there is none technologically as they are all path parameter strings anyway). However, in either case, the DGGS ID would be different, as in my previous example:

• /collections/manitoba-rcm-ard/dggs/IGEO7/...
• /collections/manitoba-rcm-ard/dggs/ISEA7H/...
• /collections/manitoba-rcm-ard/dggs/ISEA7H-Z7/...
• /collections/manitoba-rcm-ard/dggs/ISEA7H-HEX/...

In this case, ISEA7H-HEX would be the DGGS where zone IDs are expected as hex-string. That representation is no different from converting to rhombuses or Z7 representations.

Basically, once the "[any]->textual" conversion from the datasources has been done (as applicable and as currently performed), the zone ID representation can be converted once again (with convert() function) to other "textual" representations using DGGAL. Which conversions apply or not would be defined in the class mapping just like in the non-equivalent grids case (H3 to IGEO7).

I do not expect the API to receive any combination of zone ID representations and handle them automatically, as there might be some ambiguities. Since the ../dggs/{dggsID} endpoint is used with a specific zone ID representation, this already provides the necessary mechanism to switch between different kinds of zone IDs from the same collection. The returned zone IDs would also align with this dggsID, since the request was performed through that endpoint. Furthermore, the API provider still has control over which DGGRS conversions they want to support by adding them explicitly in their configuration.

The dggrs id used and defined in this API implementation is tightly coupled with the library that supports it. [...]
does this usage of dggrs id violate the API standard on the definition of dggrs id ?

The DGGRS ID is per defined by the /dggs/{dggsID} endpoint according to their respective [ogc-rel:dggrs-definition] reference. Therefore, it is perfectly valid to provide different implementations and representations.

For example, /collections/example/dggs/ISEA7H-authalic/... and /collections/example/dggs/ISEA7H-geodetic/... could both be returned, as long as their DGGRS definitions highlight this distinction (which also allows different ZIRs to be used).

In the specific case of ISEAH7H [DGGAL or DGGRID], ISEA7H_Z7 [DGGAL] and IGEO7 [DGGRID], they should all align (given the ZIRs conversion and the same vert0 configuration). I suggest using DGGAL for the ZIRs conversion simply because the functions are already there, and Jérôme put a lot of effort into validating them. The allowed conversions would still be restricted to the ones handled by convert().

[tik65536]

@fmigneault . Thanks for the input, it becomes clearer in the direction. But I would like to clarify the details regarding the DGGRs ISEA7H-HEX. I assume it is the IGEO7 z7 hex string using dggrid, as dggal doesn't support hex string representation.

If so, as I suggested above for the enhancement, this endpoint could be removed and handled by the IGEO7 endpoint. The client could send over the zone representation type (e.g. hexstring for this case) they used for the path parameter (.../dggs/IGEO7/zones/<zone ID in hexstring>) in the body, to trigger the conversion using the zone_id_to_textual for completing the query. And the server uses he exact zone ID representation for return.

And for conversion between different indexing schemas, the API can use the current convert logic, for instance, using dggal with the following dggrs :

/collections/manitoba-rcm-ard/dggs/ISEA7H/...
/collections/manitoba-rcm-ard/dggs/ISEA7H-Z7/...

The dggrs provider can declare the conversion by :

# DGGALProvider
# The list represents which grids are allowed for conversion.  
self.dggrs_conversion = {'ISEA7H':  ['ISEA7H-Z7'],
                                        'ISEA7H-Z7': ['ISEA7H'] }

Then, inside the convert function :

if (self.mygrid is in self.dggrs_conversion[targetdggrs.upper()]  ):
    # allow to convert
else:
   # error

The above logic is intended to illustrate the general idea; the actual implementation would be more complex, as the DGGALProvider aims to support multiple grids with a single class.

Furthermore, for a collection with dggrs_id set with ISEA7H, the collection dggrs would return two dggrs entries under the dggrs attributes for the query: /collections/manitoba-rcm-ard/dggs. The return is similar to the example from the conversion between non-equivalent grids]

Thanks,
Tik

[fmigneault]

Yes, that is what I had in mind.

Furthermore, for a collection with dggrs_id set with ISEA7H, the collection dggrs would return two dggrs entries under the dggrs attributes for the query: /collections/manitoba-rcm-ard/dggs.

There could be multiple more listed:

DGGRS	Example Zone ID
ISEA7H	C0-29-A
ISEA7H-Z7	0234
IGEO7	0234 (alias of ISEA7H-Z7, relevant if the client doesn't know "IGEO7" specifically, no-op conversion)
ISEA7H-HEX	0x273FFFFFFFFFFFFF (or just 273FFFFFFFFFFFFF)
ISEA7H-STR	2828260565988671487 (cast of the uint64 value without leading zeros, could allow with leading zeros as well)

If the original zone IDs from datasources is aligned with H3 DGGRS, then there could even be the additional H3 DGGS listed (doing the existing H3->IGEO7 conversion first, and then converting just the ZIR part of the IGEO7->ISEA7* variants).

You only need DGGAL to do the ISEA7H/ISEA7H-Z7 conversion. All others are simply typing conversion done in python directly:

assert int(0x273FFFFFFFFFFFFF) == 2828260565988671487
assert hex(2828260565988671487) == '0x273fffffffffffff'

[tik65536]

Okay, then, I will start the enhancement as soon as possible.

[fmigneault]

Thanks.
Let me know if there is something else you would like for me to synchronize implementations with you.
I have finished #61, so I can start looking at other capabilities/conformance classes of the standard.

[tik65536]

Thx for the fixes ~ The PR is merged to the main.
I have several tasks on hand currently, and will focus on dataset conversion for this week.
However, I will initiate the change for this issue later this week, as I have updated the DGGRS IGEO7 conversion to use int as the zone ID representation, along with the fix in #60.

May I ask you a question regarding the RCM dataset? I have several RCM TIFFs in my hands and would like to add a datetime column after the conversion. But I don't know the exact datetime for those TIFFs. The filenames of those TIFFs seem to include the datetime as a suffix, for example: rcm_2080248_HV_reproject_2022-05_05.tiff. Do you know where I can get the info?

Here is the URL for the above example: https://storage.googleapis.com/geo-assets/dggs-dev/ogc-dggs-pilot/staging/hartis/rcm_2080248_HV_reproject_2022-05_05.tif

Thx . Tik

[fmigneault]

I don't know if Hartis took the images from the same place or not, but I have been using the URIs from https://www.eodms-sgdot.nrcan-rncan.gc.ca/stac/collections/rcm-ard

[jerstlouis]

@fmigneault @tik65536
Regarding ISEA7H-HEX, from the OGC API - DGGS perspective, the {zoneId} (and the values for parent-zone query parameter, and the strings in the JSON response for Zone Queries) is always the textual ZIRS representation of the DGGRS.

It is fine to define a new DGGRS or variant which uses a stringification of hexadecimal integers to strings in a particular way (e.g., lowercase, uppercase, with a 0x prefix or a h prefix, or nothing at all), but that will need its own DGGRS definition with the zirs.textZIRS property explaining that.

[fmigneault]

@jerstlouis
Indeed.
All listed variants assume that some DGGRS definition is published at some point detailing this information, but we cannot wait until OGC publishes a registry to start working on them.

[jerstlouis]

@fmigneault

but we cannot wait until OGC publishes a registry to start working on them.

That is fine -- there is no requirement to link to an OGC registry definition -- the definition can be local.

I do still need to push the register forward ;)

[fmigneault]

the definition can be local.

Would that definition be somewhere else (eg: exposed via some file server) or provided through DGGS-API?
I think it can be anywhere as long as it resolves, but just double checking since I have not seen a DGGRS-specific endpoint.

[jerstlouis]

@fmigneault It can be anywhere. The Requirement (1) is that .../dggs/{dggrsId} includes a link with relation [ogc-rel:dggrs-definition].

There are suggestions for either /dggrs/{dggrsId} or /dggs/{dggrsId}/definition, but it could be anywhere, including on a different server or on the RAINBOW once the OGC register is operational (but having it local avoids reliance on a 3rd party, even though in practice clients are more likely to recognize the URI than be able to parse the complex DGGRS definition -- unless they use AI to try to make sense of it automatically.