replace federatedcatalogues with distributedsearch

Author: tomkralidis

default-sample.yml

@@ -54,21 +54,23 @@ logging:
 profiles:
     - apiso
 
-federatedcatalogues:
-    - id: arctic-sdi-csw
-      type: CSW
-      title: Arctic SDI
-      url: https://catalogue.arctic-sdi.org/csw
-    - id: pycsw-cite-demo
-      type: OARec
-      title: pycsw OGC CITE demo and Reference Implementation
-      uresultsrl: https://demo.pycsw.org/cite
-    - id: fedcat03
-      type: STAC-API
-      title: Copernicus Data Space Ecosystem (CDSE) asset-level STAC catalogue
-      url: https://stac.dataspace.copernicus.eu/v1
-      collections:
-          - daymet-annual-pr
+distributedsearch:
+    merge_results: false
+    catalogues:
+        - id: arctic-sdi-csw
+          type: CSW
+          title: Arctic SDI
+          url: https://catalogue.arctic-sdi.org/csw
+        - id: pycsw-cite-demo
+          type: OARec
+          title: pycsw OGC CITE demo and Reference Implementation
+          uresultsrl: https://demo.pycsw.org/cite
+        - id: fedcat03
+          type: STAC-API
+          title: Copernicus Data Space Ecosystem (CDSE) asset-level STAC catalogue
+          url: https://stac.dataspace.copernicus.eu/v1
+          collections:
+              - daymet-annual-pr
 
 manager:
     transactions: false

docker/compose/pycsw.yml

@@ -51,11 +51,12 @@ logging:
 profiles:
     - apiso
 
-federatedcatalogues:
-    - id: fedcat01
-      type: CSW
-      title: Arctic SDI
-      url: https://catalogue.arctic-sdi.org/csw
+distributedsearch:
+    catalogues:
+        - id: fedcat01
+          type: CSW
+          title: Arctic SDI
+          url: https://catalogue.arctic-sdi.org/csw
 
 manager:
     transactions: false

docker/helm/values.yaml

@@ -47,11 +47,12 @@ pycsw:
       # logfile: /tmp/pycsw.log
     profiles:
       - apiso
-    # federatedcatalogues:
-    #  - id: fedcat01
-    #    type: CSW 
-    #    title: Arctic SDI 
-    #    url: https://catalogue.arctic-sdi.org/csw
+    # distributedsearch:
+    #     catalogues:
+    #         - id: fedcat01
+    #           type: CSW 
+    #           title: Arctic SDI 
+    #           url: https://catalogue.arctic-sdi.org/csw
     manager:
       transactions: "false"
       allowed_ips:

docker/kubernetes/pycsw-configmap.yaml

@@ -56,11 +56,12 @@ data:
     profiles:
         - apiso
     
-    federatedcatalogues:
-        - id: fedcat01
-          type: CSW 
-          title: Arctic SDI 
-          url: https://catalogue.arctic-sdi.org/csw
+    distributedsearch:
+        catalogues:
+            - id: fedcat01
+              type: CSW 
+              title: Arctic SDI 
+              url: https://catalogue.arctic-sdi.org/csw
     
     manager:
         transactions: false

docker/pycsw.yml

@@ -51,11 +51,12 @@ logging:
 profiles:
     - apiso
 
-federatedcatalogues:
-    - id: fedcat01
-      type: CSW
-      title: Arctic SDI
-      url: https://catalogue.arctic-sdi.org/csw
+distributedsearch:
+    catalogues:
+        - id: fedcat01
+          type: CSW
+          title: Arctic SDI
+          url: https://catalogue.arctic-sdi.org/csw
 
 manager:
     transactions: false

docs/configuration.rst

@@ -16,7 +16,7 @@ pycsw's runtime configuration is defined by ``default.yml``.  pycsw ships with a
 - **level**: the logging level (see https://docs.python.org/library/logging.html#logging-levels)
 - **logfile**: the full file path to the logfile
 - **ogc_schemas_base**: base URL of OGC XML schemas tree file structure (default is http://schemas.opengis.net)
-- **federatedcatalogues**: arrray of distributed catalogue endpoints to be used for distributed searching, if requested by the client (see :ref:`distributedsearching`)
+- **distributedsearch**: distributed catalogue endpoints to be used for distributed searching, if requested by the client (see :ref:`distributedsearching`)
 - **pretty_print**: whether to pretty print the output (``true`` or ``false``).  Default is ``false``
 - **gzip_compresslevel**: gzip compression level, lowest is ``1``, highest is ``9``.  Default is off.  **NOTE**: if gzip compression is already enabled via your web server, do not enable this directive (or else the server will try to compress the response twice, resulting in degraded performance)
 - **domainquerytype**: for GetDomain operations, how to output domain values.  Accepted values are ``list`` and ``range`` (min/max). Default is ``list``

docs/distributedsearching.rst

@@ -15,7 +15,7 @@ Distributed Searching
 CSW 2 / 3
 ---------
 
-pycsw has the ability to perform distributed searching against other CSW servers.  Distributed searching is disabled by default; to enable, ``federatedcatalogues`` must be set.  A CSW client must issue a GetRecords request with ``csw:DistributedSearch`` specified, along with an optional ``hopCount`` attribute (see subclause 10.8.4.13 of the CSW specification).  When enabled, pycsw will search all specified catalogues and return a unified set of search results to the client.  Due to the distributed nature of this functionality, requests will take extra time to process compared to queries against the local repository.
+pycsw has the ability to perform distributed searching against other CSW servers.  Distributed searching is disabled by default; to enable, ``distributedsearch`` must be set.  A CSW client must issue a GetRecords request with ``csw:DistributedSearch`` specified, along with an optional ``hopCount`` attribute (see subclause 10.8.4.13 of the CSW specification).  When enabled, pycsw will search all specified catalogues and return a unified set of search results to the client.  Due to the distributed nature of this functionality, requests will take extra time to process compared to queries against the local repository.
 
 Scenario: Federated Search
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -56,19 +56,21 @@ in the CSW-all configuration:
 
 .. code-block:: yaml
 
-  federatedcatalogues:
-      - id: fedcat01
-        type: CSW
-        title: Federated catalogue 1
-        url: http://localhost/pycsw/csw.py?config=CSW-1.yml
-      - id: fedcat02
-        type: CSW
-        title: Federated catalogue 2
-        url: http://localhost/pycsw/csw.py?config=CSW-2.yml
-      - id: fedcat03
-        type: CSW
-        title: Federated catalogue 3
-        url: http://localhost/pycsw/csw.py?config=CSW-3.yml
+  distributedsearch:
+      merge_results: false
+      catalogues:
+          - id: fedcat01
+            type: CSW
+            title: Federated catalogue 1
+            url: http://localhost/pycsw/csw.py?config=CSW-1.yml
+          - id: fedcat02
+            type: CSW
+            title: Federated catalogue 2
+            url: http://localhost/pycsw/csw.py?config=CSW-2.yml
+          - id: fedcat03
+            type: CSW
+            title: Federated catalogue 3
+            url: http://localhost/pycsw/csw.py?config=CSW-3.yml
  
 At which point a CSW client request to CSW-all with ``distributedsearch=TRUE``, while specifying an optional ``hopCount``.  Query network topology:
 
@@ -101,42 +103,49 @@ Experimental support for distibuted searching is available in pycsw's OGC API -
 
 .. note::
 
-   The ``federatedcatalogues`` directives must point to an OGC API - Records **collections** endpoint.
+   The ``distributedsearch.catalogues`` directives must point to an OGC API - Records **collections** endpoint.
 
 .. code-block:: yaml
 
-  federatedcatalogues:
-      - id: fedcat01
-        type: OARec
-        title: Federated catalogue 1
-        url: https://example.org/collections/collection1
-      - id: fedcat02
-        type: OARec
-        title: Federated catalogue 2
-        url: https://example.org/collections/collection2
+  distributedsearch
+      catalogues:
+          - id: fedcat01
+            type: OARec
+            title: Federated catalogue 1
+            url: https://example.org/collections/collection1
+          - id: fedcat02
+            type: OARec
+            title: Federated catalogue 2
+            url: https://example.org/collections/collection2
  
 With the above configured, a distributed search can be invoked as follows:
 
 http://localhost/collections/metadata:main/items?distributedSearch=true
 
+Merging results
+^^^^^^^^^^^^^^^
+
+When `distributedsearch.merge_results` exists and is set to ``true``, pycsw will merge all results in ``federatedSearchResults``.  To prevent identifier collision, merged federated search results will have identifiers prefixed by their catalogue ``id`` (as defined in ``distributedsearch.catalogues[*].id``.  In addition, a ``federatedCatalogueId`` property is added to the feature with the catalogue id.
+
 STAC API
 --------
 
 Experimental support for distibuted searching is available in pycsw's STAC API support to allow for searching remote services.  The implementation uses the same approach as described above.
 
 .. note::
 
-   The ``federatedcatalogues`` directives must point to a STAC API endpoint.
+   The ``distributedsearch.catalogues`` directives must point to a STAC API endpoint.
 
 .. code-block:: yaml
 
-  federatedcatalogues:
-    - id: fedcat03
-      type: STAC-API
-      title: Copernicus Data Space Ecosystem (CDSE) asset-level STAC catalogue
-      url: https://stac.dataspace.copernicus.eu/v1
-      collections:
-          - daymet-annual-pr
+  distributedsearch:
+      catalogues:
+          - id: fedcat03
+            type: STAC-API
+            title: Copernicus Data Space Ecosystem (CDSE) asset-level STAC catalogue
+            url: https://stac.dataspace.copernicus.eu/v1
+            collections:
+                - daymet-annual-pr
 
 
 .. note::
@@ -148,4 +157,10 @@ With the above configured, a distributed search can be invoked as follows:
 
 http://localhost/stac/search?distributedSearch=true
 
+Merging results
+^^^^^^^^^^^^^^^
+
+Merging behaviour is implemented in the same manner as OGC API - Records support.
+
+
 .. _`OGC API - Records - Part 4: Federated Search`: https://github.com/opengeospatial/ogcapi-records/blob/master/extensions/federated-search/document.adoc

pycsw/core/admin.py

@@ -618,7 +618,9 @@ def cli_migrate_config(ctx, config, verbosity):
             'inspire': {}
         },
         'profiles': [],
-        'federatedcatalogues': [],
+        'distributedsearch': {
+            'catalogues': []
+        },
         'repository': {}
     }
 
@@ -634,7 +636,7 @@ def cli_migrate_config(ctx, config, verbosity):
         elif name == 'federatedcatalogues':
             dict_[name] = []
             for count, fc in enumerate(value.split(',')):
-                dict_[name].append({'id': f'fedcat{count}', 'url': fc})
+                dict_['distributedsearch']['catalogues'].append({'id': f'fedcat{count}', 'url': fc})
         else:
             dict_['server'][name] = get_typed_value(value)
 

pycsw/ogc/api/records.py

@@ -483,7 +483,7 @@ def collection(self, headers_, args, collection='metadata:main'):
             'hreflang': self.config['server']['language']
         }]
 
-        if collection == 'metadata:main' and 'federatedcatalogues' in self.config:
+        if collection == 'metadata:main' and 'distributedsearch' in self.config:
             LOGGER.debug('Adding federated catalogues')
             response['links'].append({
                 'rel': 'http://www.opengis.net/def/rel/ogc/1.0/federatedCatalogues',
@@ -818,7 +818,10 @@ def items(self, headers_, json_post_data, args, collection='metadata:main'):
         distributed = str2bool(args.get('distributedsearch', False))
 
         if distributed:
-            for fc in self.config.get('federatedcatalogues', []):
+            distributed_search = self.config.get('distributedsearch', {})
+            merge_results = distributed_search.get('merge_results', False)
+
+            for fc in distributed_search.get('catalogues', []):
                 if fc['type'] != 'OARec':
                     LOGGER.debug(f"Federated catalogue type {fc['type']} not supported; skipping")
                     continue
@@ -833,10 +836,19 @@ def items(self, headers_, json_post_data, args, collection='metadata:main'):
                     args.pop('distributedsearch')
                     fc_results = w.collection_items(fc_collection, **args)
                     for feature in fc_results['features']:
+                        if merge_results:
+                            feature['id'] = f"{fc['id']}::{feature['id']}"
+                            feature['federatedCatalogueId'] = fc['id']
                         response['federatedSearchResults'][fc['id']]['features'].append(feature)
                 except Exception as err:
                     LOGGER.warning(err)
 
+            if merge_results:
+                LOGGER.debug('Merging federated search results into main items')
+                fsrs = response.pop('federatedSearchResults', None)
+                for key, value in fsrs.items():
+                    response['features'].extend(value['features'])
+
         LOGGER.debug('Creating links')
 
         link_args = {**args}
@@ -970,7 +982,7 @@ def item(self, headers_, args, collection, item):
             distributed = str2bool(args.get('distributed', False))
 
             if distributed:
-                for fc in self.config.get('federatedcatalogues', []):
+                for fc in self.config.get('distributedsearch', {}).get('catalogues', []):
                     if fc['type'] != 'OARec':
                         LOGGER.debug(f"Federated catalogue type {fc['type']} not supported; skipping")
                         continue
@@ -1180,7 +1192,7 @@ def federated_catalogues(self, headers_, args, collection):
         fedcats = []
 
         if collection == 'metadata:main':
-            for fedcat_ in self.config.get('federatedcatalogues', []):
+            for fedcat_ in self.config.get('distributedsearch', {}).get('catalogues', []):
                 if fedcat_['type'] == 'OARec':
                     fedcats.append(fedcat_)
 
@@ -1213,7 +1225,7 @@ def federated_catalogue(self, headers_, args, collection, catalogue):
         fedcat = None
 
         if collection == 'metadata:main':
-            fedcats = self.config.get('federatedcatalogues')
+            fedcats = self.config.get('distributedsearch', {}).get('catalogues', [])
             for fedcat_ in fedcats:
                 if fedcat_['id'] == catalogue and fedcat_['type'] == 'OARec':
                     fedcat = fedcat_

pycsw/ogc/csw/csw2.py

@@ -825,15 +825,15 @@ def getrecords(self):
 
         hopcount = int(self.parent.kvp.get('hopcount', 2)) - 1
 
-        if ('federatedcatalogues' in self.parent.config and
+        if ('distributedsearch' in self.parent.config and
                 self.parent.kvp.get('distributedsearch') and
                 hopcount > 0):
 
             LOGGER.debug('DistributedSearch specified (hopCount: %s).', hopcount)
 
             from owslib.csw import CatalogueServiceWeb
             from owslib.ows import ExceptionReport
-            for fedcat in self.parent.config.get('federatedcatalogues', []):
+            for fedcat in self.parent.config.get('distributedsearch', {}).get('catalogues', []):
                 if fedcat['type'] != 'CSW':
                     LOGGER.debug(f"Federated catalogue type {fedcat['type']} not supported; skipping")
                     continue