Merge pull request #242 from brazil-data-cube/master

Merge 0.8.3 with 1.0

Author: raphaelrpl

CHANGES.rst

@@ -21,6 +21,13 @@ Changes
 =======
 
 
+Version 0.8.3 (2022-10-03)
+--------------------------
+
+- Add support to customize data cube path and data cube item (`#236 <https://github.com/brazil-data-cube/cube-builder/issues/236>`_)
+- Review docs related with new path format cubes
+
+
 Version 1.0.0a1 (2022-09-27)
 ----------------------------
 

INSTALL.rst

@@ -79,6 +79,7 @@ Install in development mode:
 
 .. code-block:: shell
 
+    $ pip3 install -U pip setuptools wheel
     $ pip3 install -e .[all]
 
 

USING.rst

@@ -126,20 +126,21 @@ The response will have status code ``201`` and the body::
 Creating data cube Landsat-8
 ----------------------------
 
-In order to create data cube ``Landsat-8`` monthly using the composite function ``Least Cloud Cover First`` (`LC8_30_1M_LCF`), use the following command to create data cube metadata::
+In order to create data cube ``Landsat-8`` monthly using the composite function ``Least Cloud Cover First`` (`LC8-1M`), use the following command to create data cube metadata::
 
     curl --location \
          --request POST '127.0.0.1:5000/cubes' \
          --header 'Content-Type: application/json' \
          --data-raw '
     {
-        "datacube": "LC8",
+        "datacube": "LC8-1M",
+        "datacube_identity": "LC8",
         "grs": "BRAZIL_MD",
         "title": "Landsat-8 (OLI) Cube Monthly - v001",
         "resolution": 30,
         "version": 1,
         "metadata": {
-            "license": "MIT",
+            "license": "proprietary",
             "platform": {
                 "code": "Landsat-8",
                 "instruments": "OLI"
@@ -230,7 +231,7 @@ Brazil Data Cube environment. If you don't have any account, please, refer to `B
 Once the data cube definition is created, you can trigger a data cube using the following command::
 
     SQLALCHEMY_DATABASE_URI="postgresql://postgres:postgres@localhost/bdc" \
-    cube-builder build LC8_30_1M_LCF \
+    cube-builder build LC8-1M \
         --stac-url https://brazildatacube.dpi.inpe.br/stac/ \
         --collections=LC8_SR-1 \
         --tiles=011009 \
@@ -244,12 +245,12 @@ Once the data cube definition is created, you can trigger a data cube using the
 
         # Using curl (Make sure to execute cube-builder run)
         curl --location \
-             --request POST '127.0.0.1:5000/start-cube' \
+             --request POST '127.0.0.1:5000/start' \
              --header 'Content-Type: application/json' \
              --data-raw '{
                 "stac_url": "https://brazildatacube.dpi.inpe.br/stac/",
                 "token": "<USER_BDC_TOKEN>",
-                "datacube": "LC8_30_1M_LCF",
+                "datacube": "LC8-1M",
                 "collections": ["LC8_SR-1"],
                 "tiles": ["011009"],
                 "start_date": "2019-01-01",
@@ -278,13 +279,14 @@ In order to create data cube Sentinel 2, use the following command to create dat
          --header 'Content-Type: application/json' \
          --data-raw '
     {
-        "datacube": "S2",
+        "datacube": "S2-16D",
+        "datacube_identity": "S2",
         "grs": "BRAZIL_SM",
         "title": "Sentinel-2 SR - Cube LCF 16 days -v001",
         "resolution": 10,
         "version": 1,
         "metadata": {
-            "license": "MIT",
+            "license": "proprietary",
             "platform": {
                 "code": "Sentinel-2",
                 "instruments": "MSI"
@@ -364,11 +366,11 @@ In order to create data cube Sentinel 2, use the following command to create dat
         }
     }'
 
-In order to trigger a data cube, we are going to use a collection `S2_10_16D_LCF-1` made with Surface Reflectance using Sen2Cor::
+In order to trigger a data cube, we are going to use a collection `S2-16-1` made with Surface Reflectance using Sen2Cor::
 
     # Using cube-builder command line
     SQLALCHEMY_DATABASE_URI="postgresql://postgres:postgres@localhost/bdc" \
-    cube-builder build S2_10_16D_LCF \
+    cube-builder build S2-16D \
         --stac-url https://brazildatacube.dpi.inpe.br/stac/ \
         --collections=S2_L2A-1 \
         --tiles=017019 \
@@ -389,12 +391,14 @@ In order to create data cube CBERS4 AWFI, use the following command to create da
          --header 'Content-Type: application/json' \
          --data-raw '
     {
-        "datacube": "CB4",
+        "datacube": "CB4-16D",
+        "datacube_identity": "CB4",
         "grs": "BRAZIL_LG",
         "title": "CBERS-4 (AWFI) SR - Data Cube LCF 16 days - v001",
         "resolution": 64,
         "version": 1,
         "metadata": {
+            "license": "cc-by-sa-3.0",
             "platform": {
               "code": "CBERS-4",
               "instruments": "AWFI"
@@ -473,7 +477,7 @@ Trigger data cube generation with following command:
 
     # Using cube-builder command line
     SQLALCHEMY_DATABASE_URI="postgresql://postgres:postgres@localhost/bdc" \
-    cube-builder build CB4_64_16D_LCF \
+    cube-builder build CB4-16D \
         --stac-url https://brazildatacube.dpi.inpe.br/stac/ \
         --collections=CBERS4_AWFI_L4_SR \
         --tiles=005004 \
@@ -489,21 +493,21 @@ When the ``Cube-Builder`` could not generate data cube for any unknown issue, yo
 with the same command you have dispatched::
 
     SQLALCHEMY_DATABASE_URI="postgresql://postgres:postgres@localhost/bdc" \
-    cube-builder build CB4_64_16D_LCF \
+    cube-builder build CB4-16D \
         --stac-url https://brazildatacube.dpi.inpe.br/stac/ \
         --collections=CBERS4_AWFI_L4_SR \
-        --tiles=022024 \
+        --tiles=005004 \
         --start=2019-01-01 \
         --end=2019-01-31 \
         --token <USER_BDC_TOKEN>
 
 It will reuse most of files that were already processed, executing only the failed tasks. If you notice anything suspicious or want to re-create theses files again, use the option ``--force``::
 
     SQLALCHEMY_DATABASE_URI="postgresql://postgres:postgres@localhost/bdc" \
-    cube-builder build CB4_64_16D_LCF \
+    cube-builder build CB4-16D \
         --stac-url https://brazildatacube.dpi.inpe.br/stac/ \
         --collections=CBERS4_AWFI_L4_SR \
-        --tiles=022024 \
+        --tiles=005004 \
         --start=2019-01-01 \
         --end=2019-01-31 \
         --token <USER_BDC_TOKEN> \
@@ -515,10 +519,10 @@ Data Cube Parameters
 
 The ``Cube-Builder`` supports a few parameters to be set during the data cube execution.
 
-In order to check the parameters associated with data cube ``CB4_64_16D_STK-1``, use the command::
+In order to check the parameters associated with data cube ``CB4-16D-1``, use the command::
 
     SQLALCHEMY_DATABASE_URI="postgresql://postgres:postgres@localhost/bdc" \
-    cube-builder show-parameters CB4_64_16D_LCF-1
+    cube-builder show-parameters CB4-16D-1
 
 
 The following output represents all the parameters related with the given data cube::
@@ -532,7 +536,7 @@ The following output represents all the parameters related with the given data c
 You can change any parameter with the command ``cube-builder configure`` with ``DataCubeName-Version``::
 
     SQLALCHEMY_DATABASE_URI="postgresql://postgres:postgres@localhost/bdc" \
-    cube-builder configure CB4_64_16D_LCF-1 --stac-url=AnySTAC
+    cube-builder configure CB4-16D-1 --stac-url=AnySTAC
 
 
 .. note::

cube_builder/celery/tasks.py

@@ -34,9 +34,8 @@
 from ..models import Activity
 from ..utils import get_srid_column
 from ..utils.image import check_file_integrity, create_empty_raster, match_histogram_with_merges
-from ..utils.processing import DataCubeFragments
 from ..utils.processing import blend as blend_processing
-from ..utils.processing import build_cube_path, compute_data_set_stats, get_cube_id, get_item_id, get_or_create_model
+from ..utils.processing import build_cube_path, compute_data_set_stats, get_item_id, get_or_create_model
 from ..utils.processing import merge as merge_processing
 from ..utils.processing import post_processing_quality, publish_datacube, publish_merge
 from ..utils.timeline import temporal_priority_timeline
@@ -107,16 +106,16 @@ def warp_merge(activity, band_map, mask, force=False, data_dir=None, **kwargs):
     collection_id = f'{record.collection_id}-{version}'
 
     if kwargs.get('reuse_data_cube'):
-        ref_cube_idt = get_cube_id(kwargs['reuse_data_cube']['name'])
+        ref_cube_idt = kwargs['reuse_data_cube']['name']
         # TODO: Should we search in Activity instead?
         merge_file_path = build_cube_path(ref_cube_idt, merge_date, tile_id,
                                           version=kwargs['reuse_data_cube']['version'], band=record.band,
-                                          prefix=data_dir)  # check published dir
+                                          prefix=data_dir, composed=False, **kwargs)  # check published dir
 
     if merge_file_path is None:
         merge_file_path = build_cube_path(record.warped_collection_id, merge_date,
                                           tile_id, version=version, band=record.band,
-                                          prefix=data_dir)
+                                          prefix=data_dir, composed=False, **kwargs)
 
         if activity['band'] == quality_band and len(activity['args']['datasets']):
             kwargs['build_provenance'] = True
@@ -267,9 +266,16 @@ def prepare_blend(merges, band_map: dict, reuse_data_cube=None, **kwargs):
 
     version = merges[0]['args']['version']
     identity_cube = merges[0]['warped_collection_id']
+    collection_ref: Collection = (
+        Collection.query()
+        .filter(Collection.name == merges[0]['collection_id'],
+                Collection.version == version)
+        .first()
+    )
+    composite_function = collection_ref.composite_function.alias
 
     if reuse_data_cube:
-        identity_cube = get_cube_id(reuse_data_cube['name'])
+        identity_cube = reuse_data_cube['name']
         version = reuse_data_cube['version']
 
     kwargs['mask'] = kwargs['mask'] or dict()
@@ -283,20 +289,20 @@ def prepare_blend(merges, band_map: dict, reuse_data_cube=None, **kwargs):
             if not was_reused:
                 logging.info(f'Applying post-processing in {str(quality_file)}')
                 post_processing_quality(quality_file, bands, identity_cube,
-                                        period, merges[0]['tile_id'], quality_band, band_map,
+                                        period, merges[0]['tile_id'],
+                                        band_map=band_map,
                                         version=version, block_size=block_size,
-                                        datasets=merges[0]['args']['datasets'])
+                                        datasets=merges[0]['args']['datasets'],
+                                        **kwargs)
             else:
                 logging.info(f'Skipping post-processing {str(quality_file)}')
 
-    def _is_not_stk(_merge):
+    def _is_not_stk(merge):
         """Control flag to generate cloud mask.
 
         This function is a utility to dispatch the cloud mask generation only for STK data cubes.
         """
-        collection_id = _merge['collection_id']
-        fragments = DataCubeFragments(collection_id)
-        return _merge['band'] == quality_band and fragments.composite_function not in ('STK', 'LCF')
+        return merge['band'] == quality_band and composite_function not in ('STK', 'LCF')
 
     for _merge in merges:
         # Skip quality generation for MEDIAN, AVG
@@ -306,6 +312,7 @@ def _is_not_stk(_merge):
 
         activity = activities.get(_merge['band'], dict(scenes=dict()))
 
+        activity['composite_function'] = composite_function
         activity['datacube'] = _merge['collection_id']
         activity['warped_datacube'] = _merge['warped_collection_id']
         activity['band'] = _merge['band']
@@ -382,10 +389,8 @@ def _is_not_stk(_merge):
     # Prepare list of activities to dispatch
     activity_list = list(activities.values())
 
-    datacube = activity_list[0]['datacube']
-
     # For IDENTITY data cube trigger, just publish
-    if DataCubeFragments(datacube).composite_function == IDENTITY:
+    if composite_function == 'IDT':
         task = publish.s(list(activities.values()), reuse_data_cube=reuse_data_cube, band_map=band_map, **kwargs)
         return task.apply_async()
 
@@ -427,9 +432,9 @@ def blend(activity, band_map, build_clear_observation=False, reuse_data_cube=Non
 
     logging.warning('Executing blend - {} - {}'.format(activity.get('datacube'), activity.get('band')))
 
-    return blend_processing(activity, band_map, kwargs['quality_band'], build_clear_observation,
-                            block_size=block_size, reuse_data_cube=reuse_data_cube,
-                            apply_valid_range=kwargs.get('apply_valid_range'))
+    return blend_processing(activity, band_map,
+                            build_clear_observation=build_clear_observation,
+                            block_size=block_size, reuse_data_cube=reuse_data_cube, **kwargs)
 
 
 @celery_app.task(queue=Config.QUEUE_PUBLISH_CUBE)
@@ -468,7 +473,7 @@ def publish(blends, band_map, quality_band: str, reuse_data_cube=None, **kwargs)
     merges = dict()
     blend_files = dict()
 
-    composite_function = DataCubeFragments(cube.name).composite_function
+    composite_function = cube.composite_function.alias
 
     quality_blend = dict(efficacy=100, cloudratio=0)
 
@@ -525,7 +530,8 @@ def publish(blends, band_map, quality_band: str, reuse_data_cube=None, **kwargs)
                 continue
 
             _merge_result[merge_date] = publish_merge(quick_look_bands, wcube, tile_id, merge_date, definition,
-                                                      reuse_data_cube=reuse_data_cube, srid=srid, data_dir=kwargs.get('data_dir'))
+                                                      reuse_data_cube=reuse_data_cube, srid=srid,
+                                                      **kwargs)
 
         try:
             db.session.commit()