Merge pull request #241 from raphaelrpl/bdc-catalog-v1

🔥 Add minimal support to generate data cube from local dir and minor changes

Author: raphaelrpl

CHANGES.rst

@@ -21,19 +21,20 @@ Changes
 =======
 
 
-Version 0.8.3 (2022-10-03)
---------------------------
-
-- Add support to customize data cube path and data cube item (`#236 <https://github.com/brazil-data-cube/cube-builder/issues/236>`_)
-- Review docs related with new path format cubes
-
-
-Version 1.0.0a1 (2022-09-27)
+Version 1.0.0a1 (2022-10-20)
 ----------------------------
 
 - Add integration with BDC-Catalog 1.0 `#233 <https://github.com/brazil-data-cube/cube-builder/issues/233>`_.
 - Add support to generate datacube from Sentinel-2 Zip (experimental) `#222 <https://github.com/brazil-data-cube/cube-builder/issues/222>`_.
 - Improve docs setup
+- Add support to generate data cube from local directories (`#25 <https://github.com/brazil-data-cube/cube-builder/issues/25>`_)
+
+
+Version 0.8.3 (2022-10-03)
+--------------------------
+
+- Add support to customize data cube path and data cube item (`#236 <https://github.com/brazil-data-cube/cube-builder/issues/236>`_)
+- Review docs related with new path format cubes
 
 
 Version 0.8.2 (2022-09-21)

MANIFEST.in

@@ -28,3 +28,4 @@ recursive-include examples *.py
 recursive-include tests *.py
 recursive-include tests *.json
 recursive-include cube_builder *.yaml
+recursive-include examples *.json

USING.rst

@@ -267,6 +267,8 @@ Once the data cube definition is created, you can trigger a data cube using the
     if you would like to generate data cube using a different STAC provider. Remember that the ``--collection`` must exists.
 
 
+.. _create_sentinel:
+
 Creating data cube Sentinel 2
 -----------------------------
 
@@ -545,3 +547,181 @@ You can change any parameter with the command ``cube-builder configure`` with ``
     Be aware of what you are changing to do not affect the integrity of data cube.
     For example, changing the masking ``clear_data`` when there is a already area generated.
     Make sure to re-generate all the periods and tiles again.
+
+
+
+Advanced User Guide
+-------------------
+
+Generate data cubes from local dir
+++++++++++++++++++++++++++++++++++
+
+.. versionadded:: 1.0.0
+
+.. note::
+
+    To proceed this step, you will need to have a set of files in disk.
+    We will not provide this files since its just a briefing of this feature. You may consider
+    to have own files individually.
+
+
+With latest change of ``Cube-Builder`` (1.0), the user can generate data cubes using local directories containing
+images. This feature is useful to generate data cubes when the user has a bunch of image files locally and would like
+to apply temporal composition function over these files. In this case, a  ``STAC Server`` is not required.
+This feature can be achieved using parameters ``--local DIRECTORY`` and ``--format PATH_TO_FORMAT.json``.
+It follows the signature of `GDALCubes Formats <https://github.com/appelmar/gdalcubes/tree/master/formats>`_ to read
+directories.
+Essentially, a format contains the following properties:
+
+- ``images`` (REQUIRED): Object context representing how to seek for any image in disk.
+
+    - ``pattern`` (REQUIRED)
+- ``datetime`` (REQUIRED): Object context describing how to identify data times from any directory path or file path.
+
+    - ``pattern`` (REQUIRED): A regex expression describing how to match datetime.
+    - ``format`` (REQUIRED): ISO Format to get data time from `str`.
+
+- ``bands`` (REQUIRED): The data set bands that will be captured while recurring disk. You can also add extra fields to increment metadata of band. The following internal props are required:
+
+    - ``pattern``: Regex pattern to identify band in disk.
+    - ``nodata``: No data value for band.
+- ``tags`` (OPTIONAL): List of keywords describing the given format.
+- ``description`` (OPTIONAL): A detailed multi-line description to fully explain the format.
+
+You can check a minimal example in ``examples/formats/bdc-sentinel-2-l2a-cogs.json``, which offers support to
+locate ``Sentinel-2`` Cloud Optimized GeoTIFF files. You may also take a look in `GDALCubes Formats <https://github.com/appelmar/gdalcubes/tree/master/formats>`_
+for others formats.
+
+For this example, lets create a simple sentinel-2 data cube called ``S2-LOCAL-16D``. The signature is similar from
+:ref:`create_sentinel`. We just need to change the cube parameters to something like::
+
+        ...
+        "parameters": {
+            "mask": {
+                "clear_data": [4, 5, 6],
+                "not_clear_data": [2, 3, 7, 8, 9, 10, 11],
+                "nodata": 0,
+                "saturated_data": [1]
+            },
+            "local": "/path/to/local/files",
+            "recursive": true,
+            "format": "examples/formats/bdc-sentinel-2-l2a-cogs.json",
+            "pattern": ".tif"
+        }
+
+So you can create a data cube with command::
+
+    curl --location \
+         --request POST '127.0.0.1:5000/cubes' \
+         --header 'Content-Type: application/json' \
+         --data-raw '
+    {
+        "datacube": "S2-LOCAL-16D",
+        "datacube_identity": "S2-LOCAL",
+        "grs": "BRAZIL_SM",
+        "title": "Sentinel-2 SR - Cube LCF 16 days -v001",
+        "resolution": 10,
+        "version": 1,
+        "metadata": {
+            "license": "MIT",
+            "platform": {
+                "code": "Sentinel-2",
+                "instruments": "MSI"
+            }
+        },
+        "temporal_composition": {
+            "schema": "Cyclic",
+            "step": 16,
+            "unit": "day",
+            "cycle": {
+                "unit": "year",
+                "step": 1
+            }
+        },
+        "composite_function": "LCF",
+        "bands_quicklook": [
+            "B04",
+            "B03",
+            "B02"
+        ],
+        "bands": [
+            {"name": "B01", "common_name": "coastal", "data_type": "int16", "nodata": 0},
+            {"name": "B02", "common_name": "blue", "data_type": "int16", "nodata": 0},
+            {"name": "B03", "common_name": "green", "data_type": "int16", "nodata": 0},
+            {"name": "B04", "common_name": "red", "data_type": "int16", "nodata": 0},
+            {"name": "B05", "common_name": "rededge", "data_type": "int16", "nodata": 0},
+            {"name": "B06", "common_name": "rededge", "data_type": "int16", "nodata": 0},
+            {"name": "B07", "common_name": "rededge", "data_type": "int16", "nodata": 0},
+            {"name": "B08", "common_name": "nir", "data_type": "int16", "nodata": 0},
+            {"name": "B8A", "common_name": "nir08", "data_type": "int16", "nodata": 0},
+            {"name": "B11", "common_name": "swir16", "data_type": "int16", "nodata": 0},
+            {"name": "B12", "common_name": "swir22", "data_type": "int16", "nodata": 0},
+            {"name": "SCL", "common_name": "quality","data_type": "uint8", "nodata": 0}
+        ],
+        "indexes": [
+            {
+                "name": "EVI",
+                "common_name": "evi",
+                "data_type": "int16",
+                "nodata": -9999,
+                "metadata": {
+                    "expression": {
+                        "bands": [
+                            "B8A",
+                            "B04",
+                            "B02"
+                        ],
+                        "value": "(10000. * 2.5 * (B8A - B04) / (B8A + 6. * B04 - 7.5 * B02 + 10000.))"
+                    }
+                }
+            },
+            {
+                "name": "NDVI",
+                "common_name": "ndvi",
+                "data_type": "int16",
+                "nodata": -9999,
+                "metadata": {
+                    "expression": {
+                        "bands": [
+                            "B8A",
+                            "B04"
+                        ],
+                        "value": "10000. * ((B8A - B04)/(B8A + B04))"
+                    }
+                }
+            }
+        ],
+        "quality_band": "SCL",
+        "description": "This data cube contains all available images from Sentinel-2, resampled to 10 meters of spatial resolution, reprojected, cropped and mosaicked to BDC_SM grid and time composed each 16 days using LCF temporal composition function.",
+        "parameters": {
+            "mask": {
+                "clear_data": [4, 5, 6],
+                "not_clear_data": [2, 3, 7, 8, 9, 10, 11],
+                "nodata": 0,
+                "saturated_data": [1]
+            },
+            "local": "/path/to/local/files",
+            "recursive": true,
+            "format": "examples/formats/bdc-sentinel-2-l2a-cogs.json",
+            "pattern": ".tif"
+        }
+    }'
+
+After cube definition created, you can just use the command line ``cube-builder build-local``::
+
+    SQLALCHEMY_DATABASE_URI="postgresql://postgres:postgres@localhost/bdc" \
+    cube-builder build-local S2-LOCAL-16D \
+        --tiles 003011 \
+        --start-date 2021-08-29 \
+        --end-date 2021-09-13 \
+        --directory /path/to/local/files \
+        --format examples/formats/bdc-sentinel-2-l2a-cogs.json
+
+
+.. note::
+
+    This example just illustrate how to trigger the data cube using local directory. You may need to
+    change these values like ``directory``, ``format``, ``start-date``, ``end-date`` and ``tiles``.
+
+    Right now, it only supports using ``--tiles`` as parameter. It will be replaced in the next release
+    to support any ``Region of Interest (ROI)`` or shapefile.

cube_builder/__init__.py

@@ -21,6 +21,7 @@
 from json import JSONEncoder
 
 from bdc_catalog.ext import BDCCatalog
+from bdc_catalog.models import db
 from flask import Flask, abort, request
 from flask_redoc import Redoc
 from werkzeug.exceptions import HTTPException, InternalServerError

cube_builder/celery/tasks.py

@@ -30,7 +30,7 @@
 
 # Cube Builder
 from ..config import Config
-from ..constants import CLEAR_OBSERVATION_NAME, DATASOURCE_NAME, PROVENANCE_NAME, TOTAL_OBSERVATION_NAME, IDENTITY
+from ..constants import CLEAR_OBSERVATION_NAME, DATASOURCE_NAME, IDENTITY, PROVENANCE_NAME, TOTAL_OBSERVATION_NAME
 from ..models import Activity
 from ..utils import get_srid_column
 from ..utils.image import check_file_integrity, create_empty_raster, match_histogram_with_merges

cube_builder/cli.py

@@ -169,6 +169,50 @@ def build(datacube: str, collections: str, tiles: str, start: str, end: str, ban
     assert res['ok']
 
 
+@cli.command()
+@click.argument('datacube')
+@click.option('--tiles', type=click.STRING, required=True, help='Comma delimited tiles')
+@click.option('-d', '--directory', type=click.Path(exists=True, readable=True), required=True,
+              help='Directory containing files to read')
+@click.option('-f', '--format', type=click.STRING, required=True, help='Pattern to seek for files')
+@click.option('-r', '--recursive', is_flag=True, default=False, help='Recursive listing files')
+@click.option('-p', '--pattern', type=click.STRING, default='.tif', help='Pattern to seek for files')
+@click.option('--start-date', type=click.STRING, required=False, help='Start date')
+@click.option('--end-date', type=click.STRING, required=False, help='End date')
+@click.option('--force', '-f', is_flag=True, help='Build data cube without cache')
+@click.option('--token', type=click.STRING, help='Token to access data from STAC.')
+@click.option('--export-files', type=click.Path(writable=True), help='Export Identity Merges in file')
+@with_appcontext
+def build_local(datacube: str, tiles: str, directory: str, format: str,
+                force=False, with_rgb=False, export_files=None, **kwargs):
+    """Build data cube from local files."""
+    from .forms import DataCubeProcessForm
+
+    mask = kwargs.get('mask')
+
+    if mask:
+        kwargs['mask'] = eval(mask)
+
+    data = dict(
+        datacube=datacube,
+        tiles=tiles.split(','),
+        local=directory,
+        format=format,
+        force=force,
+        with_rgb=with_rgb,
+        **kwargs
+    )
+
+    parser = DataCubeProcessForm()
+    parsed_data = parser.load(data)
+    parsed_data['export_files'] = export_files
+    parsed_data['collections'] = None
+
+    res = CubeController.maestro(**parsed_data)
+
+    assert res['ok']
+
+
 @cli.command('configure')
 @click.argument('datacube')
 @click.option('--stac-url', type=click.STRING, help='STAC to search')

cube_builder/controller.py

@@ -31,8 +31,8 @@
 from werkzeug.exceptions import NotFound, abort
 
 from .constants import (CLEAR_OBSERVATION_ATTRIBUTES, CLEAR_OBSERVATION_NAME, COG_MIME_TYPE, DATASOURCE_ATTRIBUTES,
-                        PROVENANCE_ATTRIBUTES, PROVENANCE_NAME, TOTAL_OBSERVATION_ATTRIBUTES, TOTAL_OBSERVATION_NAME,
-                        IDENTITY)
+                        IDENTITY, PROVENANCE_ATTRIBUTES, PROVENANCE_NAME, TOTAL_OBSERVATION_ATTRIBUTES,
+                        TOTAL_OBSERVATION_NAME)
 from .forms import CollectionForm
 from .grids import create_grids
 from .models import Activity, CubeParameters
@@ -328,7 +328,7 @@ def get_cube(cls, cube_id: int):
             list(filter(lambda b: b.id == cube.quicklook[0].green, cube.bands))[0].name,
             list(filter(lambda b: b.id == cube.quicklook[0].blue, cube.bands))[0].name
         ]
-        dump_cube['extent'] = None
+        dump_cube['spatial_extent'] = None
         dump_cube['grid'] = cube.grs.name
         dump_cube['composite_function'] = dict(
             name=cube.composite_function.name,