Merge branch 'master' into windows

marioba · marioba · commit 8e390a3579b7 · 2022-06-10T14:41:57.000+02:00
diff --git a/README.md b/README.md
@@ -49,201 +49,9 @@ export QFIELDCLOUD_TOKEN=017478ee2464440cb8d3e98080df5e5a
 qfieldcloud-cli --json list-projects
 ```
 
-### Global options overview
 
-```
--U, --url TEXT                  URL to the QFieldCloud API endpoint. If not
-                                passed, gets the value from QFIELDCLOUD_URL
-                                environment variable. Default:
-                                https://app.qfield.cloud/api/v1/
--u, --username TEXT             Username or email.
--p, --password TEXT
--t, --token TEXT                Session token.
---json / --human                Output the result as newline formatted json. Default: False
---verify-ssl / --no-verify-ssl  Verify SSL. Default: True
---help                          Show this message and exit.
-```
-
-Environment variables can be used instead of passing some common global options.
-
-- `QFIELDCLOUD_API` - QFieldCloud API endpoint URL
-- `QFIELDCLOUD_USERNAME` - QFieldCloud username or email. Requires `QFIELDCLOUD_PASSWORD` to be set.
-- `QFIELDCLOUD_PASSWORD` - Password. Requires `QFIELDCLOUD_USERNAME` to be set.
-- `QFIELDCLOUD_TOKEN` - Token that can be used instead of passing username and password. It can be obtained by running `qfieldcloud-cli login`.
-- `QFIELDCLOUD_VERIFY_SSL` - When set to `0` has the same effect as passing `--no-verify-ssl`.
-
-### Commands overview
-
-```
-  login             Login to QFieldCloud.
-  logout            Logout and expire the token.
-  list-projects     List QFieldCloud projects.
-  list-files        List QFieldCloud project files.
-  create-project    Creates a new empty QFieldCloud project.
-  delete-project    Deletes a QFieldCloud project.
-  upload-files      Upload files to a QFieldCloud project.
-  download-files    Download QFieldCloud project files.
-  delete-files      Delete QFieldCloud project files.
-  list-jobs         List project jobs.
-  job-trigger       Triggers a new job.
-  job-status        Get job status.
-  package-latest    Check project packaging status.
-  package-download  Download packaged QFieldCloud project files.
-```
-
-#### login
-
-Login to QFieldCloud.
-
-```
-qfieldcloud-cli login [OPTIONS] USERNAME PASSWORD
-```
-
-#### logout
-
-Logout from QFieldCloud.
-
-```
-qfieldcloud-cli logout
-```
-
-#### list-projects
-
-List QFieldCloud projects.
-
-```
-qfieldcloud-cli list-projects [OPTIONS]
-
-Options:
-  --include-public / --no-public  Includes the public project in the list. Default: False
-```
-
-#### list-files
-
-List QFieldCloud project files.
-
-```
-qfieldcloud-cli list-files [OPTIONS] PROJECT_ID
-```
-
-#### create-project
-
-Creates a new empty QFieldCloud project.
-
-```
-qfieldcloud-cli create-project [OPTIONS] NAME
-
-Options:
-  --owner TEXT                Owner of the project. If omitted, the current
-                              user is the owner.
-  --description TEXT          Description of the project.
-  --is-public / --is-private  Mark the project as public.
-```
-
-#### delete-project
-
-Deletes a QFieldCloud project.
-
-```
-qfieldcloud-cli delete-project [OPTIONS] PROJECT_ID
-```
-
-#### upload-files
+More detailed documentation can be found [here](https://docs.qfield.org/reference/qfieldcloud/sdk/)
 
-Upload files to a QFieldCloud project.
-
-```
-qfieldcloud-cli upload-files [OPTIONS] PROJECT_ID PROJECT_PATH
-
-Options:
-  --filter TEXT                   Do not upload the whole project, but only
-                                  the files which match the glob.
-  --throw-on-error / --no-throw-on-error
-                                  If any project file upload fails stop
-                                  uploading the rest. Default: False
-```
-
-#### download-files
-
-Download QFieldCloud project files.
-
-```
-qfieldcloud-cli download-files [OPTIONS] PROJECT_ID LOCAL_DIR
-
-Options:
-  --filter TEXT                   Do not download the whole project, but only
-                                  the files which match the glob.
-  --throw-on-error / --no-throw-on-error
-                                  If any project file downloads fails stop
-                                  downloading the rest. Default: False
-```
-
-#### delete-files
-
-Delete QFieldCloud project files.
-
-```
-qfieldcloud-cli delete-files [OPTIONS] PROJECT_ID PATHS...
-
-Options:
-  --throw-on-error / --no-throw-on-error
-                                  If any project file delete operations fails
-                                  stop, stop deleting the rest. Default: False
-```
-
-#### job-list
-
-List project jobs.
-
-```
-qfieldcloud-cli list-jobs [OPTIONS] PROJECT_ID
-
-Options:
-  --type JOBTYPES  Job type. One of package, delta_apply or
-                   process_projectfile.
-```
-
-#### job-trigger
-
-Triggers a new job.
-
-```
-qfieldcloud-cli job-trigger [OPTIONS] PROJECT_ID JOB_TYPE
-
-Options:
-  --force / --no-force  Should force creating a new job. Default: False
-```
-
-#### job-status
-
-Get job status.
-
-```
-qfieldcloud-cli job-status [OPTIONS] JOB_ID
-```
-
-#### package-latest
-
-Check project packaging status.
-
-```
-qfieldcloud-cli package-latest [OPTIONS] PROJECT_ID
-```
-
-#### package-download
-
-Download packaged QFieldCloud project files.
-
-```
-qfieldcloud-cli package-download [OPTIONS] PROJECT_ID LOCAL_DIR
-
-Options:
-  --filter TEXT                   Do not download the whole packaged project,
-                                  but only the files which match the glob.
-  --throw-on-error / --no-throw-on-error
-                                  If any packaged file downloads fails stop
-                                  downloading the rest. Default: False
-```
 
 ## Development
 
diff --git a/src/bin/qfieldcloud_cli.py b/src/bin/qfieldcloud_cli.py
@@ -303,14 +303,18 @@ def upload_files(ctx, project_id, project_path, filter_glob, throw_on_error):
     "--throw-on-error/--no-throw-on-error",
     help="If any project file downloads fails stop downloading the rest. Default: False",
 )
+@click.option(
+    "--force-download/--no-force-download",
+    help="Download file even if it already exists locally. Default: False",
+)
 @click.pass_context
-def download_files(ctx, project_id, local_dir, filter_glob, throw_on_error):
+def download_files(ctx, project_id, local_dir, filter_glob, throw_on_error, force_download):
     """Download QFieldCloud project files."""
 
     log(f'Downloading project "{project_id}" files to {local_dir}…')
 
     files = ctx.obj["client"].download_project(
-        project_id, local_dir, filter_glob, throw_on_error, show_progress=True
+        project_id, local_dir, filter_glob, throw_on_error, show_progress=True, force_download=force_download,
     )
 
     if ctx.obj["format_json"]:
@@ -459,14 +463,18 @@ def package_latest(ctx, project_id):
     "--throw-on-error/--no-throw-on-error",
     help="If any packaged file downloads fails stop downloading the rest. Default: False",
 )
+@click.option(
+    "--force-download/--no-force-download",
+    help="Download file even if it already exists locally. Default: False",
+)
 @click.pass_context
-def package_download(ctx, project_id, local_dir, filter_glob, throw_on_error):
+def package_download(ctx, project_id, local_dir, filter_glob, throw_on_error, force_download):
     """Download packaged QFieldCloud project files."""
 
     log(f'Downloading the latest project "{project_id}" package files to {local_dir}…')
 
     files = ctx.obj["client"].package_download(
-        project_id, local_dir, filter_glob, throw_on_error, show_progress=True
+        project_id, local_dir, filter_glob, throw_on_error, show_progress=True, force_download=force_download,
     )
 
     if ctx.obj["format_json"]:
diff --git a/src/qfieldcloud_sdk/sdk.py b/src/qfieldcloud_sdk/sdk.py
@@ -279,13 +279,15 @@ def download_project(
         filter_glob: str = None,
         throw_on_error: bool = False,
         show_progress: bool = False,
+        force_download: bool = False,
     ) -> List[Dict]:
         """Download the specified project files into the destination dir.
 
         Args:
             project_id: id of the project to be downloaded
             local_dir: destination directory where the files will be downloaded
             filter_glob: if specified, download only the files which match the glob, otherwise download all
+            force_download (bool, optional): Download file even if it already exists locally. Defaults to False.
         """
 
         files = self.list_remote_files(project_id)
@@ -298,6 +300,7 @@ def download_project(
             filter_glob,
             throw_on_error,
             show_progress,
+            force_download,
         )
 
     def list_jobs(self, project_id: str, job_type: JobTypes = None) -> Dict[str, Any]:
@@ -442,13 +445,15 @@ def package_download(
         filter_glob: str = None,
         throw_on_error: bool = False,
         show_progress: bool = False,
+        force_download: bool = False,
     ) -> List[Dict]:
         """Download the specified project packaged files into the destination dir.
 
         Args:
             project_id: id of the project to be downloaded
             local_dir: destination directory where the files will be downloaded
             filter_glob: if specified, download only packaged files which match the glob, otherwise download all
+            force_download (bool, optional): Download file even if it already exists locally. Defaults to False.
         """
         project_status = self.package_latest(project_id)
 
@@ -467,6 +472,7 @@ def package_download(
             filter_glob,
             throw_on_error,
             show_progress,
+            force_download,
         )
 
     def download_files(
@@ -478,6 +484,7 @@ def download_files(
         filter_glob: str = None,
         throw_on_error: bool = False,
         show_progress: bool = False,
+        force_download: bool = False,
     ) -> List[Dict]:
         """Download project files.
 
@@ -489,7 +496,7 @@ def download_files(
             filter_glob (str, optional): Download only files matching the glob pattern. If None download all. Defaults to None.
             throw_on_error (bool, optional): Throw if download error occurres. Defaults to False.
             show_progress (bool, optional): Show progress bar in the console. Defaults to False.
-
+            force_download (bool, optional): Download file even if it already exists locally. Defaults to False.
         Raises:
             QFieldCloudException: if throw_on_error is True, throw an error if a download request fails.
 
@@ -508,6 +515,9 @@ def download_files(
 
         for file in files_to_download:
             local_filename = Path(f'{local_dir}/{file["name"]}')
+            md5sum = None
+            if not force_download:
+                md5sum = file.get("md5sum", None)
 
             try:
                 self.download_file(
@@ -516,6 +526,7 @@ def download_files(
                     local_filename,
                     file["name"],
                     show_progress,
+                    md5sum,
                 )
                 file["status"] = FileTransferStatus.SUCCESS
             except QfcRequestException as err:
@@ -542,6 +553,7 @@ def download_file(
         local_filename: Path,
         remote_filename: Path,
         show_progress: bool,
+        remote_md5sum: str = None,
     ) -> requests.Response:
         """Download a single project file.
 
@@ -551,13 +563,27 @@ def download_file(
             local_filename (Path): Local filename
             remote_filename (Path): Remote filename
             show_progress (bool): Show progressbar in the console
+            remote_md5sum (str, optional): The md5sum of the remote file. If is None, the download of the file happens even if it already exists locally. Defaults to None.
 
         Raises:
             NotImplementedError: Raised if unknown `download_type` is passed
 
         Returns:
             requests.Response: the response object
         """
+
+        if remote_md5sum and local_filename.exists():
+            if self._get_md5sum(str(local_filename)) == remote_md5sum:
+                if show_progress:
+                    print(
+                        f"{remote_filename}: Already present locally. Download skipped."
+                    )
+                else:
+                    logging.info(
+                        f'Skipping download of "{remote_filename}" because it is already present locally'
+                    )
+                return
+
         if download_type == FileTransferType.PROJECT:
             url = f"files/{project_id}/{remote_filename}"
         elif download_type == FileTransferType.PACKAGE:
