v1.1.0 - KMS Support

Author: pscheri

.gitignore

@@ -5,11 +5,12 @@ __pycache__
 .venv
 *.egg-info
 .vscode
+.DS_Store
 
 # CDK asset staging directory
 .cdk.staging
 cdk.out
 auth
-transfer_sync_service/lambda/boto3_lambda_layer/*
+transfer_sync_service/lambda/boto3_*/*
 transfer_sync_service/lambda/sync_files/*
 !transfer_sync_service/lambda/sync_files/sync_files.py

CHANGELOG.md

@@ -2,6 +2,22 @@
 
 All notable changes to this project will be documented in this file.
 
+## 1.1.0 - 2024-10-25
+### Added features
+- Added the ability to define an optional KMS Key for the target buckets in the configuration files. This allows you to set up a default encryption with KMS on the target buckets.
+- Added KMS Encryption for the reporting bucket with a new dedicated key.
+- Added the ability to define Permission Boundaries for all the created roles.
+
+### Changed
+- Modified `transfer_sync_service_stack` and how the lambda layer is created for boto3.
+- Centralized all the solution parameters in `configuration/solution_parameters/parameters.json`.
+- Updated CLI to support the new KMS Configuration.
+
+### Dependencies
+- Bump lambda-powertools from 2.40.1 to 3.2.0.
+- Bump boto3 from 1.34.134 to 1.35.47 for lambda layers and environment.
+- Bump cdk-monitoring-constructs from 8.1.0 to 8.3.2.
+
 ## 1.0.2 - 2024-09-25
 
 ### Added features

README.md

@@ -45,7 +45,7 @@ A combination of Lambda, Step Functions and Transfer Family features facilitates
    - The connector uses the configured security policy, trusted public keys and secrets stored in AWS Secrets Manager to ensure secure communication with the remote server.
 
 4. **S3 Buckets**
-   - Only one bucket is created by this solution to store the results generated by the Transfer Family SFTP Connector when Listing the remote SFTP directories.
+   - Only one bucket is created by this solution to store the results generated by the Transfer Family SFTP Connector when Listing the remote SFTP directories. This bucket is encrypted using a KMS Customer managed keys created by the solution.
    - The solution can use as many S3 Buckets as needed for a target for the Transfer Family SFTP Connector Sync process, when the files are copied from the remote SFTP to local the local S3 Bucket. These S3 Buckets are defined in the Configuration Files.
 
 5. **AWS Secrets Manager**
@@ -81,7 +81,8 @@ The configuration file structure and content needs the following data:
         {
             "LocalRepository": {
                 "BucketName": <Local Bucket Name>,
-                "Prefix": <Local Prefix>
+                "Prefix": <Local Prefix>,
+                (OPTIONAL) "KmsKeyArn": <KMS Key ARN used for the Bucket default encryption configuration>
             },
             "RemoteFolders": {
                 "Folder": <Remote Folder to Sync>,
@@ -118,6 +119,13 @@ For the Cron expression, you can use any of the pre-defined TAGs for simplicity
 | @saturday | 0 0 ? * 7 * |
 | @every10min | 0/10 * * * ? * |
 
+### Target Bucket KMS Encryption
+
+The solution supports target S3 Buckets that use [server-side encryption with AWS KMS (SSE-KMS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-kms-encryption.html).
+If your target S3 Bucket is encrypted using KMS, you must specify the ARN of the KMS Key used for encryption in your configuration file under: `SyncSettings > LocalRepository > KmsKeyArn`
+
+**Note:** The `KmsKeyArn` parameter is optional. **Only include it if your target bucket uses KMS encryption.**
+
 ### Replaceable Tags in Remote Folder Paths
 
 The solution supports the use of replaceable tags in the remote folder paths. This feature allows for dynamic folder selection based on the current date (in UTC). The following tags are available:
@@ -184,6 +192,19 @@ This project is built using Python3 and CDK, before you start, make sure to have
 * Python venv
 
 ## Deployment
+
+### Permission Boundaries
+If you are enforcing the usage of IAM Permission Boundaries for IAM Roles created in the account, you can update the [solution parameters file](configuration/solution_parameters/parameters.json) and add the managed policy ARN you are using to the `permission_boundary_policy_arn` parameter.
+
+**Note:** 
+- This step is optional. Only add this parameter if you're enforcing IAM Permission Boundaries in your account.
+- Ensure you have the correct ARN for your permission boundary policy.
+- If you're not using permission boundaries, you can omit this parameter or leave it as an empty string.
+
+By setting this parameter, all IAM roles created by this solution will adhere to the specified permission boundary, enhancing your security posture and compliance with organizational policies.
+
+### Step-by-Step deployment
+
 Deploying the solution is easy, you just need to
 
 MacOS and Linux:

cli.py

@@ -89,9 +89,17 @@ def validate_schedule(answers, current):
 
     raise errors.ValidationError('', reason='Invalid schedule. Use predefined tags or a valid AWS Cron expression.')
 
+def validate_kms_key_arn(answers, current):
+    if not current:
+        return True
+    kms_arn_pattern = r'^arn:aws:kms:[a-z0-9-]+:\d{12}:key/[a-f0-9-]{36}$'
+    if not re.match(kms_arn_pattern, current):
+        raise errors.ValidationError('', reason='Invalid KMS Key ARN format. Please enter a valid ARN.')
+    return True
+
 def edit_sync_settings(sync_settings):
     while True:
-        choices = [f"{s['LocalRepository']['BucketName']} -> {s['RemoteFolders']['Folder']}" for s in sync_settings]
+        choices = [f"{s['LocalRepository']['BucketName']} -> {s['RemoteFolders']['Folder']} (KMS: {s['LocalRepository'].get('KmsKeyArn', 'None')})" for s in sync_settings]
         choices.append("Add new sync setting")
         choices.append("Finish editing")
 
@@ -120,18 +128,21 @@ def edit_sync_settings(sync_settings):
 
 def prompt_sync_setting(existing=None):
     questions = [
-        inquirer.Text('bucket_name', message="Enter BucketName",
+        inquirer.Text('bucket_name', message="Enter the target S3 Bucket name",
                       default=existing['LocalRepository']['BucketName'] if existing else None),
-        inquirer.Text('prefix', message="Enter Prefix",
+        inquirer.Text('prefix', message="Enter the target Prefix",
                       default=existing['LocalRepository']['Prefix'] if existing else None),
+        inquirer.Text('kms_key_arn', message="Enter KMS Key ARN (optional)",
+                      default=existing['LocalRepository'].get('KmsKeyArn', '') if existing else '',
+                      validate=validate_kms_key_arn),
         inquirer.Text('remote_folder', message="Enter Remote Folder (%year%, %month% and %day% tags are supported)",
                       default=existing['RemoteFolders']['Folder'] if existing else None),
         inquirer.Confirm('recursive', message="Is it recursive?",
                          default=existing['RemoteFolders']['Recursive'] if existing else True),
     ]
 
     answers = inquirer.prompt(questions)
-    return {
+    sync_setting = {
         "LocalRepository": {
             "BucketName": safe_bucket_name(answers['bucket_name']),
             "Prefix": safe_prefix(answers['prefix'])
@@ -141,6 +152,11 @@ def prompt_sync_setting(existing=None):
             "Recursive": answers['recursive']
         }
     }
+    
+    if answers['kms_key_arn']:
+        sync_setting['LocalRepository']['KmsKeyArn'] = answers['kms_key_arn']
+    
+    return sync_setting
 
 def fetch_host_key(hostname, port=22):
     try:
@@ -171,12 +187,13 @@ def confirm_config(config):
 
     print_colored("\nSyncSettings:", Fore.CYAN)
     sync_table = PrettyTable()
-    sync_table.field_names = ["Local", "Remote"]
+    sync_table.field_names = ["Local", "Remote", "KMS Key ARN"]
     sync_table.align = "l"
     for setting in config.get('SyncSettings', []):
         local = f"{setting['LocalRepository']['BucketName']}/{setting['LocalRepository']['Prefix']}"
         remote = setting['RemoteFolders']['Folder']
-        sync_table.add_row([local, remote])
+        kms_key_arn = setting['LocalRepository'].get('KmsKeyArn', 'None')
+        sync_table.add_row([local, remote, kms_key_arn])
     print(sync_table)
 
     return inquirer.confirm("Do you want to save this configuration?", default=True)
@@ -241,12 +258,13 @@ def main():
                 print(table)
                 print_colored("\nSyncSettings:", Fore.CYAN)
                 sync_table = PrettyTable()
-                sync_table.field_names = ["Local", "Remote"]
+                sync_table.field_names = ["Local", "Remote", "KMS Key ARN"]
                 sync_table.align = "l"
                 for setting in config.get('SyncSettings', []):
                     local = f"{setting['LocalRepository']['BucketName']}/{setting['LocalRepository']['Prefix']}"
                     remote = setting['RemoteFolders']['Folder']
-                    sync_table.add_row([local, remote])
+                    kms_key_arn = setting['LocalRepository'].get('KmsKeyArn', 'None')
+                    sync_table.add_row([local, remote, kms_key_arn])
                 print(sync_table)
             continue
         elif answers['action'] == 'Delete configuration':
@@ -337,6 +355,7 @@ def main():
                     else:
                         manual_key = inquirer.text(message="Enter the public key manually")
                         if manual_key and manual_key not in public_keys:
+                            public_keys
                             public_keys.append(manual_key)
                         elif manual_key in public_keys:
                             print_colored("This key already exists in the configuration.", Fore.YELLOW)
@@ -375,4 +394,5 @@ def main():
     try:
         main()
     except Exception as e:
-        print_colored(f"An error occurred. Please check the log file for details.", Fore.RED)
+        print_colored(f"An error occurred: {str(e)}", Fore.RED)
+        print_colored("Please check the log file for details.", Fore.RED)

configuration/examples/example-sftp-sync.json

@@ -8,7 +8,8 @@
         {
             "LocalRepository": {
                 "BucketName": "my-local-bucket",
-                "Prefix": "sftp-provider-1/full-sync"
+                "Prefix": "sftp-provider-1/full-sync",
+                "KmsKeyArn": "arn:aws:kms:<region>:<accountID>:key/<keyId>"
             },
             "RemoteFolders": {
                 "Folder": "/home/folder-1",

configuration/solution_parameters/parameters.json

@@ -0,0 +1,7 @@
+{
+    "permission_boundary_policy_arn": "",
+    "powertools_service_name": "transfer-sync-service",
+    "powertools_log_level": "INFO",
+    "boto_version": "1.35.47",
+    "pyawscron_version": "1.0.7"   
+}

requirements-dev.txt

@@ -1,4 +1,4 @@
 pytest==6.2.5
-aws-lambda-powertools~=2.40.1
-boto3~=1.34.134
+aws-lambda-powertools~=3.2.0
+boto3~=1.35.47
 pyawscron==1.0.7

requirements.txt

@@ -1,7 +1,7 @@
-aws-cdk-lib~=2.150.0
+aws-cdk-lib~=2.163.1
 constructs>=10.0.0,<11.0.0
 paramiko~=3.4.0
 inquirer~=3.4.0
 colorama~=0.4.6
 prettytable~=3.11.0
-cdk-monitoring-constructs~=8.1.0
+cdk-monitoring-constructs~=8.3.2

transfer_sync_service/lambda/get_list_status/get_list_status.py

@@ -1,5 +1,4 @@
 import boto3
-import botocore
 import json
 import copy
 from aws_lambda_powertools import Logger