|
| 1 | +# AWS Lambda durable functions to Amazon ECS with Python |
| 2 | + |
| 3 | +This pattern demonstrates how to invoke Amazon ECS tasks from AWS Lambda durable functions using Python. The workflow starts an ECS task, waits for a callback, and resumes based on the task result while maintaining state across the pause/resume cycle. |
| 4 | + |
| 5 | +Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/lambda-ecs-python-sam |
| 6 | + |
| 7 | +Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. |
| 8 | + |
| 9 | +## Requirements |
| 10 | + |
| 11 | +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. |
| 12 | +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured |
| 13 | +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) |
| 14 | +* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed |
| 15 | +* [Docker](https://docs.docker.com/get-docker/) installed (for building Lambda container images) |
| 16 | +* [Python 3.13](https://www.python.org/downloads/) or later |
| 17 | + |
| 18 | +## Deployment Instructions |
| 19 | + |
| 20 | +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: |
| 21 | + ``` |
| 22 | + git clone https://github.com/aws-samples/serverless-patterns |
| 23 | + ``` |
| 24 | +1. Change directory to the pattern directory: |
| 25 | + ``` |
| 26 | + cd lambda-ecs-durable-python-sam |
| 27 | + ``` |
| 28 | +1. From the command line, use AWS SAM to build the application: |
| 29 | + ``` |
| 30 | + sam build |
| 31 | + ``` |
| 32 | +1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yaml file: |
| 33 | + ``` |
| 34 | + sam deploy --guided |
| 35 | + ``` |
| 36 | +1. During the prompts: |
| 37 | + * Enter a stack name |
| 38 | + * Enter the desired AWS Region |
| 39 | + * Enter the VpcCIDR parameter (default: 10.0.0.0/16) |
| 40 | + * Allow SAM CLI to create IAM roles with the required permissions. |
| 41 | + * Create managed ECR repositories for all functions (required for container images) |
| 42 | +
|
| 43 | + Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults. |
| 44 | +
|
| 45 | +1. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing. |
| 46 | +
|
| 47 | +## How it works |
| 48 | +
|
| 49 | +This pattern implements an ECS task orchestration workflow using Lambda durable functions with callback pattern: |
| 50 | +
|
| 51 | +1. **Sync Lambda** starts an ECS task and polls for completion using durable waits (no compute charges during waits) |
| 52 | +2. **Callback Lambda** starts an ECS task, pauses execution using `callback.result()`, and waits for a callback |
| 53 | +3. The ECS task processes work and calls Lambda durable execution callback API when complete |
| 54 | +4. The Lambda function resumes automatically when the callback is invoked and returns the result |
| 55 | +
|
| 56 | +The pattern uses the AWS Durable Execution SDK for Python with the `@durable_execution` decorator to maintain state across the pause/resume cycle. The callback pattern ensures no compute charges while waiting for ECS task completion. |
| 57 | +
|
| 58 | +### Architecture Components |
| 59 | +
|
| 60 | +- **Sync Lambda**: Orchestrates ECS tasks using Lambda durable functions SDK with polling pattern and durable waits |
| 61 | +- **Callback Lambda**: Orchestrates ECS tasks using Lambda durable functions SDK with callback pattern |
| 62 | +- **ECS Tasks**: Process work and send callbacks to Lambda using durable execution callback APIs |
| 63 | +- **VPC and Networking**: Provides network connectivity for ECS tasks to pull Docker images and call AWS APIs |
| 64 | +- **CloudWatch Logs**: Stores execution logs for Lambda functions and ECS tasks |
| 65 | +
|
| 66 | +## Testing |
| 67 | +
|
| 68 | +### Set Environment Variables |
| 69 | +
|
| 70 | +```bash |
| 71 | +export AWS_DEFAULT_REGION=us-east-1 |
| 72 | +export STACK_NAME=<your-stack-name> |
| 73 | +
|
| 74 | +# Get function names from CloudFormation outputs |
| 75 | +export SYNC_FUNCTION=$(aws cloudformation describe-stacks \ |
| 76 | + --stack-name $STACK_NAME \ |
| 77 | + --query 'Stacks[0].Outputs[?OutputKey==`SyncLambdaFunctionArn`].OutputValue' \ |
| 78 | + --output text | awk -F: '{print $NF}') |
| 79 | +
|
| 80 | +export CALLBACK_FUNCTION=$(aws cloudformation describe-stacks \ |
| 81 | + --stack-name $STACK_NAME \ |
| 82 | + --query 'Stacks[0].Outputs[?OutputKey==`CallbackLambdaFunctionArn`].OutputValue' \ |
| 83 | + --output text | awk -F: '{print $NF}') |
| 84 | +``` |
| 85 | + |
| 86 | +### Test Synchronous Pattern |
| 87 | + |
| 88 | +```bash |
| 89 | +# Invoke the sync function (must use qualified ARN with :$LATEST) |
| 90 | +aws lambda invoke \ |
| 91 | + --function-name $SYNC_FUNCTION:\$LATEST \ |
| 92 | + --invocation-type Event \ |
| 93 | + --cli-binary-format raw-in-base64-out \ |
| 94 | + --payload '{"message": "Hello from sync pattern", "processingTime": 10}' \ |
| 95 | + response.json |
| 96 | + |
| 97 | +# Monitor ECS task logs |
| 98 | +aws logs tail /ecs/$STACK_NAME --follow |
| 99 | +``` |
| 100 | + |
| 101 | +A successful sync test shows these log messages: |
| 102 | +- ECS task logs: `[SYNC] Completed: {"status": "success", ...}` |
| 103 | +- Lambda logs: `Durable execution completed` with the ECS task result |
| 104 | + |
| 105 | +### Test Callback Pattern |
| 106 | + |
| 107 | +```bash |
| 108 | +# Invoke the callback function (must use qualified ARN with :$LATEST) |
| 109 | +aws lambda invoke \ |
| 110 | + --function-name $CALLBACK_FUNCTION:\$LATEST \ |
| 111 | + --invocation-type Event \ |
| 112 | + --cli-binary-format raw-in-base64-out \ |
| 113 | + --payload '{"message": "Hello from callback pattern", "processingTime": 30}' \ |
| 114 | + response.json |
| 115 | + |
| 116 | +# Monitor Lambda logs |
| 117 | +aws logs tail /aws/lambda/$CALLBACK_FUNCTION --follow |
| 118 | + |
| 119 | +# Monitor ECS task logs |
| 120 | +aws logs tail /ecs/$STACK_NAME --follow |
| 121 | +``` |
| 122 | + |
| 123 | +A successful callback test shows these log messages: |
| 124 | +- Lambda logs: `Callback created` followed by `Waiting for callback from ECS task` |
| 125 | +- ECS task logs: `[CALLBACK] Success callback sent!` |
| 126 | +- Lambda logs: `Callback received` followed by `Durable execution completed` with the ECS task result |
| 127 | + |
| 128 | +## Cleanup |
| 129 | + |
| 130 | +1. Delete the stack |
| 131 | + ```bash |
| 132 | + sam delete |
| 133 | + ``` |
| 134 | +1. Confirm the stack has been deleted |
| 135 | + ```bash |
| 136 | + aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'$STACK_NAME')].StackStatus" |
| 137 | + ``` |
| 138 | + |
| 139 | +---- |
| 140 | +Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved. |
| 141 | + |
| 142 | +SPDX-License-Identifier: MIT-0 |
0 commit comments