Skip to content

Commit 9ba4b7f

Browse files
julianwoodjulianwood
authored
Merge pull request #2994 from marcojahn/cdk-sfn-dmap-df
added cdk-sfn-dmap-df pattern
2 parents 4ed7683 + 1479b1f commit 9ba4b7f

14 files changed

Lines changed: 1208 additions & 0 deletions

‎cdk-sfn-dmap-df/README.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# AWS Step Functions Distributed Map with AWS Lambda durable functions
2+
3+
This pattern demonstrates how to use AWS Step Functions Distributed Map with an Amazon S3 JSON input to fan out across 50 product catalog items, invoking a Lambda durable function for each item. The key technique is using the AWS Step Functions AWS SDK service integration (`CallAwsService` targeting `lambda:invoke`) instead of the optimized Lambda integration. This is currently necessary because only the raw SDK integration exposes the `DurableExecutionName` parameter, which enables per-item idempotency, derived from each product's `itemId`, showcased in this example.
4+
5+
Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/cdk-sfn-dmap-df](https://serverlessland.com/patterns/cdk-sfn-dmap-df)
6+
7+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
8+
9+
## Requirements
10+
11+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
12+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
13+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
14+
* [Node.js and npm](https://nodejs.org/) installed (Node.js 22+)
15+
* [AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) installed
16+
* CDK bootstrapped in your target account/region
17+
18+
## Deployment Instructions
19+
20+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
21+
22+
```bash
23+
git clone https://github.com/aws-samples/serverless-patterns
24+
```
25+
26+
2. Change directory to the pattern directory:
27+
28+
```bash
29+
cd cdk-sfn-dmap-df
30+
```
31+
32+
3. Install dependencies:
33+
34+
```bash
35+
npm install
36+
```
37+
38+
4. Run tests:
39+
40+
```bash
41+
npm test
42+
```
43+
44+
This runs the CDK stack assertions and the local durable function tests using `LocalDurableTestRunner`. The durable function tests verify the three-operation workflow (validate → wait → update) and price tier assignment without deploying to AWS.
45+
46+
5. Deploy the CDK stack to your default AWS account and region:
47+
48+
```bash
49+
cdk deploy
50+
```
51+
52+
6. Note the outputs from the CDK deployment process. These contain the resource names and ARNs used for testing.
53+
54+
## How it works
55+
56+
![Architecture Diagram](cdk-sfn-dmap-df.png)
57+
58+
Architecture flow:
59+
1. AWS Step Functions Distributed Map reads 50 product items from an Amazon S3 JSON file
60+
2. For each item, the map invokes a Lambda durable function via the AWS SDK service integration (`lambda:invoke`)
61+
3. The `DurableExecutionName` is derived from each item's `itemId` using `States.Format`, providing per-item idempotency
62+
4. Each durable function executes a three-operation workflow:
63+
- **`validate-item`** (step) — Validates required fields, checks price > 0, computes a pricing tier (budget / standard / premium)
64+
- **`rate-limit-delay`** (wait) — Pauses 5 seconds to simulate downstream rate limiting. No compute charges during this wait
65+
- **`update-catalog`** (step) — Writes the enriched catalog entry with processing timestamps and a `completed` status
66+
5. Results are written back to Amazon S3 under the `results/` prefix
67+
68+
### Why the AWS SDK Service Integration?
69+
70+
AWS Step Functions offers two ways to invoke AWS Lambda:
71+
72+
| Integration | ARN Pattern | `DurableExecutionName` Support |
73+
|---|---|---|
74+
| Optimized Lambda | `arn:aws:states:::lambda:invoke` | No |
75+
| AWS SDK | `arn:aws:states:::aws-sdk:lambda:invoke` | Yes |
76+
77+
The optimized integration is simpler but only exposes a subset of the `Lambda.Invoke` API parameters. The AWS SDK integration maps directly to the full `Lambda.Invoke` API, giving access to `DurableExecutionName`. In CDK, this is expressed with `CallAwsService`:
78+
79+
```typescript
80+
new tasks.CallAwsService(this, 'InvokeDurableFunction', {
81+
service: 'lambda',
82+
action: 'invoke',
83+
parameters: {
84+
'FunctionName': `${itemProcessor.functionArn}:$LATEST`,
85+
'InvocationType': 'RequestResponse',
86+
'DurableExecutionName.$': "States.Format('dmap-item-{}', $.itemId)",
87+
'Payload.$': '$',
88+
},
89+
iamResources: [
90+
`${itemProcessor.functionArn}:$LATEST`,
91+
itemProcessor.functionArn,
92+
],
93+
});
94+
```
95+
96+
The `DurableExecutionName` is derived from each item's `itemId` using `States.Format`, so re-running the state machine with the same input file produces the same execution names. This means the durable functions return their previously checkpointed results instead of re-executing, providing end-to-end idempotency from AWS Step Functions through to durable function state.
97+
98+
## Testing
99+
100+
### Start the State Machine
101+
102+
```bash
103+
aws stepfunctions start-execution \
104+
--state-machine-arn <StateMachineArn from stack output> \
105+
--name "catalog-update-$(date +%s)"
106+
```
107+
108+
### Monitor Execution
109+
110+
Monitor the execution in the [AWS Step Functions console](https://console.aws.amazon.com/states). The Distributed Map view shows all 50 child executions and their status. Each child invokes the durable function synchronously with a unique `DurableExecutionName`. Results are written back to the Amazon S3 bucket under the `results/` prefix.
111+
112+
### Verify Deployment
113+
114+
```bash
115+
# Confirm items.json was deployed to S3
116+
aws s3 ls s3://<DataBucketName from stack output>/items.json
117+
118+
# Confirm the state machine exists
119+
aws stepfunctions describe-state-machine \
120+
--state-machine-arn <StateMachineArn from stack output>
121+
```
122+
123+
### Re-running for Idempotency
124+
125+
To demonstrate idempotency, start another execution with the same input:
126+
127+
```bash
128+
aws stepfunctions start-execution \
129+
--state-machine-arn <StateMachineArn from stack output> \
130+
--name "catalog-update-rerun-$(date +%s)"
131+
```
132+
133+
Since the `DurableExecutionName` values are derived from the `itemId` (which does not change between runs), the durable functions detect that executions with those names already completed and return the previously checkpointed results without re-executing any steps.
134+
135+
## Cleanup
136+
137+
1. Delete the stack:
138+
139+
```bash
140+
cdk destroy
141+
```
142+
143+
2. Confirm the deletion when prompted. This removes all resources including the Amazon S3 bucket (configured with `autoDeleteObjects` and `RemovalPolicy.DESTROY`) and Amazon CloudWatch log groups.
144+
145+
---
146+
147+
Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.
148+
149+
SPDX-License-Identifier: MIT-0

‎cdk-sfn-dmap-df/bin/cdk-sfn-dmap-df.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/usr/bin/env node
2+
import * as cdk from 'aws-cdk-lib/core';
3+
import { CdkSfnDmapDfStack } from '../lib/cdk-sfn-dmap-df-stack';
4+
5+
const app = new cdk.App();
6+
const description = "Sample app (uksb-1tthgi812) (tag:cdk-sfn-dmap-df)";
7+
new CdkSfnDmapDfStack(app, 'CdkSfnDmapDfStack', {description
8+
/* If you don't specify 'env', this stack will be environment-agnostic.
9+
* Account/Region-dependent features and context lookups will not work,
10+
* but a single synthesized template can be deployed anywhere. */
11+
12+
/* Uncomment the next line to specialize this stack for the AWS Account
13+
* and Region that are implied by the current CLI configuration. */
14+
// env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
15+
16+
/* Uncomment the next line if you know exactly what Account and Region you
17+
* want to deploy the stack to. */
18+
// env: { account: '123456789012', region: 'us-east-1' },
19+
20+
/* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */
21+
});

‎cdk-sfn-dmap-df/cdk-sfn-dmap-df.json‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
{
2+
"title": "AWS Step Functions Distributed Map with AWS Lambda durable functions",
3+
"description": "Fan out across Amazon S3 items using Distributed Map, invoking a Lambda durable function per item with per-item idempotency.",
4+
"language": "TypeScript",
5+
"level": "300",
6+
"framework": "AWS CDK",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This pattern uses AWS Step Functions Distributed Map to read items from an Amazon S3 JSON file and fan out processing across Lambda durable functions.",
11+
"Each item is processed by a durable function invoked via the AWS SDK service integration (CallAwsService targeting lambda:invoke).",
12+
"The DurableExecutionName parameter is derived from each item's ID, providing per-item idempotency across re-runs.",
13+
"The durable function executes a three-step workflow: validate the item, wait for rate limiting, and update the catalog entry."
14+
]
15+
},
16+
"gitHub": {
17+
"template": {
18+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/cdk-sfn-dmap-df",
19+
"templateURL": "serverless-patterns/cdk-sfn-dmap-df",
20+
"projectFolder": "cdk-sfn-dmap-df",
21+
"templateFile": "lib/cdk-sfn-dmap-df-stack.ts"
22+
}
23+
},
24+
"resources": {
25+
"bullets": [
26+
{
27+
"text": "AWS Step Functions Distributed Map",
28+
"link": "https://docs.aws.amazon.com/step-functions/latest/dg/concepts-asl-use-map-state-distributed.html"
29+
},
30+
{
31+
"text": "Lambda durable functions Documentation",
32+
"link": "https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html"
33+
},
34+
{
35+
"text": "AWS SDK Service Integration for Step Functions",
36+
"link": "https://docs.aws.amazon.com/step-functions/latest/dg/supported-services-awssdk.html"
37+
}
38+
]
39+
},
40+
"deploy": {
41+
"text": [
42+
"Clone the repository: <code>git clone https://github.com/aws-samples/serverless-patterns</code>",
43+
"Change directory: <code>cd cdk-sfn-dmap-df</code>",
44+
"Install dependencies: <code>npm install</code>",
45+
"Run tests: <code>npm test</code>",
46+
"Deploy the CDK stack: <code>cdk deploy</code>"
47+
]
48+
},
49+
"testing": {
50+
"text": [
51+
"Run local tests: <code>npm test</code> — validates CDK stack assertions and runs the durable function locally using LocalDurableTestRunner (no AWS deployment needed).",
52+
"Start the state machine: <code>aws stepfunctions start-execution --state-machine-arn &lt;StateMachineArn from stack output&gt; --name \"catalog-update-$(date +%s)\"</code>",
53+
"Monitor the execution in the AWS Step Functions console to see all 50 child executions.",
54+
"Re-run with the same input to demonstrate idempotency — durable functions return cached results."
55+
]
56+
},
57+
"cleanup": {
58+
"text": [
59+
"Delete the stack: <code>cdk destroy</code>"
60+
]
61+
},
62+
"authors": [
63+
{
64+
"name": "Marco Jahn",
65+
"image": "https://sessionize.com/image/e99b-400o400o2-pqR4BacUSzHrq4fgZ4wwEQ.png",
66+
"bio": "Senior Solutions Architect, Amazon Web Services",
67+
"linkedin": "marcojahn"
68+
}
69+
],
70+
"patternArch": {
71+
"icon1": {
72+
"x": 10,
73+
"y": 50,
74+
"service": "s3",
75+
"label": "Amazon S3"
76+
},
77+
"icon2": {
78+
"x": 30,
79+
"y": 50,
80+
"service": "sfn",
81+
"label": "AWS Step Functions"
82+
},
83+
"icon3": {
84+
"x": 60,
85+
"y": 50,
86+
"service": "lambda",
87+
"label": "AWS Lambda durable function"
88+
},
89+
"icon4": {
90+
"x": 90,
91+
"y": 50,
92+
"service": "s3",
93+
"label": "Amazon S3"
94+
},
95+
"line1": {
96+
"from": "icon1",
97+
"to": "icon2"
98+
},
99+
"line2": {
100+
"from": "icon2",
101+
"to": "icon3"
102+
},
103+
"line3": {
104+
"from": "icon3",
105+
"to": "icon4"
106+
}
107+
}
108+
}

‎cdk-sfn-dmap-df/cdk-sfn-dmap-df.png‎

45.1 KB
Loading

‎cdk-sfn-dmap-df/cdk.json‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/cdk-sfn-dmap-df.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-signer:signingProfileNamePassedToCfn": true,
21+
"@aws-cdk/aws-ecs-patterns:secGroupsDisablesImplicitOpenListener": true,
22+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
23+
"@aws-cdk/core:checkSecretUsage": true,
24+
"@aws-cdk/core:target-partitions": [
25+
"aws",
26+
"aws-cn"
27+
],
28+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
29+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
30+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
31+
"@aws-cdk/aws-iam:minimizePolicies": true,
32+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
33+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
34+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
35+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
36+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
37+
"@aws-cdk/core:enablePartitionLiterals": true,
38+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
39+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
40+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
41+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
42+
"@aws-cdk/aws-route53-patters:useCertificate": true,
43+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
44+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
45+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
46+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
47+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
48+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
49+
"@aws-cdk/aws-redshift:columnId": true,
50+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
51+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
52+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
53+
"@aws-cdk/aws-kms:aliasNameRef": true,
54+
"@aws-cdk/aws-kms:applyImportedAliasPermissionsToPrincipal": true,
55+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
56+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
57+
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
58+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
59+
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
60+
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
61+
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
62+
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
63+
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
64+
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
65+
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
66+
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
67+
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
68+
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
69+
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
70+
"@aws-cdk/aws-eks:useNativeOidcProvider": true,
71+
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
72+
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
73+
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
74+
"@aws-cdk/aws-s3:keepNotificationInImportedBucket": false,
75+
"@aws-cdk/core:explicitStackTags": true,
76+
"@aws-cdk/aws-ecs:reduceEc2FargateCloudWatchPermissions": true,
77+
"@aws-cdk/aws-dynamodb:resourcePolicyPerReplica": true,
78+
"@aws-cdk/aws-ec2:ec2SumTImeoutEnabled": true,
79+
"@aws-cdk/aws-appsync:appSyncGraphQLAPIScopeLambdaPermission": true,
80+
"@aws-cdk/aws-rds:setCorrectValueForDatabaseInstanceReadReplicaInstanceResourceId": true,
81+
"@aws-cdk/core:cfnIncludeRejectComplexResourceUpdateCreatePolicyIntrinsics": true,
82+
"@aws-cdk/aws-lambda-nodejs:sdkV3ExcludeSmithyPackages": true,
83+
"@aws-cdk/aws-stepfunctions-tasks:fixRunEcsTaskPolicy": true,
84+
"@aws-cdk/aws-ec2:bastionHostUseAmazonLinux2023ByDefault": true,
85+
"@aws-cdk/aws-route53-targets:userPoolDomainNameMethodWithoutCustomResource": true,
86+
"@aws-cdk/aws-elasticloadbalancingV2:albDualstackWithoutPublicIpv4SecurityGroupRulesDefault": true,
87+
"@aws-cdk/aws-iam:oidcRejectUnauthorizedConnections": true,
88+
"@aws-cdk/core:enableAdditionalMetadataCollection": true,
89+
"@aws-cdk/aws-lambda:createNewPoliciesWithAddToRolePolicy": false,
90+
"@aws-cdk/aws-s3:setUniqueReplicationRoleName": true,
91+
"@aws-cdk/aws-events:requireEventBusPolicySid": true,
92+
"@aws-cdk/core:aspectPrioritiesMutating": true,
93+
"@aws-cdk/aws-dynamodb:retainTableReplica": true,
94+
"@aws-cdk/aws-stepfunctions:useDistributedMapResultWriterV2": true,
95+
"@aws-cdk/s3-notifications:addS3TrustKeyPolicyForSnsSubscriptions": true,
96+
"@aws-cdk/aws-ec2:requirePrivateSubnetsForEgressOnlyInternetGateway": true,
97+
"@aws-cdk/aws-s3:publicAccessBlockedByDefault": true,
98+
"@aws-cdk/aws-lambda:useCdkManagedLogGroup": true,
99+
"@aws-cdk/aws-elasticloadbalancingv2:networkLoadBalancerWithSecurityGroupByDefault": true,
100+
"@aws-cdk/aws-ecs-patterns:uniqueTargetGroupId": true,
101+
"@aws-cdk/aws-route53-patterns:useDistribution": true
102+
}
103+
}

0 commit comments

Comments
 (0)