diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index f0a2a84..1d510d4 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -57,7 +57,7 @@ jobs: - name: Integration Tests run: | - npm test + make test - name: Show Logs if: always() diff --git a/Makefile b/Makefile index 64e9801..022b19e 100644 --- a/Makefile +++ b/Makefile @@ -3,12 +3,12 @@ export AWS_SECRET_ACCESS_KEY ?= test export AWS_DEFAULT_REGION=us-east-1 SHELL := /bin/bash -## Show this help -usage: - @fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//' +usage: ## Show this help in table format + @echo "| Target | Description |" + @echo "|------------------------|-------------------------------------------------------------------|" + @fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/:.*##\s*/##/g' | awk -F'##' '{ printf "| %-22s | %-65s |\n", $$1, $$2 }' -## Check if all required prerequisites are installed -check: +check: ## Check if all required prerequisites are installed @command -v docker > /dev/null 2>&1 || { echo "Docker is not installed. Please install Docker and try again."; exit 1; } @command -v node > /dev/null 2>&1 || { echo "Node.js is not installed. Please install Node.js and try again."; exit 1; } @command -v aws > /dev/null 2>&1 || { echo "AWS CLI is not installed. Please install AWS CLI and try again."; exit 1; } @@ -19,8 +19,7 @@ check: @command -v awslocal > /dev/null 2>&1 || { echo "awslocal is not installed. Please install awslocal and try again."; exit 1; } @echo "All required prerequisites are available." -## Install dependencies -install: +install: ## Install all required dependencies @if [ ! -d "node_modules" ]; then \ echo "node_modules not found. Running npm install..."; \ npm install; \ @@ -31,39 +30,51 @@ install: fi @echo "All required dependencies are available." -## Deploy the infrastructure -deploy: +deploy: ## Deploy the CDK stack @echo "Bootstrapping CDK..." cdklocal bootstrap @echo "Deploying CDK..." cdklocal deploy --require-approval never @echo "CDK deployed successfully." -## Run the tests -test: +test: ## Run the tests @echo "Running tests..." npm test @echo "Tests completed successfully." -## Start LocalStack in detached mode -start: +run: ## Execute a SQL query through the Lambda function + @echo "Executing SQL query through Lambda function..." + @aws_version=$$(aws --version | grep -o "aws-cli/[0-9]*" | cut -d'/' -f2); \ + if [ "$$aws_version" = "2" ]; then \ + echo "Using AWS CLI version 2 command format..."; \ + awslocal lambda invoke \ + --cli-binary-format raw-in-base64-out \ + --function-name my-lambda-rds-query-helper \ + --payload '{"sqlQuery": "select Author from books", "secretName":"/rdsinitexample/rds/creds/mysql-01"}' output; \ + else \ + echo "Using AWS CLI version 1 command format..."; \ + awslocal lambda invoke \ + --function-name my-lambda-rds-query-helper \ + --payload '{"sqlQuery": "select Author from books", "secretName":"/rdsinitexample/rds/creds/mysql-01"}' output; \ + fi + @echo "Query execution completed. Results:" + @cat output + +start: ## Start LocalStack in detached mode @echo "Starting LocalStack..." @LOCALSTACK_AUTH_TOKEN=$(LOCALSTACK_AUTH_TOKEN) localstack start -d @echo "LocalStack started successfully." -## Stop the Running LocalStack container -stop: +stop: ## Stop LocalStack @echo "Stopping LocalStack..." @localstack stop @echo "LocalStack stopped successfully." -## Make sure the LocalStack container is up -ready: +ready: ## Wait for LocalStack to be ready @echo Waiting on the LocalStack container... @localstack wait -t 30 && echo LocalStack is ready to use! || (echo Gave up waiting on LocalStack, exiting. && exit 1) -## Save the logs in a separate file -logs: +logs: ## Get LocalStack logs @localstack logs > logs.txt .PHONY: usage check start ready install deploy test logs stop diff --git a/README.md b/README.md index 7e3be33..15778c9 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,20 @@ # Amazon RDS initialization using CDK -| Key | Value | -| ------------ | ------------------------------------------------------------------------------------- | -| Environment | | -| Services | RDS, Lambda, SecretsManager, ECR | -| Integrations | AWS CDK, AWS SDK for JavaScript | -| Categories | Databases | -| Level | Intermediate | -| GitHub | [Repository link](https://github.com/localstack/amazon-rds-init-cdk) | +| Key | Value | +| ------------ | -------------------------------------------------------------------- | +| Environment | LocalStack, AWS | +| Services | RDS, Lambda, SecretsManager, ECR | +| Integrations | AWS CDK, AWS SDK for JavaScript | +| Categories | Databases | +| Level | Intermediate | +| Use Case | Cloud Pods, Pre-Seeding Databases | +| GitHub | [Repository link](https://github.com/localstack/amazon-rds-init-cdk) | ## Introduction -The Amazon RDS initialization using CDK sample application demonstrates how LocalStack supports RDS instances initialization using CDK and CloudFormation Custom Resources. The sample application implements a Node.js Lambda function for compute layer, which is able to run custom SQL scripts. It also executes custom commands supported by the [Node.js client for MySQL2](https://www.npmjs.com/package/mysql2). To test this application sample, we will demonstrate how you use LocalStack to deploy the infrastructure on your developer machine and your CI environment and use a Lambda function to run queries against the RDS database after successful deployment. +The Amazon RDS initialization using CDK sample application demonstrates how LocalStack supports RDS instances initialization using CDK and CloudFormation Custom Resources. The sample application implements a Node.js Lambda function for compute layer, which is able to run custom SQL scripts. It also executes custom commands supported by the [Node.js client for MySQL2](https://www.npmjs.com/package/mysql2). To test this application sample, we will demonstrate how you use LocalStack to deploy the infrastructure on your developer machine and your CI environment and use a Lambda function to run queries against the RDS database after successful deployment. We will also show how to pre-seed the RDS database with dummy data and use Cloud Pods to load resources instantly without full redeploys, reducing test execution time. -## Architecture Diagram +## Architecture The following diagram shows the architecture that this sample application builds and deploys: @@ -21,104 +22,103 @@ The following diagram shows the architecture that this sample application builds * [RDS](https://docs.localstack.cloud/user-guide/aws/rds/) as the central part of the sample application which is initialized and pre-filled with data. * [Lambda](https://docs.localstack.cloud/user-guide/aws/lambda/) to initialize the database, and query data -* [SecretsManager](https://docs.localstack.cloud/references/coverage/coverage_secretsmanager/) to store the credentials and configuration of the RDS database. +* [Secrets Manager](https://docs.localstack.cloud/user-guide/aws/secretsmanager/) to store the credentials and configuration of the RDS database. ## Prerequisites -- LocalStack Pro with the [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli). +- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli) with a [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/). - [AWS CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/) with the [`awslocal` wrapper](https://docs.localstack.cloud/user-guide/integrations/aws-cli/#localstack-aws-cli-awslocal). - [CDK](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/) with the [`cdklocal`](https://www.npmjs.com/package/aws-cdk-local) wrapper. - [Node.js](https://nodejs.org/en/download/) +- [`make`](https://www.gnu.org/software/make/) (**optional**, but recommended for running the sample application) -## Instructions +## Installation -You can build and deploy the sample application on LocalStack by running our `Makefile` commands: -`build`, `bootstrap`, and `deploy`. Alternatively, here are instructions to deploy it manually step-by-step. +To run the sample application, you need to install the required dependencies. -### Run LocalStack - -Start LocalStack Pro with the `LOCALSTACK_AUTH_TOKEN` pre-configured: +First, clone the repository: ```shell -export LOCALSTACK_AUTH_TOKEN= -localstack start +git clone https://github.com/localstack-samples/sample-cdk-rds.git ``` -The sample application uses RDS with a MySQL Engine. Currently, by default LocalStack will use a MariaDB engine instead (check details in our [RDS documentation](https://docs.localstack.cloud/user-guide/aws/rds/#mysql-engine)). You can enable the use of real MySQL engine, which will start a MySQL instance in a separate docker container, by setting the env `RDS_MYSQL_DOCKER=1`. Run the following command to start LocalStack with MySQL engine: +Then, navigate to the project directory: ```shell -export LOCALSTACK_AUTH_TOKEN= -RDS_MYSQL_DOCKER=1 localstack start +cd sample-cdk-rds ``` - -### Installation - -Install the project dependencies by running the following command: +Next, install the project dependencies by running the following command: ```shell -npm install +make install ``` -Additionally, there is a Node.js Lambda located in `demos/rds-query-fn-code` that also requires external dependencies. Install these dependencies by navigating to the `demos/rds-query-fn-code` directory and running: +## Deployment + +Start LocalStack with the `LOCALSTACK_AUTH_TOKEN` pre-configured: ```shell -npm install && cd demos/rds-query-fn-code && npm install && cd ../../ +localstack auth set-token +localstack start ``` -## Deploying the application +> By default, LocalStack uses the MariaDB engine (see [RDS documentation](https://docs.localstack.cloud/user-guide/aws/rds/#mysql-engine)). To use the real MySQL engine in a separate Docker container, set the environment variable `RDS_MYSQL_DOCKER=1`. -Once the dependencies are installed, run the following commands in the project's root directory: +To deploy the sample application, run the following command: ```shell -cdklocal bootstrap -cdklocal deploy --require-approval never +make deploy ``` -This command deploys the application to LocalStack without requiring manual approval for each deployment step. Wait for the deployment to complete. Once finished, you will see some `Outputs` similar to the following: +The output will be similar to the following: ```shell -RdsInitExample.RdsInitFnResponse = {"status":"OK","results":[{"fieldCount":0,"affectedRows":0,"insertId":0,"info":"","serverStatus":10,"warningStatus":0},{"fieldCount":0,"affectedRows":0,"insertId":0,"info":"","serverStatus":10,"warningStatus":0},{"fieldCount":0,"affectedRows":0,"insertId":0,"info":"","serverStatus":10,"warningStatus":0},{"fieldCount":0,"affectedRows":0,"insertId":0,"info":"","serverStatus":10,"warningStatus":0},{"fieldCount":0,"affectedRows":1,"insertId":1,"info":"","serverStatus":10,"warningStatus":0},{"fieldCount":0,"affectedRows":1,"insertId":2,"info":"","serverStatus":10,"warningStatus":0},{"fieldCount":0,"affectedRows":1,"insertId":3,"info":"","serverStatus":10,"warningStatus":0},{"fieldCount":0,"affectedRows":1,"insertId":1,"info":"","serverStatus":10,"warningStatus":0},{"fieldCount":0,"affectedRows":1,"insertId":2,"info":"","serverStatus":2,"warningStatus":0}]} +Outputs: +RdsInitExample.RdsInitFnResponse = {"status":"OK","results":[/*...SQL operations...*/]} RdsInitExample.functionName = my-lambda-rds-query-helper RdsInitExample.secretName = /rdsinitexample/rds/creds/mysql-01 -``` +Stack ARN: +arn:aws:cloudformation:us-east-1:000000000000:stack/RdsInitExample/3f53b7bd -* `RdsInitExample.RdsInitFnResponse` shows the execution result of the SQL script (`demos/rds-init-fn-code/script.sql`). -* `RdsInitExample.functionName` is the name of the function that can be used to run test queries against RDS. -* `RdsInitExample.secretName` is the name of the secret that contains information about the database. This name is required as input for the Lambda to run queries. +✨ Total time: 80.21s -## Test the Sample +CDK deployed successfully. +``` -The sample application initializes the database with tables and dummy data. Additionally, we have included a Lambda function called `my-lambda-rds-query-helper`, which allows you to run queries against the RDS database. +- `RdsInitExample.RdsInitFnResponse` shows the result of running the SQL script at `demos/rds-init-fn-code/script.sql`. +- `RdsInitExample.functionName` is the function name used to run test queries on RDS. +- `RdsInitExample.secretName` is the secret name with database info, required by the Lambda to run queries. -The Lambda function expects two parameters: `sqlQuery` for the query itself, and `secretName` for the secret containing the database connection details. +## Testing -For example, you query the authors of books like this for AWS CLI v1: +The sample application sets up the database by creating tables and inserting dummy data. It includes a Lambda function named `my-lambda-rds-query-helper`, which is used to run SQL queries against the RDS database. -```shell -awslocal lambda invoke --function-name my-lambda-rds-query-helper --payload '{"sqlQuery": "select Author from books", "secretName":"/rdsinitexample/rds/creds/mysql-01"}' output -``` +This function requires two parameters: -If you are using AWS CLI v2, please use the following: +- `sqlQuery`, which specifies the SQL command to execute +- `secretName`, which provides the name of the secret containing the database connection details. + +To run a query using AWS CLI, you can invoke the Lambda function with the following command: ```shell -awslocal lambda invoke --cli-binary-format raw-in-base64-out --function-name my-lambda-rds-query-helper --payload '{"sqlQuery": "select Author from books", "secretName":"/rdsinitexample/rds/creds/mysql-01"}' output +make run ``` -This command invokes the `my-lambda-rds-query-helper` function with the specified query and secret. - -To view the result, use the following command: +The output will contain the execution status and results, for example: ```shell -cat output -```` +{"status":"SUCCESS","results":[{"Author":"Jane Doe"},{"Author":"Jane Doe"},{"Author":"LocalStack"}]} +``` -The result will be displayed in the output, similar to the following: +You can run full end-to-end integration tests using the following command: ```shell -{"status":"SUCCESS","results":[{"Author":"Jane Doe"},{"Author":"Jane Doe"},{"Author":"LocalStack"}]} +make test ``` -### Use Cases +## Use Cases + +### Pre-seeding Databases In the sample, we set up the database by creating tables and adding dummy data using the `demos/rds-init-fn-code/scripts.sql` script. @@ -132,72 +132,35 @@ Improving segregation of duties and least privilege by providing a flexible hook - Initializing database tables (see note below). - Seeding database tables with initial datasets (see note below). -> **NOTE**: Please note that application-specific initialization logic, such as defining the structure of database tables and seeding them with initial data, is typically managed on the application side. It is generally recommended to keep infrastructure initialization/management separate from application-specific initialization. - -## Technical Implementation Details - -In order to achieve custom logic execution during the deployment flow of a CDK stack, we make use of CloudFormation Custom Resources. In the context of CDK, we use the `AwsCustomResource` construct to invoke a deployed Lambda containing the RDS initialization logic (execute SQL scripts). - -> Optionally you can read more about making custom AWS API calls using the [AwsCustomResource construct](https://docs.aws.amazon.com/cdk/api/v1/docs/custom-resources-readme.html#custom-resources-for-aws-apis). +> [!NOTE] +> Application-specific initialization logic, such as defining the structure of database tables and seeding them with initial data, is typically managed on the application side. It is generally recommended to keep infrastructure initialization/management separate from application-specific initialization. -### Client implementation based on Node.js +### Cloud Pods -To execute SQL scripts on the provisioned Amazon RDS instance we make use of the `mysql` NPM module, it allow us to easily execute custom SQL scripts or any other support `client` **->** `server` command: +Cloud Pods are persistent state snapshots of your LocalStack instance that can be stored, versioned, shared, and restored. They enable reproducible environments for development and testing. -```js -const mysql = require('mysql2') -const connection = mysql.createConnection({ - host, - user, - password, - multipleStatements: true -}) -connection.connect() +In this sample, Cloud Pods are utilized to streamline the CI workflow by pre-seeding the LocalStack environment with the necessary infrastructure and data. This approach eliminates the need to redeploy and reinitialize resources from scratch in every CI job, thereby reducing setup time and ensuring consistency across test runs. -connection.query("SELECT 'Hello World!';", (err, res) => { - // ... -}) -``` -> Full Node.js implementation example for MySQL is available at `./demos/rds-init-fn-code/index.js` - -### Docker container images for Lambda functions +The `.github/workflows/cloud-pods.yml` file demonstrates this workflow: -To avoid unnecessary overhead dealing with software dependencies, we promote the usage of Docker container images to package the RDS initialization Lambda function code. +- The `create-cloud-pod` job deploys the infrastructure using CDK and saves the LocalStack state as a Cloud Pod. +- The `test-cloud-pod` job loads the saved Cloud Pod and runs integration tests against the pre-seeded environment. -Docker container images are automatically managed by CDK and there is no need to interact with ECR repositories, simply use: +This setup ensures that your CI pipeline can quickly spin up a fully-initialized environment, leading to faster and more reliable test executions. -```js -const fnCode = DockerImageCode.fromImageAsset(`${__dirname}/your-fn-code-directory`, {}) -``` -> You can see a Lambda function code example inside the `./demos/rds-init-fn-code` directory. - -### Lambda function for querying RDS - -An additional Lambda function to show case querying of RDS database, uses code from assets. -This requires to run `npm install` in the directory first, so that the node-modules are available. - -```js -// create a new Lambda to run queries against the database for testing purpose after init -const lambdaQuery = new lambda.Function(this, 'MyLambdaRDSQueryHelper', { - code: new lambda.AssetCode(`${__dirname}/rds-query-fn-code`), - handler: 'index.handler', - runtime: lambda.Runtime.NODEJS_16_X, - memorySize: 1024, - timeout: cdk.Duration.seconds(300), - functionName: "my-lambda-rds-query-helper" -}) -``` +## Summary -### The CdkResourceInitializer construct +This sample helps you locally provision, initialize, and test an Amazon RDS database using AWS CDK and LocalStack. This showcases the following patterns: -The `CDKResourceInitializer` CDK construct generalizes the proposed solution, it encapsulates the integration requirements behind `CloudFormation Custom Resources` and `CDK`, to support the execution of AWS Lambda functions with custom initialization logic. -The implementation can be found in `lib/resource-initializer.ts`. +- Defining and deploying RDS, Lambda, and Secrets Manager resources using AWS CDK. +- Executing SQL scripts during deployment via a Lambda function to set up the database schema and seed data. +- Leveraging LocalStack to emulate AWS services locally, enabling offline development and testing. +- Utilizing Cloud Pods to save and load LocalStack states, facilitating faster and more consistent CI test runs. +- Ensuring consistent & reproducible development and testing environments by restoring from Cloud Pods. ## Learn More -More details about the original sample can be found in the AWS blog post — [Use AWS CDK to initialize Amazon RDS instances](https://aws.amazon.com/blogs/infrastructure-and-automation/use-aws-cdk-to-initialize-amazon-rds-instances/). - -## Contributing - -We appreciate your interest in contributing to our project and are always looking for new ways to improve the developer experience. We welcome feedback, bug reports, and even feature ideas from the community. -Please refer to the [contributing file](CONTRIBUTING.md) for more details on how to get started. +- Details about the original sample are available in the AWS blog post: [Use AWS CDK to initialize Amazon RDS instances](https://aws.amazon.com/blogs/infrastructure-and-automation/use-aws-cdk-to-initialize-amazon-rds-instances/). +- Additional technical implementation details are provided in the [original sample’s documentation](https://github.com/aws-samples/amazon-rds-init-cdk?tab=readme-ov-file#technical-implementation). +- For more information about Cloud Pods, visit the [LocalStack documentation](https://docs.localstack.cloud/user-guide/state-management/cloud-pods/). +- For more information about deploying CDK applications on LocalStack, visit the [LocalStack documentation](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/). diff --git a/solution-overview.png b/solution-overview.png deleted file mode 100644 index a04e5ad..0000000 Binary files a/solution-overview.png and /dev/null differ