Skip to content

Commit 69fee9b

Browse files
HarshCaspercloutierMat
HarshCasper
and
cloutierMat
authored
setup the sample application (#1)
* migrate the sample * update the .gitignore * format the code * setup the README * add a simple CI workflow --------- Co-authored-by: Author: Mathieu Cloutier <79954947+cloutierMat@users.noreply.github.com>
1 parent 9ff8428 commit 69fee9b

12 files changed

Lines changed: 1193 additions & 162 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
name: Deploy on LocalStack
2+
3+
on:
4+
push:
5+
paths-ignore:
6+
- 'README.md'
7+
branches:
8+
- main
9+
pull_request:
10+
branches:
11+
- main
12+
schedule:
13+
# “At 00:00 on Sunday.”
14+
- cron: "0 0 * * 0"
15+
workflow_dispatch:
16+
17+
jobs:
18+
cdk:
19+
name: Setup infrastructure using CDK
20+
runs-on: ubuntu-latest
21+
steps:
22+
- name: Checkout
23+
uses: actions/checkout@v3
24+
25+
- name: Setup Node.js
26+
uses: actions/setup-node@v3
27+
with:
28+
node-version: 20
29+
30+
- name: Install Python
31+
uses: actions/setup-python@v4
32+
with:
33+
python-version: '3.10'
34+
35+
- name: Install CDK
36+
run: |
37+
npm install -g aws-cdk-local aws-cdk
38+
cdklocal --version
39+
40+
- name: Install dependencies
41+
run: |
42+
make install
43+
44+
- name: Start LocalStack
45+
env:
46+
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
47+
run: |
48+
export LOCALSTACK_AUTH_TOKEN=$LOCALSTACK_AUTH_TOKEN
49+
make start
50+
sleep 30
51+
52+
- name: Deploy the infrastructure
53+
run: |
54+
make deploy
55+
56+
- name: Run tests
57+
run: |
58+
make run

‎.gitignore‎

Lines changed: 2 additions & 160 deletions
Original file line numberDiff line numberDiff line change
@@ -1,160 +1,2 @@
1-
# Byte-compiled / optimized / DLL files
2-
__pycache__/
3-
*.py[cod]
4-
*$py.class
5-
6-
# C extensions
7-
*.so
8-
9-
# Distribution / packaging
10-
.Python
11-
build/
12-
develop-eggs/
13-
dist/
14-
downloads/
15-
eggs/
16-
.eggs/
17-
lib/
18-
lib64/
19-
parts/
20-
sdist/
21-
var/
22-
wheels/
23-
share/python-wheels/
24-
*.egg-info/
25-
.installed.cfg
26-
*.egg
27-
MANIFEST
28-
29-
# PyInstaller
30-
# Usually these files are written by a python script from a template
31-
# before PyInstaller builds the exe, so as to inject date/other infos into it.
32-
*.manifest
33-
*.spec
34-
35-
# Installer logs
36-
pip-log.txt
37-
pip-delete-this-directory.txt
38-
39-
# Unit test / coverage reports
40-
htmlcov/
41-
.tox/
42-
.nox/
43-
.coverage
44-
.coverage.*
45-
.cache
46-
nosetests.xml
47-
coverage.xml
48-
*.cover
49-
*.py,cover
50-
.hypothesis/
51-
.pytest_cache/
52-
cover/
53-
54-
# Translations
55-
*.mo
56-
*.pot
57-
58-
# Django stuff:
59-
*.log
60-
local_settings.py
61-
db.sqlite3
62-
db.sqlite3-journal
63-
64-
# Flask stuff:
65-
instance/
66-
.webassets-cache
67-
68-
# Scrapy stuff:
69-
.scrapy
70-
71-
# Sphinx documentation
72-
docs/_build/
73-
74-
# PyBuilder
75-
.pybuilder/
76-
target/
77-
78-
# Jupyter Notebook
79-
.ipynb_checkpoints
80-
81-
# IPython
82-
profile_default/
83-
ipython_config.py
84-
85-
# pyenv
86-
# For a library or package, you might want to ignore these files since the code is
87-
# intended to run in multiple environments; otherwise, check them in:
88-
# .python-version
89-
90-
# pipenv
91-
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
92-
# However, in case of collaboration, if having platform-specific dependencies or dependencies
93-
# having no cross-platform support, pipenv may install dependencies that don't work, or not
94-
# install all needed dependencies.
95-
#Pipfile.lock
96-
97-
# poetry
98-
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
99-
# This is especially recommended for binary packages to ensure reproducibility, and is more
100-
# commonly ignored for libraries.
101-
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
102-
#poetry.lock
103-
104-
# pdm
105-
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
106-
#pdm.lock
107-
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
108-
# in version control.
109-
# https://pdm.fming.dev/#use-with-ide
110-
.pdm.toml
111-
112-
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
113-
__pypackages__/
114-
115-
# Celery stuff
116-
celerybeat-schedule
117-
celerybeat.pid
118-
119-
# SageMath parsed files
120-
*.sage.py
121-
122-
# Environments
123-
.env
124-
.venv
125-
env/
126-
venv/
127-
ENV/
128-
env.bak/
129-
venv.bak/
130-
131-
# Spyder project settings
132-
.spyderproject
133-
.spyproject
134-
135-
# Rope project settings
136-
.ropeproject
137-
138-
# mkdocs documentation
139-
/site
140-
141-
# mypy
142-
.mypy_cache/
143-
.dmypy.json
144-
dmypy.json
145-
146-
# Pyre type checker
147-
.pyre/
148-
149-
# pytype static type analyzer
150-
.pytype/
151-
152-
# Cython debug symbols
153-
cython_debug/
154-
155-
# PyCharm
156-
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
157-
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
158-
# and can be added to the global gitignore or merged into this file. For a more nuclear
159-
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
160-
#.idea/
1+
.venv/
2+
volume/

‎Makefile‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
VENV_BIN ?= python3 -m venv
2+
VENV_DIR ?= .venv
3+
PIP_CMD ?= pip3
4+
5+
USERNAME ?= admin
6+
DB_NAME ?= dms_sample
7+
USERPWD ?= 1Wp2Aide=z=,eLX3RrD4gJ4o54puex
8+
STACK_NAME ?= DMsSampleSetupStack
9+
DB_ENDPOINT ?= mariadb_server
10+
DB_PORT ?= 3306
11+
ENDPOINT_URL = http://localhost.localstack.cloud:4566
12+
export AWS_ACCESS_KEY_ID ?= test
13+
export AWS_SECRET_ACCESS_KEY ?= test
14+
export AWS_DEFAULT_REGION ?= us-east-1
15+
16+
VENV_RUN = . $(VENV_ACTIVATE)
17+
18+
CLOUD_ENV = USERNAME=$(USERNAME) DB_NAME=$(DB_NAME) USERPWD=$(USERPWD) STACK_NAME=$(STACK_NAME)
19+
LOCAL_ENV = USERNAME=$(USERNAME) DB_NAME=$(DB_NAME) USERPWD=$(USERPWD) STACK_NAME=$(STACK_NAME) DB_ENDPOINT=$(DB_ENDPOINT) DB_PORT=$(DB_PORT) ENDPOINT_URL=$(ENDPOINT_URL)
20+
21+
ifeq ($(OS), Windows_NT)
22+
VENV_ACTIVATE = $(VENV_DIR)/Scripts/activate
23+
else
24+
VENV_ACTIVATE = $(VENV_DIR)/bin/activate
25+
endif
26+
27+
usage: ## Show this help
28+
@grep -Fh "##" $(MAKEFILE_LIST) | grep -Fv fgrep | sed -e 's/:.*##\s*/##/g' | awk -F'##' '{ printf "%-25s %s\n", $$1, $$2 }'
29+
30+
$(VENV_ACTIVATE):
31+
test -d $(VENV_DIR) || $(VENV_BIN) $(VENV_DIR)
32+
$(VENV_RUN); touch $(VENV_ACTIVATE)
33+
34+
venv: $(VENV_ACTIVATE) ## Create a new (empty) virtual environment
35+
36+
start:
37+
$(LOCAL_ENV) docker compose up --build --detach --wait
38+
39+
install: venv
40+
$(VENV_RUN); $(PIP_CMD) install -r requirements.txt
41+
42+
deploy:
43+
$(VENV_RUN); $(LOCAL_ENV) cdklocal bootstrap --output ./cdk.local.out
44+
$(VENV_RUN); $(LOCAL_ENV) cdklocal deploy --require-approval never --output ./cdk.local.out
45+
46+
deploy-aws:
47+
$(VENV_RUN); $(CLOUD_ENV) cdk bootstrap
48+
$(VENV_RUN); $(CLOUD_ENV) cdk deploy --require-approval never
49+
50+
destroy:
51+
docker-compose down
52+
53+
destroy-aws: venv
54+
$(VENV_RUN); $(CLOUD_ENV) cdk destroy --require-approval never
55+
56+
run:
57+
$(VENV_RUN); $(LOCAL_ENV) python run.py
58+
59+
run-aws:
60+
$(VENV_RUN); $(CLOUD_ENV) python run.py

‎README.md‎

Lines changed: 108 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,108 @@
1-
# sample-dms-kinesis-rds-mariadb
2-
Sample Application showcasing how to use DMS to create CDC and full load tasks using the CDK in Python.
1+
# Sample Application showcasing how to use DMS to create CDC
2+
3+
## Introduction
4+
5+
This scenario demonstrates how to use Database Migration Service (DMS) to create change data capture (CDC) and full load tasks using the Cloud Development Kit in Python. It is a self-contained setup that will create a VPC to host 2 databases, a Kinesis stream, and 4 replication tasks.
6+
7+
## Pre-requisites
8+
9+
- [LocalStack Auth Token](https://docs.localstack.cloud/getting-started/auth-token/)
10+
- [Python 3.10](https://www.python.org/downloads/) & `pip`
11+
- [Docker Compose](https://docs.docker.com/compose/install/)
12+
- [CDK](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/) with the [`cdklocal`](https://github.com/localstack/aws-cdk-local) wrapper.
13+
14+
15+
Start LocalStack Pro with the `LOCALSTACK_AUTH_TOKEN` pre-configured:
16+
17+
```bash
18+
export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
19+
docker-compose up
20+
```
21+
22+
The Docker Compose file will start LocalStack Pro container and a MariaDB container. The MariaDB container will be used to showcase how to reach a database external to LocalStack.
23+
24+
## Instructions
25+
26+
### Install the dependencies
27+
28+
Install all the dependencies by running the following command:
29+
30+
```bash
31+
make install
32+
```
33+
34+
### Creating the infrastructure
35+
36+
To deploy the infrastructure, you can run the following command:
37+
38+
```bash
39+
make deploy
40+
```
41+
42+
After successful deployment, you will see the following output:
43+
44+
```bash
45+
Outputs:
46+
DMsSampleSetupStack.cdcTask1 = arn:aws:dms:us-east-1:000000000000:task:A001NYMR4Z0NK45ZBJT6954RNMGEKL2PQ9XQYR4
47+
DMsSampleSetupStack.cdcTask2 = arn:aws:dms:us-east-1:000000000000:task:GO5RC4J6CKZWSJKF4CGB6ZV3ZEMGI38DFPJF2ZU
48+
DMsSampleSetupStack.cdcTaskSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-rdsinstanceSecret07FEB42-907ed0cf-RSPkZq
49+
DMsSampleSetupStack.fullTask1 = arn:aws:dms:us-east-1:000000000000:task:BCZLANJP9WFXKNTYBEWTAQ1YHIVJ5C2ZUIHDPB2
50+
DMsSampleSetupStack.fullTask2 = arn:aws:dms:us-east-1:000000000000:task:ZO7WPZTTAKOA1CONK2Y3Y0H6FXLAFWUYX1OPGPM
51+
DMsSampleSetupStack.fullTaskSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-mariadbaccesssecret40AD7-611fcbcd-IKWDDh
52+
DMsSampleSetupStack.kinesisStream = arn:aws:kinesis:us-east-1:000000000000:stream/DMsSampleSetupStack-TargetStream3B4B2880-02dd0371
53+
Stack ARN:
54+
arn:aws:cloudformation:us-east-1:000000000000:stack/DMsSampleSetupStack/b8298866
55+
56+
✨ Total time: 49.33s
57+
```
58+
59+
### Running the tasks
60+
61+
You can run the tasks by executing the following command:
62+
63+
```bash
64+
make run
65+
```
66+
67+
## Developer Notes
68+
69+
Four tasks are deployed with the stack, split into two parts.
70+
71+
First, a full load replication task runs against the external DB:
72+
73+
- Creates three tables: `authors`, `accounts`, `novels`
74+
- Makes four inserts
75+
- Starts full load task 1 targeting tables starting with 'a' (`a%` table mapping)
76+
- Captures and logs six Kinesis events: 2 drop tables, 2 create tables, 2 inserts
77+
- Starts full load task 2 targeting the `novels` table (`novels` table mapping)
78+
- Captures and logs four Kinesis events: 1 drop table, 1 create table, 2 inserts
79+
- Logs `table_statistics` for both tasks
80+
81+
Next, a CDC replication task runs against the RDS database:
82+
83+
- Creates three tables: `authors`, `accounts`, `novels`
84+
- Starts CDC task 1 targeting tables starting with 'a' (`a%` table mapping)
85+
- Starts CDC task 2 targeting the `novels` table (`novels` table mapping)
86+
- Captures and logs five Kinesis events: 2 for `awsdms_apply_exceptions` table, 3 for our tables
87+
- Makes four inserts
88+
- Captures and logs four Kinesis events: 2 for tables in task 1, 2 for table in task 2
89+
- Makes three table alterations, one per table
90+
- Captures and logs three Kinesis events
91+
- Logs `table_statistics` for both tasks
92+
93+
Two tasks perform full load replication on Dockerized MariaDB. The other two perform CDC replication on a MariaDB RDS database.
94+
95+
All tasks target the same Kinesis Stream.
96+
97+
## Deploying on AWS
98+
99+
You can deploy and run the stack on AWS by running the following commands:
100+
101+
```bash
102+
make deploy-aws
103+
make run-aws
104+
```
105+
106+
## License
107+
108+
This project is licensed under the Apache 2.0 License.

0 commit comments

Comments
 (0)