Skip to content

Commit 2e25b71

Browse files
juanpardojuanpardo
authored
Added the DynamoDB integration (#120)
* Added the DynamoDB integration documentation
1 parent c46fc7d commit 2e25b71

File tree

4 files changed

+154
-7
lines changed

4 files changed

+154
-7
lines changed

‎docs/cluster/import.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ Currently the following data formats are supported:
1313
- JSON (JSON-Lines, JSON Arrays, and JSON Documents)
1414
- Parquet
1515
- MongoDB collection
16+
- DynamoDB
1617

1718
:::{note}
1819
If you don't have a dataset prepared, we also provide sample data to let

‎docs/cluster/integrations/dynamo-cdc.md‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
(integrations-dynamo-cdc)=
2+
# DynamoDB CDC
3+
4+
CrateDB Cloud enables continuous data ingestion from DynamoDB using Change Data
5+
Capture (CDC), providing real-time synchronization of your data.
6+
7+
## Key Concepts
8+
9+
The DynamoDB CDC integration in CrateDB Cloud allows you to keep your data
10+
synchronized between a DynamoDB table and your CrateDB Cloud cluster
11+
in real-time.
12+
13+
### How It Works
14+
15+
The integration functions in two main stages:
16+
17+
1. **Initial Sync:**
18+
The integration performs a complete scan of your DynamoDB table.
19+
20+
2. **Continuous Sync:**
21+
The integration uses a Kinesis Data Stream to read changes from your DynamoDB table.
22+
Please note the Kinesis Data Stream needs to be set up before you can use this stage.
23+
You can refer to the AWS documentation page
24+
[Getting started with Kinesis Data Streams for Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds_gettingstarted.html)
25+
to complete your setup.
26+
27+
### Data Consistency and Mode
28+
29+
For continuous sync, CrateDB Cloud uses a Kinesis Data Stream. This provides full items
30+
that are then written as either full inserts or full upserts to ensure data consistency.
31+
Directly reading CDC events from DynamoDB is limited to 1 day of history. By using
32+
a Kinesis Data Stream you can choose the retention policy that best suits your needs.
33+
34+
---
35+
36+
## Create a new Integration
37+
A DynamoDB integration allows you to sync a single table from DynamoDB. You can reuse an existing Cloud secret across multiple integrations
38+
to continuously sync data from multiple DynamoDB tables.
39+
40+
Supported authentication methods:
41+
- AWS Access Key and Access secret pair.
42+
43+
### Optional setup for CDC
44+
This setup is only required if you want to enable the Continuous Sync.
45+
Please note that running a Kinesis data stream has additional costs that AWS will charge you directly.
46+
47+
#### Set up a Kinesis Data Stream for CDC in AWS
48+
1. In the **AWS console**, go to **DynamoDB**.
49+
2. Go to **Tables**.
50+
3. Click on the table name you want to use.
51+
4. Click on the **Exports and streams** tab.
52+
5. On the **Amazon Kinesis data stream details** section, click the button **Turn On**.
53+
6. Click the **Create new** button right to **Destination Kinesis data stream**.
54+
7. Fill in the **Data stream name**. Please note you will need this name later when setting up the integration in CrateDB Cloud.
55+
8. Choose the capacity that best suits your needs.
56+
8. Choose a **Maximum record size** that can hold more than two times your biggest DynamoDB table items.
57+
9. Click the button **Create data stream**.
58+
10. The Kinesis data stream has now been created. If you want to change its retention policy (recommended), click on the tab **Configuration**, then click on **Edit** under the **Data retention** section.
59+
11. Select the data retention that suits you. At least 7 days is recommended. Please note the data retention period will have an impact on the cost of the Kinesis Data Stream.
60+
12. Back to the **Stream to an Amazon Kinesis data stream** DynamoDB page, select the newly created Kinesis Stream from the list.
61+
13. Click on **Turn on stream**.
62+
63+
64+
### Set Up Integration in CrateDB Cloud
65+
66+
Follow these steps in the CrateDB Cloud Console to set up the DynamoDB CDC integration:
67+
68+
:::::{stepper}
69+
#### Create an Integration
70+
1. Navigate to the **Import** section in the CrateDB Cloud Console.
71+
2. Click on the **DynamoDB** button.
72+
73+
#### Configure Secret
74+
1. Choose **Add Secret** or select an existing one.
75+
2. Fill in the following details:
76+
:::{tab} AWS Secret
77+
- **Secret Name**: Provide a unique name for the secret.
78+
- **Access key**: The AWS access key ID that will be used. Please note this access key requires access to your DynamoDB table and, if CDC is enabled, also to the Kinesis data stream.
79+
- **Secret access key**: The AWS secret key.
80+
:::
81+
82+
#### Configure Integration Settings
83+
1. Enter a name for the integration.
84+
2. Select the integration mode:
85+
- **Full load only**: Imports the data once but doesn’t sync changes.
86+
- **Full load and CDC**: Imports the data and syncs changes in real-time.
87+
- **CDC only**: Syncs only new changes in real-time without importing existing data.
88+
89+
#### Select source
90+
1. Enter the AWS region name the DynamoDB table is in.
91+
2. Enter the DynamoDB table name.
92+
3. If CDC is enabled, please also enter the Kinesis stream name.
93+
94+
#### Select Target Table
95+
1. Specify the target table in your CrateDB Cloud cluster where the data will be synced.
96+
2. DynamoDB items will be inserted into an object column called `document`.
97+
3. Select the object type for the column:
98+
- **`dynamic`**: Allows indexing and columnar storage for faster querying.
99+
- **`ignored`**: Prevents type conflicts in CrateDB if your source data lacks a strict schema.
100+
101+
:::{note}
102+
If your source data doesn't follow a strict schema, select `ignored` to avoid type conflicts.
103+
However, selecting `dynamic` provides faster query performance by utilizing indexes and columnar storage.
104+
:::
105+
106+
107+
#### Create the Integration
108+
Click **Start import** to finalize the setup. CrateDB Cloud will now sync
109+
your DynamoDB data based on the selected settings.
110+
:::::
111+
112+
### Column Name Restrictions
113+
114+
Column or property names containing square brackets `[]` are not supported and
115+
are replaced with `_obkt_` and `_cbkt_` respectively. Likewise, column
116+
names containing dots `.` are not supported and are replaced with `_dot_`.
117+
118+
:::{warning}
119+
This behavior may change in future releases.
120+
:::

‎docs/cluster/integrations/index.md‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,9 @@ setup or maintenance of separate ETL pipelines.
2020
:titlesonly:
2121
:hidden:
2222

23-
MongoDB CDC (Preview) <mongo-cdc>
23+
MongoDB CDC <mongo-cdc>
24+
DynamoDB CDC <dynamo-cdc>
25+
2426
:::
2527

2628
---
@@ -40,11 +42,20 @@ in real time. You can create multiple integrations for the same data source to
4042
support different tables or configurations.
4143

4244
Currently, CrateDB Cloud supports ingestion from the following data source:
43-
- {ref}`MongoDB CDC (Preview) <integrations-mongo-cdc>`
45+
- {ref}`MongoDB CDC <integrations-mongo-cdc>`
46+
- {ref}`DynamoDB CDC <integrations-dynamo-cdc>`
4447

4548
More integrations are planned for future releases to expand the range of
4649
supported data sources and use cases.
4750

51+
The integration automatically make field names compatible with CrateDB naming
52+
restrictions.
53+
54+
When a new data field is discovered, the integration will infer its data type.
55+
To cast data to specific types, simply create the column in CrateDB with the target type
56+
and. The integration will always try to cast the data to the type defined in the
57+
CrateDB destination table.
58+
4859
:::
4960
### Connection
5061
:::
@@ -54,3 +65,18 @@ specific data source. This allows secure access to the external system and is
5465
reusable across multiple integrations. By setting up a connection once, you can
5566
streamline the process of creating and managing integrations without having to
5667
re-enter credentials for each one.
68+
69+
:::
70+
### Integration types
71+
:::
72+
73+
There are 3 different integration types:
74+
- **Full load only**: Imports all your data and immediately ends after.
75+
- **CDC only**: It indefinitely listens to CDC (Change Data Capture) events on the
76+
source and applies them into your CrateDB table. Once it reaches the last CDC event
77+
it waits for new events to come.
78+
- **Full load and CDC**
79+
It imports all the data like the type **full load only**, but once that phase finishes
80+
it starts processing CDC events. If the source supports it, it will try to read CDC
81+
events starting from right when the import phase started. This way any data alteration
82+
during the import phase will be picked up and processed.

‎docs/cluster/integrations/mongo-cdc.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -174,10 +174,10 @@ Follow these steps in the CrateDB Cloud Console to set up the MongoDB CDC integr
174174
:::::{stepper}
175175
#### Create an Integration
176176
1. Navigate to the **Import** section in the CrateDB Cloud Console.
177-
2. Click **Create Integration** and select **MongoDB** as the source type.
177+
2. Click on the **MongoDB** button.
178178

179179
#### Configure Connection
180-
1. Choose **Create New Connection** or select an existing one.
180+
1. Choose which connection type to create or select an existing one.
181181
2. Fill in the following details:
182182
:::{tab} SCRAM Authentication
183183
- **Connection Name**: Provide a unique name for the connection.
@@ -216,9 +216,9 @@ want to sync with CrateDB Cloud.
216216
#### Configure Integration Settings
217217
1. Enter a name for the integration.
218218
2. Select the integration mode:
219-
- **Full Load Only**: Imports the data once but doesn’t sync changes.
220-
- **Full Load and CDC**: Imports the data and syncs changes in real-time.
221-
- **CDC Only**: Syncs only new changes in real-time without importing existing data.
219+
- **Full load only**: Imports the data once but doesn’t sync changes.
220+
- **Full load and CDC**: Imports the data and syncs changes in real-time.
221+
- **CDC only**: Syncs only new changes in real-time without importing existing data.
222222

223223
#### Create the Integration
224224
Click **Create Integration** to finalize the setup. CrateDB Cloud will now sync

0 commit comments

Comments
 (0)