You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/en/docs/refguide/modeling/integration/change-data-capture/_index.md
+27-18Lines changed: 27 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,23 +1,25 @@
1
1
---
2
2
title: "Change Data Capture"
3
3
url: /refguide/change-data-capture/
4
-
weight: 45
4
+
weight: 85
5
5
description: "Describes Change Data Capture (CDC) services in Studio Pro, which publish domain model entity changes as Kafka streams for data warehouse and analytics pipelines."
6
6
---
7
7
8
8
## Introduction
9
9
10
-
Change Data Capture (CDC) lets you stream domain model data out of a Mendix app in near real time. When objects in a tracked entity are created, updated, or deleted, the Mendix runtime captures those changes and publishes them as events to a Kafka topic. Downstream systems — such as data warehouses or analytics pipelines — can then consume the stream without polling the Mendix database.
10
+
{{% alert color="warning" %}} This feature is in beta. For more information, see [Release Status](/releasenotes/release-status/). {{% /alert %}}
11
+
12
+
Change Data Capture (CDC) lets you stream domain model data out of a Mendix app in near real time. When objects in a tracked entity are created, updated, or deleted, the Mendix Runtime captures those changes and publishes them as events to a Kafka topic. Downstream systems, such as data warehouses or analytics pipelines, can then consume the stream without polling the Mendix database.
11
13
12
14
CDC is intended for developers who need to move Mendix domain model data to external data stores, such as a data warehouse, data lake, or blob storage.
13
15
14
16
## How It Works
15
17
16
-
A CDC service document in Studio Pro defines which entities the runtime should track. On deployment, each tracked entity gets its own Kafka topic. Every time a committed object change occurs — create, update, or delete — the runtime publishes an event to the corresponding topic.
18
+
A CDC service document in Studio Pro defines which entities the Mendix Runtime should track. On deployment, each tracked entity gets its own Kafka topic. Every time a committed object change occurs (create, update, or delete) the Mendix Runtime publishes an event to the corresponding topic.
17
19
18
-
Read events are triggered as snapshots during startup or when a stream changes in a way that constitutes a breaking change. Snapshots populate the new topic with the current data of the tracked entity. See [Revisions](/refguide/published-cdc-services/#revisions).
20
+
Read events are triggered as snapshots during startup or when a stream changes in a way that constitutes a breaking change. Snapshots populate the new topic with the current data of the tracked entity. For more information, see the [Revisions](/refguide/published-cdc-services/#revisions) section below.
19
21
20
-
The Kafka broker is either the [Mendix Event Broker](/appstore/services/event-broker/) or a Bring Your Own Kafka (BYOK) cluster. For details on configuring BYOK, see the [Mendix Event Broker](/appstore/services/event-broker/) page.
22
+
The broker that receives these events is either the [Mendix Event Broker](/appstore/services/event-broker/) or a Bring Your Own Kafka (BYOK) cluster. For BYOK configuration details, see [Mendix Event Broker](/appstore/services/event-broker/).
21
23
22
24
To move the streamed data to a destination such as Azure Blob Storage or AWS S3, you configure an [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) separately in the Event Broker Manager after deployment.
23
25
@@ -27,41 +29,48 @@ To move the streamed data to a destination such as Azure Blob Storage or AWS S3,
27
29
Change Data Capture is not available for Free Apps. A licensed Mendix Cloud environment is required.
28
30
{{% /alert %}}
29
31
30
-
* A licensed Mendix Cloud environment.
31
-
* A [Mendix Event Broker](/appstore/services/event-broker/) license, or a BYOK Kafka cluster configured as described in [Mendix Event Broker](/appstore/services/event-broker/).
32
-
* An [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) configured in the Event Broker Manager if you want to route CDC events to external storage.
32
+
* A licensed Mendix Cloud environment
33
+
* A [Mendix Event Broker](/appstore/services/event-broker/) license, or a BYOK Kafka cluster configured as described in [Mendix Event Broker](/appstore/services/event-broker/)
34
+
* An [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) configured in the Event Broker Manager if you want to route CDC events to external storage
33
35
34
36
## Setting Up Change Data Capture
35
37
38
+
Set up a CDC service in Studio Pro by following the steps below:
39
+
36
40
1. In Studio Pro, right-click a module in the App Explorer and choose **Add other** > **Change data capture service**.
37
-
2.Configure the entities to track and their exposed names. See [Published CDC Services](/refguide/published-cdc-services/) for details.
38
-
3. Deploy the app. The runtime creates Kafka topics for each tracked entity automatically.
41
+
2.Select the entities to track and set an exposed name for each. For more information on configuring entities, see the [Entities to Track](/refguide/published-cdc-services/#entities) section of *Published CDC Services*.
42
+
3. Deploy the app. The Runtime creates Kafka topics for each tracked entity automatically.
39
43
4. In the [Event Broker Manager](https://broker.mendix.com/), configure an [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) to route CDC events to your destination.
40
44
41
45
## Runtime Configuration {#runtime-configuration}
42
46
43
-
CDC requires runtime settings to connect to a Kafka broker. In Studio Pro, open **App** > **Settings**, go to the **Configurations** tab, select or create a configuration, and open the **Custom** tab to add the settings below. For deployed environments, set these via your environment's custom runtime settings.
47
+
CDC requires runtime settings to connect to a Kafka broker. Follow the steps below:
48
+
49
+
1. In Studio Pro, open **App** > **Settings**.
50
+
2. Open the **Configurations** tab.
51
+
3. Select an existing configuration or click **New** to make a new one.
52
+
4. In the configuration, open the **Custom** tab to add the settings documented in the section below. For deployed environments, set these via your environment's custom runtime settings.
44
53
45
54
{{< figure src="/attachments/refguide/modeling/integration/change-data-capture/cdc-custom-configuration.png" alt="Edit Configuration dialog in Studio Pro showing the Custom tab with Kafka.BootstrapServers set to an IP address and port" >}}
46
55
47
56
### Running Locally {#local-configuration}
48
57
49
-
When running the app locally, only the bootstrap server address is required:
58
+
When running the app locally, only the bootstrap server address is required.
50
59
51
60
| Name | Description | Default Value |
52
61
| --- | --- | --- |
53
62
|`Kafka.BootstrapServers`| The address of the Kafka broker, in the format `host:port`. ||
54
63
55
64
### Bring Your Own Kafka (BYOK) {#byok-configuration}
56
65
57
-
When connecting to a BYOK Kafka cluster, provide the bootstrap server address and credentials for authentication. The supported authentication method is SASL/SCRAM-SHA-512.
66
+
When connecting to a BYOK Kafka cluster, provide the bootstrap server address and credentials for authentication. The supported authentication method is `SASL/SCRAM-SHA-512`.
58
67
59
68
| Name | Description | Default Value |
60
-
| --- | --- | --- | --- |
69
+
| --- | --- | --- |
61
70
|`Kafka.BootstrapServers`| The address of the Kafka broker, in the format `host:port`. ||
62
-
|`Kafka.Username`| The username for SASL/SCRAM-SHA-512 authentication with the Kafka cluster. ||
63
-
|`Kafka.Password`| The password for SASL/SCRAM-SHA-512 authentication with the Kafka cluster. ||
64
-
|`EventBroker.Space`| The Event Broker space in which the app is placed. Defines which other applications can exchange events with this app. |`local`|
71
+
|`Kafka.Username`| The username for `SASL/SCRAM-SHA-512` authentication with the Kafka cluster. ||
72
+
|`Kafka.Password`| The password for `SASL/SCRAM-SHA-512` authentication with the Kafka cluster. ||
73
+
|`EventBroker.Space`| The Event Broker space in which the app is placed. This defines which other applications can exchange events with this app. |`local`|
@@ -73,7 +82,7 @@ If the active instance stops sending heartbeats (for example, because it crashed
73
82
74
83
The `EventBroker.CdcProducerTransactionId` setting enables Kafka transactions on the active producer, which provides exactly-once delivery guarantees during normal operation and clean handover during failover.
75
84
76
-
If any of the three settings is missing, the runtime logs a warning and falls back to single-instance mode, where all instances publish independently.
85
+
If any of the three settings is missing, the Mendix Runtime logs a warning and falls back to single-instance mode, where all instances publish independently.
Copy file name to clipboardExpand all lines: content/en/docs/refguide/modeling/integration/change-data-capture/published-cdc-services.md
+26-22Lines changed: 26 additions & 22 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,11 +7,9 @@ description: "Describes how to configure a Published CDC Service document in Stu
7
7
8
8
## Introduction
9
9
10
-
A Published CDC Service document defines the entities whose object changes the Mendix runtime tracks and publishes as Kafka events. Each tracked entity produces a stream of create, update, and delete events on its own Kafka topic.
10
+
{{% alert color="warning" %}} This feature is in beta. For more information, see [Release Status](/releasenotes/release-status/). {{% /alert %}}
11
11
12
-
{{% alert color="warning" %}}
13
-
Change Data Capture is a beta feature in Mendix 11.12. Its behavior and configuration options may change in future releases.
14
-
{{% /alert %}}
12
+
A Published CDC Service document defines the entities whose object changes the Mendix Runtime tracks and publishes as Kafka events. Each tracked entity produces a stream of create, update, and delete events on its own Kafka topic.
15
13
16
14
## Creating a Published CDC Service {#create}
17
15
@@ -23,7 +21,7 @@ You can have multiple CDC service documents in an app — for example, to group
23
21
24
22
### Service Name {#service-name}
25
23
26
-
The service name uniquely identifies the CDC service within the app. The App Name is used as part of the topic to ensure uniqueness.
24
+
The service name uniquely identifies the CDC service within the app. The app name is used as part of the topic to ensure uniqueness.
27
25
28
26
### Description {#description}
29
27
@@ -37,28 +35,34 @@ The **Entities to track** table lists the entities whose object changes are publ
37
35
38
36
Use the toolbar to manage tracked entities:
39
37
40
-
***+ Add** — add an entity from the domain model
41
-
***Remove** — stop tracking a selected entity
42
-
***Accept changes** — lock in the current revision numbers after reviewing modifications (see [Revisions](#revisions))
38
+
*{{% icon name="add-filled" %}} **Add** — add an entity from the domain model
*{{% icon name="refresh" %}} **Accept changes** — lock in the current revision numbers after reviewing modifications (see [Revisions](#revisions))
43
41
44
42
Each row in the table has the following columns:
45
43
46
-
| Column | Description |
47
-
| --- | --- |
48
-
|**Entities**| The domain model entity being tracked. Expand the row to view and select individual attributes and associations. |
49
-
|**Exposed name**| The name used for this entity in the Kafka topic and event payload. Defaults to the entity name. |
50
-
|**Modification**| The pending change state: **Added**, **Changed**, or **Removed**. Blank if the entity is unchanged since the last accepted revision. |
51
-
|**Revision**| The schema revision of the entity's event payload. See [Revisions](#revisions). |
52
-
|**Topic**| The Kafka topic name for this entity, in the format `cdc.<app-name>.<ExposedName>.<revision>.{space}`, where `{space}` is replaced at runtime by the Event Broker space name. See [Bring Your Own Kafka (BYOK)](/refguide/change-data-capture/#byok-configuration)|
44
+
### Entities
45
+
46
+
The domain model entity being tracked. Expand the row to view and select individual attributes and associations
47
+
48
+
### Exposed Name
49
+
50
+
The name used for this entity in the Kafka topic and event payload. Defaults to the entity name.
51
+
52
+
### Modification
53
+
54
+
The pending change state: **Added**, **Changed**, or **Removed**. Blank if the entity is unchanged since the last accepted revision.
55
+
56
+
### Revision
57
+
58
+
The schema revision of the entity's event payload. See [Revisions](#revisions).
59
+
60
+
### Topic
53
61
54
-
### Adding an Entity {#add-entity}
62
+
The Kafka topic name for this entity, in the format `cdc.<app-name>.<ExposedName>.<revision>.{space}`, where `{space}` is replaced at runtime by the Event Broker space name. See [Bring Your Own Kafka (BYOK)](/refguide/change-data-capture/#byok-configuration).
55
63
56
-
1. Click **+ Add** in the toolbar.
57
-
2. Select a persistable entity from the domain model.
58
-
3. Optionally edit the **Exposed name**.
59
-
4. Expand the row to deselect any attributes or associations you do not want included in the event payload.
60
64
61
-
###Attribute and Association Selection {#attributes}
65
+
## Attribute and Association Selection {#attributes}
62
66
63
67
Expand an entity row to see each attribute and association with a checkbox. Uncheck an item to exclude it from the event payload. The **Exposed name** column lets you rename individual attributes in the payload independently of their domain model names.
64
68
@@ -82,7 +86,7 @@ Pending modifications are not finalized until you click **Accept changes** in th
82
86
Accepting changes confirms the new revision numbers and clears the modification states, leaving the document in a clean state ready for deployment.
83
87
84
88
{{% alert color="warning" %}}
85
-
A major revision creates a new Kafka topic. Consumers subscribed to the previous topic will no longer receive events after deployment. Ensure downstream systems are updated in coordination of deploying a major revision change.
89
+
A major revision creates a new Kafka topic. Consumers subscribed to the previous topic will no longer receive events after deployment. Ensure downstream systems are updated before or alongside deploying a major revision change.
0 commit comments