Skip to content

Commit 275c3c6

Browse files
committed
Start review
1 parent caa6ce1 commit 275c3c6

2 files changed

Lines changed: 53 additions & 40 deletions

File tree

content/en/docs/refguide/modeling/integration/change-data-capture/_index.md

Lines changed: 27 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,25 @@
11
---
22
title: "Change Data Capture"
33
url: /refguide/change-data-capture/
4-
weight: 45
4+
weight: 85
55
description: "Describes Change Data Capture (CDC) services in Studio Pro, which publish domain model entity changes as Kafka streams for data warehouse and analytics pipelines."
66
---
77

88
## Introduction
99

10-
Change Data Capture (CDC) lets you stream domain model data out of a Mendix app in near real time. When objects in a tracked entity are created, updated, or deleted, the Mendix runtime captures those changes and publishes them as events to a Kafka topic. Downstream systems — such as data warehouses or analytics pipelines — can then consume the stream without polling the Mendix database.
10+
{{% alert color="warning" %}} This feature is in beta. For more information, see [Release Status](/releasenotes/release-status/). {{% /alert %}}
11+
12+
Change Data Capture (CDC) lets you stream domain model data out of a Mendix app in near real time. When objects in a tracked entity are created, updated, or deleted, the Mendix Runtime captures those changes and publishes them as events to a Kafka topic. Downstream systems, such as data warehouses or analytics pipelines, can then consume the stream without polling the Mendix database.
1113

1214
CDC is intended for developers who need to move Mendix domain model data to external data stores, such as a data warehouse, data lake, or blob storage.
1315

1416
## How It Works
1517

16-
A CDC service document in Studio Pro defines which entities the runtime should track. On deployment, each tracked entity gets its own Kafka topic. Every time a committed object change occurs create, update, or deletethe runtime publishes an event to the corresponding topic.
18+
A CDC service document in Studio Pro defines which entities the Mendix Runtime should track. On deployment, each tracked entity gets its own Kafka topic. Every time a committed object change occurs (create, update, or delete) the Mendix Runtime publishes an event to the corresponding topic.
1719

18-
Read events are triggered as snapshots during startup or when a stream changes in a way that constitutes a breaking change. Snapshots populate the new topic with the current data of the tracked entity. See [Revisions](/refguide/published-cdc-services/#revisions).
20+
Read events are triggered as snapshots during startup or when a stream changes in a way that constitutes a breaking change. Snapshots populate the new topic with the current data of the tracked entity. For more information, see the [Revisions](/refguide/published-cdc-services/#revisions) section below.
1921

20-
The Kafka broker is either the [Mendix Event Broker](/appstore/services/event-broker/) or a Bring Your Own Kafka (BYOK) cluster. For details on configuring BYOK, see the [Mendix Event Broker](/appstore/services/event-broker/) page.
22+
The broker that receives these events is either the [Mendix Event Broker](/appstore/services/event-broker/) or a Bring Your Own Kafka (BYOK) cluster. For BYOK configuration details, see [Mendix Event Broker](/appstore/services/event-broker/).
2123

2224
To move the streamed data to a destination such as Azure Blob Storage or AWS S3, you configure an [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) separately in the Event Broker Manager after deployment.
2325

@@ -27,41 +29,48 @@ To move the streamed data to a destination such as Azure Blob Storage or AWS S3,
2729
Change Data Capture is not available for Free Apps. A licensed Mendix Cloud environment is required.
2830
{{% /alert %}}
2931

30-
* A licensed Mendix Cloud environment.
31-
* A [Mendix Event Broker](/appstore/services/event-broker/) license, or a BYOK Kafka cluster configured as described in [Mendix Event Broker](/appstore/services/event-broker/).
32-
* An [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) configured in the Event Broker Manager if you want to route CDC events to external storage.
32+
* A licensed Mendix Cloud environment
33+
* A [Mendix Event Broker](/appstore/services/event-broker/) license, or a BYOK Kafka cluster configured as described in [Mendix Event Broker](/appstore/services/event-broker/)
34+
* An [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) configured in the Event Broker Manager if you want to route CDC events to external storage
3335

3436
## Setting Up Change Data Capture
3537

38+
Set up a CDC service in Studio Pro by following the steps below:
39+
3640
1. In Studio Pro, right-click a module in the App Explorer and choose **Add other** > **Change data capture service**.
37-
2. Configure the entities to track and their exposed names. See [Published CDC Services](/refguide/published-cdc-services/) for details.
38-
3. Deploy the app. The runtime creates Kafka topics for each tracked entity automatically.
41+
2. Select the entities to track and set an exposed name for each. For more information on configuring entities, see the [Entities to Track](/refguide/published-cdc-services/#entities) section of *Published CDC Services*.
42+
3. Deploy the app. The Runtime creates Kafka topics for each tracked entity automatically.
3943
4. In the [Event Broker Manager](https://broker.mendix.com/), configure an [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) to route CDC events to your destination.
4044

4145
## Runtime Configuration {#runtime-configuration}
4246

43-
CDC requires runtime settings to connect to a Kafka broker. In Studio Pro, open **App** > **Settings**, go to the **Configurations** tab, select or create a configuration, and open the **Custom** tab to add the settings below. For deployed environments, set these via your environment's custom runtime settings.
47+
CDC requires runtime settings to connect to a Kafka broker. Follow the steps below:
48+
49+
1. In Studio Pro, open **App** > **Settings**.
50+
2. Open the **Configurations** tab.
51+
3. Select an existing configuration or click **New** to make a new one.
52+
4. In the configuration, open the **Custom** tab to add the settings documented in the section below. For deployed environments, set these via your environment's custom runtime settings.
4453

4554
{{< figure src="/attachments/refguide/modeling/integration/change-data-capture/cdc-custom-configuration.png" alt="Edit Configuration dialog in Studio Pro showing the Custom tab with Kafka.BootstrapServers set to an IP address and port" >}}
4655

4756
### Running Locally {#local-configuration}
4857

49-
When running the app locally, only the bootstrap server address is required:
58+
When running the app locally, only the bootstrap server address is required.
5059

5160
| Name | Description | Default Value |
5261
| --- | --- | --- |
5362
| `Kafka.BootstrapServers` | The address of the Kafka broker, in the format `host:port`. | |
5463

5564
### Bring Your Own Kafka (BYOK) {#byok-configuration}
5665

57-
When connecting to a BYOK Kafka cluster, provide the bootstrap server address and credentials for authentication. The supported authentication method is SASL/SCRAM-SHA-512.
66+
When connecting to a BYOK Kafka cluster, provide the bootstrap server address and credentials for authentication. The supported authentication method is `SASL/SCRAM-SHA-512`.
5867

5968
| Name | Description | Default Value |
60-
| --- | --- | --- | --- |
69+
| --- | --- | --- |
6170
| `Kafka.BootstrapServers` | The address of the Kafka broker, in the format `host:port`. | |
62-
| `Kafka.Username` | The username for SASL/SCRAM-SHA-512 authentication with the Kafka cluster. | |
63-
| `Kafka.Password` | The password for SASL/SCRAM-SHA-512 authentication with the Kafka cluster. | |
64-
| `EventBroker.Space` | The Event Broker space in which the app is placed. Defines which other applications can exchange events with this app. | `local` |
71+
| `Kafka.Username` | The username for `SASL/SCRAM-SHA-512` authentication with the Kafka cluster. | |
72+
| `Kafka.Password` | The password for `SASL/SCRAM-SHA-512` authentication with the Kafka cluster. | |
73+
| `EventBroker.Space` | The Event Broker space in which the app is placed. This defines which other applications can exchange events with this app. | `local` |
6574

6675
### Active-Producer Election {#active-producer-election}
6776

@@ -73,7 +82,7 @@ If the active instance stops sending heartbeats (for example, because it crashed
7382

7483
The `EventBroker.CdcProducerTransactionId` setting enables Kafka transactions on the active producer, which provides exactly-once delivery guarantees during normal operation and clean handover during failover.
7584

76-
If any of the three settings is missing, the runtime logs a warning and falls back to single-instance mode, where all instances publish independently.
85+
If any of the three settings is missing, the Mendix Runtime logs a warning and falls back to single-instance mode, where all instances publish independently.
7786

7887
### Message Format {#message-format}
7988

content/en/docs/refguide/modeling/integration/change-data-capture/published-cdc-services.md

Lines changed: 26 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,9 @@ description: "Describes how to configure a Published CDC Service document in Stu
77

88
## Introduction
99

10-
A Published CDC Service document defines the entities whose object changes the Mendix runtime tracks and publishes as Kafka events. Each tracked entity produces a stream of create, update, and delete events on its own Kafka topic.
10+
{{% alert color="warning" %}} This feature is in beta. For more information, see [Release Status](/releasenotes/release-status/). {{% /alert %}}
1111

12-
{{% alert color="warning" %}}
13-
Change Data Capture is a beta feature in Mendix 11.12. Its behavior and configuration options may change in future releases.
14-
{{% /alert %}}
12+
A Published CDC Service document defines the entities whose object changes the Mendix Runtime tracks and publishes as Kafka events. Each tracked entity produces a stream of create, update, and delete events on its own Kafka topic.
1513

1614
## Creating a Published CDC Service {#create}
1715

@@ -23,7 +21,7 @@ You can have multiple CDC service documents in an app — for example, to group
2321

2422
### Service Name {#service-name}
2523

26-
The service name uniquely identifies the CDC service within the app. The App Name is used as part of the topic to ensure uniqueness.
24+
The service name uniquely identifies the CDC service within the app. The app name is used as part of the topic to ensure uniqueness.
2725

2826
### Description {#description}
2927

@@ -37,28 +35,34 @@ The **Entities to track** table lists the entities whose object changes are publ
3735

3836
Use the toolbar to manage tracked entities:
3937

40-
* **+ Add** — add an entity from the domain model
41-
* **Remove** — stop tracking a selected entity
42-
* **Accept changes** — lock in the current revision numbers after reviewing modifications (see [Revisions](#revisions))
38+
* {{% icon name="add-filled" %}} **Add** — add an entity from the domain model
39+
* {{% icon name="subtract-circle" %}} **Remove** — stop tracking a selected entity
40+
* {{% icon name="refresh" %}} **Accept changes** — lock in the current revision numbers after reviewing modifications (see [Revisions](#revisions))
4341

4442
Each row in the table has the following columns:
4543

46-
| Column | Description |
47-
| --- | --- |
48-
| **Entities** | The domain model entity being tracked. Expand the row to view and select individual attributes and associations. |
49-
| **Exposed name** | The name used for this entity in the Kafka topic and event payload. Defaults to the entity name. |
50-
| **Modification** | The pending change state: **Added**, **Changed**, or **Removed**. Blank if the entity is unchanged since the last accepted revision. |
51-
| **Revision** | The schema revision of the entity's event payload. See [Revisions](#revisions). |
52-
| **Topic** | The Kafka topic name for this entity, in the format `cdc.<app-name>.<ExposedName>.<revision>.{space}`, where `{space}` is replaced at runtime by the Event Broker space name. See [Bring Your Own Kafka (BYOK)](/refguide/change-data-capture/#byok-configuration) |
44+
### Entities
45+
46+
The domain model entity being tracked. Expand the row to view and select individual attributes and associations
47+
48+
### Exposed Name
49+
50+
The name used for this entity in the Kafka topic and event payload. Defaults to the entity name.
51+
52+
### Modification
53+
54+
The pending change state: **Added**, **Changed**, or **Removed**. Blank if the entity is unchanged since the last accepted revision.
55+
56+
### Revision
57+
58+
The schema revision of the entity's event payload. See [Revisions](#revisions).
59+
60+
### Topic
5361

54-
### Adding an Entity {#add-entity}
62+
The Kafka topic name for this entity, in the format `cdc.<app-name>.<ExposedName>.<revision>.{space}`, where `{space}` is replaced at runtime by the Event Broker space name. See [Bring Your Own Kafka (BYOK)](/refguide/change-data-capture/#byok-configuration).
5563

56-
1. Click **+ Add** in the toolbar.
57-
2. Select a persistable entity from the domain model.
58-
3. Optionally edit the **Exposed name**.
59-
4. Expand the row to deselect any attributes or associations you do not want included in the event payload.
6064

61-
### Attribute and Association Selection {#attributes}
65+
## Attribute and Association Selection {#attributes}
6266

6367
Expand an entity row to see each attribute and association with a checkbox. Uncheck an item to exclude it from the event payload. The **Exposed name** column lets you rename individual attributes in the payload independently of their domain model names.
6468

@@ -82,7 +86,7 @@ Pending modifications are not finalized until you click **Accept changes** in th
8286
Accepting changes confirms the new revision numbers and clears the modification states, leaving the document in a clean state ready for deployment.
8387

8488
{{% alert color="warning" %}}
85-
A major revision creates a new Kafka topic. Consumers subscribed to the previous topic will no longer receive events after deployment. Ensure downstream systems are updated in coordination of deploying a major revision change.
89+
A major revision creates a new Kafka topic. Consumers subscribed to the previous topic will no longer receive events after deployment. Ensure downstream systems are updated before or alongside deploying a major revision change.
8690
{{% /alert %}}
8791

8892
## Runtime Behavior {#runtime}

0 commit comments

Comments
 (0)