Merge pull request #11464 from stefanus-heath/cdc-documentation

DRAFT: Change Data Capture documentation

Author: quinntracy

content/en/docs/refguide/modeling/integration/_index.md

@@ -9,19 +9,20 @@ description: "Introduces the services that Mendix uses for application integrati
 
 ## Introduction
 
-Integration with other applications (other Mendix apps or applications built with different technology) can be done using REST, OData, GraphQL, SOAP/Web Services, or Business Events. Mendix can also import and export data from XML and JSON and supports integration with machine learning models.
+Integration with other applications (other Mendix apps or applications built with different technology) can be done using REST, OData, GraphQL, SOAP/Web Services, Change Data Capture, and Business Events. Mendix can also import and export data from XML and JSON and supports integration with machine learning models.
 
 Data integration capabilities in Studio Pro is part of [Mendix Connect](#integration-mx-connect).
 
 ## Integration and Mendix Connect {#integration-mx-connect}
 
 [Mendix Connect](https://www.mendix.com/data-hub/) is a collection of functionalities available in the Mendix Platform that allows people in your organization to discover, understand, connect, and govern data securely. The following published and consumed integration services in Studio Pro allow you connect and share data between and among apps using APIs:
 
-* [OData services](/refguide/integration/odata-services/) (view and manage in your company's [Catalog](/catalog/))
 * [REST services](/refguide/integration/rest-services/)
+* [OData services](/refguide/integration/odata-services/) (view and manage in your company's [Catalog](/catalog/))
+* [Published GraphQL services](/refguide/published-odata-services/) (experimental)
 * [Web services](/refguide/integration/web-services/)
+* [Change Data Capture services](/refguide/change-data-capture/)
 * [Business Event services](/refguide/business-event-services/)
-* [Published GraphQL services](/refguide/published-odata-services/) (experimental)
 
 In addition to integration capabilities in Studio Pro, Mendix Connect functionalities include:
 
@@ -30,9 +31,11 @@ In addition to integration capabilities in Studio Pro, Mendix Connect functional
 
 The following services require you to download an external module from the Mendix Marketplace prior to use:
 
+* [Data Importer](https://marketplace.mendix.com/link/component/219833)
 * [External Database Connector](https://marketplace.mendix.com/link/component/219862)
 * [Business Event services](https://marketplace.mendix.com/link/component/202649)
 
+
 ## Integrating RapidMiner Graph Studio
 
 In addition to the integration capabilities available in the Studio Pro, Mendix also supports integration with external systems through RapidMiner Graph Studio. The RapidMiner Graph Studio provides a scalable environment for connecting, managing, and analyzing graph-based data. For more information, see [Integrating Mendix with RapidMiner Graph Studio](/refguide/graph-studio/).

content/en/docs/refguide/modeling/integration/change-data-capture/_index.md

@@ -0,0 +1,105 @@
+---
+title: "Change Data Capture"
+url: /refguide/change-data-capture/
+weight: 85
+description: "Describes Change Data Capture (CDC) services in Studio Pro, which publish domain model entity changes as Kafka streams for data warehouse and analytics pipelines."
+---
+
+## Introduction
+
+{{% alert color="warning" %}} This feature is in beta. For more information, see [Release Status](/releasenotes/release-status/). {{% /alert %}}
+
+Change Data Capture (CDC) lets you stream domain model data out of a Mendix app in near real time. When objects in a tracked entity are created, updated, or deleted, the Mendix Runtime captures those changes and publishes them as events to a Kafka topic. Downstream systems, such as data warehouses or analytics pipelines, can then consume the stream without polling the Mendix database.
+
+CDC is intended for developers who need to move Mendix domain model data to external data stores, such as a data warehouse, data lake, or blob storage.
+
+## How It Works
+
+A CDC service document in Studio Pro defines which entities the Mendix Runtime should track. On deployment, each tracked entity gets its own Kafka topic. Every time a committed object change occurs (create, update, or delete) the Mendix Runtime publishes an event to the corresponding topic.
+
+Read events are triggered as snapshots during startup or when a stream changes in a way that constitutes a breaking change. Snapshots populate the new topic with the current data of the tracked entity. For more information, see the [Revisions](/refguide/published-cdc-services/#revisions) section of *Published CDC Services*.
+
+The broker that receives these events is either the [Mendix Event Broker](/appstore/services/event-broker/) or a Bring Your Own Kafka (BYOK) cluster. For BYOK configuration details, see [Mendix Event Broker](/appstore/services/event-broker/).
+
+To move the streamed data to a destination (such as Azure Blob Storage or AWS S3), you configure an [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) separately in the Event Broker Manager after deployment.
+
+## Prerequisites
+
+{{% alert color="warning" %}}
+Change Data Capture is not available for Free Apps. A licensed Mendix Cloud environment is required.
+{{% /alert %}}
+
+* A licensed Mendix Cloud environment
+* A [Mendix Event Broker](/appstore/services/event-broker/) license, or a BYOK Kafka cluster configured as described in [Mendix Event Broker](/appstore/services/event-broker/)
+* An [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) configured in the Event Broker Manager if you want to route CDC events to external storage
+
+## Setting Up Change Data Capture
+
+Set up a CDC service in Studio Pro by following the steps below:
+
+1. In Studio Pro, right-click a module in the App Explorer and choose **Add other** > **Change data capture service**.
+2. Select the entities to track and set an exposed name for each. For more information on configuring entities, see the [Entities to Track](/refguide/published-cdc-services/#entities) section of *Published CDC Services*.
+3. Deploy the app. The Runtime creates Kafka topics for each tracked entity automatically.
+4. In the [Event Broker Manager](https://broker.mendix.com/), configure an [Event Broker Bridge](/appstore/services/event-broker/#manage-mx-broker-bridge) to route CDC events to your destination.
+
+## Runtime Configuration {#runtime-configuration}
+
+CDC requires runtime settings to connect to a Kafka broker. Follow the steps below:
+
+1. In Studio Pro, open **App** > **Settings**.
+2. Open the **Configurations** tab.
+3. Select an existing configuration or click **New** to make a new one.
+4. In the configuration, open the **Custom** tab to add the settings documented in the section below. For deployed environments, set these via your environment's custom runtime settings.
+
+{{< figure src="/attachments/refguide/modeling/integration/change-data-capture/cdc-custom-configuration.png" alt="Edit Configuration dialog in Studio Pro showing the Custom tab with Kafka.BootstrapServers set to an IP address and port" >}}
+
+### Running Locally {#local-configuration}
+
+When running the app locally, only the bootstrap server address is required.
+
+| Name | Description | Default Value |
+| --- | --- | --- |
+| `Kafka.BootstrapServers` | The address of the Kafka broker, in the format `host:port`. | |
+
+### Bring Your Own Kafka (BYOK) {#byok-configuration}
+
+When connecting to a BYOK Kafka cluster, provide the bootstrap server address and credentials for authentication. The supported authentication method is `SASL/SCRAM-SHA-512`.
+
+| Name | Description | Default Value |
+| --- | --- | --- |
+| `Kafka.BootstrapServers` | The address of the Kafka broker, in the format `host:port`. | |
+| `Kafka.Username` | The username for `SASL/SCRAM-SHA-512` authentication with the Kafka cluster. | |
+| `Kafka.Password` | The password for `SASL/SCRAM-SHA-512` authentication with the Kafka cluster. | |
+| `EventBroker.Space` | The Event Broker space in which the app is placed. This defines which other applications can exchange events with this app. | `local` |
+
+### Active-Producer Election {#active-producer-election}
+
+In a horizontally-scaled deployment, multiple app instances would all publish CDC events to the same Kafka topics, producing duplicates. Active-producer election ensures that only one instance publishes at a time.
+
+When all three election settings are configured, each instance joins the same Kafka consumer group and subscribes to the election topic. Kafka assigns partition 0 of that topic to exactly one group member, and that instance becomes the active producer. All other instances stand by without publishing.
+
+If the active instance stops sending heartbeats (for example, because it crashed or was scaled down), Kafka triggers a rebalance and assigns the partition to another instance, which then starts producing. Before relinquishing the partition, the departing instance drains any in-flight events to prevent gaps.
+
+The `EventBroker.CdcProducerTransactionId` setting enables Kafka transactions on the active producer, providing exactly-once delivery guarantees during normal operation and a clean handover during failover.
+
+If any of the three election settings is missing, the Mendix Runtime logs a warning and falls back to single-instance mode, where all instances publish independently.
+
+### Message Format {#message-format}
+
+CDC events follow the [Debezium](https://debezium.io/) envelope format. Each event includes the following fields:
+
+* An `op` field indicating the operation type (`c` for create, `u` for update, `d` for delete, `r` for read/snapshot)
+* A `before` and `after` payload with the entity attribute values
+* A `source` block with metadata such as the timestamp and Mendix version
+
+Because the format is Debezium-compatible, BYOK consumers that already support Debezium (such as Kafka Connect sink connectors for databases or data warehouses) can consume CDC events without any message transformation.
+
+To parse messages, the schema for each tracked entity is available as an AsyncAPI document. You can download it from the [Event Broker Manager](https://broker.mendix.com/) on the service details page for the CDC service.
+
+For details on setting up a BYOK cluster with the Mendix Event Broker, see [Mendix Event Broker](/appstore/services/event-broker/).
+
+## Read More
+
+* [Published CDC Services](/refguide/published-cdc-services/)
+* [Mendix Event Broker](/appstore/services/event-broker/)
+* [Event Broker Bridges](/appstore/services/event-broker/#manage-mx-broker-bridge)

content/en/docs/refguide/modeling/integration/change-data-capture/published-cdc-services.md

@@ -0,0 +1,104 @@
+---
+title: "Published CDC Services"
+url: /refguide/published-cdc-services/
+weight: 10
+description: "Describes how to configure a Published CDC Service document in Studio Pro to stream entity changes to Kafka topics."
+---
+
+## Introduction
+
+{{% alert color="warning" %}} This feature is in beta. For more information, see [Release Status](/releasenotes/release-status/). {{% /alert %}}
+
+A Published CDC Service document defines which entities the Mendix Runtime tracks for object changes and publishes as Kafka events. Each tracked entity produces a stream of create, update, and delete events on its own Kafka topic.
+
+## Creating a Published CDC Service {#create}
+
+To create a published CDC service, right-click a module in the **App Explorer** and choose **Add other** > **Change data capture service**. Studio Pro adds the document to that module for logical grouping, but the service operates at app level.
+
+You can have multiple CDC service documents in an app, for example, to group entities by domain area or team ownership.
+
+## General {#general}
+
+### Service Name {#service-name}
+
+The service name identifies the CDC service within the app. The app name is included in the topic name to ensure uniqueness across apps.
+
+### Description {#description}
+
+This is an optional description for the CDC service.
+
+## Entities to Track {#entities}
+
+The **Entities to track** table lists the entities whose object changes are published to Kafka.
+
+Use the toolbar to manage tracked entities:
+
+* {{% icon name="add-filled" %}} **Add** – add an entity from the domain model
+* {{% icon name="subtract-circle" %}} **Remove** – stop tracking a selected entity
+* {{% icon name="refresh" %}} **Accept changes** – lock in the current [Revision](#revisions) numbers after reviewing modifications 
+
+
+{{< figure src="/attachments/refguide/modeling/integration/change-data-capture/published-cdc-service.png" alt="Published CDC Service document showing the Entities to track table with columns for Exposed name, Modification, Revision, and Topic" >}}
+
+### Entities
+
+The **Entities** column refers to the domain model entity being tracked. Expand the row to view and select individual attributes and associations.
+
+### Exposed Name
+
+The **Exposed name** column is the name used for this entity in the Kafka topic and event payload. It defaults to the entity name.
+
+### Modification
+
+The **Modification** column refers to the pending change state: **Added**, **Changed**, or **Removed**. This will remain blank if the entity is unchanged since the last accepted revision.
+
+### Revision
+
+The **Revision** column is the schema revision of the entity's event payload. See the [Revisions](#revisions) section below for more information.
+
+### Topic
+
+The **Topic** column is the Kafka topic name for this entity, in the format `cdc.<app-name>.<ExposedName>.<revision>.{space}`, where `{space}` is replaced at runtime by the Event Broker space name. See the [Bring Your Own Kafka (BYOK)](/refguide/change-data-capture/#byok-configuration) section of *Change Data Capture* for more information.
+
+
+## Attribute and Association Selection {#attributes}
+
+Expand an entity row to see each attribute and association with a checkbox. Uncheck an item to exclude it from the event payload. The **Exposed name** column allows you to rename individual attributes in the payload independently of their domain model names.
+
+Associations appear as a list of identifiers within the parent entity's event payload. Attributes and associations do not produce separate Kafka topics and show no **Revision** or **Topic** values of their own.
+
+## Revisions {#revisions}
+
+Each tracked entity has a **Revision** number that identifies the schema of its event payload. Downstream consumers use the revision to detect and respond to schema changes. The major revision is also embedded in the Kafka topic name, so only breaking changes result in a new topic.
+
+Studio Pro manages revisions automatically. When you modify the tracked configuration of an entity, Studio Pro marks it as **Changed** and calculates the new revision based on whether the change is breaking or non-breaking:
+
+* **Major revision** (for example, `1.0` → `2.0`) – a breaking change such as removing an attribute, renaming an entity's exposed name, or removing an entity from tracking. Consumers must be updated to use the new topic.
+* **Minor revision** (for example, `1.0` → `1.1`) – a non-breaking change such as adding a new attribute. Existing consumers can continue reading the topic without modification.
+
+{{< figure src="/attachments/refguide/modeling/integration/change-data-capture/published-cdc-service-changes.png" alt="Published CDC Service document showing entities with Changed and Removed modification states and updated revision numbers" >}}
+
+### Accepting Changes {#accepting-changes}
+
+Pending modifications are not finalized until you click **Accept changes** in the toolbar. Until you do, Studio Pro shows a consistency error on the document. You must resolve this error by accepting the changes before you can deploy the app.
+
+Accepting changes confirms the new revision numbers and clears the modification states, leaving the document in a clean state ready for deployment.
+
+{{% alert color="warning" %}}
+A major revision creates a new Kafka topic. Consumers subscribed to the previous topic will no longer receive events after deployment. Ensure downstream systems are updated before or alongside deploying a major revision change.
+{{% /alert %}}
+
+## Runtime Behavior {#runtime}
+
+The following describe how the CDC service behaves at runtime:
+
+* The CDC service runs in the system context, so no user security applies
+* Events are published for every committed object change (create, update, and delete)
+* Kafka topics are created automatically on deployment for each tracked entity
+* Events use [CloudEvents](https://cloudevents.io/) payload format, consistent with other Mendix Event Broker services
+
+## Read More
+
+* [Change Data Capture](/refguide/change-data-capture/)
+* [Mendix Event Broker](/appstore/services/event-broker/)
+* [Event Broker Bridges](/appstore/services/event-broker/#manage-mx-broker-bridge)