You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/build/kafka-consumer/index.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ By using the [cmem-plugin-kafka](https://pypi.org/project/cmem-plugin-kafka/) [P
17
17
18
18
## Installation
19
19
20
-
In order to use the Kafka Consumer workflow task, you need to extend your Corporate Memory instance with the `cmem-plugin-kafka` package.
20
+
In order to use the [Kafka Consumer](../reference/customtask/cmem_plugin_kafka-ReceiveMessages.md) or [Kafka Producer](../reference/customtask/cmem_plugin_kafka-SendMessages.md) workflow task, you need to extend your Corporate Memory instance with the `cmem-plugin-kafka` package.
21
21
This can be done by using cmemc:
22
22
23
23
```shell-session title="Installing cmem-plugin-kafka on the instance 'my-cmem'"
@@ -74,7 +74,7 @@ There are two main modes how the consumer handles received messages: either the
74
74
75
75
### Write Messages to a Dataset
76
76
77
-
In order to write the received messages to a dataset, the option **Messages Dataset** needs to be set. Only JSON and XML message formats are supported in this mode. So depending on the message format a [JSON](../../deploy-and-configure/configuration/dataintegration/plugin-reference/index.md#json) or [XML Dataset](../../deploy-and-configure/configuration/dataintegration/plugin-reference/index.md#xml) needs to be created and configured as the **Messages Dataset**.
77
+
In order to write the received messages to a dataset, the option **Messages Dataset** needs to be set. Only JSON and XML message formats are supported in this mode. So depending on the message format a [JSON](../reference/dataset/json.md) or [XML Dataset](../reference/dataset/xml.md) needs to be created and configured as the **Messages Dataset**.
78
78
79
79
{ class="bordered" }<!-- 24.1 -->
80
80
@@ -92,15 +92,15 @@ In the "message streaming mode" (**Messages Dataset** is not set) the received m
92
92
-**ts-production** — the timestamp when the message was written to the topic,
93
93
-**ts-consumption** — the timestamp when the message was consumed from the topic.
94
94
95
-
Connect the output of Kafka Consumer inside a Workflow to a tabular dataset (e.g. a [CSV Dataset](../../deploy-and-configure/configuration/dataintegration/plugin-reference/index.md#csv)) or directly to a transformation task.
95
+
Connect the output of Kafka Consumer inside a Workflow to a tabular dataset (e.g. a [CSV Dataset](../reference/dataset/csv.md)) or directly to a transformation task.
The message content is captured as plain text. In order to process complex message content, the `content` path needs to be parsed with operators such as [Parse JSON](../../deploy-and-configure/configuration/dataintegration/plugin-reference/index.md#parse-json) or [Parse XML](../../deploy-and-configure/configuration/dataintegration/plugin-reference/index.md#parse-xml) to process the message content in a transformation.
99
+
The message content is captured as plain text. In order to process complex message content, the `content` path needs to be parsed with operators such as [Parse JSON](../reference/customtask/JsonParserOperator.md) or [Parse XML](../reference/customtask/XmlParserOperator.md) to process the message content in a transformation.
Any modifications to the message set, such as filtering, can be done prior to parsing the content. One could for example remove duplicates (according to the message key) from the messages by using the [Distinct-by task](../../deploy-and-configure/configuration/dataintegration/plugin-reference/index.md).
103
+
Any modifications to the message set, such as filtering, can be done prior to parsing the content. One could for example remove duplicates (according to the message key) from the messages by using the [Distinct-by task](../reference/customtask/DistinctBy.md).
0 commit comments