Skip to content
This repository was archived by the owner on Jan 24, 2024. It is now read-only.

Commit 5398f77

Browse files
authored
[Doc] - Add entryFormat description and performance test class (#1694)
Fixes #1693 ### Motivation - Add `entryFormat` description and performance test class. This description will help the user understands each value of `entryFormat` in the KoP. ### Modifications - Add `entryFormat` description and performance test class ### Verifying this change - Add `entryFormat` description and performance test class in the `~/docs/configuration.md#performance`
1 parent 786d0a2 commit 5398f77

1 file changed

Lines changed: 10 additions & 0 deletions

File tree

docs/configuration.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,16 @@ This section lists configurations that may affect the performance.
7070

7171
### Choose the proper `entryFormat`
7272

73+
This table lists `entryFormat` values that are supported in KoP.
74+
75+
| Name | Description |
76+
|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
77+
| pulsar | `pulsar` is the default `entryFormat` in KoP. It is used to encode or decode formats between the Kafka message and the Pulsar message. Therefore, the performance is the worst. The benefit is that both the Kafka client and the Pulsar client consumers can consume the messages from the Pulsar cluster. |
78+
| kafka | When you set the `entryFormat` option to `kafka`, KoP does not encode or decode Kafka messages. The messages will be directly stored in the bookie cluster in entries format, and the Pulsar client can not parse these messages. Therefore, the performance is the best. |
79+
| mixed_kafka | The `mixed_kafka` format works similarly to the `kafka` format. You can set this option for some non-official Kafka clients for encoding or decoding Kafka messages. The performance is medium. |
80+
81+
You can run the `io.streamnative.pulsar.handlers.kop.format.EncodePerformanceTest.java` to get the performance result among the above formats.
82+
7383
Generally, if you don't have Pulsar consumers that consume messages from Kafka producers, `kafka` format is perferred because **it has much higher performance** when Kafka consumers interact with Kafka producers.
7484

7585
However, some non-official Kafka clients might not work for `kafka` format. For example, old [Golang Sarama](https://github.com/Shopify/sarama) client didn't assign relative offsets in compressed message sets before [Shopify/sarama #1002](https://github.com/Shopify/sarama/pull/1002). In this case, the broker has to assign relative offsets and then do recompression. Since this behavior leads to some performance loss, KoP adds the `mixed_kafka` format to perform the conversion. The `mixed_kafka` format should be chosen when you have such an old Kafka client. Like `kafka` format, in this case, Pulsar consumers still cannot consume messages from Kafka producers.

0 commit comments

Comments
 (0)