You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: modules/develop/pages/transactions.adoc
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -274,18 +274,18 @@ To help avoid common pitfalls and optimize performance, consider the following w
274
274
275
275
For production environments with heavy producer usage, configure both xref:reference:properties/cluster-properties.adoc#max_concurrent_producer_ids[`max_concurrent_producer_ids`] and xref:reference:properties/cluster-properties.adoc#transactional_id_expiration_ms[`transactional_id_expiration_ms`] to prevent out-of-memory (OOM) crashes. Setting limits on producer IDs helps manage memory usage in high-throughput environments, particularly when using transactions or idempotent producers.
276
276
277
-
When setting `max_concurrent_producer_ids`, you can determine an appropriate value based on your connection patterns if you have `kafka_connections_max` configured.
277
+
If you have`kafka_connections_max` configured, you can determine an appropriate value for `max_concurrent_producer_ids` based on your connection patterns.
278
278
279
-
Lower bound: `kafka_connections_max` / `number_of_shards`, assuming each producer connects to only one shard.
280
-
Upper bound: `topic_partitions_per_shard` * `kafka_connections_max`, assuming producers connect to all shards.
279
+
* Lower bound: `kafka_connections_max` / `number_of_shards`, assuming each producer connects to only one shard.
280
+
* Upper bound: `topic_partitions_per_shard` * `kafka_connections_max`, assuming producers connect to all shards.
281
281
282
-
If `kafka_connections_max` is not configured in your environment, estimate based on your application patterns. A conservative approach is to start with 1000-5000 per shard, then monitor and adjust as needed. Applications with many partitions per producer typically require higher values, such as 10000 or more per shard.
282
+
If `kafka_connections_max` is not configured, estimate the value for `max_concurrent_producer_ids` based on your application patterns. A conservative approach is to start with 1000-5000 per shard, then monitor and adjust as needed. Applications with many partitions per producer typically require higher values, such as 10000 or more per shard.
283
283
284
-
The `transactional_id_expiration_ms` setting should be tuned based on your application's transaction patterns. Calculate this value by taking your longest expected transaction time and adding a safety buffer. For example, if transactions typically run for 30 minutes, consider setting this to 2-4 hours. Short-lived transactions can use values between 1-4 hours, while batch processing applications should match their batch interval plus buffer time. Interactive applications may benefit from shorter values to free up memory faster.
284
+
Tune `transactional_id_expiration_ms` based on your application's transaction patterns. Calculate this value by taking your longest expected transaction time and adding a safety buffer. For example, if transactions typically run for 30 minutes, consider setting this to 2-4 hours. Short-lived transactions can use values between 1-4 hours, while batch processing applications should match their batch interval plus buffer time. Interactive applications may benefit from shorter values to free up memory faster.
285
285
286
-
Client applications should minimize producer ID churn. Reuse producer instances when possible instead of creating new ones for each operation. Avoid using random transactional IDs, as some Flink configurations do, since this creates excessive producer ID churn. Instead, use consistent transactional IDs that can be resumed across application restarts.
286
+
Client applications should minimize producer ID churn. Reuse producer instances when possible, instead of creating new ones for each operation. Avoid using random transactional IDs, as some Flink configurations do, because this creates excessive producer ID churn. Instead, use consistent transactional IDs that can be resumed across application restarts.
287
287
288
-
Monitor these metrics to determine if the limit is being reached:
288
+
Monitor the following metrics to determine if the limit is being reached:
289
289
290
290
* xref:reference:internal-metrics-reference.adoc#vectorized_cluster_producer_state_manager_evicted_producers[`vectorized_cluster_producer_state_manager_evicted_producers`]: Number of evicted producers (should be 0 in steady state)
291
291
* xref:reference:internal-metrics-reference.adoc#vectorized_cluster_producer_state_manager_producer_manager_total_active_producers[`vectorized_cluster_producer_state_manager_producer_manager_total_active_producers`]: Current number of active producers per shard
Copy file name to clipboardExpand all lines: modules/reference/partials/properties/cluster-properties.adoc
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12377,9 +12377,9 @@ endif::[]
12377
12377
12378
12378
Maximum number of active producer sessions per shard. Each shard tracks producer IDs using an LRU (Least Recently Used) eviction policy. When the configured limit is exceeded, the least recently used producer IDs are evicted from the cache.
12379
12379
12380
-
If upgrading from 23.2.x to 23.3.x and experiencing `OUT_OF_SEQUENCE` errors, consider increasing this value. The configuration changed from per-partition to per-shard basis in 23.3.x.
12380
+
If you upgrade from 23.2.x to 23.3.x and encounter `OUT_OF_SEQUENCE` errors, consider increasing this value. In 23.3.x, the configuration changed from a per-partition basis to a per-shard basis.
12381
12381
12382
-
IMPORTANT: The default value is unlimited, which can lead to unbounded memory growth and out-of-memory (OOM) crashes in production environments with heavy producer usage, especially when using transactions or idempotent producers. It is strongly recommended to set a reasonable limit in production deployments.
12382
+
IMPORTANT: The default value is unlimited, which can lead to unbounded memory growth and out-of-memory (OOM) crashes in production environments with heavy producer usage, especially when using transactions or idempotent producers. Set a reasonable limit in production deployments.
0 commit comments