You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
:description: Guidelines for configuring Redpanda clusters for optimal availability.
2
+
:description: Learn about guidelines for configuring client connections in Redpanda clusters for optimal availability.
3
3
:page-categories: Management, Networking
4
+
// tag::single-source[]
4
5
5
6
Optimize the availability of your clusters by configuring and tuning properties.
6
7
@@ -10,26 +11,68 @@ A malicious Kafka client application may create many network connections to exec
10
11
11
12
The following Redpanda cluster properties limit the number of connections:
12
13
13
-
* xref:reference:cluster-properties.adoc#kafka_connections_max[`kafka_connections_max`]: Similar to Kafka's `max.connections`, this sets the maximum number of connections per broker.
14
-
* xref:reference:cluster-properties.adoc#kafka_connections_max_per_ip[`kafka_connections_max_per_ip`]: Similar to Kafka's `max.connections.per.ip`, this sets the maximum number of connections accepted per IP address by a broker.
15
-
* xref:reference:cluster-properties.adoc#kafka_connections_max_overrides[`kafka_connections_max_overrides`]: A list of IP addresses for which `kafka_connections_max_per_ip` is overridden and doesn't apply.
14
+
* xref:reference:properties/cluster-properties.adoc#kafka_connections_max_per_ip[`kafka_connections_max_per_ip`]: Similar to Kafka's `max.connections.per.ip`, this sets the maximum number of connections accepted per IP address by a broker.
15
+
* xref:reference:properties/cluster-properties.adoc#kafka_connections_max_overrides[`kafka_connections_max_overrides`]: A list of IP addresses for which `kafka_connections_max_per_ip` is overridden and doesn't apply.
16
+
ifndef::env-cloud[]
17
+
* xref:reference:properties/cluster-properties.adoc#kafka_connections_max[`kafka_connections_max`]: Similar to Kafka's `max.connections`, this sets the maximum number of connections per broker.
16
18
17
19
Redpanda also provides properties to manage the rate of connection creation:
18
20
19
-
* xref:reference:cluster-properties.adoc#kafka_connection_rate_limit[`kafka_connection_rate_limit`]: This property limits the maximum rate of connections created per second. It applies to each CPU core.
20
-
* xref:reference:cluster-properties.adoc#kafka_connection_rate_limit_overrides[`kafka_connection_rate_limit_overrides`]: A list of IP addresses for which `kafka_connection_rate_limit` is overridden and doesn't apply.
21
+
* xref:reference:properties/cluster-properties.adoc#kafka_connection_rate_limit[`kafka_connection_rate_limit`]: This property limits the maximum rate of connections created per second. It applies to each CPU core.
22
+
* xref:reference:properties/cluster-properties.adoc#kafka_connection_rate_limit_overrides[`kafka_connection_rate_limit_overrides`]: A list of IP addresses for which `kafka_connection_rate_limit` is overridden and doesn't apply.
23
+
endif::[]
21
24
22
25
[NOTE]
23
26
====
24
27
* These connection limit properties are disabled by default. You must manually enable them.
25
-
* Typically, a client opens two or three connections, so the total number of connections is not equal to the number of clients. For example, to support 100 clients, you might set your connection limit to 300.
28
+
* The total number of connections is not equal to the number of clients, because a client can open multiple connections. As a conservative estimate, for a cluster with N brokers, plan for N + 2 connections per client.
26
29
====
27
30
31
+
ifdef::env-cloud[]
32
+
=== Configure connection count limit by client IP
33
+
34
+
Use the `kafka_connections_max_per_ip` property to limit the number of connections from each client IP address.
35
+
36
+
IMPORTANT: Per-IP connection controls require Redpanda to see individual client IPs. If clients connect through private link endpoints, NAT gateways, or other shared-IP egress, the per-IP limit applies to the shared IP, affecting all clients behind it and preventing isolation of a single offending client. Similarly, multiple clients running on the same host will share the same IP address, and the limit applies collectively to all those clients.
37
+
38
+
==== Configure the limit
39
+
40
+
To configure `kafka_connections_max_per_ip` safely without disrupting legitimate clients, follow these steps:
41
+
42
+
. Set up your monitoring stack for your cluster. See xref:manage:monitor-cloud.adoc[].
43
+
44
+
. Monitor current connection patterns using the `redpanda_rpc_active_connections` metric with the `redpanda_server="kafka"` filter:
. Analyze the connection data to identify the normal range of connections for each broker during typical traffic cycles. For example, in the following Grafana screenshot, the normal range is around 200-300 connections:
51
+
+
52
+
image::shared:monitor_connections.png[Range of active connections over time]
53
+
54
+
. Set the `kafka_connections_max_per_ip` value based on your analysis. Use the upper bound of normal connections from step 3, or use a lower value if you know how many connections per client IP are being opened.
55
+
56
+
. Continue monitoring the connection metrics after applying the limit to ensure that legitimate clients are not affected and that the problematic client is properly controlled.
57
+
58
+
==== Limitations
59
+
60
+
* Decreasing the limit does not terminate any currently open Kafka API connections.
61
+
* This limit does not apply to Kafka HTTP Proxy connections.
62
+
* Clients behind NAT gateways or private links share the same IP address as seen by Redpanda brokers.
63
+
* The limit may negatively affect tail latencies across all client connections.
64
+
* All clients behind the shared IP are collectively subject to the single `kafka_connections_max_per_ip` limit.
65
+
* Connection rejections occur randomly among clients when the limit is reached. For example, suppose `kafka_connections_max_per_ip` is set to 100, but clients behind a NAT gateway collectively need 150 connections. When the limit is reached, clients can make only some of the connections while others get rejected, leaving the client in a not-working state.
66
+
* Redpanda may modify this property during internal operations.
67
+
* Availability incidents caused by misconfiguring this feature are excluded from the Redpanda Cloud SLA.
68
+
69
+
endif::env-cloud[]
70
+
28
71
== Configure client reconnections
29
72
30
73
You can configure the Kafka client backoff and retry properties to change the default behavior of the clients to suit your failure requirements.
31
74
32
-
The following Kafka properties let you manage client reconnections:
75
+
Set the following Kafka client properties on your application's producer or consumer to manage client reconnections:
33
76
34
77
* `reconnect.backoff.ms`: Amount of time to wait before attempting to reconnect to the broker. The default is 50 milliseconds.
35
78
* `reconnect.backoff.max.ms`: Maximum amount of time in milliseconds to wait when reconnecting to a broker. The backoff increases exponentially for each consecutive connection failure, up to this maximum. The default is 1000 milliseconds (1 second).
@@ -42,21 +85,4 @@ Additionally, you can use Kafka properties to control message retry behavior. De
42
85
43
86
See also: xref:develop:produce-data/configure-producers.adoc[Configure Producers]
44
87
45
-
== Prevent crash loops
46
-
47
-
A Redpanda broker may create log segments at startup. If a broker crashes after startup, and if it gets stuck in a crash loop, it could produce progressively more stored state that uses more disk space and takes more time for each restart to process.
48
-
49
-
To prevent infinite crash loops, the Redpanda broker property xref:reference:node-properties.adoc#crash_loop_limit[`crash_loop_limit`] sets an upper limit on the number of consecutive crashes that can happen within one hour of each other. After it reaches the limit, a broker cannot restart until its internal consecutive crash counter is reset to zero by one of the following conditions:
50
-
51
-
* The `redpanda.yaml` configuration file is updated.
52
-
* The `startup_log` file in the broker's xref:reference:node-properties.adoc#data_directory[data_directory] is manually deleted.
53
-
* One hour has elapsed since the last crash.
54
-
* The broker is properly shut down. (This is not possible after `crash_loop_limit` has been reached and the broker cannot be restarted.)
55
-
56
-
[NOTE]
57
-
====
58
-
* The `crash_loop_limit` property is disabled by default. You must manually enable it by setting it to a non-zero value.
59
-
* If the limit is less than two, the broker is blocked from restarting after every crash, until one of the reset conditions is met.
60
-
====
61
-
62
-
To facilitate debugging in environments where a broker is stuck in a crash loop, set the xref:reference:properties/broker-properties.adoc#crash_loop_sleep_sec[`crash_loop_sleep_sec` configuration]. This setting determines how long the broker sleeps before terminating the process after reaching the crash loop limit. The window during which the broker remains available allows you to troubleshoot the issue. This setting is most useful when xref:troubleshoot:errors-solutions/k-resolve-errors.adoc[troubleshooting in Kubernetes environments].
Flag to require authorization for Kafka connections. If `null`, the property is disabled, and authorization is instead enabled by <<enable_sasl,`enable_sasl`>>.
0 commit comments