Skip to content

Commit 5a571e2

Browse files
micheleRPkbatuigas
andauthored
DOC-1587 Conditionalize Manage Throughput for Cloud (#1309)
Co-authored-by: Kat Batuigas <36839689+kbatuigas@users.noreply.github.com>
1 parent b98bf1b commit 5a571e2

5 files changed

Lines changed: 47 additions & 13 deletions

File tree

modules/manage/pages/cluster-maintenance/manage-throughput.adoc

Lines changed: 31 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,32 @@
11
= Manage Throughput
22
:description: Manage the throughput of Kafka traffic with configurable properties.
33
:page-categories: Management, Networking
4+
// tag::single-source[]
45

5-
Redpanda supports applying throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. The purpose of this is to prevent unbounded network and disk usage of the broker by clients. Broker-wide limits apply to all clients connected to the broker and restrict total traffic on the broker. Client limits apply to a set of clients defined by their `client_id` and help prevent a set of clients from starving other clients using the same broker.
6+
Redpanda supports throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. This helps prevent clients from causing unbounded network and disk usage on brokers.
7+
8+
* Broker-wide limits apply to all clients connected to the broker and restrict total traffic on the broker.
9+
* Client limits apply to a set of clients defined by their `client_id` and help prevent a set of clients from starving other clients using the same broker. You can manage client quotas with xref:reference:rpk/rpk-cluster/rpk-cluster-quotas.adoc[`rpk cluster quotas`] or with the Kafka API. When no quotas apply, the client has unlimited throughput.
10+
11+
ifdef::env-cloud[]
12+
NOTE: Throughput throttling is supported for BYOC and Dedicated clusters.
13+
endif::[]
614

715
== Throughput throttling enforcement
816

917
NOTE: As of v24.2, Redpanda enforces all throughput limits per broker, including client throughput.
1018

11-
Throughput limits are enforced by applying backpressure to clients. When a connection is in breach of the throughput limit, the throttler advises the client about the delay (throttle time) that would bring the rate back to the allowed level. Redpanda starts by adding a `throttle_time_ms` field to responses. If that isn't honored, delays are inserted on the connection's next read operation. The throttling delay may not exceed the limit set by xref:reference:tunable-properties.adoc#max_kafka_throttle_delay_ms[`max_kafka_throttle_delay_ms`].
19+
Throughput limits are enforced by applying backpressure to clients. When a connection is in breach of the throughput limit, the throttler advises the client about the delay (throttle time) that would bring the rate back to the allowed level. Redpanda starts by adding a `throttle_time_ms` field to responses. If that isn't honored, delays are inserted on the connection's next read operation.
20+
21+
ifndef::env-cloud[]
22+
The throttling delay may not exceed the limit set by xref:reference:tunable-properties.adoc#max_kafka_throttle_delay_ms[`max_kafka_throttle_delay_ms`].
23+
endif::[]
1224

1325
== Broker-wide throughput limits
1426

1527
Broker-wide throughput limits account for all Kafka API traffic going into or out of the broker, as data is produced to or consumed from a topic. The limit values represent the allowed rate of data in bytes per second passing through in each direction. Redpanda also provides administrators the ability to exclude clients from throughput throttling and to fine-tune which Kafka request types are subject to throttling limits.
1628

29+
ifndef::env-cloud[]
1730
=== Broker-wide throughput limit properties
1831

1932
The properties for broker-wide throughput quota balancing are configured at the cluster level, for all brokers in a cluster:
@@ -40,8 +53,9 @@ The properties for broker-wide throughput quota balancing are configured at the
4053

4154
[NOTE]
4255
====
43-
* By default, both `kafka_throughput_limit_node_in_bps` and `kafka_throughput_limit_node_out_bps` are disabled, and no throughput limits are applied. You must manually set them to enable throughput throttling.
56+
By default, both `kafka_throughput_limit_node_in_bps` and `kafka_throughput_limit_node_out_bps` are disabled, and no throughput limits are applied. You must manually set them to enable throughput throttling.
4457
====
58+
endif::[]
4559

4660
== Client throughput limits
4761

@@ -71,9 +85,13 @@ It is possible to create conflicting quotas if you configure the same quotas thr
7185

7286
. Quota configured through the Kafka API for an exact match on `client_id`
7387
. Quota configured through the Kafka API for a prefix match on `client_id`
74-
. Quota configured through cluster configuration properties (`kafka_client_group_byte_rate_quota`, `kafka_client_group_fetch_byte_rate_quota`, xref:upgrade:deprecated/index.adoc[deprecated starting in v24.2]) for a prefix match on `client_id`
88+
ifndef::env-cloud[]
89+
. Quota configured through cluster configuration properties (`kafka_client_group_byte_rate_quota`, `kafka_client_group_fetch_byte_rate_quota`-deprecated in v24.2) for a prefix match on `client_id`
90+
endif::[]
7591
. Default quota configured through the Kafka API on `client_id`
76-
. Default quota configured through cluster configuration properties (`target_quota_byte_rate`, `target_fetch_quota_byte_rate`, `kafka_admin_topic_api_rate`, xref:upgrade:deprecated/index.adoc[deprecated starting in v24.2]) on `client_id`
92+
ifndef::env-cloud[]
93+
. Default quota configured through cluster configuration properties (`target_quota_byte_rate`, `target_fetch_quota_byte_rate`, `kafka_admin_topic_api_rate`-deprecated in v24.2) on `client_id`
94+
endif::[]
7795

7896
Redpanda recommends <<migrate,migrating>> over from cluster configuration-managed quotas to Kafka-compatible quotas. You can re-create the configuration-based quotas with `rpk`, and then remove the cluster configurations.
7997

@@ -94,6 +112,7 @@ client-id=consumer-1
94112
producer_byte_rate=140000
95113
----
96114

115+
97116
To set a throughput quota for a single client, use the xref:reference:rpk/rpk-cluster/rpk-cluster-quotas-alter.adoc[`rpk cluster quotas alter`] command.
98117

99118
[,bash]
@@ -118,6 +137,7 @@ rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id-prefix
118137

119138
NOTE: A client group specified with `client-id-prefix` is not the equivalent of a Kafka consumer group. It is used only to match requests based on the `client_id` prefix. The `client_id` field is typically a configurable property when you create a client with Kafka libraries.
120139

140+
121141
=== Default client throughput limit
122142

123143
You can apply default throughput limits to clients. Redpanda applies the default limits if no quotas are configured for a specific `client_id` or prefix.
@@ -222,14 +242,18 @@ You can also use Redpanda Console to view enforced limits. In the menu, go to **
222242

223243
=== Monitor client throughput
224244

225-
The following metrics are available on both the `/public_metrics` and `/metrics` endpoints to provide insights into client throughput quota usage:
245+
The following metrics provide insights into client throughput quota usage:
226246

227247
* Client quota throughput per rule and quota type:
228248
** `/public_metrics` - xref:reference:public-metrics-reference.adoc#redpanda_kafka_quotas_client_quota_throughput[`redpanda_kafka_quotas_client_quota_throughput`]
249+
ifndef::env-cloud[]
229250
** `/metrics` - xref:reference:internal-metrics-reference.adoc#vectorized_kafka_quotas_client_quota_throughput[`vectorized_kafka_quotas_client_quota_throughput`]
251+
endif::[]
230252
* Client quota throttling delay per rule and quota type, in seconds:
231253
** `/public_metrics` - xref:reference:public-metrics-reference.adoc#redpanda_kafka_quotas_client_quota_throttle_time[`redpanda_kafka_quotas_client_quota_throttle_time`]
254+
ifndef::env-cloud[]
232255
** `/metrics` - xref:reference:internal-metrics-reference.adoc#vectorized_kafka_quotas_client_quota_throttle_time[`vectorized_kafka_quotas_client_quota_throttle_time`]
256+
endif::[]
233257

234258
The `kafka_quotas` logger provides details at the trace level on client quota throttling:
235259

@@ -242,3 +266,4 @@ TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - quota_manager.cc:36
242266
TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - connection_context.cc:605 - [127.0.0.1:58636] throttle request:{snc:0, client:184}, enforce:{snc:-14359, client:-14359}, key:0, request_size:1316
243267
----
244268

269+
// end::single-source[]

modules/reference/pages/rpk/rpk-cluster/rpk-cluster-quotas-alter.adoc

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
= rpk cluster quotas alter
2-
:description: rpk cluster quotas alter
2+
// tag::single-source[]
33

44
Add or delete a client quota.
55

@@ -74,4 +74,6 @@ Remove quota (producer_byte_rate) from client ID `foo`:
7474
[,bash]
7575
----
7676
rpk cluster quotas alter --delete producer_byte_rate --name client-id=<foo>
77-
----
77+
----
78+
79+
// end::single-source[]

modules/reference/pages/rpk/rpk-cluster/rpk-cluster-quotas-describe.adoc

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
= rpk cluster quotas describe
2-
:description: rpk cluster quotas describe
2+
// tag::single-source[]
33

44
Describe client quotas.
55

@@ -64,4 +64,6 @@ Describe client quotas for a given client ID prefix `<bar>.`:
6464
[,bash]
6565
----
6666
rpk cluster quotas describe --name client-id=<bar>.
67-
----
67+
----
68+
69+
// end::single-source[]

modules/reference/pages/rpk/rpk-cluster/rpk-cluster-quotas-import.adoc

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
= rpk cluster quotas import
2+
// tag::single-source[]
23

34
Use this command to import client quotas in the format produced by `rpk cluster quotas describe --format json/yaml`.
45

@@ -43,7 +44,7 @@ JSON::
4344
----
4445
======
4546

46-
Use the '--no-confirm' flag if you wish to avoid the confirmation prompt.
47+
Use the `--no-confirm` flag to avoid the confirmation prompt.
4748

4849
== Usage
4950

@@ -150,3 +151,5 @@ quotas:
150151
value: "140000"
151152
'
152153
----
154+
155+
// end::single-source[]

modules/reference/pages/rpk/rpk-cluster/rpk-cluster-quotas.adoc

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
= rpk cluster quotas
2-
:description: rpk cluster quotas
2+
// tag::single-source[]
33

44
Manage Redpanda client quotas.
55

@@ -34,4 +34,6 @@ quotas, quota
3434
|--profile |string |Profile to use. See xref:reference:rpk/rpk-profile.adoc[`rpk profile`] for more details.
3535

3636
|-v, --verbose |- |Enable verbose logging.
37-
|===
37+
|===
38+
39+
// end::single-source[]

0 commit comments

Comments
 (0)