Skip to content

Commit 1f51d30

Browse files
micheleRPclaude
andcommitted
DOC-2130 Address CodeRabbit feedback
* Shared `max_in_flight_batched` description now says "message batches" instead of "messages" — `max_in_flight` controls parallel batches, not parallel messages, matching the upstream benthos description correction (commit 77a2bba44). * Drop the dead `max_in_flight` override on `zmq4`. Although the connector source registers via `MustRegisterBatchOutput`, the generated configspec does not expose `max_in_flight`, so the override would never render. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent f62df66 commit 1f51d30

32 files changed

Lines changed: 33 additions & 41 deletions

docs-data/overrides.json

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@
8888
"description": "The Elasticsearch index where messages are published. This field supports xref:configuration:interpolation.adoc#bloblang-queries[interpolation functions]."
8989
},
9090
"max_in_flight_batched": {
91-
"description": "The maximum number of messages to have in flight at a given time. Increase this to improve throughput.\n\nNOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance."
91+
"description": "The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.\n\nNOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance."
9292
},
9393
"elasticsearch_retry_on_conflict": {
9494
"description": "The number of times to retry an update operation when a version conflict occurs."
@@ -6288,15 +6288,7 @@
62886288
},
62896289
{
62906290
"name": "zmq4",
6291-
"summary": "Writes messages to a ZeroMQ socket.",
6292-
"config": {
6293-
"children": [
6294-
{
6295-
"name": "max_in_flight",
6296-
"$ref": "#/definitions/max_in_flight_batched"
6297-
}
6298-
]
6299-
}
6291+
"summary": "Writes messages to a ZeroMQ socket."
63006292
},
63016293
{
63026294
"name": "arc",

modules/components/partials/fields/outputs/arc.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -392,7 +392,7 @@ Size in bytes of the per-connection write buffer.
392392

393393
=== `max_in_flight`
394394

395-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
395+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
396396

397397
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
398398

modules/components/partials/fields/outputs/aws_dynamodb.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ json_map_columns:
225225

226226
=== `max_in_flight`
227227

228-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
228+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
229229

230230
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
231231

modules/components/partials/fields/outputs/aws_kinesis.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,7 +211,7 @@ This field supports xref:configuration:interpolation.adoc#bloblang-queries[inter
211211

212212
=== `max_in_flight`
213213

214-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
214+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
215215

216216
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
217217

modules/components/partials/fields/outputs/aws_kinesis_firehose.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,7 @@ Allows you to specify a custom endpoint for the AWS API.
204204

205205
=== `max_in_flight`
206206

207-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
207+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
208208

209209
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
210210

modules/components/partials/fields/outputs/aws_s3.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,7 @@ An optional server-side encryption key.
249249

250250
=== `max_in_flight`
251251

252-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
252+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
253253

254254
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
255255

modules/components/partials/fields/outputs/aws_sqs.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,7 +211,7 @@ Allows you to specify a custom endpoint for the AWS API.
211211

212212
=== `max_in_flight`
213213

214-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
214+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
215215

216216
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
217217

modules/components/partials/fields/outputs/azure_cosmosdb.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ item_id: ${! json("id") }
199199

200200
=== `max_in_flight`
201201

202-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
202+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
203203

204204
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
205205

modules/components/partials/fields/outputs/azure_queue_storage.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ processors:
114114

115115
=== `max_in_flight`
116116

117-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
117+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
118118

119119
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
120120

modules/components/partials/fields/outputs/azure_table_storage.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ processors:
114114

115115
=== `max_in_flight`
116116

117-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
117+
The maximum number of message batches to have in flight at a given time. Increase this to improve throughput.
118118

119119
NOTE: Set `max_in_flight: 1` when a `batching` block with `processors` is configured on this output. If a batching processor (for example `parquet_encode`, `archive`, or `compress`) errors at runtime, Redpanda Connect can write raw, unprocessed messages to the output instead of the encoded batch, producing corrupt downstream data (for example, a `.parquet` file containing raw JSON). Because the batching processor is single threaded, higher values do not improve performance.
120120

0 commit comments

Comments
 (0)