Skip to content

Commit 4d5f898

Browse files
committed
Implement suggestion from doc review
1 parent 2e50aa2 commit 4d5f898

4 files changed

Lines changed: 23 additions & 15 deletions

File tree

modules/manage/pages/iceberg/specify-iceberg-schema.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ CREATE TABLE ClickEvent (
162162

163163
As you produce records to the topic, the data also becomes available in object storage for Iceberg-compatible clients to consume. You can use the same analytical tools to xref:manage:iceberg/query-iceberg-topics.adoc[read the Iceberg topic data] in a data lake as you would for a relational database.
164164

165-
If Redpanda fails to translate the record to the columnar format as defined by the schema, it writes the record to a dead-letter queue (DLQ) table. See xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue] for more information.
165+
If Redpanda fails to translate the record to the columnar format as defined by the schema, it writes the record to a dead-letter queue (DLQ) table. See xref:manage:iceberg/about-iceberg-topics.adoc#troubleshoot-errors[Troubleshoot errors] for more information.
166166

167167
=== Schema types translation
168168

modules/manage/partials/iceberg/about-iceberg-topics.adoc

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -312,24 +312,21 @@ Querying the Iceberg table for `demo-topic` includes the new column `ts`:
312312
+---------+--------------+--------------------------+
313313
----
314314

315-
== Manage dead-letter queue
315+
== Troubleshoot errors
316316

317-
Errors may occur when translating records in the `value_schema_id_prefix` or `value_schema_latest` modes to the Iceberg table format; for example, if you do not use the Schema Registry wire format with the magic byte, if the schema ID in the record is not found in the Schema Registry, or if a schema data type cannot be translated to an Iceberg type.
317+
If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda by default writes the record to a separate dead-letter queue (DLQ) Iceberg table named `<topic-name>~dlq`. The following can cause errors to occur when translating records in the `value_schema_id_prefix` and `value_schema_latest` modes to the Iceberg table format:
318318

319-
ifndef::env-cloud[]
320-
If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda writes the record to a separate dead-letter queue (DLQ) Iceberg table named `<topic-name>~dlq`. To disable the default behavior for a topic and drop the record, set the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-invalid-record-action[`redpanda.iceberg.invalid.record.action`] topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property.
321-
endif::[]
322-
ifdef::env-cloud[]
323-
If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda writes the record to a separate dead-letter queue (DLQ) Iceberg table named `<topic-name>~dlq`. To disable the default behavior for a topic and drop the record, set the `redpanda.iceberg.invalid.record.action` topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property.
324-
endif::[]
319+
- Redpanda cannot find the embedded schema ID in the Schema Registry.
320+
- Redpanda fails to translate one or more schema data types to an Iceberg type.
321+
- In `value_schema_id_prefix` mode, you do not use the Schema Registry wire format with the magic byte.
325322

326323
The DLQ table itself uses the `key_value` schema, consisting of two columns: the record metadata including the key, and a binary column for the record's value.
327324

328-
You can inspect the DLQ table for records that failed to write to the Iceberg table, and you can take further action on these records, such as transforming and reprocessing them, or debugging issues that occurred upstream.
329-
330325
NOTE: Topic property misconfiguration, such as xref:manage:iceberg/specify-iceberg-schema.adoc#override-value-schema-latest-default[overriding the default behavior of `value_schema_latest` mode] but not specifying the fully qualified Protobuf message name, does not cause records to be written to the DLQ table. Instead, Redpanda pauses the topic data translation to the Iceberg table until you fix the misconfiguration.
331326

332-
=== Reprocess DLQ records
327+
=== Inspect DLQ table
328+
329+
You can inspect the DLQ table for records that failed to write to the Iceberg table, and you can take further action on these records, such as transforming and reprocessing them, or debugging issues that occurred upstream.
333330

334331
The following example produces a record to a topic named `ClickEvent` and does not use the Schema Registry wire format that includes the magic byte and schema ID:
335332

@@ -360,7 +357,9 @@ FROM <catalog-name>."ClickEvent~dlq"; -- Fully qualified table name
360357
+-------------------------------------------------+
361358
----
362359

363-
The data is in binary format, and the first byte is not `0x00`, indicating that it was not produced with a schema.
360+
The data is in binary format, and the first byte is not `0x00`, indicating that it was not produced with a schema.
361+
362+
=== Reprocess DLQ records
364363

365364
You can apply a transformation and reprocess the record in your data lakehouse to the original Iceberg table. In this case, you have a JSON value represented as a UTF-8 binary. Depending on your query engine, you might need to decode the binary value first before extracting the JSON fields. Some engines may automatically decode the binary value for you:
366365

@@ -389,6 +388,15 @@ FROM (
389388

390389
You can now insert the transformed record back into the main Iceberg table. Redpanda recommends employing a strategy for exactly-once processing to avoid duplicates when reprocessing records.
391390

391+
=== Drop invalid records
392+
393+
ifndef::env-cloud[]
394+
To disable the default behavior and drop an invalid record, set the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-invalid-record-action[`redpanda.iceberg.invalid.record.action`] topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property.
395+
endif::[]
396+
ifdef::env-cloud[]
397+
To disable the default behavior and drop an invalid record, set the `redpanda.iceberg.invalid.record.action` topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property.
398+
endif::[]
399+
392400
== Performance considerations
393401

394402
When you enable Iceberg for any substantial workload and start translating topic data to the Iceberg format, you may see most of your cluster's CPU utilization increase. If this additional workload overwhelms the brokers and causes the Iceberg table lag to exceed the configured target lag, Redpanda automatically applies backpressure to producers to prevent Iceberg tables from lagging further. This ensures that Iceberg tables keep up with the volume of incoming data, but sacrifices ingress throughput of the cluster.

modules/reference/pages/properties/cluster-properties.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2059,7 +2059,7 @@ endif::[]
20592059

20602060
**Related topics**:
20612061

2062-
- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue]
2062+
- xref:manage:iceberg/about-iceberg-topics.adoc#troubleshoot-errors[Troubleshoot errors]
20632063

20642064
---
20652065

modules/reference/pages/properties/topic-properties.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -611,7 +611,7 @@ Whether to write invalid records to a dead-letter queue (DLQ).
611611

612612
**Related topics**:
613613

614-
- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue]
614+
- xref:manage:iceberg/about-iceberg-topics.adoc#troubleshoot-errors[Troubleshoot errors]
615615

616616
---
617617

0 commit comments

Comments
 (0)