From 39527cd8dae419ca4be19278ba2ebefdcacf15e1 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 11 Jul 2025 16:55:05 -0700 Subject: [PATCH 01/20] Rename to Specify Iceberg Schema for better discoverability --- modules/ROOT/nav.adoc | 2 +- .../{choose-iceberg-mode.adoc => specify-iceberg-schema.adoc} | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) rename modules/manage/pages/iceberg/{choose-iceberg-mode.adoc => specify-iceberg-schema.adoc} (99%) diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 1b6f62f4ab..f6ba2c317a 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -170,7 +170,7 @@ *** xref:manage:whole-cluster-restore.adoc[Whole Cluster Restore] ** xref:manage:iceberg/index.adoc[Iceberg] *** xref:manage:iceberg/about-iceberg-topics.adoc[About Iceberg Topics] -*** xref:manage:iceberg/choose-iceberg-mode.adoc[Choose Iceberg Mode] +*** xref:manage:iceberg/specify-iceberg-schema.adoc[Specify Iceberg Schema] *** xref:manage:iceberg/use-iceberg-catalogs.adoc[Use Iceberg Catalogs] *** xref:manage:iceberg/query-iceberg-topics.adoc[Query Iceberg Topics] *** xref:manage:iceberg/iceberg-topics-databricks-unity.adoc[Query Iceberg Topics with Databricks Unity Catalog] diff --git a/modules/manage/pages/iceberg/choose-iceberg-mode.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc similarity index 99% rename from modules/manage/pages/iceberg/choose-iceberg-mode.adoc rename to modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 0c7e8888d6..8745296350 100644 --- a/modules/manage/pages/iceberg/choose-iceberg-mode.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -1,6 +1,7 @@ -= Choose an Iceberg Mode += Specify Iceberg Schema :description: Learn about supported Iceberg modes and how you can integrate schemas with Iceberg topics. :page-categories: Iceberg, Tiered Storage, Management, High Availability, Data Replication, Integration +:page-aliases: manage:iceberg/choose-iceberg-mode.adoc :schema-id-val-doc: manage:schema-reg/schema-id-validation.adoc // tag::single-source[] From d4fe57bed505da8c8ece81b6eb6453bb51cc395e Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 11 Jul 2025 21:21:53 -0700 Subject: [PATCH 02/20] JSON Schema support for Iceberg topics --- .../pages/iceberg/specify-iceberg-schema.adoc | 59 +++++++++++++++++-- .../iceberg/about-iceberg-topics.adoc | 2 +- 2 files changed, 56 insertions(+), 5 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 8745296350..0d99c1e310 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -12,9 +12,7 @@ include::shared:partial$enterprise-license.adoc[] ==== endif::[] -In xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Iceberg-enabled clusters], the `redpanda.iceberg.mode` topic property determines how Redpanda maps topic data to the Iceberg table structure. You can have the generated Iceberg table match the structure of an Avro or Protobuf schema in the Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table. - -NOTE: The JSON Schema format is not supported for Iceberg topics. If your topic data is in JSON, use the `key_value` mode. +In xref:manage:iceberg/about-iceberg-topics.adoc#enable-iceberg-integration[Iceberg-enabled clusters], the `redpanda.iceberg.mode` topic property determines how Redpanda maps topic data to the Iceberg table structure. You can have the generated Iceberg table match the structure of a schema in the Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table. == Supported Iceberg modes @@ -225,7 +223,7 @@ Protobuf:: There are some cases where the Protobuf type does not map directly to an Iceberg type and Redpanda applies the following transformations: -* Repeated values are translated into Iceberg `array` types. +* Repeated values are translated into Iceberg `list` types. * Enums are translated into Iceberg `int` types based on the integer value of the enumerated type. * `uint32` and `fixed32` are translated into Iceberg `long` types as that is the existing semantic for unsigned 32-bit values in Iceberg. * `uint64` and `fixed64` values are translated into their Base-10 string representation. @@ -233,6 +231,59 @@ There are some cases where the Protobuf type does not map directly to an Iceberg Recursive types are not supported. -- + +JSON Schema:: ++ +-- +Requirements: + +- You must use a JSON Schema that constrains JSON documents to a strict type in order for Redpanda to translate to Iceberg. +- You must declare the JSON Schema dialect using the `$schema` keyword, for example `"$schema": "http://json-schema.org/draft-07/schema#"`. +- Only JSON Schema Draft-07 is supported. + + +|=== +| JSON type | Iceberg type | Notes + +| array +| list +| The keywords `items` and `additionalItems` are used to define element types. + +| boolean +| boolean +| + +| null +| null +| + +| number +| +| + +| integer +| int +| + +| string +| string +| + +| object +| +| + +|=== + +The following are not supported for JSON Schema: + +* Relative and absolute (including external) references using `$ref` and `$dynamicRef` keywords +* The `default` keyword +* Conditional typing (`if`, `then`, `else`, `dependent` keywords) +* Boolean JSON Schema combinations (`allOf`, `anyOf`, o`neOf` keywords) +* Dynamic object members (`patternProperties`, `additionalProperties` (except when it is set to `false`) keywords) +-- + ====== // end::single-source[] diff --git a/modules/manage/partials/iceberg/about-iceberg-topics.adoc b/modules/manage/partials/iceberg/about-iceberg-topics.adoc index cd4ef3aefd..147f916f77 100644 --- a/modules/manage/partials/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/about-iceberg-topics.adoc @@ -52,7 +52,7 @@ endif::[] * It is not possible to append topic data to an existing Iceberg table that is not created by Redpanda. * If you enable the Iceberg integration on an existing Redpanda topic, Redpanda does not backfill the generated Iceberg table with topic data. -* JSON schemas are not currently supported. If the topic data is in JSON, use the `key_value` mode to store the JSON in Iceberg, which then can be parsed by most query engines. +* JSON schemas are supported starting with Redpanda version 25.2. == Enable Iceberg integration From 65c90c2f1594810432005e0d3b92efd1e40565a8 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 11 Jul 2025 21:37:58 -0700 Subject: [PATCH 03/20] Apply suggestions from automated review --- modules/manage/pages/iceberg/specify-iceberg-schema.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 0d99c1e310..923c346e39 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -174,8 +174,8 @@ Avro:: | string | string | record | struct | array | list -| maps | map -| fixed | fixed* +| map | list +| fixed | fixed | decimal | decimal | uuid | uuid* | date | date @@ -280,7 +280,7 @@ The following are not supported for JSON Schema: * Relative and absolute (including external) references using `$ref` and `$dynamicRef` keywords * The `default` keyword * Conditional typing (`if`, `then`, `else`, `dependent` keywords) -* Boolean JSON Schema combinations (`allOf`, `anyOf`, o`neOf` keywords) +* Boolean JSON Schema combinations (`allOf`, `anyOf`, `oneOf` keywords) * Dynamic object members (`patternProperties`, `additionalProperties` (except when it is set to `false`) keywords) -- From 802fe8a7b78605796e66cd8071cf6a4b4db05aef Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 11 Jul 2025 22:11:55 -0700 Subject: [PATCH 04/20] Deploy preview --- local-antora-playbook.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index c0a904527a..67f8695e6e 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -17,7 +17,7 @@ content: - url: https://github.com/redpanda-data/docs branches: [main, v/*, api, shared, site-search,'!v-end-of-life/*'] - url: https://github.com/redpanda-data/cloud-docs - branches: 'main' + branches: 'DOC-1379-cloud-iceberg-json-schema' - url: https://github.com/redpanda-data/redpanda-labs branches: main start_paths: [docs,'*/docs'] From e006064816b466597c41224c9d6732515f8caa5c Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Wed, 16 Jul 2025 13:07:27 -0700 Subject: [PATCH 05/20] Revert change to Avro mapping --- modules/manage/pages/iceberg/specify-iceberg-schema.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 923c346e39..228a6acced 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -174,8 +174,8 @@ Avro:: | string | string | record | struct | array | list -| map | list -| fixed | fixed +| map | map +| fixed | fixed* | decimal | decimal | uuid | uuid* | date | date From dccc07e976d68709074f66aea79af694659b0ddc Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Thu, 17 Jul 2025 22:43:24 -0700 Subject: [PATCH 06/20] JSON schema translation and value_schema_latest mode details --- .../pages/iceberg/specify-iceberg-schema.adoc | 47 ++++++++++++++----- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 228a6acced..1653a38ddc 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -36,7 +36,9 @@ In the xref:manage:schema-reg/schema-reg-overview.adoc#wire-format[Schema Regist === value_schema_latest -Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. You must register a schema in the Schema Registry. Unlike the `value_schema_id_prefix` mode, `value_schema_latest` does not require that producers use the wire format. +Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. You must register a schema in the Schema Registry. For Protobuf, you must use the fully qualified schema name, which includes the package name, for example `com.example.manufacturing.SensorData`. + +Producers cannot use the wire format in `value_schema_latest` mode. The latest schema is cached periodically. The cache period is defined by the cluster property `iceberg_latest_schema_cache_ttl_ms` (default: 5 minutes). @@ -129,7 +131,7 @@ CREATE TABLE ClickEvent ( ) ---- -Use `key_value` mode if the topic data is in JSON or if you are able to use the Iceberg data in its semi-structured format. +Use `key_value` mode if you're able to use the Iceberg data in its semi-structured format. The `value_schema_id_prefix` and `value_schema_latest` modes can use the schema to translate to the following table format: @@ -237,41 +239,60 @@ JSON Schema:: -- Requirements: -- You must use a JSON Schema that constrains JSON documents to a strict type in order for Redpanda to translate to Iceberg. -- You must declare the JSON Schema dialect using the `$schema` keyword, for example `"$schema": "http://json-schema.org/draft-07/schema#"`. - Only JSON Schema Draft-07 is supported. +- You must declare the JSON Schema dialect using the `$schema` keyword, for example `"$schema": "http://json-schema.org/draft-07/schema#"`. +- You must use a JSON Schema that constrains JSON documents to a strict type in order for Redpanda to translate to Iceberg; that is, each subschema must use the `type` keyword. +.Valid JSON Schema example +[,json] +---- +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "properties": { + "productId": { + "type": "integer" + }, + "tags": { + "type": "array", + "items": { + "type": "string" + } + } + } +} +---- |=== | JSON type | Iceberg type | Notes | array | list -| The keywords `items` and `additionalItems` are used to define element types. +| The keywords `items` and `additionalItems` must be used to constrain element types. | boolean | boolean | | null -| null -| +| +| The `null` type is not supported except when it is paired with another type to indicate nullability. | number -| +| double | | integer -| int +| long | | string | string -| +| The `format` keyword can be used for custom Iceberg types. | object -| -| +| struct +| The `properties` keyword must be used to define `struct` fields and constrain their types. The `additionalProperties` keyword is accepted only when it is set to `false`. |=== @@ -281,7 +302,7 @@ The following are not supported for JSON Schema: * The `default` keyword * Conditional typing (`if`, `then`, `else`, `dependent` keywords) * Boolean JSON Schema combinations (`allOf`, `anyOf`, `oneOf` keywords) -* Dynamic object members (`patternProperties`, `additionalProperties` (except when it is set to `false`) keywords) +* Dynamic object members (`patternProperties` and `additionalProperties` (except when it is set to `false`) keywords) -- ====== From 6da91ed25712a7955d32fa68a0e79e5a54ba6c8b Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Thu, 17 Jul 2025 22:51:32 -0700 Subject: [PATCH 07/20] Apply suggestion from automated review --- modules/manage/pages/iceberg/specify-iceberg-schema.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 1653a38ddc..ba03688635 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -59,7 +59,7 @@ rpk topic create --topic-config=redpanda.iceberg.mode= --set redpanda.iceberg.mode= +rpk topic alter-config --set redpanda.iceberg.mode= ---- [[override-value-schema-latest-default]] From 489b6686bc62cc2ab7e642e59cbc407d2366ea99 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Thu, 17 Jul 2025 22:53:33 -0700 Subject: [PATCH 08/20] DOC-1502 do not use wire format with value_schema_latest --- modules/manage/partials/iceberg/query-iceberg-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/partials/iceberg/query-iceberg-topics.adoc b/modules/manage/partials/iceberg/query-iceberg-topics.adoc index 34cad3ccb5..c7ee424cb0 100644 --- a/modules/manage/partials/iceberg/query-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/query-iceberg-topics.adoc @@ -77,7 +77,7 @@ endif::[] === Topic with schema (`value_schema_id_prefix` mode) -NOTE: The steps in this section also apply to the `value_schema_latest` mode, except for step 2. The `value_schema_latest` mode doesn't require the Schema Registry wire format, so you'll use your own producer code instead of xref:reference:rpk/rpk-topic/rpk-topic-produce[`rpk topic produce`]. +NOTE: The steps in this section also apply to the `value_schema_latest` mode, except for step 2. The `value_schema_latest` mode is not compatible with the Schema Registry wire format, so you'll use your own producer code instead of xref:reference:rpk/rpk-topic/rpk-topic-produce[`rpk topic produce`]. Assume that you have created the `ClickEvent` topic, set `redpanda.iceberg.mode` to `value_schema_id_prefix`, and are connecting to a REST-based Iceberg catalog. The following is an Avro schema for `ClickEvent`: From b21156bb36d56e142d643a11631df060873e6505 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Thu, 17 Jul 2025 23:03:46 -0700 Subject: [PATCH 09/20] DOC-1403 topic misconfiguration pauses Iceberg translation --- modules/manage/partials/iceberg/about-iceberg-topics.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/manage/partials/iceberg/about-iceberg-topics.adoc b/modules/manage/partials/iceberg/about-iceberg-topics.adoc index 147f916f77..b1722eae2a 100644 --- a/modules/manage/partials/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/about-iceberg-topics.adoc @@ -314,7 +314,7 @@ Querying the Iceberg table for `demo-topic` includes the new column `ts`: == Manage dead-letter queue -Errors may occur when translating records in the `value_schema_id_prefix` mode to the Iceberg table format; for example, if you do not use the Schema Registry wire format with the magic byte, if the schema ID in the record is not found in the Schema Registry, or if an Avro or Protobuf data type cannot be translated to an Iceberg type. +Errors may occur when translating records in the `value_schema_id_prefix` or `value_schema_latest` modes to the Iceberg table format; for example, if you do not use the Schema Registry wire format with the magic byte, if the schema ID in the record is not found in the Schema Registry, or if an Avro or Protobuf data type cannot be translated to an Iceberg type. ifndef::env-cloud[] If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda writes the record to a separate dead-letter queue (DLQ) Iceberg table named `~dlq`. To disable the default behavior for a topic and drop the record, set the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-invalid-record-action[`redpanda.iceberg.invalid.record.action`] topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. @@ -327,6 +327,8 @@ The DLQ table itself uses the `key_value` schema, consisting of two columns: the You can inspect the DLQ table for records that failed to write to the Iceberg table, and you can take further action on these records, such as transforming and reprocessing them, or debugging issues that occurred upstream. +NOTE: Topic property misconfiguration, such as setting `redpanda.iceberg.mode` to `value_schema_latest` but not specifying the fully qualified schema name, does not cause records to be written to the DLQ table. Instead, Redpanda pauses the topic data translation to the Iceberg table until you fix the misconfiguration. + === Reprocess DLQ records The following example produces a record to a topic named `ClickEvent` and does not use the Schema Registry wire format that includes the magic byte and schema ID: From 2c9ec97cd41e0355a852f540fce3c425abedcf0f Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Thu, 17 Jul 2025 23:13:23 -0700 Subject: [PATCH 10/20] Apply suggestions from automated review --- modules/manage/partials/iceberg/query-iceberg-topics.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/manage/partials/iceberg/query-iceberg-topics.adoc b/modules/manage/partials/iceberg/query-iceberg-topics.adoc index c7ee424cb0..5f9fe5a37f 100644 --- a/modules/manage/partials/iceberg/query-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/query-iceberg-topics.adoc @@ -77,7 +77,7 @@ endif::[] === Topic with schema (`value_schema_id_prefix` mode) -NOTE: The steps in this section also apply to the `value_schema_latest` mode, except for step 2. The `value_schema_latest` mode is not compatible with the Schema Registry wire format, so you'll use your own producer code instead of xref:reference:rpk/rpk-topic/rpk-topic-produce[`rpk topic produce`]. +NOTE: The steps in this section also apply to the `value_schema_latest` mode, except for step 2. The `value_schema_latest` mode is not compatible with the Schema Registry wire format. The xref:reference:rpk/rpk-topic/rpk-topic-produce[`rpk topic produce`] command embeds the wire format header, so you must use your own producer code with `value_schema_latest`. Assume that you have created the `ClickEvent` topic, set `redpanda.iceberg.mode` to `value_schema_id_prefix`, and are connecting to a REST-based Iceberg catalog. The following is an Avro schema for `ClickEvent`: @@ -139,7 +139,7 @@ In this example, assume that you have created the `ClickEvent_key_value` topic, + [,bash] ---- -echo 'key1 {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"}' | rpk topic produce ClickEvent_key_value --format='%k %v\n' +echo '"key1" {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"}' | rpk topic produce ClickEvent_key_value --format='%k %v\n' ---- . The following Spark SQL query returns the semi-structured data in the `ClickEvent_key_value` table. The table consists of two columns: one named `redpanda`, containing the record key and other metadata, and another binary column named `value` for the record's value: From c8c58c32150e6fc7a471b7333fb337a68b91fc22 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 18 Jul 2025 15:46:54 -0700 Subject: [PATCH 11/20] Apply suggestions from SME review --- .../pages/iceberg/specify-iceberg-schema.adoc | 19 +++++++++++++++---- .../iceberg/about-iceberg-topics.adoc | 2 +- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index ba03688635..806469eab2 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -36,9 +36,9 @@ In the xref:manage:schema-reg/schema-reg-overview.adoc#wire-format[Schema Regist === value_schema_latest -Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. You must register a schema in the Schema Registry. For Protobuf, you must use the fully qualified schema name, which includes the package name, for example `com.example.manufacturing.SensorData`. +Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. You must register a schema in the Schema Registry. -Producers cannot use the wire format in `value_schema_latest` mode. +Producers cannot use the wire format in `value_schema_latest` mode. Redpanda expects the serialized message as-is in the record value. The latest schema is cached periodically. The cache period is defined by the cluster property `iceberg_latest_schema_cache_ttl_ms` (default: 5 minutes). @@ -84,7 +84,7 @@ value_schema_latest:subject=,protobuf_name=`, for example `value_schema_latest:subject=sensor-data`. * For Protobuf only: -** Specify a different message definition by using a key-value pair `protobuf_name=`, for example: `value_schema_latest:protobuf_name=com.example.manufacturing.SensorData`. +** Specify a different message definition by using a key-value pair `protobuf_name=`. You must use the fully qualified name, which includes the package name, for example, `value_schema_latest:protobuf_name=com.example.manufacturing.SensorData`. ** To specify both a different subject and message definition, separate the key-value pairs with a comma, for example: `value_schema_latest:subject=my_protobuf_schema,protobuf_name=com.example.manufacturing.SensorData`. == How Iceberg modes translate to table format @@ -288,7 +288,7 @@ Requirements: | string | string -| The `format` keyword can be used for custom Iceberg types. +| The `format` keyword can be used for custom Iceberg types. See <> for details. | object | struct @@ -296,6 +296,17 @@ Requirements: |=== +[[format-translation]] +.`format` annotation translation +|=== +| `format` value | Iceberg type + +| date-time | timestampz +| date | date +| time | time + +|=== + The following are not supported for JSON Schema: * Relative and absolute (including external) references using `$ref` and `$dynamicRef` keywords diff --git a/modules/manage/partials/iceberg/about-iceberg-topics.adoc b/modules/manage/partials/iceberg/about-iceberg-topics.adoc index b1722eae2a..7de97d195f 100644 --- a/modules/manage/partials/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/about-iceberg-topics.adoc @@ -314,7 +314,7 @@ Querying the Iceberg table for `demo-topic` includes the new column `ts`: == Manage dead-letter queue -Errors may occur when translating records in the `value_schema_id_prefix` or `value_schema_latest` modes to the Iceberg table format; for example, if you do not use the Schema Registry wire format with the magic byte, if the schema ID in the record is not found in the Schema Registry, or if an Avro or Protobuf data type cannot be translated to an Iceberg type. +Errors may occur when translating records in the `value_schema_id_prefix` or `value_schema_latest` modes to the Iceberg table format; for example, if you do not use the Schema Registry wire format with the magic byte, if the schema ID in the record is not found in the Schema Registry, or if a schema data type cannot be translated to an Iceberg type. ifndef::env-cloud[] If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda writes the record to a separate dead-letter queue (DLQ) Iceberg table named `~dlq`. To disable the default behavior for a topic and drop the record, set the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-invalid-record-action[`redpanda.iceberg.invalid.record.action`] topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. From ad711ee1442e71f6e22661046fe93a821d42732a Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 18 Jul 2025 15:53:08 -0700 Subject: [PATCH 12/20] Apply suggestions from automated review --- modules/manage/pages/iceberg/specify-iceberg-schema.adoc | 2 +- modules/manage/partials/iceberg/query-iceberg-topics.adoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 806469eab2..01f621a9e1 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -301,7 +301,7 @@ Requirements: |=== | `format` value | Iceberg type -| date-time | timestampz +| date-time | timestamptz | date | date | time | time diff --git a/modules/manage/partials/iceberg/query-iceberg-topics.adoc b/modules/manage/partials/iceberg/query-iceberg-topics.adoc index 5f9fe5a37f..47b353feb0 100644 --- a/modules/manage/partials/iceberg/query-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/query-iceberg-topics.adoc @@ -77,7 +77,7 @@ endif::[] === Topic with schema (`value_schema_id_prefix` mode) -NOTE: The steps in this section also apply to the `value_schema_latest` mode, except for step 2. The `value_schema_latest` mode is not compatible with the Schema Registry wire format. The xref:reference:rpk/rpk-topic/rpk-topic-produce[`rpk topic produce`] command embeds the wire format header, so you must use your own producer code with `value_schema_latest`. +NOTE: The steps in this section also apply to the `value_schema_latest` mode, except the produce step. The `value_schema_latest` mode is not compatible with the Schema Registry wire format. The xref:reference:rpk/rpk-topic/rpk-topic-produce[`rpk topic produce`] command embeds the wire format header, so you must use your own producer code with `value_schema_latest`. Assume that you have created the `ClickEvent` topic, set `redpanda.iceberg.mode` to `value_schema_id_prefix`, and are connecting to a REST-based Iceberg catalog. The following is an Avro schema for `ClickEvent`: From 2a1d3960c2a1496e289287ac7da11279c7aa3062 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 18 Jul 2025 16:14:20 -0700 Subject: [PATCH 13/20] Edits per review --- modules/manage/pages/iceberg/specify-iceberg-schema.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 01f621a9e1..a61c980f9a 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -131,7 +131,7 @@ CREATE TABLE ClickEvent ( ) ---- -Use `key_value` mode if you're able to use the Iceberg data in its semi-structured format. +Use `key_value` mode if you want to use the Iceberg data in its semi-structured format. The `value_schema_id_prefix` and `value_schema_latest` modes can use the schema to translate to the following table format: @@ -239,7 +239,7 @@ JSON Schema:: -- Requirements: -- Only JSON Schema Draft-07 is supported. +- Only JSON Schema Draft-07 is currently supported. - You must declare the JSON Schema dialect using the `$schema` keyword, for example `"$schema": "http://json-schema.org/draft-07/schema#"`. - You must use a JSON Schema that constrains JSON documents to a strict type in order for Redpanda to translate to Iceberg; that is, each subschema must use the `type` keyword. From 97f3605809ee0eb0c646dfffd2a16f6ee05e10ca Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 18 Jul 2025 16:30:09 -0700 Subject: [PATCH 14/20] Make sure to have line break before tag --- modules/manage/pages/iceberg/specify-iceberg-schema.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index a61c980f9a..2222a4ebcb 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -3,6 +3,7 @@ :page-categories: Iceberg, Tiered Storage, Management, High Availability, Data Replication, Integration :page-aliases: manage:iceberg/choose-iceberg-mode.adoc :schema-id-val-doc: manage:schema-reg/schema-id-validation.adoc + // tag::single-source[] ifndef::env-cloud[] From 5d6fc1312a1a012e14e6227da8788ba34a75fa6e Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 22 Jul 2025 14:54:53 -0700 Subject: [PATCH 15/20] Add JSON Schema for Iceberg topics in What's New --- modules/get-started/pages/release-notes/redpanda.adoc | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 22bbffdf09..4f9a06a2e6 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -7,6 +7,10 @@ This topic includes new content added in version {page-component-version}. For a * xref:redpanda-cloud:get-started:whats-new-cloud.adoc[] * xref:redpanda-cloud:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility] +== JSON Schema support for Iceberg topics + +Redpanda now supports JSON Schema for Iceberg topics. This allows you to use all supported schema types (Protobuf, Avro, and JSON Schema) for Iceberg topics. For more information, see xref:manage:iceberg/specify-iceberg-schema.adoc[]. + == Manage SASL users with Kafka APIs Redpanda now supports the following Kafka APIs for managing SASL user credentials as described in https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API[KIP-554^]: From 5a8045ed4f1debe65c879a4b9364f345a1fbce7f Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Tue, 22 Jul 2025 15:29:43 -0700 Subject: [PATCH 16/20] Apply suggested from automated review --- modules/manage/pages/iceberg/specify-iceberg-schema.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 2222a4ebcb..8bfd3c6aeb 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -41,6 +41,8 @@ Creates an Iceberg table whose structure matches the latest schema registered fo Producers cannot use the wire format in `value_schema_latest` mode. Redpanda expects the serialized message as-is in the record value. +NOTE: The `value_schema_latest` mode is not compatible with the xref:reference:rpk/rpk-topic/rpk-topic-produce[`rpk topic produce`] command which embeds the wire format header. You must use your own producer code to produce to topics in `value_schema_latest` mode. + The latest schema is cached periodically. The cache period is defined by the cluster property `iceberg_latest_schema_cache_ttl_ms` (default: 5 minutes). === disabled From 4ae7e7f51c8240bd31ee057df46ca7937c804bce Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Wed, 23 Jul 2025 09:00:05 -0700 Subject: [PATCH 17/20] Schema evolution is also supported with JSON --- modules/manage/partials/iceberg/about-iceberg-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/partials/iceberg/about-iceberg-topics.adoc b/modules/manage/partials/iceberg/about-iceberg-topics.adoc index 7de97d195f..efd38aca3a 100644 --- a/modules/manage/partials/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/about-iceberg-topics.adoc @@ -233,7 +233,7 @@ Data in an Iceberg-enabled topic is consumable from Kafka based on the configure == Schema evolution -Redpanda supports schema evolution for Avro and Protobuf schemas in accordance with the https://iceberg.apache.org/spec/#schema-evolution[Iceberg specification^]. Permitted schema evolutions include reordering fields and promoting field types. When you update the schema in Schema Registry, Redpanda automatically updates the Iceberg table schema to match the new schema. +Redpanda supports schema evolution in accordance with the https://iceberg.apache.org/spec/#schema-evolution[Iceberg specification^]. Permitted schema evolutions include reordering fields and promoting field types. When you update the schema in Schema Registry, Redpanda automatically updates the Iceberg table schema to match the new schema. For example, if you produce records to a topic `demo-topic` with the following Avro schema: From 2e50aa28600b4d274d97fa66d29a4a2e0ffac56a Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 25 Jul 2025 07:06:02 -0700 Subject: [PATCH 18/20] Apply suggestions from SME review --- modules/manage/pages/iceberg/specify-iceberg-schema.adoc | 8 +++++++- modules/manage/partials/iceberg/about-iceberg-topics.adoc | 2 +- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 8bfd3c6aeb..4e5902c51a 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -39,7 +39,7 @@ In the xref:manage:schema-reg/schema-reg-overview.adoc#wire-format[Schema Regist Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. You must register a schema in the Schema Registry. -Producers cannot use the wire format in `value_schema_latest` mode. Redpanda expects the serialized message as-is in the record value. +Producers cannot use the wire format in `value_schema_latest` mode. Redpanda expects the serialized message as-is without the magic byte or schema ID prefix in the record value. NOTE: The `value_schema_latest` mode is not compatible with the xref:reference:rpk/rpk-topic/rpk-topic-produce[`rpk topic produce`] command which embeds the wire format header. You must use your own producer code to produce to topics in `value_schema_latest` mode. @@ -87,8 +87,14 @@ value_schema_latest:subject=,protobuf_name=`, for example `value_schema_latest:subject=sensor-data`. * For Protobuf only: ++ +-- ** Specify a different message definition by using a key-value pair `protobuf_name=`. You must use the fully qualified name, which includes the package name, for example, `value_schema_latest:protobuf_name=com.example.manufacturing.SensorData`. ** To specify both a different subject and message definition, separate the key-value pairs with a comma, for example: `value_schema_latest:subject=my_protobuf_schema,protobuf_name=com.example.manufacturing.SensorData`. +-- ++ +NOTE: If you don't specify the fully qualified Protobuf message name, Redpanda pauses the data translation to the Iceberg table until you fix the topic misconfiguration. + == How Iceberg modes translate to table format diff --git a/modules/manage/partials/iceberg/about-iceberg-topics.adoc b/modules/manage/partials/iceberg/about-iceberg-topics.adoc index efd38aca3a..1ca4b76e8b 100644 --- a/modules/manage/partials/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/about-iceberg-topics.adoc @@ -327,7 +327,7 @@ The DLQ table itself uses the `key_value` schema, consisting of two columns: the You can inspect the DLQ table for records that failed to write to the Iceberg table, and you can take further action on these records, such as transforming and reprocessing them, or debugging issues that occurred upstream. -NOTE: Topic property misconfiguration, such as setting `redpanda.iceberg.mode` to `value_schema_latest` but not specifying the fully qualified schema name, does not cause records to be written to the DLQ table. Instead, Redpanda pauses the topic data translation to the Iceberg table until you fix the misconfiguration. +NOTE: Topic property misconfiguration, such as xref:manage:iceberg/specify-iceberg-schema.adoc#override-value-schema-latest-default[overriding the default behavior of `value_schema_latest` mode] but not specifying the fully qualified Protobuf message name, does not cause records to be written to the DLQ table. Instead, Redpanda pauses the topic data translation to the Iceberg table until you fix the misconfiguration. === Reprocess DLQ records From 4d5f8986c062aabb06b96c7a1d17f7da9b73ad11 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 25 Jul 2025 18:34:56 -0700 Subject: [PATCH 19/20] Implement suggestion from doc review --- .../pages/iceberg/specify-iceberg-schema.adoc | 2 +- .../iceberg/about-iceberg-topics.adoc | 32 ++++++++++++------- .../pages/properties/cluster-properties.adoc | 2 +- .../pages/properties/topic-properties.adoc | 2 +- 4 files changed, 23 insertions(+), 15 deletions(-) diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 4e5902c51a..ffcfb55f2c 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -162,7 +162,7 @@ CREATE TABLE ClickEvent ( As you produce records to the topic, the data also becomes available in object storage for Iceberg-compatible clients to consume. You can use the same analytical tools to xref:manage:iceberg/query-iceberg-topics.adoc[read the Iceberg topic data] in a data lake as you would for a relational database. -If Redpanda fails to translate the record to the columnar format as defined by the schema, it writes the record to a dead-letter queue (DLQ) table. See xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue] for more information. +If Redpanda fails to translate the record to the columnar format as defined by the schema, it writes the record to a dead-letter queue (DLQ) table. See xref:manage:iceberg/about-iceberg-topics.adoc#troubleshoot-errors[Troubleshoot errors] for more information. === Schema types translation diff --git a/modules/manage/partials/iceberg/about-iceberg-topics.adoc b/modules/manage/partials/iceberg/about-iceberg-topics.adoc index 1ca4b76e8b..03d9ec1206 100644 --- a/modules/manage/partials/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/partials/iceberg/about-iceberg-topics.adoc @@ -312,24 +312,21 @@ Querying the Iceberg table for `demo-topic` includes the new column `ts`: +---------+--------------+--------------------------+ ---- -== Manage dead-letter queue +== Troubleshoot errors -Errors may occur when translating records in the `value_schema_id_prefix` or `value_schema_latest` modes to the Iceberg table format; for example, if you do not use the Schema Registry wire format with the magic byte, if the schema ID in the record is not found in the Schema Registry, or if a schema data type cannot be translated to an Iceberg type. +If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda by default writes the record to a separate dead-letter queue (DLQ) Iceberg table named `~dlq`. The following can cause errors to occur when translating records in the `value_schema_id_prefix` and `value_schema_latest` modes to the Iceberg table format: -ifndef::env-cloud[] -If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda writes the record to a separate dead-letter queue (DLQ) Iceberg table named `~dlq`. To disable the default behavior for a topic and drop the record, set the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-invalid-record-action[`redpanda.iceberg.invalid.record.action`] topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. -endif::[] -ifdef::env-cloud[] -If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda writes the record to a separate dead-letter queue (DLQ) Iceberg table named `~dlq`. To disable the default behavior for a topic and drop the record, set the `redpanda.iceberg.invalid.record.action` topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. -endif::[] +- Redpanda cannot find the embedded schema ID in the Schema Registry. +- Redpanda fails to translate one or more schema data types to an Iceberg type. +- In `value_schema_id_prefix` mode, you do not use the Schema Registry wire format with the magic byte. The DLQ table itself uses the `key_value` schema, consisting of two columns: the record metadata including the key, and a binary column for the record's value. -You can inspect the DLQ table for records that failed to write to the Iceberg table, and you can take further action on these records, such as transforming and reprocessing them, or debugging issues that occurred upstream. - NOTE: Topic property misconfiguration, such as xref:manage:iceberg/specify-iceberg-schema.adoc#override-value-schema-latest-default[overriding the default behavior of `value_schema_latest` mode] but not specifying the fully qualified Protobuf message name, does not cause records to be written to the DLQ table. Instead, Redpanda pauses the topic data translation to the Iceberg table until you fix the misconfiguration. -=== Reprocess DLQ records +=== Inspect DLQ table + +You can inspect the DLQ table for records that failed to write to the Iceberg table, and you can take further action on these records, such as transforming and reprocessing them, or debugging issues that occurred upstream. The following example produces a record to a topic named `ClickEvent` and does not use the Schema Registry wire format that includes the magic byte and schema ID: @@ -360,7 +357,9 @@ FROM ."ClickEvent~dlq"; -- Fully qualified table name +-------------------------------------------------+ ---- -The data is in binary format, and the first byte is not `0x00`, indicating that it was not produced with a schema. +The data is in binary format, and the first byte is not `0x00`, indicating that it was not produced with a schema. + +=== Reprocess DLQ records You can apply a transformation and reprocess the record in your data lakehouse to the original Iceberg table. In this case, you have a JSON value represented as a UTF-8 binary. Depending on your query engine, you might need to decode the binary value first before extracting the JSON fields. Some engines may automatically decode the binary value for you: @@ -389,6 +388,15 @@ FROM ( You can now insert the transformed record back into the main Iceberg table. Redpanda recommends employing a strategy for exactly-once processing to avoid duplicates when reprocessing records. +=== Drop invalid records + +ifndef::env-cloud[] +To disable the default behavior and drop an invalid record, set the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-invalid-record-action[`redpanda.iceberg.invalid.record.action`] topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. +endif::[] +ifdef::env-cloud[] +To disable the default behavior and drop an invalid record, set the `redpanda.iceberg.invalid.record.action` topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. +endif::[] + == Performance considerations When you enable Iceberg for any substantial workload and start translating topic data to the Iceberg format, you may see most of your cluster's CPU utilization increase. If this additional workload overwhelms the brokers and causes the Iceberg table lag to exceed the configured target lag, Redpanda automatically applies backpressure to producers to prevent Iceberg tables from lagging further. This ensures that Iceberg tables keep up with the volume of incoming data, but sacrifices ingress throughput of the cluster. diff --git a/modules/reference/pages/properties/cluster-properties.adoc b/modules/reference/pages/properties/cluster-properties.adoc index 7a5df5640c..9f3c92bf76 100644 --- a/modules/reference/pages/properties/cluster-properties.adoc +++ b/modules/reference/pages/properties/cluster-properties.adoc @@ -2059,7 +2059,7 @@ endif::[] **Related topics**: -- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue] +- xref:manage:iceberg/about-iceberg-topics.adoc#troubleshoot-errors[Troubleshoot errors] --- diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 17f2240dbb..80edaca031 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -611,7 +611,7 @@ Whether to write invalid records to a dead-letter queue (DLQ). **Related topics**: -- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue] +- xref:manage:iceberg/about-iceberg-topics.adoc#troubleshoot-errors[Troubleshoot errors] --- From 09811b66eef812f75c70b937654704fb543222fd Mon Sep 17 00:00:00 2001 From: Kat Batuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 25 Jul 2025 19:11:15 -0700 Subject: [PATCH 20/20] Update local-antora-playbook.yml Co-authored-by: Jake Cahill <45230295+JakeSCahill@users.noreply.github.com> --- local-antora-playbook.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index 67f8695e6e..c0a904527a 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -17,7 +17,7 @@ content: - url: https://github.com/redpanda-data/docs branches: [main, v/*, api, shared, site-search,'!v-end-of-life/*'] - url: https://github.com/redpanda-data/cloud-docs - branches: 'DOC-1379-cloud-iceberg-json-schema' + branches: 'main' - url: https://github.com/redpanda-data/redpanda-labs branches: main start_paths: [docs,'*/docs']