You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: modules/manage/partials/iceberg/about-iceberg-topics.adoc
+6-5Lines changed: 6 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -133,7 +133,7 @@ For details on the partitioning specification, including allowed transforms, see
133
133
--
134
134
* `key_value`: Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record's value.
135
135
* `value_schema_id_prefix`: Creates an Iceberg table whose structure matches the Redpanda schema for this topic, with columns corresponding to each field. You must register a schema in the Schema Registry (see next step), and producers must write to the topic using the Schema Registry wire format. Redpanda parses the schema used by the record based on the schema ID encoded in the payload header and stores the topic values in the corresponding table columns.
136
-
* `value_subject_latest`: Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry.
136
+
* `value_schema_latest`: Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry.
137
137
* `disabled` (default): Disables writing to an Iceberg table for this topic.
138
138
--
139
139
+
@@ -177,12 +177,13 @@ The Iceberg table resides in a namespace called `redpanda` and has the same name
177
177
178
178
== About schema support and translation to Iceberg format
179
179
180
-
The `redpanda.iceberg.mode` topic property determines how Redpanda maps the topic data to the Iceberg table structure. You can use the `value_schema_id_prefix` and `value_subject_latest` modes to generate an Iceberg table that matches the structure of an Avro or Protobuf schema in the Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table.
180
+
The `redpanda.iceberg.mode` topic property determines how Redpanda maps the topic data to the Iceberg table structure. You can use the `value_schema_id_prefix` and `value_schema_latest` modes to generate an Iceberg table that matches the structure of an Avro or Protobuf schema in the Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table.
181
181
182
182
The JSON Schema format is not supported. If your topic data is in JSON, use the `key_value` mode.
183
183
184
+
// TODO: Remove conditional once topic property page is available in Cloud
184
185
ifndef::env-cloud[]
185
-
See the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-mode[`redpanda.iceberg.mode` property reference] for more details, including allowed syntax for `value_subject_latest`.
186
+
See the xref:reference:properties/topic-properties.adoc#redpanda-iceberg-mode[`redpanda.iceberg.mode` property reference] for more details, including allowed syntax for `value_schema_latest`.
186
187
endif::[]
187
188
188
189
=== Iceberg modes and table schemas
@@ -231,7 +232,7 @@ CREATE TABLE ClickEvent (
231
232
232
233
Consider this approach if the topic data is in JSON, or if you can use the Iceberg data in its semi-structured format.
233
234
234
-
The `value_schema_id_prefix` and `value_subject_latest` modes translate to the following table format:
235
+
The `value_schema_id_prefix` and `value_schema_latest` modes translate to the following table format:
235
236
236
237
[,sql]
237
238
----
@@ -257,7 +258,7 @@ ifdef::env-cloud[]
257
258
With schema integration using the `value_schema_id_prefix` mode, Redpanda uses the Schema Registry wire format, which embeds the magic byte and schema ID in the message payload header. Producers to the topic must use the wire format in the serialization process so Redpanda can determine the schema used for each record, parse the record according to that schema, and use the schema to define the Iceberg table. See also: the https://www.redpanda.com/blog/schema-registry-kafka-streaming#how-does-serialization-work-with-schema-registry-in-kafka[Understanding Apache Kafka Schema Registry^] blog post.
258
259
endif::[]
259
260
260
-
The `value_subject_latest` mode uses the latest schema registered for the subject in the Schema Registry. This mode is similar to the `value_schema_id_prefix` mode, but it does not require that producers use the wire format.
261
+
The `value_schema_latest` mode uses the latest schema registered for the subject in the Schema Registry. This mode is similar to the `value_schema_id_prefix` mode, but it does not require that producers use the wire format.
261
262
262
263
If Redpanda fails to translate the record to the columnar format as defined by the schema, it writes the record to a dead-letter queue (DLQ) table. See <<manage-dead-letter-queue,Manage dead-letter queue>> for more information.
=== Topic with schema (`value_schema_id_prefix` or `value_subject_latest` mode)
46
+
=== Topic with schema (`value_schema_id_prefix` or `value_schema_latest` mode)
47
47
48
48
Assume that you have created the `ClickEvent` topic, set `redpanda.iceberg.mode` to `value_schema_id_prefix`, and are connecting to a REST-based Iceberg catalog. The following is an Avro schema for `ClickEvent`:
Copy file name to clipboardExpand all lines: modules/reference/pages/properties/topic-properties.adoc
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -538,24 +538,24 @@ Enable the Iceberg integration for the topic. You can choose one of three modes.
538
538
539
539
- `key_value`: Creates an Iceberg table with a `Key` column and a `Value` column. Redpanda stores the raw topic content in the `Value` column.
540
540
- `value_schema_id_prefix`: Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. Redpanda uses the Schema Registry wire format, consisting of the "magic byte" and schema ID encoded in the payload header, to parse the topic values per field and store them in the corresponding table columns.
541
-
- `value_subject_latest`: Creates an Iceberg table whose structure matches the latest schema version in Schema Registry that matches the subject name. This mode is compatible with Avro and Protobuf schemas and is used when you don't produce to the topic using the wire format.
541
+
- `value_schema_latest`: Creates an Iceberg table whose structure matches the latest schema version in Schema Registry that matches the subject name. This mode is compatible with Avro and Protobuf schemas and is used when you don't produce to the topic using the wire format.
542
542
- `disabled`: Disables writing to an Iceberg table for the topic.
543
543
544
-
For `value_subject_latet` mode, use the following syntax:
544
+
For `value_schema_latest` mode, use the following syntax:
* Only the string `value_subject_latest` is required in the property value.
551
+
* Only the string `value_schema_latest` is required in the property value.
552
552
* Key-value pairs as indicated by `<key>=<value>` inside the square brackets `[]` are optional. The key-value pairs override this mode's default behavior for the topic.
553
553
** For Protobuf data, the mode's default behavior derives the suject name using xref:manage:schema-reg/schema-id-validation.adoc#set-subject-name-strategy-per-topic[TopicNameStrategy] and deserializes records using the first message definition in the associated Protobuf schema in the Schema Registry.
554
554
+
555
555
--
556
-
* Specify a different subject name using a key-value pair `subject=<subject-name>`, for example `value_subject_latest:subject=my_protobuf_schema`.
557
-
* Specify a different message definition using a key-value pair `protobuf_name=<message-name>`, for example `value_subject_latest:protobuf_name=com.example.manufacturing.SensorData`.
558
-
* To specify both the subject and message definition, separate the key-value pairs with a comma, for example `value_subject_latest:protobuf_name=com.example.manufacturing.SensorData,subject=my_protobuf_schema`.
556
+
* Specify a different subject name using a key-value pair `subject=<subject-name>`, for example `value_schema_latest:subject=my_protobuf_schema`.
557
+
* Specify a different message definition using a key-value pair `protobuf_name=<message-name>`, for example `value_schema_latest:protobuf_name=com.example.manufacturing.SensorData`.
558
+
* To specify both the subject and message definition, separate the key-value pairs with a comma, for example `value_schema_latest:protobuf_name=com.example.manufacturing.SensorData,subject=my_protobuf_schema`.
0 commit comments