Skip to content

Commit ddae5f1

Browse files
authored
Iceberg updates for REST catalog and wire format compatible modes (#1238)
1 parent cdc67ae commit ddae5f1

4 files changed

Lines changed: 41 additions & 10 deletions

File tree

modules/manage/pages/iceberg/choose-iceberg-mode.adoc

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,16 @@ The latest schema is cached periodically. The cache period is defined by the clu
4545

4646
Default for `redpanda.iceberg.mode`. Disables writing to an Iceberg table for the topic.
4747

48+
[NOTE]
49+
====
50+
The following modes are compatible with producing to an Iceberg topic using Redpanda Console:
51+
52+
- `key_value`
53+
- Starting in version 25.2, `value_schema_latest` with a JSON schema
54+
55+
Otherwise, records may fail to write to the Iceberg table and instead write to the xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[dead-letter queue].
56+
====
57+
4858
== Configure Iceberg mode for a topic
4959

5060
You can set the Iceberg mode for a topic when you create the topic, or you can update the mode for an existing topic.

modules/manage/partials/iceberg/about-iceberg-topics.adoc

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ TOPIC STATUS
135135
<new-topic-name> OK
136136
----
137137

138-
. Configure `redpanda.iceberg.mode` for the topic. You can choose one of the following modes:
138+
. Configure `redpanda.iceberg.mode` for the topic. You can choose one of the following xref:manage:iceberg/choose-iceberg-mode.adoc[Iceberg modes]:
139139
+
140140
--
141141
* `key_value`: Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record's value.
@@ -154,8 +154,6 @@ rpk topic alter-config <new-topic-name> --set redpanda.iceberg.mode=<topic-icebe
154154
TOPIC STATUS
155155
<new-topic-name> OK
156156
----
157-
+
158-
See also: xref:manage:iceberg/choose-iceberg-mode.adoc[].
159157

160158
. Register a schema for the topic. This step is required for the `value_schema_id_prefix` and `value_schema_latest` modes.
161159
+

modules/manage/partials/iceberg/use-iceberg-catalogs.adoc

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,11 @@ The Iceberg integration for Redpanda Cloud supports multiple Iceberg catalogs ac
2828
The following matrix shows the current status of Iceberg integrations across different cloud providers and catalogs. Check this matrix regularly as Redpanda Cloud continues to expand GA coverage for Iceberg topics.
2929

3030
|===
31-
| |Databricks Unity Catalog |Snowflake Open Catalog |Dremio Nessie
31+
| | Databricks Unity Catalog | Snowflake Open Catalog | AWS Glue Data Catalog | Google BigQuery
3232

33-
|AWS |Beta |Beta |Beta
34-
|Azure |Beta |Beta |Beta
35-
|GCP |Beta |Beta |Beta
33+
|AWS |Supported |Beta |Beta |N/A
34+
|GCP |Supported |Beta |N/A |Beta
35+
|Azure |Beta |Beta |N/A |N/A
3636
|===
3737
endif::[]
3838

@@ -46,13 +46,16 @@ The following shows the current status of Iceberg catalog integrations. Check th
4646
|===
4747
|Catalog service | Status
4848

49-
|Databricks Unity Catalog |Tested
50-
|Snowflake Open Catalog |Tested
51-
|Dremio Nessie |Tested
49+
|Databricks Unity Catalog |Supported
50+
|Snowflake Open Catalog |Supported
51+
|AWS Glue Data Catalog | Beta
52+
|Google BigQuery |Beta
5253

5354
|===
5455
endif::[]
5556

57+
Other REST catalogs such as Dremio Nessie, Apache Polaris, and the Apache reference implementation have been tested but are not regularly verified. For more information, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^].
58+
5659
=== Set cluster properties
5760

5861
To connect to a REST catalog, set the following cluster configuration properties:

modules/reference/pages/properties/cluster-properties.adoc

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2090,6 +2090,26 @@ endif::[]
20902090

20912091
// end::iceberg_rest_catalog_authentication_mode[]
20922092

2093+
// tag::iceberg_rest_catalog_base_location[]
2094+
2095+
=== iceberg_rest_catalog_base_location
2096+
2097+
NOTE: This property is available in Redpanda version 25.1.7 and later.
2098+
2099+
Base URI for the Iceberg REST catalog. If unset, the REST catalog server determines the location. Some REST catalogs, like AWS Glue, require the client to set this. After Iceberg is enabled, do not change this value.
2100+
2101+
*Requires restart:* Yes
2102+
2103+
*Visibility:* `user`
2104+
2105+
*Type:* string
2106+
2107+
*Default:* `null`
2108+
2109+
---
2110+
2111+
// end::iceberg_rest_catalog_base_location[]
2112+
20932113
// tag::iceberg_rest_catalog_client_id[]
20942114
=== iceberg_rest_catalog_client_id
20952115

0 commit comments

Comments
 (0)