-
Notifications
You must be signed in to change notification settings - Fork 50
Iceberg updates for REST catalog and wire format compatible modes #1238
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
9f00e7d
d7b2477
5e2f574
fac4d9c
714d812
57597da
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -45,6 +45,16 @@ The latest schema is cached periodically. The cache period is defined by the clu | |
|
|
||
| Default for `redpanda.iceberg.mode`. Disables writing to an Iceberg table for the topic. | ||
|
|
||
| [NOTE] | ||
| ==== | ||
| The following modes are compatible with producing to an Iceberg topic using Redpanda Console: | ||
|
|
||
| - `key_value` | ||
| - Starting in version 25.2, `value_schema_latest` with a JSON schema | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note: The PR to add JSON Schema support is open and will be merged when 25.2 is GA. |
||
|
|
||
| Otherwise, records may fail to write to the Iceberg table and instead write to the xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[dead-letter queue]. | ||
| ==== | ||
|
|
||
| == Configure Iceberg mode for a topic | ||
|
|
||
| You can set the Iceberg mode for a topic when you create the topic, or you can update the mode for an existing topic. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -28,11 +28,11 @@ The Iceberg integration for Redpanda Cloud supports multiple Iceberg catalogs ac | |
| The following matrix shows the current status of Iceberg integrations across different cloud providers and catalogs. Check this matrix regularly as Redpanda Cloud continues to expand GA coverage for Iceberg topics. | ||
|
|
||
| |=== | ||
| | |Databricks Unity Catalog |Snowflake Open Catalog |Dremio Nessie | ||
| | | Databricks Unity Catalog | Snowflake Open Catalog | AWS Glue Data Catalog | Google BigQuery | ||
|
|
||
| |AWS |Beta |Beta |Beta | ||
| |Azure |Beta |Beta |Beta | ||
| |GCP |Beta |Beta |Beta | ||
| |AWS |Supported |Beta |Beta |N/A | ||
| |GCP |Supported |Beta |N/A |Beta | ||
| |Azure |Beta |Beta |N/A |N/A | ||
| |=== | ||
| endif::[] | ||
|
|
||
|
|
@@ -46,13 +46,16 @@ The following shows the current status of Iceberg catalog integrations. Check th | |
| |=== | ||
| |Catalog service | Status | ||
|
|
||
| |Databricks Unity Catalog |Tested | ||
| |Snowflake Open Catalog |Tested | ||
| |Dremio Nessie |Tested | ||
| |Databricks Unity Catalog |Supported | ||
| |Snowflake Open Catalog |Supported | ||
| |AWS Glue Data Catalog | Beta | ||
| |Google BigQuery |Beta | ||
|
|
||
| |=== | ||
| endif::[] | ||
|
|
||
| Other REST catalogs such as Dremio Nessie, Apache Polaris, and the Apache reference implementation have been tested but are not regularly verified. For more information, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @wzzzrd86 mind signing off on this? Thank you |
||
|
|
||
| === Set cluster properties | ||
|
|
||
| To connect to a REST catalog, set the following cluster configuration properties: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2090,6 +2090,26 @@ endif::[] | |
|
|
||
| // end::iceberg_rest_catalog_authentication_mode[] | ||
|
|
||
| // tag::iceberg_rest_catalog_base_location[] | ||
|
|
||
| === iceberg_rest_catalog_base_location | ||
|
|
||
| NOTE: This property is available in Redpanda version 25.1.7 and later. | ||
|
|
||
| Base URI for the Iceberg REST catalog. If unset, the REST catalog server determines the location. Some REST catalogs, like AWS Glue, require the client to set this. After Iceberg is enabled, do not change this value. | ||
|
|
||
| *Requires restart:* Yes | ||
|
|
||
| *Visibility:* `user` | ||
|
|
||
| *Type:* string | ||
|
|
||
| *Default:* `null` | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @wdberkeley @alenkacz does this default also apply to Cloud? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes |
||
|
|
||
| --- | ||
|
|
||
| // end::iceberg_rest_catalog_base_location[] | ||
|
|
||
| // tag::iceberg_rest_catalog_client_id[] | ||
| === iceberg_rest_catalog_client_id | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rockwotj mind signing off on this? Thank you
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes this information is correct
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rockwotj did we confirm its impossible to get Console to product using the wire format (whereas rpk always does!) cc @r-vasquez
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Console only produces raw values, it does no encoding, so no support for protobuf/avro with or without schema registry.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be cool to support json or textproto for console testing (with or without wire format)