Skip to content

Commit eca357b

Browse files
authored
Iceberg data retention (#1171)
1 parent b4cdd4a commit eca357b

2 files changed

Lines changed: 11 additions & 2 deletions

File tree

modules/ROOT/nav.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,7 @@
175175
*** xref:manage:iceberg/use-iceberg-catalogs.adoc[Use Iceberg Catalogs]
176176
*** xref:manage:iceberg/query-iceberg-topics.adoc[Query Iceberg Topics]
177177
*** xref:manage:iceberg/iceberg-topics-databricks-unity.adoc[Query Iceberg Topics with Databricks Unity Catalog]
178-
*** xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[Query Iceberg Topics with Snowflake]
178+
*** xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[Query Iceberg Topics with Snowflake and Open Catalog]
179179
** xref:manage:schema-reg/index.adoc[Schema Registry]
180180
*** xref:manage:schema-reg/schema-reg-overview.adoc[Overview]
181181
*** xref:manage:schema-reg/manage-schema-reg.adoc[]

modules/manage/partials/iceberg/about-iceberg-topics.adoc

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
1-
:schema-id-val-doc: manage:schema-reg/schema-id-validation.adoc
1+
ifdef::env-cloud[:manage-topic-doc: get-started:create-topic.adoc]
2+
ifndef::env-cloud[:manage-topic-doc: manage:cluster-maintenance/disk-utilization.adoc]
23

34
The Apache Iceberg integration for Redpanda allows you to store topic data in the cloud in the Iceberg open table format. This makes your streaming data immediately available in downstream analytical systems, including data warehouses like Snowflake, Databricks, ClickHouse, and Redshift, without setting up and maintaining additional ETL pipelines. You can also integrate your data directly into commonly-used big data processing frameworks, such as Apache Spark and Flink, standardizing and simplifying the consumption of streams as tables in a wide variety of data analytics pipelines.
45

@@ -222,6 +223,14 @@ As you produce records to the topic, the data also becomes available in object s
222223

223224
See also: xref:manage:iceberg/choose-iceberg-mode.adoc#schema-types-translation[Schema types translation].
224225

226+
=== Iceberg data retention
227+
228+
Data in an Iceberg-enabled topic is consumable from Kafka based on the configured xref:{manage-topic-doc}[topic retention policy]. Conversely, data written to Iceberg remains queryable as Iceberg tables indefinitely. The Iceberg table persists unless you:
229+
230+
- Delete the Redpanda topic associated with the Iceberg table. This is the default behavior set by the config_ref:iceberg_delete,true,properties/cluster-properties[`iceberg_delete`] cluster property and the `redpanda.iceberg.delete` topic property. If you set this property to `false`, the Iceberg table remains even after you delete the topic.
231+
- Explicitly delete data from the Iceberg table using a query engine.
232+
- Disable the Iceberg integration for the topic and delete the Parquet files in object storage.
233+
225234
== Schema evolution
226235

227236
Redpanda supports schema evolution for Avro and Protobuf schemas in accordance with the https://iceberg.apache.org/spec/#schema-evolution[Iceberg specification^]. Permitted schema evolutions include reordering fields and promoting field types. When you update the schema in Schema Registry, Redpanda automatically updates the Iceberg table schema to match the new schema.

0 commit comments

Comments
 (0)