From 92060160d9fc7fee313ddcbe61c69667e9a1f16f Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Thu, 21 Aug 2025 20:41:43 -0300 Subject: [PATCH 01/22] topic properties: reorganize content --- .../topic-property-configuration.adoc | 202 ++++++++++++++++++ .../pages/properties/topic-properties.adoc | 126 +---------- 2 files changed, 212 insertions(+), 116 deletions(-) create mode 100644 modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc new file mode 100644 index 000000000..8f9fa8c8d --- /dev/null +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -0,0 +1,202 @@ += Configure Topic Properties +:description: Learn how to configure topic properties to control Redpanda's behavior for individual topics, including retention, cleanup policies, and Tiered Storage settings. +:page-categories: Management + +Topic properties control Redpanda's behavior for individual topics, including data retention, cleanup policies, compression settings, and Tiered Storage configurations. These properties let you customize how Redpanda stores, processes, and manages data for each topic, overriding cluster-wide defaults when needed. + +Redpanda stores topic properties as metadata associated with each topic and replicates them across the cluster to ensure consistency. Many topic properties correspond to xref:manage:cluster-maintenance/cluster-property-configuration.adoc[cluster properties] that establish default values for all topics. When you set a topic property, it overrides the corresponding cluster default for that specific topic. + +NOTE: All topic properties take effect immediately after being set. + +For a complete reference of available topic properties and their corresponding cluster properties, see xref:reference:properties/topic-properties.adoc[Topic Configuration Properties]. + +== Configuration methods + +You can configure topic properties through multiple interfaces: + +* **rpk commands** - Use xref:reference:rpk/rpk-topic/rpk-topic-create.adoc[`rpk topic create`] with the `-c` flag to set properties during topic creation, or xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`] to modify existing topics. +* **Kafka Admin API** - Any Kafka-compatible client can use the standard Admin API to configure topic properties. +* **Kubernetes Topic resources** - In Kubernetes deployments, configure topic properties using the `additionalConfig` field in Topic resources. See xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics in Kubernetes]. +* **Redpanda Console** - Use the web interface to xref:console:ui/edit-topic-configuration.adoc[edit topic configuration] through a graphical interface. + +== Verify topic property configuration + +The `SOURCE` output of the xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe `] command shows how properties are configured for the topic: + +* `DEFAULT_CONFIG` is set by a Redpanda default. +* `DYNAMIC_TOPIC_CONFIG` is set by the user specifically for the topic and overrides inherited default configurations, such as a default or a cluster-level property. + +Although xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`] doesn't report `replication.factor` as a configuration, `replication.factor` can indeed be set by using the xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`] command. + +== Examples + +The following examples show how to configure topic-level properties. Set a topic-level property for a topic to override the value of corresponding cluster property. + +=== Create topic with topic properties + +To set topic properties when creating a topic, use the xref:reference:rpk/rpk-topic/rpk-topic-create.adoc[rpk topic create] command with the `-c` option. + +For example, to create a topic with the `cleanup.policy` property set to `compact`: + +[tabs] +==== +Local:: ++ +-- + +```bash +rpk topic create -c cleanup.policy=compact +``` + +-- +Kubernetes:: ++ +-- + +```bash +kubectl exec -- rpk topic create -c cleanup.policy=compact +``` + +-- +==== + +To configure multiple properties for a topic, use the `-c` option for each property. + +For example, to create a topic with all necessary properties for Tiered Storage: + +[tabs] +==== +Local:: ++ +-- + +```bash +rpk topic create -c redpanda.remote.recovery=true -c redpanda.remote.write=true -c redpanda.remote.read=true +``` + +-- +Kubernetes:: ++ +-- + +```bash +kubectl exec -- rpk topic create -c redpanda.remote.recovery=true -c redpanda.remote.write=true -c redpanda.remote.read=true +``` + +-- +==== + +=== Modify topic properties + +To modify topic properties of an existing topic, use the xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[rpk topic alter-config] command. + +For example, to modify a topic's `retention.ms` property: + +[tabs] +==== +Local:: ++ +-- + +```bash +rpk topic alter-config --set retention.ms= +``` + +-- +Kubernetes:: ++ +-- + +```bash +kubectl exec -- rpk topic alter-config --set retention.ms= +``` + +-- +==== + +=== Configure topic properties with Kubernetes + +In Kubernetes deployments, configure topic properties using the `additionalConfig` field in Topic resources: + +```yaml +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Topic +metadata: + name: example-topic +spec: + partitions: 3 + replicationFactor: 3 + additionalConfig: + cleanup.policy: "compact" + retention.ms: "604800000" + segment.ms: "86400000" +``` + +Apply the configuration: + +```bash +kubectl apply -f topic-config.yaml +``` + +== Common topic property categories + +Topic properties are organized into several categories based on their functionality: + +=== Disk space management + +Configure properties to manage disk space usage for a topic: + +- Clean up log segments by deletion and/or compaction (xref:reference:properties/topic-properties.adoc#cleanuppolicy[`cleanup.policy`]). +- Control compaction timing with maximum and minimum lag settings (xref:reference:properties/topic-properties.adoc#maxcompactionlagms[`max.compaction.lag.ms`] and xref:reference:properties/topic-properties.adoc#mincompactionlagms[`min.compaction.lag.ms`]). +- Retain logs up to a maximum size per partition before cleanup (xref:reference:properties/topic-properties.adoc#retentionbytes[`retention.bytes`]). +- Retain logs for a maximum duration before cleanup (xref:reference:properties/topic-properties.adoc#retentionms[`retention.ms`]). +- Periodically close an active log segment (xref:reference:properties/topic-properties.adoc#segmentms[`segment.ms`]). +- Limit the maximum size of an active log segment (xref:reference:properties/topic-properties.adoc#segmentbytes[`segment.bytes`]). + +=== Write performance and caching + +Control how Redpanda writes and syncs data to disk: + +- Cache batches until the segment appender chunk is full, instead of fsyncing for every `acks=all` write (xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`]). +- Configure the maximum delay between two subsequent fsyncs (xref:reference:properties/topic-properties.adoc#flushms[`flush.ms`]). +- Set the maximum bytes not fsynced per partition (xref:reference:properties/topic-properties.adoc#flushbytes[`flush.bytes`]). + +=== Message properties + +Configure message-specific settings: + +- Set the source of a message's timestamp (xref:reference:properties/topic-properties.adoc#messagetimestamptype[`message.timestamp.type`]). +- Set the maximum size of a message (xref:reference:properties/topic-properties.adoc#maxmessagebytes[`max.message.bytes`]). +- Configure compression for stored data (xref:reference:properties/topic-properties.adoc#compressiontype[`compression.type`]). + +=== Tiered Storage + +For topics using Tiered Storage, configure: + +- Upload and fetch data to and from object storage (xref:reference:properties/topic-properties.adoc#redpandaremotewrite[`redpanda.remote.write`] and xref:reference:properties/topic-properties.adoc#redpandaremoteread[`redpanda.remote.read`]). +- Recover data from object storage (xref:reference:properties/topic-properties.adoc#redpandaremoterecovery[`redpanda.remote.recovery`]). +- Configure local storage retention limits (xref:reference:properties/topic-properties.adoc#retentionlocaltargetbytes[`retention.local.target.bytes`] and xref:reference:properties/topic-properties.adoc#retentionlocaltargetms[`retention.local.target.ms`]). + +=== Compaction settings + +For topics with compacted cleanup policies, configure: + +- Minimum ratio of dirty segments required before compaction (xref:reference:properties/topic-properties.adoc#mincleanabledirtyratio[`min.cleanable.dirty.ratio`]). +- Retention time for tombstone records (xref:reference:properties/topic-properties.adoc#deleteretentionms[`delete.retention.ms`]). + +=== Replication and leadership + +Configure replication and leadership settings: + +- Number of replicas for the topic (xref:reference:properties/topic-properties.adoc#replicationfactor[`replication.factor`]). +- Preferred location for partition leaders (xref:reference:properties/topic-properties.adoc#redpandaleaderspreference[`redpanda.leaders.preference`]). + +For complete details about all available topic properties, see xref:reference:properties/topic-properties.adoc[Topic Configuration Properties]. + +== Related topics + +* xref:reference:properties/topic-properties.adoc[Topic Configuration Properties] - Complete reference of all available topic properties +* xref:manage:cluster-maintenance/cluster-property-configuration.adoc[Configure Cluster Properties] - Configure cluster-wide defaults +* xref:develop:config-topics.adoc[Manage Topics] - Create and manage topics +* xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics in Kubernetes] - Topic management in Kubernetes deployments +* xref:console:ui/edit-topic-configuration.adoc[Edit Topic Configuration in Redpanda Console] - Graphical topic configuration \ No newline at end of file diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 10e249121..0215f5981 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -6,9 +6,12 @@ A topic-level property sets a Redpanda or Kafka configuration for a particular t Many topic-level properties have corresponding xref:manage:cluster-maintenance/cluster-property-configuration.adoc[cluster properties] that set a default value for all topics of a cluster. To customize the value for a topic, you can set a topic-level property that overrides the value of the corresponding cluster property. +For information on how to configure topic properties, see xref:manage:cluster-maintenance/topic-property-configuration.adoc[]. NOTE: All topic properties take effect immediately after being set. +== Topic property mappings + |=== | Topic property | Corresponding cluster property @@ -61,115 +64,7 @@ NOTE: All topic properties take effect immediately after being set. | xref:./cluster-properties.adoc#write_caching_default[`write_caching_default`] |=== -[NOTE] -==== -The `SOURCE` output of the xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe `] command describes how the property is set for the topic: - -* `DEFAULT_CONFIG` is set by a Redpanda default. -* `DYNAMIC_TOPIC_CONFIG` is set by the user specifically for the topic and overrides inherited default configurations, such as a default or a cluster-level property. - -Although xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`] doesn't report `replication.factor` as a configuration, `replication.factor` can indeed be set by using the xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`] command. -==== - -== Examples - -The following examples show how to configure topic-level properties. Set a topic-level property for a topic to override the value of corresponding cluster property. - -=== Create topic with topic properties - -To set topic properties when creating a topic, use the xref:reference:rpk/rpk-topic/rpk-topic-create.adoc[rpk topic create] command with the `-c` option. - -For example, to create a topic with the `cleanup.policy` property set to `compact`: - -[tabs] -==== -Local:: -+ --- - -```bash -rpk topic create -c cleanup.policy=compact -``` - --- -Kubernetes:: -+ --- - -```bash -kubectl exec -- rpk topic create -c cleanup.policy=compact -``` - --- -==== - -To configure multiple properties for a topic, use the `-c` option for each property. - -For example, to create a topic with all necessary properties for Tiered Storage: - -[tabs] -==== -Local:: -+ --- - -```bash -rpk topic create -c redpanda.remote.recovery=true -c redpanda.remote.write=true -c redpanda.remote.read=true -``` - --- -Kubernetes:: -+ --- - -```bash -kubectl exec -- rpk topic create -c redpanda.remote.recovery=true -c redpanda.remote.write=true -c redpanda.remote.read=true -``` - --- -==== - -=== Modify topic properties - -To modify topic properties of an existing topic, use the xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[rpk topic alter-config] command. - -For example, to modify a topic's `retention.ms` property: - -[tabs] -==== -Local:: -+ --- - -```bash -rpk topic alter-config --set retention.ms= -``` - --- -Kubernetes:: -+ --- - -```bash -kubectl exec -- rpk topic alter-config --set retention.ms= -``` - --- -==== - -== Disk space properties - -Configure properties to manage the disk space used by a topic: - -- Clean up log segments by deletion and/or compaction (<>). -- Control compaction timing with maximum and minimum lag settings (<> and <>). -- Retain logs up to a maximum size per partition before cleanup (<>). -- Retain logs for a maximum duration before cleanup (<>). -- Periodically close an active log segment (<>). -- Limit the maximum size of an active log segment (<>). -- Cache batches until the segment appender chunk is full, instead of fsyncing for every `acks=all` write (<>). With `write.caching` enabled, fsyncs follow <> and <>, whichever is reached first. - ---- +== Topic properties [[cleanuppolicy]] === cleanup.policy @@ -189,7 +84,6 @@ include::develop:partial$topic-properties-warning.adoc[] - `[compact,delete]` - The latest values are kept for each key, while the remaining data is deleted according to retention limits. - **Related topics**: - xref:manage:cluster-maintenance/disk-utilization.adoc#configure-segment-size[Configure segment size] @@ -356,7 +250,7 @@ When `write.caching` is set, it overrides the cluster property xref:cluster-prop --- -== Message properties +=== Message properties Configure properties for the messages of a topic: @@ -419,7 +313,7 @@ If `max.message.bytes` is set to a positive value, it overrides the cluster prop --- -== Tiered Storage properties +=== Tiered Storage properties Configure properties to manage topics for xref:manage:tiered-storage.adoc[Tiered Storage]: @@ -539,7 +433,7 @@ NOTE: `redpanda.remote.delete` doesn't apply to Remote Read Replica topics: a Re --- -== Remote Read Replica properties +=== Remote Read Replica properties Configure properties to manage topics for xref:manage:remote-read-replicas.adoc[Remote Read Replicas]. @@ -557,7 +451,7 @@ CAUTION: Setting `redpanda.remote.readreplica` together with either `redpanda.re --- -== Apache Iceberg integration properties +=== Apache Iceberg integration properties Integrate Redpanda topics as Iceberg tables. @@ -634,7 +528,7 @@ Controls how often the data in the Iceberg table is refreshed with new data from --- -== Redpanda topic properties +=== Redpanda topic properties Configure Redpanda-specific topic properties. @@ -700,7 +594,7 @@ NOTE: Although `replication.factor` isn't returned or displayed by xref:referenc -== Related topics +=== Related topics - xref:develop:produce-data/configure-producers.adoc[Configure Producers] - xref:develop:config-topics.adoc[Manage Topics] From 8cc0138aa6d49732b54890d7f1547770e65f3856 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Thu, 21 Aug 2025 21:12:36 -0300 Subject: [PATCH 02/22] reorganize and update table --- .../pages/properties/topic-properties.adoc | 70 ++++++++++++------- 1 file changed, 43 insertions(+), 27 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 0215f5981..d5347f73e 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -8,7 +8,7 @@ Many topic-level properties have corresponding xref:manage:cluster-maintenance/c For information on how to configure topic properties, see xref:manage:cluster-maintenance/topic-property-configuration.adoc[]. -NOTE: All topic properties take effect immediately after being set. +NOTE: All topic properties take effect immediately after being set. == Topic property mappings @@ -18,50 +18,66 @@ NOTE: All topic properties take effect immediately after being set. | <> | xref:./cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] -| <> -| xref:./cluster-properties.adoc#raft_replica_max_pending_flush_bytes[`raft_replica_max_pending_flush_bytes`] - -| <> -| xref:./cluster-properties.adoc#raft_replica_max_flush_delay_ms[`raft_replica_max_flush_delay_ms`] - -| <> -| xref:./cluster-properties.adoc#initial_retention_local_target_ms_default[`initial_retention_local_target_ms_default`] +| <> +| xref:./cluster-properties.adoc#compaction_strategy[`compaction_strategy`] -| <> -| xref:reference:cluster-properties.adoc#min_cleanable_dirty_ratio[`min_cleanable_dirty_ratio`] +| <> +| xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] -| <> -| xref:./cluster-properties.adoc#retention_bytes[`retention_bytes`] +| <> +| xref:./cluster-properties.adoc#delete_retention_ms[`delete_retention_ms`] -| <> -| xref:./cluster-properties.adoc#log_retention_ms[`log_retention_ms`] +| <> +| xref:./cluster-properties.adoc#flush_bytes[`flush_bytes`] -| <> -| xref:./cluster-properties.adoc#log_segment_ms[`log_segment_ms`] +| <> +| xref:./cluster-properties.adoc#flush_ms[`flush_ms`] -| <> -| xref:reference:properties/cluster-properties.adoc#log_segment_size[`log_segment_size`] +| <> +| xref:./cluster-properties.adoc#initial_retention_local_target_bytes[`initial_retention_local_target_bytes`] -| <> -| xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] +| <> +| xref:./cluster-properties.adoc#initial_retention_local_target_ms[`initial_retention_local_target_ms`] | <> | xref:./cluster-properties.adoc#max_compaction_lag_ms[`max_compaction_lag_ms`] -| <> -| xref:./cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`] +| <> +| xref:./cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] | <> | xref:./cluster-properties.adoc#log_message_timestamp_type[`log_message_timestamp_type`] -| <> -| xref:reference:properties/cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] +| <> +| xref:./cluster-properties.adoc#min_cleanable_dirty_ratio[`min_cleanable_dirty_ratio`] + +| <> +| xref:./cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`] | <> -| xref:./cluster-properties.adoc#default_topic_replication[`default_topic_replication`] +| xref:./cluster-properties.adoc#replication_factor[`replication_factor`] + +| <> +| xref:./cluster-properties.adoc#retention_bytes[`retention_bytes`] + +| <> +| xref:./cluster-properties.adoc#retention_local_target_bytes[`retention_local_target_bytes`] + +| <> +| xref:./cluster-properties.adoc#retention_local_target_ms[`retention_local_target_ms`] + +| <> +| xref:./cluster-properties.adoc#retention_ms[`retention_ms`] + +| <> +| xref:./cluster-properties.adoc#log_segment_size[`log_segment_size`] + +| <> +| xref:./cluster-properties.adoc#segment_ms[`segment_ms`] | <> -| xref:./cluster-properties.adoc#write_caching_default[`write_caching_default`] +| xref:./cluster-properties.adoc#write_caching[`write_caching`] + |=== == Topic properties From 41fe7fb6649c0df30ffffb2e900a1de7816b8c03 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 10:46:04 -0300 Subject: [PATCH 03/22] reorganize topic props --- .../pages/properties/topic-properties.adoc | 526 ++++++++++-------- 1 file changed, 305 insertions(+), 221 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index d5347f73e..be81439b9 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -82,8 +82,15 @@ NOTE: All topic properties take effect immediately after being set. == Topic properties +Properties are grouped by feature area. Within each group, properties are listed in alphabetical order. + +--- +=== Retention and Compaction Properties + +These properties control how data is stored, for how long, and when it is deleted or compacted. + [[cleanuppolicy]] -=== cleanup.policy +==== cleanup.policy The cleanup policy to apply for log segments of a topic. @@ -91,14 +98,19 @@ When `cleanup.policy` is set, it overrides the cluster property xref:cluster-pro include::develop:partial$topic-properties-warning.adoc[] -**Default**: `[delete]` +*Type:* string + +*Accepted values:* [`delete`, `compact`, `compact,delete`] + +**Default**: `delete` **Values**: -- `[delete]` - Deletes data according to size-based or time-based retention limits, or both. -- `[compact]` - Deletes data according to a key-based retention policy, discarding all but the latest value for each key. -- `[compact,delete]` - The latest values are kept for each key, while the remaining data is deleted according to retention limits. +- `delete` - Deletes data according to size-based or time-based retention limits, or both. +- `compact` - Deletes data according to a key-based retention policy, discarding all but the latest value for each key. +- `compact,delete` - The latest values are kept for each key, while the remaining data is deleted according to retention limits. +*Related cluster property:* xref:./cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] **Related topics**: @@ -106,82 +118,100 @@ include::develop:partial$topic-properties-warning.adoc[] - xref:manage:tiered-storage.adoc#compacted-topics-in-tiered-storage[Compacted topics in Tiered Storage] --- +[[compactionstrategy]] +==== compaction.strategy -[[flushms]] -=== flush.ms +Specifies the strategy used to determine which records to remove during log compaction. The compaction strategy controls how Redpanda identifies and removes duplicate records while preserving the latest value for each key. -The maximum delay (in ms) between two subsequent fsyncs. After this delay, the log is automatically fsynced. +*Type:* string -**Default**: `100` - -**Related topics**: +*Default:* `offset` -- xref:develop:produce-data/configure-producers.adoc[] +*Accepted values:* ---- +* `offset` - Uses record offsets to determine which records to compact (default and currently the only supported strategy) -[[flushbytes]] -=== flush.bytes +*Related cluster property:* xref:./cluster-properties.adoc#compaction_strategy[`compaction_strategy`] -The maximum bytes not fsynced per partition. If this configured threshold is reached, the log is automatically fsynced, even though it wasn't explicitly requested. +--- +[[deleteretentionms]] +==== delete.retention.ms -**Default**: `262144` +The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded. -**Related topics**: +If you have enabled Tiered Storage and set <> or <> for the topic, you cannot enable tombstone removal. -- xref:develop:produce-data/configure-producers.adoc[] +If both `delete.retention.ms` and the cluster property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] are set, `delete.retention.ms` overrides the cluster level tombstone retention for an individual topic. ---- +*Type:* integer -[[mincleanabledirtyratio]] -=== min.cleanable.dirty.ratio +*Accepted values:* [1, 9223372036854775] milliseconds -The minimum ratio between the number of bytes in dirty segments and the total number of bytes in closed segments that must be reached before a partition's log is eligible for compaction in a compact topic. +*Default*: null -**Default**: null +*Related cluster property:* xref:./cluster-properties.adoc#tombstone_retention_ms[`tombstone_retention_ms`] **Related topics**: -- xref:manage:cluster-maintenance/compaction-settings.adoc[] +- xref:manage:cluster-maintenance/compaction-settings.adoc#tombstone-record-removal[Tombstone record removal] --- - [[maxcompactionlagms]] -=== max.compaction.lag.ms +==== max.compaction.lag.ms The maximum amount of time (in ms) that a log segment can remain unaltered before it is eligible for compaction in a compact topic. Overrides the cluster property xref:cluster-properties.adoc#max_compaction_lag_ms[`max_compaction_lag_ms`] for the topic. *Type:* integer -*Accepted values:* [`1`, `9223372036854`] +*Accepted values:* [`1`, `9223372036854775`] + +**Default:* `9223372036854775` -*Default:* `9223372036854` +*Related cluster property:* xref:./cluster-properties.adoc#max_compaction_lag_ms[`max_compaction_lag_ms`] **Related topics**: - xref:manage:cluster-maintenance/compaction-settings.adoc#configuration-options[Configure maximum compaction lag] --- +[[mincleanabledirtyratio]] +==== min.cleanable.dirty.ratio + +The minimum ratio between the number of bytes in dirty segments and the total number of bytes in closed segments that must be reached before a partition's log is eligible for compaction in a compact topic. + +*Type:* number + +*Accepted values:* [`0`, `1.0`] + +**Default**: null + +*Related cluster property:* xref:./cluster-properties.adoc#min_cleanable_dirty_ratio[`min_cleanable_dirty_ratio`] +**Related topics**: + +- xref:manage:cluster-maintenance/compaction-settings.adoc[] + +--- [[mincompactionlagms]] -=== min.compaction.lag.ms +==== min.compaction.lag.ms The minimum amount of time (in ms) that a log segment must remain unaltered before it can be compacted in a compact topic. Overrides the cluster property xref:cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`] for the topic. *Type:* integer -*Accepted values:* [`0`, `9223372036854`] +*Accepted values:* [`0`, `9223372036854775`] -*Default:* `0` +**Default:* `0` + +*Related cluster property:* xref:./cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`] **Related topics**: - xref:manage:cluster-maintenance/compaction-settings.adoc#configure-min-compaction-lag[Configure minimum compaction lag] --- - [[retentionbytes]] -=== retention.bytes +==== retention.bytes A size-based retention limit that configures the maximum size that a topic partition can grow before becoming eligible for cleanup. @@ -189,16 +219,21 @@ If `retention.bytes` is set to a positive value, it overrides the cluster proper When both size-based (`retention.bytes`) and time-based (`retention.ms`) retention limits are set, cleanup occurs when either limit is reached. +*Type:* integer + +*Accepted values:* [1, 9223372036854775807] bytes + **Default**: null +*Related cluster property:* xref:./cluster-properties.adoc#retention_bytes[`retention_bytes`] + **Related topics**: - xref:manage:cluster-maintenance/disk-utilization.adoc#configure-message-retention[Configure message retention] --- - [[retentionms]] -=== retention.ms +==== retention.ms A time-based retention limit that configures the maximum duration that a log's segment file for a topic is retained before it becomes eligible to be cleaned up. To consume all data, a consumer of the topic must read from a segment before its `retention.ms` elapses, otherwise the segment may be compacted and/or deleted. If a non-positive value, no per-topic limit is applied. @@ -206,233 +241,282 @@ If `retention.ms` is set to a positive value, it overrides the cluster property When both size-based (`retention.bytes`) and time-based (`retention.ms`) retention limits are set, the earliest occurring limit applies. +*Type:* integer + +*Accepted values:* [1, 9223372036854775] milliseconds + **Default**: null +*Related cluster property:* xref:./cluster-properties.adoc#log_retention_ms[`log_retention_ms`] + **Related topics**: - xref:manage:cluster-maintenance/disk-utilization.adoc#configure-message-retention[Configure message retention] --- +### Segment and Message Properties -[[segmentms]] -=== segment.ms +These properties control the size and lifecycle of log segment files and settings for individual messages. -The maximum duration that a log segment of a topic is active (open for writes and not deletable). A periodic event, with `segment.ms` as its period, forcibly closes the active segment and transitions, or rolls, to a new active segment. The closed (inactive) segment is then eligible to be cleaned up according to cleanup and retention properties. +[[compressiontype]] +==== compression.type -If set to a positive duration, `segment.ms` overrides the cluster property xref:./cluster-properties.adoc#log_segment_ms[`log_segment_ms`] and its lower and upper bounds set by xref:./cluster-properties.adoc#log_segment_ms_min[`log_segment_ms_min`] and xref:./cluster-properties.adoc#log_segment_ms_max[`log_segment_ms_max`], respectively. +IMPORTANT: This property is ignored regardless of the value specified. The behavior is always the same as the `producer` value. If producers send compressed data, Redpanda stores and serves it as-is; if producers send uncompressed data, Redpanda stores it uncompressed. Other listed values are accepted for Apache Kafka compatibility but are ignored by the broker. This property may appear in Admin API and `rpk topic describe` outputs for compatibility. -**Default**: null +Redpanda does not enforce or change compression for stored data; configure compression in your producers. -**Related topics**: +Enabling compression reduces message size, which improves throughput and decreases storage for messages with repetitive values and data structures. The trade-off is increased CPU utilization and network latency to perform the compression. You can also enable producer batching to increase compression efficiency, since the messages in a batch likely have repeated data that can be compressed. -- xref:manage:cluster-maintenance/disk-utilization.adoc#log-rolling[Log rolling] +When `compression.type` is set, it overrides the cluster property xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] for the topic. ---- +NOTE: The valid values of `compression.type` are taken from `log_compression_type` and differ from Kafka's compression types. -[[segmentbytes]] -=== segment.bytes +*Type:* string -The maximum size of an active log segment for a topic. When the size of an active segment exceeds `segment.bytes`, the segment is closed and a new active segment is created. The closed, inactive segment is then eligible to be cleaned up according to retention properties. +*Accepted Values:* `producer`. The following values are accepted for Kafka compatibility but ignored by the broker: `gzip`, `snappy`, `lz4`, `zstd`, `none`. -When `segment.bytes` is set to a positive value, it overrides the cluster property xref:reference:properties/cluster-properties.adoc#log_segment_size[`log_segment_size`] for the topic. +**Default**: `none` -**Default**: null +*Related cluster property:* xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] **Related topics**: -- xref:manage:cluster-maintenance/disk-utilization.adoc#configure-segment-size[Configure segment size] -- xref:manage:cluster-maintenance/disk-utilization.adoc#configure-message-retention[Configure message retention] -- xref:manage:remote-read-replicas.adoc[Remote Read Replicas] +- xref:develop:produce-data/configure-producers.adoc#message-batching[Message batching] +- xref:develop:produce-data/configure-producers.adoc#commonly-used-producer-configuration-options[Common producer configuration options] --- +[[maxmessagebytes]] +==== max.message.bytes -[[writecaching]] -=== write.caching +The maximum size of a message or batch of a topic. If a compression type is enabled, `max.message.bytes` sets the maximum size of the compressed message or batch. -The write caching mode to apply to a topic. +If `max.message.bytes` is set to a positive value, it overrides the cluster property xref:reference:properties/cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] for the topic. -When `write.caching` is set, it overrides the cluster property xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. Fsyncs follow <> and <>, whichever is reached first. +*Type:* integer -**Default**: `false` +*Accepted values:* [1, 2147483647] bytes -**Values**: +**Default**: null -- `true` - Enables write caching for a topic, according to <> and <>. -- `false` - Disables write caching for a topic, according to <> and <>. +*Related cluster property:* xref:./cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] **Related topics**: -- xref:develop:config-topics.adoc#configure-write-caching[Write caching] +- xref:develop:produce-data/configure-producers.adoc#message-batching[Message batching] --- +[[messagetimestamptype]] +==== message.timestamp.type -=== Message properties - -Configure properties for the messages of a topic: - -- Compress a message or batch to reduce storage space and increase throughput (<>). -- Set the source of a message's timestamp (<>). -- Set the maximum size of a message (<>). - -[[compressiontype]] -=== compression.type +The source of a message's timestamp: either the message's creation time or its log append time. -IMPORTANT: This property is ignored regardless of the value specified. The behavior is always the same as the `producer` value. If producers send compressed data, Redpanda stores and serves it as-is; if producers send uncompressed data, Redpanda stores it uncompressed. Other listed values are accepted for Apache Kafka compatibility but are ignored by the broker. This property may appear in Admin API and `rpk topic describe` outputs for compatibility. +When `message.timestamp.type` is set, it overrides the cluster property xref:./cluster-properties.adoc#log_message_timestamp_type[`log_message_timestamp_type`] for the topic. -Redpanda does not enforce or change compression for stored data; configure compression in your producers. +*Type:* string -Enabling compression reduces message size, which improves throughput and decreases storage for messages with repetitive values and data structures. The trade-off is increased CPU utilization and network latency to perform the compression. You can also enable producer batching to increase compression efficiency, since the messages in a batch likely have repeated data that can be compressed. +*Accepted values:* [`CreateTime`, `LogAppendTime`] -When `compression.type` is set, it overrides the cluster property xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] for the topic. +**Default**: `CreateTime` -NOTE: The valid values of `compression.type` are taken from `log_compression_type` and differ from Kafka's compression types. +**Values**: -**Default**: `none` +- `CreateTime` +- `LogAppendTime` -*Accepted Values:* `producer`. The following values are accepted for Kafka compatibility but ignored by the broker: `gzip`, `snappy`, `lz4`, `zstd`, `none`. +*Related cluster property:* xref:./cluster-properties.adoc#log_message_timestamp_type[`log_message_timestamp_type`] -**Related topics**: +--- +[[segmentbytes]] +==== segment.bytes -- xref:develop:produce-data/configure-producers.adoc#message-batching[Message batching] -- xref:develop:produce-data/configure-producers.adoc#commonly-used-producer-configuration-options[Common producer configuration options] +The maximum size of an active log segment for a topic. When the size of an active segment exceeds `segment.bytes`, the segment is closed and a new active segment is created. The closed, inactive segment is then eligible to be cleaned up according to retention properties. ---- +When `segment.bytes` is set to a positive value, it overrides the cluster property xref:reference:properties/cluster-properties.adoc#log_segment_size[`log_segment_size`] for the topic. -[[messagetimestamptype]] -=== message.timestamp.type +*Type:* integer -The source of a message's timestamp: either the message's creation time or its log append time. +*Accepted values:* [1, 9223372036854775807] bytes -When `message.timestamp.type` is set, it overrides the cluster property xref:./cluster-properties.adoc#log_message_timestamp_type[`log_message_timestamp_type`] for the topic. +**Default**: null -**Default**: `CreateTime` +*Related cluster property:* xref:./cluster-properties.adoc#log_segment_size[`log_segment_size`] -**Values**: +**Related topics**: -- `CreateTime` -- `LogAppendTime` +- xref:manage:cluster-maintenance/disk-utilization.adoc#configure-segment-size[Configure segment size] +- xref:manage:cluster-maintenance/disk-utilization.adoc#configure-message-retention[Configure message retention] +- xref:manage:remote-read-replicas.adoc[Remote Read Replicas] --- +[[segmentms]] +==== segment.ms -[[maxmessagebytes]] -=== max.message.bytes +The maximum duration that a log segment of a topic is active (open for writes and not deletable). A periodic event, with `segment.ms` as its period, forcibly closes the active segment and transitions, or rolls, to a new active segment. The closed (inactive) segment is then eligible to be cleaned up according to cleanup and retention properties. -The maximum size of a message or batch of a topic. If a compression type is enabled, `max.message.bytes` sets the maximum size of the compressed message or batch. +If set to a positive duration, `segment.ms` overrides the cluster property xref:./cluster-properties.adoc#log_segment_ms[`log_segment_ms`] and its lower and upper bounds set by xref:./cluster-properties.adoc#log_segment_ms_min[`log_segment_ms_min`] and xref:./cluster-properties.adoc#log_segment_ms_max[`log_segment_ms_max`], respectively. -If `max.message.bytes` is set to a positive value, it overrides the cluster property xref:reference:properties/cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] for the topic. +*Type:* integer + +*Accepted values:* [600000, 31536000000] milliseconds (clamped by cluster bounds) **Default**: null +*Related cluster property:* xref:./cluster-properties.adoc#segment_ms[`segment_ms`] + **Related topics**: -- xref:develop:produce-data/configure-producers.adoc#message-batching[Message batching] +- xref:manage:cluster-maintenance/disk-utilization.adoc#log-rolling[Log rolling] --- +### Performance and Cluster Properties -=== Tiered Storage properties +These properties control disk flushing, replication, and how topics interact with the cluster. -Configure properties to manage topics for xref:manage:tiered-storage.adoc[Tiered Storage]: +[[flushbytes]] +==== flush.bytes -- Upload and fetch data to and from object storage for a topic (<> and <>). -- Configure size-based and time-based retention properties for local storage of a topic (<> and <>). -- Recover or reproduce data for a topic from object storage (<>). -- Delete data from object storage for a topic when it's deleted from local storage (<>). +The maximum bytes not fsynced per partition. If this configured threshold is reached, the log is automatically fsynced, even though it wasn't explicitly requested. -[[redpandaremotewrite]] -=== redpanda.remote.write +*Type:* integer -A flag for enabling Redpanda to upload data for a topic from local storage to object storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. +*Accepted values:* [1, 9223372036854775807] bytes -**Default**: false +**Default**: `262144` + +*Related cluster property:* xref:./cluster-properties.adoc#flush_bytes[`flush_bytes`] **Related topics**: -- xref:manage:tiered-storage.adoc[Tiered Storage] +- xref:develop:produce-data/configure-producers.adoc[] --- +[[flushms]] +==== flush.ms -[[redpandaremoteread]] -=== redpanda.remote.read +The maximum delay (in ms) between two subsequent fsyncs. After this delay, the log is automatically fsynced. -A flag for enabling Redpanda to fetch data for a topic from object storage to local storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. +*Type:* integer -**Default**: false +*Accepted values:* [1, 9223372036854775] milliseconds + +**Default**: `100` + +*Related cluster property:* xref:./cluster-properties.adoc#flush_ms[`flush_ms`] **Related topics**: -- xref:manage:tiered-storage.adoc[Tiered Storage] +- xref:develop:produce-data/configure-producers.adoc[] --- +[[redpandaleaderspreference]] +==== redpanda.leaders.preference -[[initialretentionlocaltargetbytes]] -=== initial.retention.local.target.bytes +The preferred location (rack) for partition leaders of a topic. -A size-based initial retention limit for Tiered Storage that determines how much data in local storage is transferred to a partition replica when a cluster is resized. If `null` (default), all locally retained data is transferred. +This property inherits the value from the config_ref:default_leaders_preference,true,properties/cluster-properties[] cluster configuration property. You may override the cluster-wide setting by specifying the value for individual topics. -**Default**: null +If the cluster configuration property config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, Leader Pinning is disabled across the cluster. + +**Default**: `none` + +**Values**: + +- `none`: Opt out the topic from Leader Pinning. +- `racks:[,,...]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more rack IDs. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks. **Related topics**: -- xref:manage:tiered-storage.adoc#fast-commission-and-decommission[Fast commission and decommission through Tiered Storage] +- xref:develop:produce-data/leader-pinning.adoc[Leader pinning] --- +[[replicationfactor]] +==== replication.factor -[[initialretentionlocaltargetms]] -=== initial.retention.local.target.ms +The number of replicas of a topic to save in different nodes (brokers) of a cluster. -A time-based initial retention limit for Tiered Storage that determines how much data in local storage is transferred to a partition replica when a cluster is resized. If `null` (default), all locally retained data is transferred. +If `replication.factor` is set to a positive value, it overrides the cluster property xref:./cluster-properties.adoc#default_topic_replication[default_topic_replication] for the topic. + +NOTE: Although `replication.factor` isn't returned or displayed by xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`] as a valid Kafka property, you can set it using xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`]. When the `replication.factor` of a topic is altered, it isn't simply a property value that's updated, but rather the actual replica sets of topic partitions that are changed. + +*Type:* integer + +*Accepted values:* [`1`, `512`] **Default**: null +*Related cluster property:* xref:./cluster-properties.adoc#default_topic_replication[`default_topic_replication`] + **Related topics**: -- xref:manage:tiered-storage.adoc#fast-commission-and-decommission[Fast commission and decommission through Tiered Storage] +- xref:develop:config-topics.adoc#choose-the-replication-factor.adoc[Choose the replication factor] +- xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor] --- +[[writecaching]] +==== write.caching -[[retentionlocaltargetbytes]] -=== retention.local.target.bytes +The write caching mode to apply to a topic. -A size-based retention limit for Tiered Storage that configures the maximum size that a topic partition in local storage can grow before becoming eligible for cleanup. It applies per partition and is equivalent to <> without Tiered Storage. +When `write.caching` is set, it overrides the cluster property xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. Fsyncs follow <> and <>, whichever is reached first. -**Default**: null +*Type:* boolean + +**Default**: `false` + +**Values**: + +- `true` - Enables write caching for a topic, according to <> and <>. +- `false` - Disables write caching for a topic, according to <> and <>. + +*Related cluster property:* xref:./cluster-properties.adoc#write_caching_default[`write_caching_default`] **Related topics**: -- xref:manage:tiered-storage.adoc[Tiered Storage] +- xref:develop:config-topics.adoc#configure-write-caching[Write caching] --- +### Tiered Storage properties -[[retentionlocaltargetms]] -=== retention.local.target.ms +Configure properties to manage topics for xref:manage:tiered-storage.adoc[Tiered Storage]. -A time-based retention limit for Tiered Storage that sets the maximum duration that a log's segment file for a topic is retained in local storage before it's eligible for cleanup. This property is equivalent to <> without Tiered Storage. +[[initialretentionlocaltargetbytes]] +==== initial.retention.local.target.bytes -**Default**: 86400000 +A size-based initial retention limit for Tiered Storage that determines how much data in local storage is transferred to a partition replica when a cluster is resized. If `null` (default), all locally retained data is transferred. + +*Type:* integer + +*Accepted values:* [1, 9223372036854775807] bytes + +**Default**: null + +*Related cluster property:* xref:./cluster-properties.adoc#initial_retention_local_target_bytes[`initial_retention_local_target_bytes`] **Related topics**: -- xref:manage:tiered-storage.adoc[Tiered Storage] +- xref:manage:tiered-storage.adoc#fast-commission-and-decommission[Fast commission and decommission through Tiered Storage] --- +[[initialretentionlocaltargetms]] +==== initial.retention.local.target.ms -[[redpandaremoterecovery]] -=== redpanda.remote.recovery +A time-based initial retention limit for Tiered Storage that determines how much data in local storage is transferred to a partition replica when a cluster is resized. If `null` (default), all locally retained data is transferred. -A flag that enables the recovery or reproduction of a topic from object storage for Tiered Storage. The recovered data is saved in local storage, and the maximum amount of recovered data is determined by the local storage retention limits of the topic. +*Type:* integer -TIP: You can only configure `redpanda.remote.recovery` when you create a topic. You cannot apply this setting to existing topics. +*Accepted values:* [1, 9223372036854775] milliseconds -**Default**: false +**Default**: null + +*Related cluster property:* xref:./cluster-properties.adoc#initial_retention_local_target_ms[`initial_retention_local_target_ms`] **Related topics**: -- xref:manage:tiered-storage.adoc[Tiered Storage] +- xref:manage:tiered-storage.adoc#fast-commission-and-decommission[Fast commission and decommission through Tiered Storage] --- - [[redpandaremotedelete]] -=== redpanda.remote.delete +==== redpanda.remote.delete A flag that enables deletion of data from object storage for Tiered Storage when it's deleted from local storage for a topic. @@ -448,170 +532,170 @@ NOTE: `redpanda.remote.delete` doesn't apply to Remote Read Replica topics: a Re - xref:manage:tiered-storage.adoc[Tiered Storage] --- +[[redpandaremoteread]] +==== redpanda.remote.read -=== Remote Read Replica properties - -Configure properties to manage topics for xref:manage:remote-read-replicas.adoc[Remote Read Replicas]. - -=== redpanda.remote.readreplica - -The name of the object storage bucket for a Remote Read Replica topic. - -CAUTION: Setting `redpanda.remote.readreplica` together with either `redpanda.remote.read` or `redpanda.remote.write` results in an error. +A flag for enabling Redpanda to fetch data for a topic from object storage to local storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. -**Default**: null +**Default**: false **Related topics**: -- xref:manage:remote-read-replicas.adoc[Remote Read Replicas] +- xref:manage:tiered-storage.adoc[Tiered Storage] --- +[[redpandaremoterecovery]] +==== redpanda.remote.recovery -=== Apache Iceberg integration properties +A flag that enables the recovery or reproduction of a topic from object storage for Tiered Storage. The recovered data is saved in local storage, and the maximum amount of recovered data is determined by the local storage retention limits of the topic. -Integrate Redpanda topics as Iceberg tables. +TIP: You can only configure `redpanda.remote.recovery` when you create a topic. You cannot apply this setting to existing topics. -=== redpanda.iceberg.mode +**Default**: false -Enable the Iceberg integration for the topic. You can choose one of four modes. +**Related topics**: -**Default**: `disabled` +- xref:manage:tiered-storage.adoc[Tiered Storage] -**Values**: +--- +[[redpandaremotewrite]] +==== redpanda.remote.write -- `key_value`: Creates an Iceberg table with a `Key` column and a `Value` column. Redpanda stores the raw topic content in the `Value` column. -- `value_schema_id_prefix`: Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. Redpanda uses the Schema Registry wire format, consisting of the "magic byte" and schema ID encoded in the payload header, to parse the topic values per field and store them in the corresponding table columns. -- `value_schema_latest`: Creates an Iceberg table whose structure matches the latest schema version in Schema Registry that matches the subject name. This mode is compatible with Avro and Protobuf schemas and is used when you don't produce to the topic using the wire format. See xref:manage:iceberg/choose-iceberg-mode.adoc#override-value-schema-latest-default[Choose an Iceberg Mode] for details on using this mode. -- `disabled`: Disables writing to an Iceberg table for the topic. +A flag for enabling Redpanda to upload data for a topic from local storage to object storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. + +**Default**: false **Related topics**: -- xref:manage:iceberg/choose-iceberg-mode.adoc[] -- xref:manage:iceberg/about-iceberg-topics.adoc[] +- xref:manage:tiered-storage.adoc[Tiered Storage] --- +[[retentionlocaltargetbytes]] +==== retention.local.target.bytes -=== redpanda.iceberg.delete +A size-based retention limit for Tiered Storage that configures the maximum size that a topic partition in local storage can grow before becoming eligible for cleanup. It applies per partition and is equivalent to <> without Tiered Storage. -Whether the corresponding Iceberg table is deleted upon deleting the topic. +*Type:* integer -**Default**: `true` +*Accepted values:* [1, 9223372036854775807] bytes + +**Default**: null + +*Related cluster property:* xref:./cluster-properties.adoc#retention_local_target_bytes[`retention_local_target_bytes`] **Related topics**: -- xref:manage:iceberg/about-iceberg-topics.adoc[] +- xref:manage:tiered-storage.adoc[Tiered Storage] --- +[[retentionlocaltargetms]] +==== retention.local.target.ms -=== redpanda.iceberg.invalid.record.action +A time-based retention limit for Tiered Storage that sets the maximum duration that a log's segment file for a topic is retained in local storage before it's eligible for cleanup. This property is equivalent to <> without Tiered Storage. -Whether to write invalid records to a dead-letter queue (DLQ). +*Type:* integer -**Default**: `dlq_table` +*Accepted values:* [1, 9223372036854775] milliseconds -**Values**: +**Default**: 86400000 -- `drop`: Disable the DLQ and drop invalid records. -- `dlq_table`: Write invalid records to a separate DLQ Iceberg table. +*Related cluster property:* xref:./cluster-properties.adoc#retention_local_target_ms[`retention_local_target_ms`] **Related topics**: -- xref:manage:iceberg/about-iceberg-topics.adoc#troubleshoot-errors[Troubleshoot errors] +- xref:manage:tiered-storage.adoc[Tiered Storage] --- +### Remote Read Replica properties -=== redpanda.iceberg.partition.spec +Configure properties to manage topics for xref:manage:remote-read-replicas.adoc[Remote Read Replicas]. -The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table. +[[redpandaremotereadreplica]] +==== redpanda.remote.readreplica -**Default**: `(hour(redpanda.timestamp))` +The name of the object storage bucket for a Remote Read Replica topic. + +CAUTION: Setting `redpanda.remote.readreplica` together with either `redpanda.remote.read` or `redpanda.remote.write` results in an error. + +**Default**: null **Related topics**: -- xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[Use custom partitioning] +- xref:manage:remote-read-replicas.adoc[Remote Read Replicas] --- +### Apache Iceberg integration properties -=== redpanda.iceberg.target.lag.ms +Integrate Redpanda topics as Iceberg tables. -Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. +[[redpandaicebergdelete]] +==== redpanda.iceberg.delete -**Default**: `60000` +Whether the corresponding Iceberg table is deleted upon deleting the topic. + +**Default**: `true` **Related topics**: - xref:manage:iceberg/about-iceberg-topics.adoc[] --- +[[redpandaiceberginvalidrecordaction]] +==== redpanda.iceberg.invalid.record.action -=== Redpanda topic properties - -Configure Redpanda-specific topic properties. - ---- - -[[deleteretentionms]] -=== delete.retention.ms - -The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded. - -If you have enabled Tiered Storage and set <> or <> for the topic, you cannot enable tombstone removal. +Whether to write invalid records to a dead-letter queue (DLQ). -If both `delete.retention.ms` and the cluster property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] are set, `delete.retention.ms` overrides the cluster level tombstone retention for an individual topic. +**Default**: `dlq_table` -*Unit:* milliseconds +**Values**: -**Default**: null +- `drop`: Disable the DLQ and drop invalid records. +- `dlq_table`: Write invalid records to a separate DLQ Iceberg table. **Related topics**: -- xref:manage:cluster-maintenance/compaction-settings.adoc#tombstone-record-removal[Tombstone record removal] +- xref:manage:iceberg/about-iceberg-topics.adoc#troubleshoot-errors[Troubleshoot errors] --- +[[redpandaicebergmode]] +==== redpanda.iceberg.mode -[[redpandaleaderspreference]] -=== redpanda.leaders.preference - -The preferred location (rack) for partition leaders of a topic. - -This property inherits the value from the config_ref:default_leaders_preference,true,properties/cluster-properties[] cluster configuration property. You may override the cluster-wide setting by specifying the value for individual topics. - -If the cluster configuration property config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, Leader Pinning is disabled across the cluster. +Enable the Iceberg integration for the topic. You can choose one of four modes. -**Default**: `none` +**Default**: `disabled` **Values**: -- `none`: Opt out the topic from Leader Pinning. -- `racks:[,,...]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more rack IDs. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks. +- `key_value`: Creates an Iceberg table with a `Key` column and a `Value` column. Redpanda stores the raw topic content in the `Value` column. +- `value_schema_id_prefix`: Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. Redpanda uses the Schema Registry wire format, consisting of the "magic byte" and schema ID encoded in the payload header, to parse the topic values per field and store them in the corresponding table columns. +- `value_schema_latest`: Creates an Iceberg table whose structure matches the latest schema version in Schema Registry that matches the subject name. This mode is compatible with Avro and Protobuf schemas and is used when you don't produce to the topic using the wire format. See xref:manage:iceberg/choose-iceberg-mode.adoc#override-value-schema-latest-default[Choose an Iceberg Mode] for details on using this mode. +- `disabled`: Disables writing to an Iceberg table for the topic. **Related topics**: -- xref:develop:produce-data/leader-pinning.adoc[Leader pinning] +- xref:manage:iceberg/choose-iceberg-mode.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc[] --- +[[redpandaicebergpartitionspec]] +==== redpanda.iceberg.partition.spec -=== replication.factor - -The number of replicas of a topic to save in different nodes (brokers) of a cluster. - -If `replication.factor` is set to a positive value, it overrides the cluster property xref:./cluster-properties.adoc#default_topic_replication[default_topic_replication] for the topic. - -NOTE: Although `replication.factor` isn't returned or displayed by xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`] as a valid Kafka property, you can set it using xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`]. When the `replication.factor` of a topic is altered, it isn't simply a property value that's updated, but rather the actual replica sets of topic partitions that are changed. +The [https://iceberg.apache.org/docs/nightly/partitioning/](https://iceberg.apache.org/docs/nightly/partitioning/)[partitioning^] specification for the Iceberg table. -**Default**: null +**Default**: `(hour(redpanda.timestamp))` **Related topics**: -- xref:develop:config-topics.adoc#choose-the-replication-factor.adoc[Choose the replication factor] -- xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor] +- xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[Use custom partitioning] --- +[[redpandaicebergtargetlagms]] +==== redpanda.iceberg.target.lag.ms +Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. +**Default**: `60000` -=== Related topics - -- xref:develop:produce-data/configure-producers.adoc[Configure Producers] -- xref:develop:config-topics.adoc[Manage Topics] +**Related topics**: +- xref:manage:iceberg/about-iceberg-topics.adoc[] \ No newline at end of file From 1587c13d5a609e82e73acffca98066e6c8f7d715 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 11:24:56 -0300 Subject: [PATCH 04/22] add to nav --- modules/ROOT/nav.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index d2b45ab9f..4a0d4568e 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -150,6 +150,7 @@ ** xref:manage:cluster-maintenance/index.adoc[Cluster Maintenance] *** xref:manage:cluster-maintenance/cluster-property-configuration.adoc[] *** xref:manage:cluster-maintenance/node-property-configuration.adoc[] +*** xref:manage:cluster-maintenance/topic-property-configuration.adoc[] *** xref:manage:cluster-maintenance/cluster-balancing.adoc[] *** xref:manage:cluster-maintenance/continuous-data-balancing.adoc[Continuous Data Balancing] *** xref:manage:cluster-maintenance/decommission-brokers.adoc[Decommission Brokers] From d1de2d9b7522436da7f157dd6eec6e50567d2eb6 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 11:33:10 -0300 Subject: [PATCH 05/22] Apply suggestions from code review Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --- .../cluster-maintenance/topic-property-configuration.adoc | 3 +-- modules/reference/pages/properties/topic-properties.adoc | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc index 8f9fa8c8d..5947122a6 100644 --- a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -54,8 +54,7 @@ Kubernetes:: -- ```bash -kubectl exec -- rpk topic create -c cleanup.policy=compact -``` +kubectl exec -- rpk topic create -c cleanup.policy=compact -- ==== diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index be81439b9..0953c1b28 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -25,8 +25,7 @@ NOTE: All topic properties take effect immediately after being set. | xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] | <> -| xref:./cluster-properties.adoc#delete_retention_ms[`delete_retention_ms`] - +| xref:./cluster-properties.adoc#tombstone_retention_ms[`tombstone_retention_ms`] | <> | xref:./cluster-properties.adoc#flush_bytes[`flush_bytes`] From 71439e10d0610f3e1304ff6a493819bdd9df7ff1 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 13:09:13 -0300 Subject: [PATCH 06/22] Update modules/reference/pages/properties/topic-properties.adoc --- modules/reference/pages/properties/topic-properties.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 0953c1b28..e8e2ad351 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -679,7 +679,7 @@ Enable the Iceberg integration for the topic. You can choose one of four modes. [[redpandaicebergpartitionspec]] ==== redpanda.iceberg.partition.spec -The [https://iceberg.apache.org/docs/nightly/partitioning/](https://iceberg.apache.org/docs/nightly/partitioning/)[partitioning^] specification for the Iceberg table. +The link:https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table. **Default**: `(hour(redpanda.timestamp))` From 121f0cd2764df4539b9fc7c01f7f1e20dcc4f9e1 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 17:20:01 -0300 Subject: [PATCH 07/22] code review 5:19 --- modules/reference/pages/properties/topic-properties.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index e8e2ad351..3883d87ca 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -66,16 +66,16 @@ NOTE: All topic properties take effect immediately after being set. | xref:./cluster-properties.adoc#retention_local_target_ms[`retention_local_target_ms`] | <> -| xref:./cluster-properties.adoc#retention_ms[`retention_ms`] +| xref:./cluster-properties.adoc#log_retention_ms[`log_retention_ms`] | <> | xref:./cluster-properties.adoc#log_segment_size[`log_segment_size`] | <> -| xref:./cluster-properties.adoc#segment_ms[`segment_ms`] +| xref:./cluster-properties.adoc#log_segment_ms[`log_segment_ms`] | <> -| xref:./cluster-properties.adoc#write_caching[`write_caching`] +| xref:./cluster-properties.adoc#write_caching_default[`write_caching_default`] |=== @@ -360,7 +360,7 @@ If set to a positive duration, `segment.ms` overrides the cluster property xref: **Default**: null -*Related cluster property:* xref:./cluster-properties.adoc#segment_ms[`segment_ms`] +*Related cluster property:* xref:./cluster-properties.adoc#log_segment_ms[`log_segment_ms`] **Related topics**: From b93217d34ca37d042028289e30a5e05dd3b55035 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 17:25:48 -0300 Subject: [PATCH 08/22] review points on linking --- modules/reference/pages/properties/topic-properties.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 3883d87ca..545a9fc82 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -289,7 +289,7 @@ NOTE: The valid values of `compression.type` are taken from `log_compression_typ The maximum size of a message or batch of a topic. If a compression type is enabled, `max.message.bytes` sets the maximum size of the compressed message or batch. -If `max.message.bytes` is set to a positive value, it overrides the cluster property xref:reference:properties/cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] for the topic. +If `max.message.bytes` is set to a positive value, it overrides the cluster property xref:./cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] for the topic. *Type:* integer @@ -330,7 +330,7 @@ When `message.timestamp.type` is set, it overrides the cluster property xref:./c The maximum size of an active log segment for a topic. When the size of an active segment exceeds `segment.bytes`, the segment is closed and a new active segment is created. The closed, inactive segment is then eligible to be cleaned up according to retention properties. -When `segment.bytes` is set to a positive value, it overrides the cluster property xref:reference:properties/cluster-properties.adoc#log_segment_size[`log_segment_size`] for the topic. +When `segment.bytes` is set to a positive value, it overrides the cluster property xref:./cluster-properties.adoc#log_segment_size[`log_segment_size`] for the topic. *Type:* integer @@ -447,7 +447,7 @@ NOTE: Although `replication.factor` isn't returned or displayed by xref:referenc **Related topics**: -- xref:develop:config-topics.adoc#choose-the-replication-factor.adoc[Choose the replication factor] +- xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor] - xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor] --- From f2b20930e247564cabaaa7890e4d97e57c1b51cd Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 17:28:34 -0300 Subject: [PATCH 09/22] Simplify entry for compression type. --- .../pages/properties/topic-properties.adoc | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 545a9fc82..66050b77d 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -260,25 +260,23 @@ These properties control the size and lifecycle of log segment files and setting [[compressiontype]] ==== compression.type -IMPORTANT: This property is ignored regardless of the value specified. The behavior is always the same as the `producer` value. If producers send compressed data, Redpanda stores and serves it as-is; if producers send uncompressed data, Redpanda stores it uncompressed. Other listed values are accepted for Apache Kafka compatibility but are ignored by the broker. This property may appear in Admin API and `rpk topic describe` outputs for compatibility. +Redpanda ignores this property and always uses producer compression semantics. If producers send compressed data, Redpanda stores and serves it as-is. If producers send uncompressed data, Redpanda stores it uncompressed. -Redpanda does not enforce or change compression for stored data; configure compression in your producers. +This property exists for Apache Kafka compatibility. Configure compression in your producers instead of using this topic property. -Enabling compression reduces message size, which improves throughput and decreases storage for messages with repetitive values and data structures. The trade-off is increased CPU utilization and network latency to perform the compression. You can also enable producer batching to increase compression efficiency, since the messages in a batch likely have repeated data that can be compressed. +Compression reduces message size and improves throughput, but increases CPU utilization. Enable producer batching to increase compression efficiency. -When `compression.type` is set, it overrides the cluster property xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] for the topic. - -NOTE: The valid values of `compression.type` are taken from `log_compression_type` and differ from Kafka's compression types. +When set, this property overrides the cluster property xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] for the topic. *Type:* string -*Accepted Values:* `producer`. The following values are accepted for Kafka compatibility but ignored by the broker: `gzip`, `snappy`, `lz4`, `zstd`, `none`. +*Accepted Values:* `producer`. The following values are accepted for Kafka compatibility but ignored: `gzip`, `snappy`, `lz4`, `zstd`, `none`. -**Default**: `none` +*Default:* `producer` *Related cluster property:* xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`] -**Related topics**: +*Related topics:* - xref:develop:produce-data/configure-producers.adoc#message-batching[Message batching] - xref:develop:produce-data/configure-producers.adoc#commonly-used-producer-configuration-options[Common producer configuration options] From 66ea5f1d8bf4111e3611c66b702cc2aa22969977 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 18:08:42 -0300 Subject: [PATCH 10/22] code review --- modules/reference/pages/properties/topic-properties.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 66050b77d..bf1678949 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -54,7 +54,7 @@ NOTE: All topic properties take effect immediately after being set. | xref:./cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`] | <> -| xref:./cluster-properties.adoc#replication_factor[`replication_factor`] +| xref:./cluster-properties.adoc#default_topic_replications[`default_topic_replications`] | <> | xref:./cluster-properties.adoc#retention_bytes[`retention_bytes`] From 9dcc8bb77bab5fff2ebbc66a2448cb21c301e17f Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 18:08:48 -0300 Subject: [PATCH 11/22] fix default # Conflicts: # modules/reference/pages/properties/topic-properties.adoc --- .../pages/properties/topic-properties.adoc | 54 +++++++++---------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index bf1678949..b1dccc294 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -101,7 +101,7 @@ include::develop:partial$topic-properties-warning.adoc[] *Accepted values:* [`delete`, `compact`, `compact,delete`] -**Default**: `delete` +*Default:* `delete` **Values**: @@ -182,7 +182,7 @@ The minimum ratio between the number of bytes in dirty segments and the total nu *Accepted values:* [`0`, `1.0`] -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#min_cleanable_dirty_ratio[`min_cleanable_dirty_ratio`] @@ -222,7 +222,7 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti *Accepted values:* [1, 9223372036854775807] bytes -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#retention_bytes[`retention_bytes`] @@ -244,7 +244,7 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti *Accepted values:* [1, 9223372036854775] milliseconds -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#log_retention_ms[`log_retention_ms`] @@ -293,7 +293,7 @@ If `max.message.bytes` is set to a positive value, it overrides the cluster prop *Accepted values:* [1, 2147483647] bytes -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] @@ -313,7 +313,7 @@ When `message.timestamp.type` is set, it overrides the cluster property xref:./c *Accepted values:* [`CreateTime`, `LogAppendTime`] -**Default**: `CreateTime` +*Default:* `CreateTime` **Values**: @@ -334,7 +334,7 @@ When `segment.bytes` is set to a positive value, it overrides the cluster proper *Accepted values:* [1, 9223372036854775807] bytes -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#log_segment_size[`log_segment_size`] @@ -356,7 +356,7 @@ If set to a positive duration, `segment.ms` overrides the cluster property xref: *Accepted values:* [600000, 31536000000] milliseconds (clamped by cluster bounds) -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#log_segment_ms[`log_segment_ms`] @@ -378,7 +378,7 @@ The maximum bytes not fsynced per partition. If this configured threshold is rea *Accepted values:* [1, 9223372036854775807] bytes -**Default**: `262144` +*Default:* `262144` *Related cluster property:* xref:./cluster-properties.adoc#flush_bytes[`flush_bytes`] @@ -396,7 +396,7 @@ The maximum delay (in ms) between two subsequent fsyncs. After this delay, the l *Accepted values:* [1, 9223372036854775] milliseconds -**Default**: `100` +*Default:* `100` *Related cluster property:* xref:./cluster-properties.adoc#flush_ms[`flush_ms`] @@ -414,7 +414,7 @@ This property inherits the value from the config_ref:default_leaders_preference, If the cluster configuration property config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, Leader Pinning is disabled across the cluster. -**Default**: `none` +*Default:* `none` **Values**: @@ -439,7 +439,7 @@ NOTE: Although `replication.factor` isn't returned or displayed by xref:referenc *Accepted values:* [`1`, `512`] -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#default_topic_replication[`default_topic_replication`] @@ -458,7 +458,7 @@ When `write.caching` is set, it overrides the cluster property xref:cluster-prop *Type:* boolean -**Default**: `false` +*Default:* `false` **Values**: @@ -485,7 +485,7 @@ A size-based initial retention limit for Tiered Storage that determines how much *Accepted values:* [1, 9223372036854775807] bytes -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#initial_retention_local_target_bytes[`initial_retention_local_target_bytes`] @@ -503,7 +503,7 @@ A time-based initial retention limit for Tiered Storage that determines how much *Accepted values:* [1, 9223372036854775] milliseconds -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#initial_retention_local_target_ms[`initial_retention_local_target_ms`] @@ -519,7 +519,7 @@ A flag that enables deletion of data from object storage for Tiered Storage when NOTE: `redpanda.remote.delete` doesn't apply to Remote Read Replica topics: a Remote Read Replica topic isn't deleted from object storage when this flag is `true`. -**Default**: +*Default:* - `false` for topics created using Redpanda 22.2 or earlier. - `true` for topics created in Redpanda 22.3 and later, including new topics on upgraded clusters. @@ -534,7 +534,7 @@ NOTE: `redpanda.remote.delete` doesn't apply to Remote Read Replica topics: a Re A flag for enabling Redpanda to fetch data for a topic from object storage to local storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. -**Default**: false +*Default:* false **Related topics**: @@ -548,7 +548,7 @@ A flag that enables the recovery or reproduction of a topic from object storage TIP: You can only configure `redpanda.remote.recovery` when you create a topic. You cannot apply this setting to existing topics. -**Default**: false +*Default:* false **Related topics**: @@ -560,7 +560,7 @@ TIP: You can only configure `redpanda.remote.recovery` when you create a topic. A flag for enabling Redpanda to upload data for a topic from local storage to object storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. -**Default**: false +*Default:* false **Related topics**: @@ -576,7 +576,7 @@ A size-based retention limit for Tiered Storage that configures the maximum size *Accepted values:* [1, 9223372036854775807] bytes -**Default**: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#retention_local_target_bytes[`retention_local_target_bytes`] @@ -594,7 +594,7 @@ A time-based retention limit for Tiered Storage that sets the maximum duration t *Accepted values:* [1, 9223372036854775] milliseconds -**Default**: 86400000 +*Default:* 86400000 *Related cluster property:* xref:./cluster-properties.adoc#retention_local_target_ms[`retention_local_target_ms`] @@ -614,7 +614,7 @@ The name of the object storage bucket for a Remote Read Replica topic. CAUTION: Setting `redpanda.remote.readreplica` together with either `redpanda.remote.read` or `redpanda.remote.write` results in an error. -**Default**: null +*Default:* null **Related topics**: @@ -630,7 +630,7 @@ Integrate Redpanda topics as Iceberg tables. Whether the corresponding Iceberg table is deleted upon deleting the topic. -**Default**: `true` +*Default:* `true` **Related topics**: @@ -642,7 +642,7 @@ Whether the corresponding Iceberg table is deleted upon deleting the topic. Whether to write invalid records to a dead-letter queue (DLQ). -**Default**: `dlq_table` +*Default:* `dlq_table` **Values**: @@ -659,7 +659,7 @@ Whether to write invalid records to a dead-letter queue (DLQ). Enable the Iceberg integration for the topic. You can choose one of four modes. -**Default**: `disabled` +*Default:* `disabled` **Values**: @@ -679,7 +679,7 @@ Enable the Iceberg integration for the topic. You can choose one of four modes. The link:https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table. -**Default**: `(hour(redpanda.timestamp))` +*Default:* `(hour(redpanda.timestamp))` **Related topics**: @@ -691,7 +691,7 @@ The link:https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] sp Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. -**Default**: `60000` +*Default:* `60000` **Related topics**: From cd33b052d46b7389f07d2fa3200fa38ebbbe44db Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 18:15:16 -0300 Subject: [PATCH 12/22] fix acceptable value --- modules/reference/pages/properties/topic-properties.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index b1dccc294..8e39f25b9 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -242,7 +242,7 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti *Type:* integer -*Accepted values:* [1, 9223372036854775] milliseconds +*Accepted values:* [-1, 9223372036854775807] milliseconds *Default:* null From d491a2724d426c1fc9984d4567682ece239f4976 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 18:31:10 -0300 Subject: [PATCH 13/22] adjust accepted values --- .../pages/properties/topic-properties.adoc | 50 +++++++++++++------ 1 file changed, 36 insertions(+), 14 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 8e39f25b9..ea3f0a9fb 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -144,9 +144,9 @@ If both `delete.retention.ms` and the cluster property config_ref:tombstone_rete *Type:* integer -*Accepted values:* [1, 9223372036854775] milliseconds +*Accepted values:* [`1`, `9223372036854775`] milliseconds -*Default*: null +*Default:* null *Related cluster property:* xref:./cluster-properties.adoc#tombstone_retention_ms[`tombstone_retention_ms`] @@ -220,7 +220,7 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti *Type:* integer -*Accepted values:* [1, 9223372036854775807] bytes +*Accepted values:* [`1`, `9223372036854775807`] bytes *Default:* null @@ -242,7 +242,7 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti *Type:* integer -*Accepted values:* [-1, 9223372036854775807] milliseconds +*Accepted values:* [`-9223372036854775808`, `9223372036854775807`] milliseconds *Default:* null @@ -270,7 +270,7 @@ When set, this property overrides the cluster property xref:./cluster-properties *Type:* string -*Accepted Values:* `producer`. The following values are accepted for Kafka compatibility but ignored: `gzip`, `snappy`, `lz4`, `zstd`, `none`. +*Accepted values:* `producer`. The following values are accepted for Kafka compatibility but ignored: `gzip`, `snappy`, `lz4`, `zstd`, `none`. *Default:* `producer` @@ -291,7 +291,7 @@ If `max.message.bytes` is set to a positive value, it overrides the cluster prop *Type:* integer -*Accepted values:* [1, 2147483647] bytes +*Accepted values:* [`1`, `2147483647`] bytes *Default:* null @@ -332,7 +332,7 @@ When `segment.bytes` is set to a positive value, it overrides the cluster proper *Type:* integer -*Accepted values:* [1, 9223372036854775807] bytes +*Accepted values:* [`1`, `9223372036854775807`] bytes *Default:* null @@ -354,7 +354,7 @@ If set to a positive duration, `segment.ms` overrides the cluster property xref: *Type:* integer -*Accepted values:* [600000, 31536000000] milliseconds (clamped by cluster bounds) +*Accepted values:* `[600000, 31536000000]` milliseconds (clamped by cluster bounds) *Default:* null @@ -376,7 +376,7 @@ The maximum bytes not fsynced per partition. If this configured threshold is rea *Type:* integer -*Accepted values:* [1, 9223372036854775807] bytes +*Accepted values:* [`1`, `9223372036854775807`] bytes *Default:* `262144` @@ -394,7 +394,7 @@ The maximum delay (in ms) between two subsequent fsyncs. After this delay, the l *Type:* integer -*Accepted values:* [1, 9223372036854775] milliseconds +*Accepted values:* [`1`, `9223372036854775`] milliseconds *Default:* `100` @@ -414,6 +414,8 @@ This property inherits the value from the config_ref:default_leaders_preference, If the cluster configuration property config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, Leader Pinning is disabled across the cluster. +*Type:* string + *Default:* `none` **Values**: @@ -483,7 +485,7 @@ A size-based initial retention limit for Tiered Storage that determines how much *Type:* integer -*Accepted values:* [1, 9223372036854775807] bytes +*Accepted values:* [`1`, `9223372036854775807`] bytes *Default:* null @@ -501,7 +503,7 @@ A time-based initial retention limit for Tiered Storage that determines how much *Type:* integer -*Accepted values:* [1, 9223372036854775] milliseconds +*Accepted values:* [`1`, `9223372036854775`] milliseconds *Default:* null @@ -519,6 +521,8 @@ A flag that enables deletion of data from object storage for Tiered Storage when NOTE: `redpanda.remote.delete` doesn't apply to Remote Read Replica topics: a Remote Read Replica topic isn't deleted from object storage when this flag is `true`. +*Type:* boolean + *Default:* - `false` for topics created using Redpanda 22.2 or earlier. @@ -534,6 +538,8 @@ NOTE: `redpanda.remote.delete` doesn't apply to Remote Read Replica topics: a Re A flag for enabling Redpanda to fetch data for a topic from object storage to local storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. +*Type:* boolean + *Default:* false **Related topics**: @@ -548,6 +554,8 @@ A flag that enables the recovery or reproduction of a topic from object storage TIP: You can only configure `redpanda.remote.recovery` when you create a topic. You cannot apply this setting to existing topics. +*Type:* boolean + *Default:* false **Related topics**: @@ -560,6 +568,8 @@ TIP: You can only configure `redpanda.remote.recovery` when you create a topic. A flag for enabling Redpanda to upload data for a topic from local storage to object storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. +*Type:* boolean + *Default:* false **Related topics**: @@ -574,7 +584,7 @@ A size-based retention limit for Tiered Storage that configures the maximum size *Type:* integer -*Accepted values:* [1, 9223372036854775807] bytes +*Accepted values:* [`1`, `9223372036854775807`] bytes *Default:* null @@ -592,7 +602,7 @@ A time-based retention limit for Tiered Storage that sets the maximum duration t *Type:* integer -*Accepted values:* [1, 9223372036854775] milliseconds +*Accepted values:* [`1`, `9223372036854775`] milliseconds *Default:* 86400000 @@ -614,6 +624,8 @@ The name of the object storage bucket for a Remote Read Replica topic. CAUTION: Setting `redpanda.remote.readreplica` together with either `redpanda.remote.read` or `redpanda.remote.write` results in an error. +*Type:* string + *Default:* null **Related topics**: @@ -630,6 +642,8 @@ Integrate Redpanda topics as Iceberg tables. Whether the corresponding Iceberg table is deleted upon deleting the topic. +*Type:* boolean + *Default:* `true` **Related topics**: @@ -642,6 +656,8 @@ Whether the corresponding Iceberg table is deleted upon deleting the topic. Whether to write invalid records to a dead-letter queue (DLQ). +*Type:* string + *Default:* `dlq_table` **Values**: @@ -659,6 +675,8 @@ Whether to write invalid records to a dead-letter queue (DLQ). Enable the Iceberg integration for the topic. You can choose one of four modes. +*Type:* string + *Default:* `disabled` **Values**: @@ -679,6 +697,8 @@ Enable the Iceberg integration for the topic. You can choose one of four modes. The link:https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table. +*Type:* string + *Default:* `(hour(redpanda.timestamp))` **Related topics**: @@ -691,6 +711,8 @@ The link:https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] sp Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. +*Type:* integer + *Default:* `60000` **Related topics**: From 92190c62ce32092ccfab7f88636bc01b654ddd66 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 18:36:45 -0300 Subject: [PATCH 14/22] adjust description of segment.ms --- modules/reference/pages/properties/topic-properties.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index ea3f0a9fb..e3c03e4cd 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -350,11 +350,11 @@ When `segment.bytes` is set to a positive value, it overrides the cluster proper The maximum duration that a log segment of a topic is active (open for writes and not deletable). A periodic event, with `segment.ms` as its period, forcibly closes the active segment and transitions, or rolls, to a new active segment. The closed (inactive) segment is then eligible to be cleaned up according to cleanup and retention properties. -If set to a positive duration, `segment.ms` overrides the cluster property xref:./cluster-properties.adoc#log_segment_ms[`log_segment_ms`] and its lower and upper bounds set by xref:./cluster-properties.adoc#log_segment_ms_min[`log_segment_ms_min`] and xref:./cluster-properties.adoc#log_segment_ms_max[`log_segment_ms_max`], respectively. +If set to a positive duration, `segment.ms` overrides the cluster property xref:./cluster-properties.adoc#log_segment_ms[`log_segment_ms`]. Values are automatically clamped between the cluster bounds set by xref:./cluster-properties.adoc#log_segment_ms_min[`log_segment_ms_min`] (default: 10 minutes) and xref:./cluster-properties.adoc#log_segment_ms_max[`log_segment_ms_max`] (default: 1 year). If your configured value exceeds these bounds, Redpanda uses the bound value and logs a warning. Check current cluster bounds with `rpk cluster config get log_segment_ms_min log_segment_ms_max`. *Type:* integer -*Accepted values:* `[600000, 31536000000]` milliseconds (clamped by cluster bounds) +*Accepted values:* [`600000`, `31536000000`] milliseconds (clamped by cluster bounds) *Default:* null From 8961916bf6bd01a72a91b7ecbeed74df8a798937 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 22 Aug 2025 18:42:25 -0300 Subject: [PATCH 15/22] split unit and acceptable values --- .../pages/properties/topic-properties.adoc | 52 ++++++++++++++----- 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index e3c03e4cd..3064ebfaf 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -144,7 +144,9 @@ If both `delete.retention.ms` and the cluster property config_ref:tombstone_rete *Type:* integer -*Accepted values:* [`1`, `9223372036854775`] milliseconds +*Unit:* milliseconds + +*Accepted values:* [`1`, `9223372036854775`] *Default:* null @@ -162,6 +164,8 @@ The maximum amount of time (in ms) that a log segment can remain unaltered befor *Type:* integer +*Unit:* milliseconds + *Accepted values:* [`1`, `9223372036854775`] **Default:* `9223372036854775` @@ -198,6 +202,8 @@ The minimum amount of time (in ms) that a log segment must remain unaltered befo *Type:* integer +*Unit:* milliseconds + *Accepted values:* [`0`, `9223372036854775`] **Default:* `0` @@ -220,7 +226,9 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti *Type:* integer -*Accepted values:* [`1`, `9223372036854775807`] bytes +*Unit:* bytes + +*Accepted values:* [`1`, `9223372036854775807`] *Default:* null @@ -242,7 +250,9 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti *Type:* integer -*Accepted values:* [`-9223372036854775808`, `9223372036854775807`] milliseconds +*Unit:* milliseconds + +*Accepted values:* [`-9223372036854775808`, `9223372036854775807`] *Default:* null @@ -291,7 +301,9 @@ If `max.message.bytes` is set to a positive value, it overrides the cluster prop *Type:* integer -*Accepted values:* [`1`, `2147483647`] bytes +*Unit:* bytes + +*Accepted values:* [`1`, `2147483647`] *Default:* null @@ -332,7 +344,9 @@ When `segment.bytes` is set to a positive value, it overrides the cluster proper *Type:* integer -*Accepted values:* [`1`, `9223372036854775807`] bytes +*Unit:* bytes + +*Accepted values:* [`1`, `9223372036854775807`] *Default:* null @@ -354,7 +368,9 @@ If set to a positive duration, `segment.ms` overrides the cluster property xref: *Type:* integer -*Accepted values:* [`600000`, `31536000000`] milliseconds (clamped by cluster bounds) +*Unit:* milliseconds + +*Accepted values:* [`600000`, `31536000000`] *Default:* null @@ -376,7 +392,9 @@ The maximum bytes not fsynced per partition. If this configured threshold is rea *Type:* integer -*Accepted values:* [`1`, `9223372036854775807`] bytes +*Unit:* bytes + +*Accepted values:* [`1`, `9223372036854775807`] *Default:* `262144` @@ -394,7 +412,9 @@ The maximum delay (in ms) between two subsequent fsyncs. After this delay, the l *Type:* integer -*Accepted values:* [`1`, `9223372036854775`] milliseconds +*Unit:* milliseconds + +*Accepted values:* [`1`, `9223372036854775`] *Default:* `100` @@ -485,7 +505,9 @@ A size-based initial retention limit for Tiered Storage that determines how much *Type:* integer -*Accepted values:* [`1`, `9223372036854775807`] bytes +*Unit:* bytes + +*Accepted values:* [`1`, `9223372036854775807`] *Default:* null @@ -503,7 +525,9 @@ A time-based initial retention limit for Tiered Storage that determines how much *Type:* integer -*Accepted values:* [`1`, `9223372036854775`] milliseconds +*Unit:* milliseconds + +*Accepted values:* [`1`, `9223372036854775`] *Default:* null @@ -584,7 +608,9 @@ A size-based retention limit for Tiered Storage that configures the maximum size *Type:* integer -*Accepted values:* [`1`, `9223372036854775807`] bytes +*Unit:* bytes + +*Accepted values:* [`1`, `9223372036854775807`] *Default:* null @@ -602,7 +628,9 @@ A time-based retention limit for Tiered Storage that sets the maximum duration t *Type:* integer -*Accepted values:* [`1`, `9223372036854775`] milliseconds +*Unit:* milliseconds + +*Accepted values:* [`1`, `9223372036854775`] *Default:* 86400000 From d0fd3ec83ccac907cc12fbef8b1c363f5137a6e1 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Tue, 26 Aug 2025 12:25:22 -0300 Subject: [PATCH 16/22] fix headings --- .../pages/properties/topic-properties.adoc | 84 +++++++++---------- 1 file changed, 40 insertions(+), 44 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index 3064ebfaf..d5e971189 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -79,17 +79,13 @@ NOTE: All topic properties take effect immediately after being set. |=== -== Topic properties - -Properties are grouped by feature area. Within each group, properties are listed in alphabetical order. - --- -=== Retention and Compaction Properties +== Retention and Compaction Properties These properties control how data is stored, for how long, and when it is deleted or compacted. [[cleanuppolicy]] -==== cleanup.policy +=== cleanup.policy The cleanup policy to apply for log segments of a topic. @@ -118,7 +114,7 @@ include::develop:partial$topic-properties-warning.adoc[] --- [[compactionstrategy]] -==== compaction.strategy +=== compaction.strategy Specifies the strategy used to determine which records to remove during log compaction. The compaction strategy controls how Redpanda identifies and removes duplicate records while preserving the latest value for each key. @@ -134,7 +130,7 @@ Specifies the strategy used to determine which records to remove during log comp --- [[deleteretentionms]] -==== delete.retention.ms +=== delete.retention.ms The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded. @@ -158,7 +154,7 @@ If both `delete.retention.ms` and the cluster property config_ref:tombstone_rete --- [[maxcompactionlagms]] -==== max.compaction.lag.ms +=== max.compaction.lag.ms The maximum amount of time (in ms) that a log segment can remain unaltered before it is eligible for compaction in a compact topic. Overrides the cluster property xref:cluster-properties.adoc#max_compaction_lag_ms[`max_compaction_lag_ms`] for the topic. @@ -168,7 +164,7 @@ The maximum amount of time (in ms) that a log segment can remain unaltered befor *Accepted values:* [`1`, `9223372036854775`] -**Default:* `9223372036854775` +*Default:* `9223372036854775` *Related cluster property:* xref:./cluster-properties.adoc#max_compaction_lag_ms[`max_compaction_lag_ms`] @@ -178,7 +174,7 @@ The maximum amount of time (in ms) that a log segment can remain unaltered befor --- [[mincleanabledirtyratio]] -==== min.cleanable.dirty.ratio +=== min.cleanable.dirty.ratio The minimum ratio between the number of bytes in dirty segments and the total number of bytes in closed segments that must be reached before a partition's log is eligible for compaction in a compact topic. @@ -196,7 +192,7 @@ The minimum ratio between the number of bytes in dirty segments and the total nu --- [[mincompactionlagms]] -==== min.compaction.lag.ms +=== min.compaction.lag.ms The minimum amount of time (in ms) that a log segment must remain unaltered before it can be compacted in a compact topic. Overrides the cluster property xref:cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`] for the topic. @@ -206,7 +202,7 @@ The minimum amount of time (in ms) that a log segment must remain unaltered befo *Accepted values:* [`0`, `9223372036854775`] -**Default:* `0` +*Default:* `0` *Related cluster property:* xref:./cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`] @@ -216,7 +212,7 @@ The minimum amount of time (in ms) that a log segment must remain unaltered befo --- [[retentionbytes]] -==== retention.bytes +=== retention.bytes A size-based retention limit that configures the maximum size that a topic partition can grow before becoming eligible for cleanup. @@ -240,7 +236,7 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti --- [[retentionms]] -==== retention.ms +=== retention.ms A time-based retention limit that configures the maximum duration that a log's segment file for a topic is retained before it becomes eligible to be cleaned up. To consume all data, a consumer of the topic must read from a segment before its `retention.ms` elapses, otherwise the segment may be compacted and/or deleted. If a non-positive value, no per-topic limit is applied. @@ -263,12 +259,12 @@ When both size-based (`retention.bytes`) and time-based (`retention.ms`) retenti - xref:manage:cluster-maintenance/disk-utilization.adoc#configure-message-retention[Configure message retention] --- -### Segment and Message Properties +== Segment and Message Properties These properties control the size and lifecycle of log segment files and settings for individual messages. [[compressiontype]] -==== compression.type +=== compression.type Redpanda ignores this property and always uses producer compression semantics. If producers send compressed data, Redpanda stores and serves it as-is. If producers send uncompressed data, Redpanda stores it uncompressed. @@ -293,7 +289,7 @@ When set, this property overrides the cluster property xref:./cluster-properties --- [[maxmessagebytes]] -==== max.message.bytes +=== max.message.bytes The maximum size of a message or batch of a topic. If a compression type is enabled, `max.message.bytes` sets the maximum size of the compressed message or batch. @@ -315,7 +311,7 @@ If `max.message.bytes` is set to a positive value, it overrides the cluster prop --- [[messagetimestamptype]] -==== message.timestamp.type +=== message.timestamp.type The source of a message's timestamp: either the message's creation time or its log append time. @@ -336,7 +332,7 @@ When `message.timestamp.type` is set, it overrides the cluster property xref:./c --- [[segmentbytes]] -==== segment.bytes +=== segment.bytes The maximum size of an active log segment for a topic. When the size of an active segment exceeds `segment.bytes`, the segment is closed and a new active segment is created. The closed, inactive segment is then eligible to be cleaned up according to retention properties. @@ -360,7 +356,7 @@ When `segment.bytes` is set to a positive value, it overrides the cluster proper --- [[segmentms]] -==== segment.ms +=== segment.ms The maximum duration that a log segment of a topic is active (open for writes and not deletable). A periodic event, with `segment.ms` as its period, forcibly closes the active segment and transitions, or rolls, to a new active segment. The closed (inactive) segment is then eligible to be cleaned up according to cleanup and retention properties. @@ -381,12 +377,12 @@ If set to a positive duration, `segment.ms` overrides the cluster property xref: - xref:manage:cluster-maintenance/disk-utilization.adoc#log-rolling[Log rolling] --- -### Performance and Cluster Properties +== Performance and Cluster Properties These properties control disk flushing, replication, and how topics interact with the cluster. [[flushbytes]] -==== flush.bytes +=== flush.bytes The maximum bytes not fsynced per partition. If this configured threshold is reached, the log is automatically fsynced, even though it wasn't explicitly requested. @@ -406,7 +402,7 @@ The maximum bytes not fsynced per partition. If this configured threshold is rea --- [[flushms]] -==== flush.ms +=== flush.ms The maximum delay (in ms) between two subsequent fsyncs. After this delay, the log is automatically fsynced. @@ -426,7 +422,7 @@ The maximum delay (in ms) between two subsequent fsyncs. After this delay, the l --- [[redpandaleaderspreference]] -==== redpanda.leaders.preference +=== redpanda.leaders.preference The preferred location (rack) for partition leaders of a topic. @@ -449,7 +445,7 @@ If the cluster configuration property config_ref:enable_rack_awareness,true,prop --- [[replicationfactor]] -==== replication.factor +=== replication.factor The number of replicas of a topic to save in different nodes (brokers) of a cluster. @@ -472,7 +468,7 @@ NOTE: Although `replication.factor` isn't returned or displayed by xref:referenc --- [[writecaching]] -==== write.caching +=== write.caching The write caching mode to apply to a topic. @@ -494,12 +490,12 @@ When `write.caching` is set, it overrides the cluster property xref:cluster-prop - xref:develop:config-topics.adoc#configure-write-caching[Write caching] --- -### Tiered Storage properties +== Tiered Storage properties Configure properties to manage topics for xref:manage:tiered-storage.adoc[Tiered Storage]. [[initialretentionlocaltargetbytes]] -==== initial.retention.local.target.bytes +=== initial.retention.local.target.bytes A size-based initial retention limit for Tiered Storage that determines how much data in local storage is transferred to a partition replica when a cluster is resized. If `null` (default), all locally retained data is transferred. @@ -519,7 +515,7 @@ A size-based initial retention limit for Tiered Storage that determines how much --- [[initialretentionlocaltargetms]] -==== initial.retention.local.target.ms +=== initial.retention.local.target.ms A time-based initial retention limit for Tiered Storage that determines how much data in local storage is transferred to a partition replica when a cluster is resized. If `null` (default), all locally retained data is transferred. @@ -539,7 +535,7 @@ A time-based initial retention limit for Tiered Storage that determines how much --- [[redpandaremotedelete]] -==== redpanda.remote.delete +=== redpanda.remote.delete A flag that enables deletion of data from object storage for Tiered Storage when it's deleted from local storage for a topic. @@ -558,7 +554,7 @@ NOTE: `redpanda.remote.delete` doesn't apply to Remote Read Replica topics: a Re --- [[redpandaremoteread]] -==== redpanda.remote.read +=== redpanda.remote.read A flag for enabling Redpanda to fetch data for a topic from object storage to local storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. @@ -572,7 +568,7 @@ A flag for enabling Redpanda to fetch data for a topic from object storage to lo --- [[redpandaremoterecovery]] -==== redpanda.remote.recovery +=== redpanda.remote.recovery A flag that enables the recovery or reproduction of a topic from object storage for Tiered Storage. The recovered data is saved in local storage, and the maximum amount of recovered data is determined by the local storage retention limits of the topic. @@ -588,7 +584,7 @@ TIP: You can only configure `redpanda.remote.recovery` when you create a topic. --- [[redpandaremotewrite]] -==== redpanda.remote.write +=== redpanda.remote.write A flag for enabling Redpanda to upload data for a topic from local storage to object storage. When set to `true` together with <>, it enables the xref:manage:tiered-storage.adoc[Tiered Storage] feature. @@ -602,7 +598,7 @@ A flag for enabling Redpanda to upload data for a topic from local storage to ob --- [[retentionlocaltargetbytes]] -==== retention.local.target.bytes +=== retention.local.target.bytes A size-based retention limit for Tiered Storage that configures the maximum size that a topic partition in local storage can grow before becoming eligible for cleanup. It applies per partition and is equivalent to <> without Tiered Storage. @@ -622,7 +618,7 @@ A size-based retention limit for Tiered Storage that configures the maximum size --- [[retentionlocaltargetms]] -==== retention.local.target.ms +=== retention.local.target.ms A time-based retention limit for Tiered Storage that sets the maximum duration that a log's segment file for a topic is retained in local storage before it's eligible for cleanup. This property is equivalent to <> without Tiered Storage. @@ -641,12 +637,12 @@ A time-based retention limit for Tiered Storage that sets the maximum duration t - xref:manage:tiered-storage.adoc[Tiered Storage] --- -### Remote Read Replica properties +== Remote Read Replica properties Configure properties to manage topics for xref:manage:remote-read-replicas.adoc[Remote Read Replicas]. [[redpandaremotereadreplica]] -==== redpanda.remote.readreplica +=== redpanda.remote.readreplica The name of the object storage bucket for a Remote Read Replica topic. @@ -661,12 +657,12 @@ CAUTION: Setting `redpanda.remote.readreplica` together with either `redpanda.re - xref:manage:remote-read-replicas.adoc[Remote Read Replicas] --- -### Apache Iceberg integration properties +== Apache Iceberg integration properties Integrate Redpanda topics as Iceberg tables. [[redpandaicebergdelete]] -==== redpanda.iceberg.delete +=== redpanda.iceberg.delete Whether the corresponding Iceberg table is deleted upon deleting the topic. @@ -680,7 +676,7 @@ Whether the corresponding Iceberg table is deleted upon deleting the topic. --- [[redpandaiceberginvalidrecordaction]] -==== redpanda.iceberg.invalid.record.action +=== redpanda.iceberg.invalid.record.action Whether to write invalid records to a dead-letter queue (DLQ). @@ -699,7 +695,7 @@ Whether to write invalid records to a dead-letter queue (DLQ). --- [[redpandaicebergmode]] -==== redpanda.iceberg.mode +=== redpanda.iceberg.mode Enable the Iceberg integration for the topic. You can choose one of four modes. @@ -721,7 +717,7 @@ Enable the Iceberg integration for the topic. You can choose one of four modes. --- [[redpandaicebergpartitionspec]] -==== redpanda.iceberg.partition.spec +=== redpanda.iceberg.partition.spec The link:https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table. @@ -735,7 +731,7 @@ The link:https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] sp --- [[redpandaicebergtargetlagms]] -==== redpanda.iceberg.target.lag.ms +=== redpanda.iceberg.target.lag.ms Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. From 62f77d0c64d4715b5c83fd4a1eed535d2babcaa0 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Thu, 28 Aug 2025 19:21:34 -0300 Subject: [PATCH 17/22] modify wording on compaction/disk space --- .../topic-property-configuration.adoc | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc index 5947122a6..96c2866a3 100644 --- a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -143,14 +143,12 @@ Topic properties are organized into several categories based on their functional === Disk space management -Configure properties to manage disk space usage for a topic: +Redpanda manages disk space through two main mechanisms: **compaction** (removing duplicate keys) and **retention** (removing old data). Configure the core compaction properties to control how Redpanda cleans up data: -- Clean up log segments by deletion and/or compaction (xref:reference:properties/topic-properties.adoc#cleanuppolicy[`cleanup.policy`]). -- Control compaction timing with maximum and minimum lag settings (xref:reference:properties/topic-properties.adoc#maxcompactionlagms[`max.compaction.lag.ms`] and xref:reference:properties/topic-properties.adoc#mincompactionlagms[`min.compaction.lag.ms`]). -- Retain logs up to a maximum size per partition before cleanup (xref:reference:properties/topic-properties.adoc#retentionbytes[`retention.bytes`]). -- Retain logs for a maximum duration before cleanup (xref:reference:properties/topic-properties.adoc#retentionms[`retention.ms`]). -- Periodically close an active log segment (xref:reference:properties/topic-properties.adoc#segmentms[`segment.ms`]). -- Limit the maximum size of an active log segment (xref:reference:properties/topic-properties.adoc#segmentbytes[`segment.bytes`]). +- Choose cleanup strategy: deletion, compaction, or both (xref:reference:properties/topic-properties.adoc#cleanuppolicy[`cleanup.policy`]). +- Control when compaction triggers based on dirty data ratio (xref:reference:properties/topic-properties.adoc#mincleanabledirtyratio[`min.cleanable.dirty.ratio`]). +- Set maximum time before compaction is forced (xref:reference:properties/topic-properties.adoc#maxcompactionlagms[`max.compaction.lag.ms`]). +- Set minimum time before compaction can occur (xref:reference:properties/topic-properties.adoc#mincompactionlagms[`min.compaction.lag.ms`]). === Write performance and caching @@ -176,13 +174,6 @@ For topics using Tiered Storage, configure: - Recover data from object storage (xref:reference:properties/topic-properties.adoc#redpandaremoterecovery[`redpanda.remote.recovery`]). - Configure local storage retention limits (xref:reference:properties/topic-properties.adoc#retentionlocaltargetbytes[`retention.local.target.bytes`] and xref:reference:properties/topic-properties.adoc#retentionlocaltargetms[`retention.local.target.ms`]). -=== Compaction settings - -For topics with compacted cleanup policies, configure: - -- Minimum ratio of dirty segments required before compaction (xref:reference:properties/topic-properties.adoc#mincleanabledirtyratio[`min.cleanable.dirty.ratio`]). -- Retention time for tombstone records (xref:reference:properties/topic-properties.adoc#deleteretentionms[`delete.retention.ms`]). - === Replication and leadership Configure replication and leadership settings: From 115ddf1f237924ce6f65ba6bd630e45e9ec7016e Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Thu, 28 Aug 2025 19:30:27 -0300 Subject: [PATCH 18/22] modify warning about internal topics --- modules/develop/partials/topic-properties-warning.adoc | 2 +- .../cluster-maintenance/topic-property-configuration.adoc | 2 +- modules/reference/pages/properties/topic-properties.adoc | 4 +--- 3 files changed, 3 insertions(+), 5 deletions(-) diff --git a/modules/develop/partials/topic-properties-warning.adoc b/modules/develop/partials/topic-properties-warning.adoc index 62fef970c..d093296c0 100644 --- a/modules/develop/partials/topic-properties-warning.adoc +++ b/modules/develop/partials/topic-properties-warning.adoc @@ -1 +1 @@ -WARNING: Modifying the properties of topics that are created and managed by Redpanda applications can cause unexpected errors. This may lead to connector and cluster failures. \ No newline at end of file +WARNING: All topic properties take effect immediately after being set. Do not modify properties on internal Redpanda topics (such as `__consumer_offsets`, `_schemas`, or other system topics) as this can cause cluster instability. diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc index 96c2866a3..df98709d1 100644 --- a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -6,7 +6,7 @@ Topic properties control Redpanda's behavior for individual topics, including da Redpanda stores topic properties as metadata associated with each topic and replicates them across the cluster to ensure consistency. Many topic properties correspond to xref:manage:cluster-maintenance/cluster-property-configuration.adoc[cluster properties] that establish default values for all topics. When you set a topic property, it overrides the corresponding cluster default for that specific topic. -NOTE: All topic properties take effect immediately after being set. +include::develop:partial$topic-properties-warning.adoc[] For a complete reference of available topic properties and their corresponding cluster properties, see xref:reference:properties/topic-properties.adoc[Topic Configuration Properties]. diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index d5e971189..630fcf774 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -8,7 +8,7 @@ Many topic-level properties have corresponding xref:manage:cluster-maintenance/c For information on how to configure topic properties, see xref:manage:cluster-maintenance/topic-property-configuration.adoc[]. -NOTE: All topic properties take effect immediately after being set. +include::develop:partial$topic-properties-warning.adoc[] == Topic property mappings @@ -91,8 +91,6 @@ The cleanup policy to apply for log segments of a topic. When `cleanup.policy` is set, it overrides the cluster property xref:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] for the topic. -include::develop:partial$topic-properties-warning.adoc[] - *Type:* string *Accepted values:* [`delete`, `compact`, `compact,delete`] From 1850302e2acbb47db4fbec2307d912b34078820c Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Thu, 28 Aug 2025 19:30:32 -0300 Subject: [PATCH 19/22] fix syntax --- .../pages/cluster-maintenance/topic-property-configuration.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc index df98709d1..83fc73f6d 100644 --- a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -55,6 +55,7 @@ Kubernetes:: ```bash kubectl exec -- rpk topic create -c cleanup.policy=compact +``` -- ==== From 2e529e9b62f0beb599bc3f3409501d578cc47c0f Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Thu, 28 Aug 2025 20:37:36 -0300 Subject: [PATCH 20/22] code review suggestions --- .../topic-property-configuration.adoc | 45 +++++++++++++++++-- 1 file changed, 41 insertions(+), 4 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc index 83fc73f6d..592183edf 100644 --- a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -21,12 +21,49 @@ You can configure topic properties through multiple interfaces: == Verify topic property configuration -The `SOURCE` output of the xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe `] command shows how properties are configured for the topic: +Use `rpk topic describe ` to view topic properties and their configuration sources: -* `DEFAULT_CONFIG` is set by a Redpanda default. -* `DYNAMIC_TOPIC_CONFIG` is set by the user specifically for the topic and overrides inherited default configurations, such as a default or a cluster-level property. +``` +rpk topic describe my-topic +``` + +The output shows two sections: + +``` +SUMMARY +======= +NAME my-topic +PARTITIONS 1 +REPLICAS 1 + +CONFIGS +======= +KEY VALUE SOURCE +cleanup.policy delete DEFAULT_CONFIG +compression.type producer DEFAULT_CONFIG +retention.ms 604800000 DEFAULT_CONFIG +write.caching true DYNAMIC_TOPIC_CONFIG +``` + +The `SOURCE` column indicates how each property is configured: + +* `DEFAULT_CONFIG` - Redpanda's default value +* `DYNAMIC_TOPIC_CONFIG` - User-configured value -Although xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`] doesn't report `replication.factor` as a configuration, `replication.factor` can indeed be set by using the xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`] command. +The replication factor appears as REPLICAS in the SUMMARY section, not as replication.factor in the CONFIGS list. However, you need to use the `replication.factor` key when modifying the value with `rpk topic alter-config`. + +For partition-level details, add the `-p` flag: + +``` +rpk topic describe my-topic -p +``` + +This shows a different output focused on partition information: + +``` +PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK +0 0 1 [0] 0 0 +``` == Examples From dcf762c49d6212612f39d85d17ffec9aea5ec4e4 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 29 Aug 2025 17:19:17 -0300 Subject: [PATCH 21/22] adjust examples --- .../cluster-maintenance/topic-property-configuration.adoc | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc index 592183edf..a571c104a 100644 --- a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -33,8 +33,8 @@ The output shows two sections: SUMMARY ======= NAME my-topic -PARTITIONS 1 -REPLICAS 1 +PARTITIONS 3 +REPLICAS 3 CONFIGS ======= @@ -62,7 +62,9 @@ This shows a different output focused on partition information: ``` PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK -0 0 1 [0] 0 0 +0 1 2 [1 2 3] 0 6 +1 2 4 [1 2 3] 0 10 +2 3 1 [1 2 3] 0 8 ``` == Examples From 399196d8d66cfe87692c8924d183d76539677a0b Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Fri, 29 Aug 2025 17:24:06 -0300 Subject: [PATCH 22/22] rework section on most common props --- .../topic-property-configuration.adoc | 49 ++++++++----------- 1 file changed, 21 insertions(+), 28 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc index a571c104a..8a182e138 100644 --- a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -179,47 +179,40 @@ kubectl apply -f topic-config.yaml == Common topic property categories -Topic properties are organized into several categories based on their functionality: +The most commonly configured topic properties fall into these main categories: === Disk space management -Redpanda manages disk space through two main mechanisms: **compaction** (removing duplicate keys) and **retention** (removing old data). Configure the core compaction properties to control how Redpanda cleans up data: +Redpanda manages disk space through two main mechanisms: **compaction** (removing duplicate keys) and **retention** (removing old data). -- Choose cleanup strategy: deletion, compaction, or both (xref:reference:properties/topic-properties.adoc#cleanuppolicy[`cleanup.policy`]). -- Control when compaction triggers based on dirty data ratio (xref:reference:properties/topic-properties.adoc#mincleanabledirtyratio[`min.cleanable.dirty.ratio`]). -- Set maximum time before compaction is forced (xref:reference:properties/topic-properties.adoc#maxcompactionlagms[`max.compaction.lag.ms`]). -- Set minimum time before compaction can occur (xref:reference:properties/topic-properties.adoc#mincompactionlagms[`min.compaction.lag.ms`]). +Choose your cleanup strategy with xref:reference:properties/topic-properties.adoc#cleanuppolicy[`cleanup.policy`]: +- `delete` - Remove old data based on time or size limits +- `compact` - Keep only the latest value for each key +- `compact,delete` - Combine both strategies -=== Write performance and caching +=== Compaction -Control how Redpanda writes and syncs data to disk: +When using `cleanup.policy=compact` or `cleanup.policy=compact,delete`, configure: -- Cache batches until the segment appender chunk is full, instead of fsyncing for every `acks=all` write (xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`]). -- Configure the maximum delay between two subsequent fsyncs (xref:reference:properties/topic-properties.adoc#flushms[`flush.ms`]). -- Set the maximum bytes not fsynced per partition (xref:reference:properties/topic-properties.adoc#flushbytes[`flush.bytes`]). +- xref:reference:properties/topic-properties.adoc#mincleanabledirtyratio[`min.cleanable.dirty.ratio`] - Control when compaction triggers based on dirty data ratio +- xref:reference:properties/topic-properties.adoc#maxcompactionlagms[`max.compaction.lag.ms`] - Set maximum time before compaction is forced +- xref:reference:properties/topic-properties.adoc#mincompactionlagms[`min.compaction.lag.ms`] - Set minimum time before compaction can occur -=== Message properties +=== Retention -Configure message-specific settings: +When using `cleanup.policy=delete` or `cleanup.policy=compact,delete`, configure: -- Set the source of a message's timestamp (xref:reference:properties/topic-properties.adoc#messagetimestamptype[`message.timestamp.type`]). -- Set the maximum size of a message (xref:reference:properties/topic-properties.adoc#maxmessagebytes[`max.message.bytes`]). -- Configure compression for stored data (xref:reference:properties/topic-properties.adoc#compressiontype[`compression.type`]). +- xref:reference:properties/topic-properties.adoc#retentionbytes[`retention.bytes`] - Maximum size before cleanup (size-based retention) +- xref:reference:properties/topic-properties.adoc#retentionms[`retention.ms`] - Maximum age before cleanup (time-based retention) +- xref:reference:properties/topic-properties.adoc#segmentbytes[`segment.bytes`] - Control how frequently cleanup can occur by setting segment size -=== Tiered Storage +=== Performance -For topics using Tiered Storage, configure: +Essential performance tuning properties: -- Upload and fetch data to and from object storage (xref:reference:properties/topic-properties.adoc#redpandaremotewrite[`redpanda.remote.write`] and xref:reference:properties/topic-properties.adoc#redpandaremoteread[`redpanda.remote.read`]). -- Recover data from object storage (xref:reference:properties/topic-properties.adoc#redpandaremoterecovery[`redpanda.remote.recovery`]). -- Configure local storage retention limits (xref:reference:properties/topic-properties.adoc#retentionlocaltargetbytes[`retention.local.target.bytes`] and xref:reference:properties/topic-properties.adoc#retentionlocaltargetms[`retention.local.target.ms`]). - -=== Replication and leadership - -Configure replication and leadership settings: - -- Number of replicas for the topic (xref:reference:properties/topic-properties.adoc#replicationfactor[`replication.factor`]). -- Preferred location for partition leaders (xref:reference:properties/topic-properties.adoc#redpandaleaderspreference[`redpanda.leaders.preference`]). +- xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`] - Cache writes for lower latency with `acks=all` +- xref:reference:properties/topic-properties.adoc#maxmessagebytes[`max.message.bytes`] - Set maximum message size +- xref:reference:properties/topic-properties.adoc#replicationfactor[`replication.factor`] - Number of replicas for durability vs. performance For complete details about all available topic properties, see xref:reference:properties/topic-properties.adoc[Topic Configuration Properties].