Skip to content

Commit f647a67

Browse files
paulohtb6micheleRP
andauthored
DOC-1380: Deletion of expired transaction markers (#1551)
Co-authored-by: Michele Cyran <michele@redpanda.com>
1 parent ea6057d commit f647a67

2 files changed

Lines changed: 22 additions & 2 deletions

File tree

modules/develop/pages/transactions.adoc

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -320,11 +320,13 @@ Different transactions require different approaches to handling failures within
320320

321321
== Transactions with compacted segments
322322

323-
Transactions are supported on topics with compaction configured. The compaction process removes the aborted transaction's data and all transactional control markers from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches).
323+
Transactions are supported on topics with compaction configured. The compaction process removes aborted transaction data from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches).
324324

325325
ifndef::env-cloud[]
326326
At a cluster-level, compaction is set when xref:reference:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] or xref:reference:topic-properties.adoc#cleanuppolicy[`cleanup.policy`] are set to either `compact` or `compact,delete`.
327327

328+
Optionally, you can enable removal of transactional control batches (commit and abort markers) during compaction by setting xref:reference:properties/cluster-properties.adoc#log_compaction_tx_batch_removal_enabled[`log_compaction_tx_batch_removal_enabled`] to `true`. When enabled, the topic's xref:reference:properties/topic-properties.adoc#delete-retention-ms[`delete.retention.ms`] setting is applied to these markers. For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level. This feature is not applied when Tiered Storage is enabled. See xref:manage:cluster-maintenance/compaction-settings.adoc#transactional-control-batch-removal[Transactional control batch removal].
329+
328330
endif::[]
329331

330332
include::shared:partial$suggested-reading.adoc[]

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,25 @@ If obtaining a complete snapshot of the log, including tombstone records, is imp
154154

155155
On the other hand, if more frequent cleanup of tombstones is important for optimizing workloads and space management, consider setting a shorter tombstone retention, for example the typical default of 24 hours (86400000 ms).
156156

157-
Compaction and tombstone removal are coordinated across replicas, preventing inconsistencies and ensuring that deleted records are properly recognized by all readers. As a result, tombstone removal on one replica may be delayed if another replica is stopped or lagging.
157+
Compaction and tombstone removal are coordinated across replicas, preventing inconsistencies and ensuring that deleted records are properly recognized by all readers. As a result, tombstone removal on one replica may be delayed if another replica is stopped or lagging.
158+
159+
== Transactional control batch removal
160+
161+
Transactional workloads write control batches (commit and abort markers) to the log. By default, these markers are retained indefinitely. You can enable their removal during compaction by setting xref:reference:properties/cluster-properties.adoc#log_compaction_tx_batch_removal_enabled[`log_compaction_tx_batch_removal_enabled`] to `true`:
162+
163+
[,bash]
164+
----
165+
rpk cluster config set log_compaction_tx_batch_removal_enabled=true
166+
----
167+
168+
When enabled, the xref:reference:properties/topic-properties.adoc#delete-retention-ms[`delete.retention.ms`] setting is applied to transactional control batches, removing them after the retention period. This uses the same coordinated compaction mechanism as tombstone removal.
169+
170+
[NOTE]
171+
====
172+
For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level to enable removal of transactional control batches.
173+
====
174+
175+
Consider enabling this feature if you have compacted topics with heavy transactional workloads and observe disk usage from accumulated transaction markers.
158176

159177
== Compaction policy settings
160178

0 commit comments

Comments
 (0)