Skip to content

Commit 80b8d13

Browse files
author
shuxu.li
committed
feat(update): add OverwriteFiles for overwrite snapshot commits
Summary: Add a production OverwriteFiles builder that brings iceberg-cpp to semantic parity with Java's BaseOverwriteFiles. It supports explicit file replacement (DeleteFile + AddFile) and range-based replacement (OverwriteByRowFilter + AddFile) with the same family of pre-commit concurrency validations. The builder is a thin subclass of MergingSnapshotUpdate and reuses the existing commit kernel (Apply/summary/retry/cleanup) unchanged. Changes: - New OverwriteFiles class (src/iceberg/update/overwrite_files.{h,cc}) and Table::NewOverwrite() / Transaction::NewOverwrite() entry points. - Builder surface: AddFile, DeleteFile, bulk DeleteFiles, OverwriteByRowFilter, ValidateFromSnapshot, ConflictDetectionFilter, ValidateNoConflictingData, ValidateNoConflictingDeletes, ValidateAddedFilesMatchOverwriteFilter, WithCaseSensitivity. - Validate(): conflict-filter resolution, concurrent add/delete conflict checks, and strict added-file range validation (projection + StrictMetricsEvaluator). - Tests (overwrite_files_test.cc, 45 cases) and CMake/meson wiring. Behavior alignment with Java: - operation() returns append/delete/overwrite from builder content. - Conflict-filter resolution mirrors BaseOverwriteFiles (explicit -> row filter -> AlwaysTrue); replaced-file delete checks honor ConflictDetectionFilter. - Strict added-file validation uses a single DataSpec(), rejecting multi-spec and empty added-file sets. - Deviations: public WithCaseSensitivity (vs caseSensitive) to avoid a protected-name clash; ValidateFromSnapshot rejects negative ids early.
1 parent d3b02bb commit 80b8d13

13 files changed

Lines changed: 1587 additions & 1 deletion

src/iceberg/CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ set(ICEBERG_SOURCES
9797
update/expire_snapshots.cc
9898
update/fast_append.cc
9999
update/merging_snapshot_update.cc
100+
update/overwrite_files.cc
100101
update/pending_update.cc
101102
update/set_snapshot.cc
102103
update/snapshot_manager.cc

src/iceberg/meson.build

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,7 @@ iceberg_sources = files(
119119
'update/expire_snapshots.cc',
120120
'update/fast_append.cc',
121121
'update/merging_snapshot_update.cc',
122+
'update/overwrite_files.cc',
122123
'update/pending_update.cc',
123124
'update/set_snapshot.cc',
124125
'update/snapshot_manager.cc',

src/iceberg/table.cc

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
#include "iceberg/transaction.h"
3434
#include "iceberg/update/expire_snapshots.h"
3535
#include "iceberg/update/fast_append.h"
36+
#include "iceberg/update/overwrite_files.h"
3637
#include "iceberg/update/set_snapshot.h"
3738
#include "iceberg/update/snapshot_manager.h"
3839
#include "iceberg/update/update_location.h"
@@ -217,6 +218,12 @@ Result<std::shared_ptr<FastAppend>> Table::NewFastAppend() {
217218
return FastAppend::Make(name().name, std::move(ctx));
218219
}
219220

221+
Result<std::shared_ptr<OverwriteFiles>> Table::NewOverwrite() {
222+
ICEBERG_ASSIGN_OR_RAISE(
223+
auto ctx, TransactionContext::Make(shared_from_this(), TransactionKind::kUpdate));
224+
return OverwriteFiles::Make(name().name, std::move(ctx));
225+
}
226+
220227
Result<std::shared_ptr<UpdateStatistics>> Table::NewUpdateStatistics() {
221228
ICEBERG_ASSIGN_OR_RAISE(
222229
auto ctx, TransactionContext::Make(shared_from_this(), TransactionKind::kUpdate));

src/iceberg/table.h

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,9 @@ class ICEBERG_EXPORT Table : public std::enable_shared_from_this<Table> {
176176
/// \brief Create a new FastAppend to append data files and commit the changes.
177177
virtual Result<std::shared_ptr<FastAppend>> NewFastAppend();
178178

179+
/// \brief Create a new OverwriteFiles to overwrite data files and commit the changes.
180+
virtual Result<std::shared_ptr<OverwriteFiles>> NewOverwrite();
181+
179182
/// \brief Create a new SnapshotManager to manage snapshots and snapshot references.
180183
virtual Result<std::shared_ptr<SnapshotManager>> NewSnapshotManager();
181184

src/iceberg/test/CMakeLists.txt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -232,7 +232,8 @@ if(ICEBERG_BUILD_BUNDLE)
232232
update_properties_test.cc
233233
update_schema_test.cc
234234
update_sort_order_test.cc
235-
update_statistics_test.cc)
235+
update_statistics_test.cc
236+
overwrite_files_test.cc)
236237

237238
add_iceberg_test(data_test
238239
USE_BUNDLE

src/iceberg/test/meson.build

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,10 @@ iceberg_tests = {
120120
),
121121
'use_data': true,
122122
},
123+
'overwrite_files_test': {
124+
'sources': files('overwrite_files_test.cc'),
125+
'use_data': true,
126+
},
123127
}
124128

125129
if get_option('rest').enabled()

0 commit comments

Comments
 (0)