feat: add ReplacePartitions core class (PR1 of 2 for #637) by shangxinli · Pull Request #776 · apache/iceberg-cpp

shangxinli · 2026-06-24T00:05:57Z

Summary

Adds the iceberg::ReplacePartitions class — the dynamic partition overwrite operation. Each AddFile() registers the file's partition for replacement of all existing data files in that partition. Produces an overwrite snapshot with "replace-partitions"="true" in the summary. Unpartitioned tables replace all existing data files.

This is PR1 of 2 for #637. See tracking issue #775 for the split rationale.

What's in PR1

iceberg::ReplacePartitions class extending MergingSnapshotUpdate:
- Builder API: AddFile, ValidateAppendOnly, ValidateFromSnapshot, ValidateNoConflictingData, ValidateNoConflictingDeletes
- Apply / Summary / CleanUncommitted overrides
- Uses ManifestFilterManager for partition-based manifest filtering
SnapshotSummaryFields::kReplacePartitions = "replace-partitions" constant
ReplacePartitions forward declaration in type_fwd.h
CMake + Meson registration

What's deferred to PR2

Transaction::NewReplacePartitions() API wiring on Table / Transaction
End-to-end test file (replace_partitions_test.cc)

Design notes

Follows the Java ReplacePartitions semantics: operation type is overwrite, summary carries "replace-partitions"="true".
Mirrors the existing C++ FastAppend / MergeAppend patterns (factory Make(), MergingSnapshotUpdate base, builder error collector).
Unpartitioned table case treated as table-wide replace via DeleteByRowFilter(AlwaysTrue()), matching Java BaseReplacePartitions.

Test plan

Clean build (cmake --build build)
clang-format clean
End-to-end behavioral tests land in PR2 alongside the Table/Transaction API wiring

Closes part of #637 (PR2 will close it).
Tracking: #775

Adds iceberg::ReplacePartitions — the dynamic partition overwrite operation. Each AddFile() registers the file's partition for replacement of all existing data and delete files in that partition. Produces an overwrite snapshot with "replace-partitions=true" in the summary. Unpartitioned tables replace all existing data files. Extends MergingSnapshotUpdate (matching Java's BaseReplacePartitions) so the full data+delete manifest pipeline, custom-summary-property handling, and conflict-validation helpers are inherited. AddFile() unconditionally calls DropPartition(spec_id, file->partition) — for unpartitioned specs the partition value is empty and the filter manager matches every file in that spec, so no separate AlwaysTrue path is needed. Touched partitions are tracked in a PartitionSet; Validate() uses the partition-scoped overloads of ValidateAddedDataFiles / ValidateNoNewDeleteFiles, or skips entirely when no partitions were staged. Changes: * iceberg::ReplacePartitions extending MergingSnapshotUpdate with builder API (AddFile, ValidateAppendOnly, ValidateFromSnapshot, ValidateNoConflictingData, ValidateNoConflictingDeletes) and Validate() override. * SnapshotSummaryFields::kReplacePartitions = "replace-partitions". * MergingSnapshotUpdate::SetSummaryProperty promoted from private to protected so subclasses can stash custom summary entries that survive commit retry via the cached-rebuild path. * Forward declaration in type_fwd.h. * CMake + Meson source registration. Public API wiring (Table::NewReplacePartitions(), Transaction::NewReplacePartitions()) and end-to-end tests are deferred to PR2. Tracking: apache#775 Related: apache#637

manuzhang · 2026-06-24T09:16:51Z

+        current_metadata, starting_snapshot_id_, replaced_partitions_, snapshot, io));
+  }
+  if (validate_conflicting_deletes_) {
+    ICEBERG_RETURN_UNEXPECTED(ValidateNoNewDeleteFiles(


Java BaseReplacePartitions.validate also calls validateDeletedDataFiles, so concurrent overwrite/delete commits in the replaced partition are rejected

manuzhang · 2026-06-24T09:19:25Z

+  // the partition values are empty and naturally match every file under that
+  // spec — no separate AlwaysTrue path is needed, and validation stays scoped
+  // to the spec rather than the whole table.
+  ICEBERG_BUILDER_RETURN_IF_ERROR(DropPartition(spec_id, file->partition));


unpartitioned replace is spec-scoped, but Java treats it as table-wide.

…aFiles - AddFile() now uses DeleteByRowFilter(AlwaysTrue()) for unpartitioned specs instead of DropPartition with empty partition values, matching Java BaseReplacePartitions which treats unpartitioned tables as a table-wide replace. - Validate() now also calls ValidateDeletedDataFiles when ValidateNoConflictingDeletes is enabled, mirroring Java where validateNewDeletes gates both checks. This rejects concurrent overwrite/delete commits in the replaced partitions. - New replace_by_row_filter_ flag drives the AlwaysTrue path in Validate() for the unpartitioned case.

manuzhang · 2026-06-25T03:44:49Z

+  // No-op update: no partitions were staged and no table-wide replace was
+  // requested, so there is nothing to conflict with. Calling the validators
+  // with AlwaysTrue here would turn an empty builder into a full-table check.
+  if (!replace_by_row_filter_ && replaced_partitions_.empty()) {


Java rejects no-data-files replace. Suggest to call DataSpec() like Java.

Not sure where it's been implemented. Can you check again?

Sorry, the previous "Done" was premature — DataSpec() wasn't actually being called at that point. It's now in place: Validate() calls DataSpec() in the partitioned path (!replace_by_row_filter_), which throws if no files were staged or if staged files span multiple partition specs, matching Java's BaseReplacePartitions.validate.

manuzhang

It looks replace_partitions.h is missing from src/iceberg/update/meson.build.

zhjwpku · 2026-06-25T07:30:56Z

In the PR description, should "the iceberg::ReplacePartitions class extending SnapshotUpdate" be changed to "the iceberg::ReplacePartitions class extending MergingSnapshotUpdate"?

zhjwpku · 2026-06-25T09:55:35Z

+
+namespace iceberg {
+
+/// \brief Replaces partitions in a table with new data files.


Do we need to mention This is provided to implement SQL compatible with Hive table operations but is not recommended. Instead, use the {@link OverwriteFiles overwrite API} to explicitly overwrite data. like java?

Changed to '/// \note This is provided to implement SQL compatible with Hive table operations but is not recommended. Instead, use OverwriteFiles to explicitly overwrite data.'

…date - Add replace_partitions.h to src/iceberg/update/meson.build (manuzhang) - Add Java-style note recommending OverwriteFiles for non-Hive use (zhjwpku) - Validate(): call DataSpec() to reject no-data-files replace, matching Java BaseReplacePartitions which throws when no file was added (manuzhang)

DataSpec() also errors on multi-spec stages, which would silently break a ReplacePartitions that touches files from more than one spec. Use the flags AddFile() sets (replace_by_row_filter_ / replaced_partitions_) instead — the same condition that already gated the no-op early-return below, now promoted to an error to match Java BaseReplacePartitions.

shangxinli · 2026-06-25T18:46:42Z

In the PR description, should "the iceberg::ReplacePartitions class extending SnapshotUpdate" be changed to "the iceberg::ReplacePartitions class extending MergingSnapshotUpdate"?

Updated the PR description — "extending SnapshotUpdate" → "extending MergingSnapshotUpdate". Thanks for catching that.

…e-spec enforcement

wgtmac

Codex was used to review Java parity against Iceberg Java's ReplacePartitions, BaseReplacePartitions, MergingSnapshotProducer, and TestReplacePartitions. The inline comments cover the remaining behavior, API documentation, and test gaps.

- Call DataSpec() unconditionally in Validate so a replace mixing an unpartitioned/all-void file with files from another spec is rejected instead of committing a table-wide replace. - Override Apply() to translate the fail-any-delete error into Java's "Cannot commit file that conflicts with existing partition" wording. - Run ValidateDeletedDataFiles before ValidateNoNewDeleteFiles in both branches to match Java's deterministic failure ordering. - Trim Java-comparison/implementation-history comments to focus on the rule; align the ValidateNoConflictingDeletes contract with Java. - Add direct ReplacePartitions tests: partitioned and unpartitioned replacement, empty and mixed specs, append-only validation, and same/different-partition data and delete concurrency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ions-core # Conflicts: # src/iceberg/update/merging_snapshot_update.h

wgtmac

Codex was used for this Java parity review against the current Iceberg Java ReplacePartitions, BaseReplacePartitions, and tests. The remaining inline comments cover public contracts, comment cleanup, and focused test gaps.

wgtmac · 2026-07-24T03:45:08Z

+/// "replace-partitions"="true" in the summary. For unpartitioned tables, all
+/// existing files are replaced.
+///
+/// When committing, these changes are applied to the latest table snapshot.


Please include Java's default idempotent-validation contract here: when neither conflict validation is enabled, the replace is reapplied and committed despite concurrent table changes. This is part of the public API.

wgtmac · 2026-07-24T03:45:08Z

+  /// \return Reference to this for method chaining
+  ReplacePartitions& AddFile(const std::shared_ptr<DataFile>& file);
+
+  /// \brief Fail the commit if any existing data file would be deleted.


ValidateAppendOnly() fails if any partition would be replaced, not only when an existing data file is deleted; FailAnyDelete() also checks delete manifests. Please use Java's contract wording here.

wgtmac · 2026-07-24T03:45:08Z

+
+  EXPECT_THAT(table_->Refresh(), IsOk());
+  ICEBERG_UNWRAP_OR_FAIL(auto snapshot, table_->current_snapshot());
+  EXPECT_EQ(snapshot->summary.at(SnapshotSummaryFields::kDeletedDataFiles), "1");


Summary counts do not prove that the other partition was preserved; deleting file_b_ and leaving file_a_ would produce the same counts. Assert the live file paths or manifest entries include file_b_ and the replacement and exclude file_a_.

wgtmac · 2026-07-24T03:45:08Z

+  op->ValidateAppendOnly();
+  auto result = op->Commit();
+  EXPECT_THAT(result, IsError(ErrorKind::kValidationFailed));
+  EXPECT_THAT(result, HasErrorMessage(


Please assert the full translated message, including the partition path. This substring would pass if Apply() dropped or corrupted the value extracted from the base error.

wgtmac · 2026-07-24T03:45:08Z

+
+  ICEBERG_UNWRAP_OR_FAIL(auto op, NewReplace());
+  op->AddFile(MakeDataFile("/data/replacement_x1.parquet", /*partition_x=*/1L));
+  op->ValidateFromSnapshot(first_id);


Please add the no-validation counterpart from Java: the same-partition concurrent replace must commit when only ValidateFromSnapshot() is set. This locks the API's default idempotent behavior, which the current conflict tests do not cover.

shangxinli force-pushed the feat/replace-partitions-core branch 4 times, most recently from ae973f6 to e33f433 Compare June 24, 2026 01:01

shangxinli mentioned this pull request Jun 24, 2026

feat: implement ReplacePartitions (two-PR plan) #775

Open

shangxinli force-pushed the feat/replace-partitions-core branch from e33f433 to 12dc230 Compare June 24, 2026 01:13

manuzhang reviewed Jun 24, 2026

View reviewed changes

manuzhang reviewed Jun 25, 2026

View reviewed changes

manuzhang suggested changes Jun 25, 2026

View reviewed changes

zhjwpku reviewed Jun 25, 2026

View reviewed changes

shangxinli added 2 commits June 25, 2026 09:36

wgtmac reviewed Jul 2, 2026

View reviewed changes

Comment thread src/iceberg/update/replace_partitions.cc Outdated

wgtmac reviewed Jul 2, 2026

View reviewed changes

Comment thread src/iceberg/update/replace_partitions.cc Outdated

shangxinli added 2 commits July 11, 2026 13:22

Address review: void-transform unpartitioned check and DataSpec singl…

b03d790

…e-spec enforcement

Merge branch 'main' into feat/replace-partitions-core

52ef133

manuzhang added this to the 0.4.0 milestone Jul 21, 2026

wgtmac reviewed Jul 23, 2026

View reviewed changes

shangxinli and others added 2 commits July 23, 2026 13:40

Merge remote-tracking branch 'upstream/main' into feat/replace-partit…

b971977

…ions-core # Conflicts: # src/iceberg/update/merging_snapshot_update.h

wgtmac reviewed Jul 24, 2026

View reviewed changes


		namespace iceberg {

		/// \brief Replaces partitions in a table with new data files.

Uh oh!

Conversation

shangxinli commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in PR1

What's deferred to PR2

Design notes

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manuzhang left a comment

Choose a reason for hiding this comment

Uh oh!

zhjwpku commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shangxinli commented Jun 25, 2026

Uh oh!

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shangxinli commented Jun 24, 2026 •

edited

Loading