feat: add ReplacePartitions core class (PR1 of 2 for #637) by shangxinli · Pull Request #776 · apache/iceberg-cpp

shangxinli · 2026-06-24T00:05:57Z

Summary

Adds the iceberg::ReplacePartitions class — the dynamic partition overwrite operation. Each AddFile() registers the file's partition for replacement of all existing data files in that partition. Produces an overwrite snapshot with "replace-partitions"="true" in the summary. Unpartitioned tables replace all existing data files.

This is PR1 of 2 for #637. See tracking issue #775 for the split rationale.

What's in PR1

iceberg::ReplacePartitions class extending MergingSnapshotUpdate:
- Builder API: AddFile, ValidateAppendOnly, ValidateFromSnapshot, ValidateNoConflictingData, ValidateNoConflictingDeletes
- Apply / Summary / CleanUncommitted overrides
- Uses ManifestFilterManager for partition-based manifest filtering
SnapshotSummaryFields::kReplacePartitions = "replace-partitions" constant
ReplacePartitions forward declaration in type_fwd.h
CMake + Meson registration

What's deferred to PR2

Transaction::NewReplacePartitions() API wiring on Table / Transaction
End-to-end test file (replace_partitions_test.cc)

Design notes

Follows the Java ReplacePartitions semantics: operation type is overwrite, summary carries "replace-partitions"="true".
Mirrors the existing C++ FastAppend / MergeAppend patterns (factory Make(), MergingSnapshotUpdate base, builder error collector).
Unpartitioned table case treated as table-wide replace via DeleteByRowFilter(AlwaysTrue()), matching Java BaseReplacePartitions.

Test plan

Clean build (cmake --build build)
clang-format clean
End-to-end behavioral tests land in PR2 alongside the Table/Transaction API wiring

Closes part of #637 (PR2 will close it).
Tracking: #775

Adds iceberg::ReplacePartitions — the dynamic partition overwrite operation. Each AddFile() registers the file's partition for replacement of all existing data and delete files in that partition. Produces an overwrite snapshot with "replace-partitions=true" in the summary. Unpartitioned tables replace all existing data files. Extends MergingSnapshotUpdate (matching Java's BaseReplacePartitions) so the full data+delete manifest pipeline, custom-summary-property handling, and conflict-validation helpers are inherited. AddFile() unconditionally calls DropPartition(spec_id, file->partition) — for unpartitioned specs the partition value is empty and the filter manager matches every file in that spec, so no separate AlwaysTrue path is needed. Touched partitions are tracked in a PartitionSet; Validate() uses the partition-scoped overloads of ValidateAddedDataFiles / ValidateNoNewDeleteFiles, or skips entirely when no partitions were staged. Changes: * iceberg::ReplacePartitions extending MergingSnapshotUpdate with builder API (AddFile, ValidateAppendOnly, ValidateFromSnapshot, ValidateNoConflictingData, ValidateNoConflictingDeletes) and Validate() override. * SnapshotSummaryFields::kReplacePartitions = "replace-partitions". * MergingSnapshotUpdate::SetSummaryProperty promoted from private to protected so subclasses can stash custom summary entries that survive commit retry via the cached-rebuild path. * Forward declaration in type_fwd.h. * CMake + Meson source registration. Public API wiring (Table::NewReplacePartitions(), Transaction::NewReplacePartitions()) and end-to-end tests are deferred to PR2. Tracking: apache#775 Related: apache#637

manuzhang · 2026-06-24T09:16:51Z

+        current_metadata, starting_snapshot_id_, replaced_partitions_, snapshot, io));
+  }
+  if (validate_conflicting_deletes_) {
+    ICEBERG_RETURN_UNEXPECTED(ValidateNoNewDeleteFiles(


Java BaseReplacePartitions.validate also calls validateDeletedDataFiles, so concurrent overwrite/delete commits in the replaced partition are rejected

manuzhang · 2026-06-24T09:19:25Z

+  // the partition values are empty and naturally match every file under that
+  // spec — no separate AlwaysTrue path is needed, and validation stays scoped
+  // to the spec rather than the whole table.
+  ICEBERG_BUILDER_RETURN_IF_ERROR(DropPartition(spec_id, file->partition));


unpartitioned replace is spec-scoped, but Java treats it as table-wide.

…aFiles - AddFile() now uses DeleteByRowFilter(AlwaysTrue()) for unpartitioned specs instead of DropPartition with empty partition values, matching Java BaseReplacePartitions which treats unpartitioned tables as a table-wide replace. - Validate() now also calls ValidateDeletedDataFiles when ValidateNoConflictingDeletes is enabled, mirroring Java where validateNewDeletes gates both checks. This rejects concurrent overwrite/delete commits in the replaced partitions. - New replace_by_row_filter_ flag drives the AlwaysTrue path in Validate() for the unpartitioned case.

manuzhang · 2026-06-25T03:44:49Z

+  // No-op update: no partitions were staged and no table-wide replace was
+  // requested, so there is nothing to conflict with. Calling the validators
+  // with AlwaysTrue here would turn an empty builder into a full-table check.
+  if (!replace_by_row_filter_ && replaced_partitions_.empty()) {


Java rejects no-data-files replace. Suggest to call DataSpec() like Java.

Not sure where it's been implemented. Can you check again?

Sorry, the previous "Done" was premature — DataSpec() wasn't actually being called at that point. It's now in place: Validate() calls DataSpec() in the partitioned path (!replace_by_row_filter_), which throws if no files were staged or if staged files span multiple partition specs, matching Java's BaseReplacePartitions.validate.

manuzhang

It looks replace_partitions.h is missing from src/iceberg/update/meson.build.

zhjwpku · 2026-06-25T07:30:56Z

In the PR description, should "the iceberg::ReplacePartitions class extending SnapshotUpdate" be changed to "the iceberg::ReplacePartitions class extending MergingSnapshotUpdate"?

zhjwpku · 2026-06-25T09:55:35Z

+
+namespace iceberg {
+
+/// \brief Replaces partitions in a table with new data files.


Do we need to mention This is provided to implement SQL compatible with Hive table operations but is not recommended. Instead, use the {@link OverwriteFiles overwrite API} to explicitly overwrite data. like java?

Changed to '/// \note This is provided to implement SQL compatible with Hive table operations but is not recommended. Instead, use OverwriteFiles to explicitly overwrite data.'

…date - Add replace_partitions.h to src/iceberg/update/meson.build (manuzhang) - Add Java-style note recommending OverwriteFiles for non-Hive use (zhjwpku) - Validate(): call DataSpec() to reject no-data-files replace, matching Java BaseReplacePartitions which throws when no file was added (manuzhang)

DataSpec() also errors on multi-spec stages, which would silently break a ReplacePartitions that touches files from more than one spec. Use the flags AddFile() sets (replace_by_row_filter_ / replaced_partitions_) instead — the same condition that already gated the no-op early-return below, now promoted to an error to match Java BaseReplacePartitions.

shangxinli · 2026-06-25T18:46:42Z

In the PR description, should "the iceberg::ReplacePartitions class extending SnapshotUpdate" be changed to "the iceberg::ReplacePartitions class extending MergingSnapshotUpdate"?

Updated the PR description — "extending SnapshotUpdate" → "extending MergingSnapshotUpdate". Thanks for catching that.

wgtmac · 2026-07-02T04:41:21Z

+  ICEBERG_BUILDER_ASSIGN_OR_RETURN(auto spec, base().PartitionSpecById(spec_id));
+
+  ICEBERG_BUILDER_RETURN_IF_ERROR(AddDataFile(file));
+  if (spec->fields().empty()) {


Java's PartitionSpec.isUnpartitioned() returns true not only for an empty field list but also for specs where all partition transforms are void/alwaysNull. With an all-void partition spec, BaseReplacePartitions.apply goes through the unpartitioned path and deletes by alwaysTrue(), replacing the whole table. This check only looks at fields().empty(), so C++ will treat an all-void spec as partitioned and call DropPartition, leaving files from other equivalent unpartitioned/all-void specs instead of doing the table-wide replace.

Fixed. Replaced the fields().empty() guard with an is_unpartitioned lambda that matches Java's PartitionSpec.isUnpartitioned() — a spec is treated as unpartitioned if it has no fields or all transforms are void. All-void specs now route through DeleteByRowFilter(AlwaysTrue()) instead of DropPartition.

wgtmac · 2026-07-02T04:41:23Z

+  // Match Java BaseReplacePartitions: require at least one staged data file.
+  // Use the flags AddFile() sets — `DataSpec()` would also error on multi-spec
+  // stages and is not the guard we want here.
+  if (!replace_by_row_filter_ && replaced_partitions_.empty()) {


Java still calls dataSpec() from validate/apply, which enforces that exactly one partition spec is represented by the staged data files. This guard only checks that at least one partition was recorded, so a replace containing files from multiple spec IDs can commit in C++ even though Java rejects it. That can produce snapshot semantics that don't align with Java after spec evolution.

Fixed. Validate() now calls DataSpec() in the partitioned path, which errors if no files were staged or if staged files span more than one partition spec — matching Java's behavior. The unpartitioned path skips it because replace_by_row_filter_ is only set after a successful AddFile(), so at least one file is already guaranteed.

…e-spec enforcement

wgtmac

Codex was used to review Java parity against Iceberg Java's ReplacePartitions, BaseReplacePartitions, MergingSnapshotProducer, and TestReplacePartitions. The inline comments cover the remaining behavior, API documentation, and test gaps.

wgtmac · 2026-07-23T07:06:00Z

+  // file and that all staged files share exactly one partition spec.
+  // DataSpec() enforces both invariants; ignore the returned spec here since
+  // replace_by_row_filter_ / replaced_partitions_ already scope validation.
+  if (!replace_by_row_filter_) {


Please call DataSpec() before this branch unconditionally. If an unpartitioned or all-void file is added first, replace_by_row_filter_ becomes true and the single-spec check is skipped; a later file from another spec then commits a table-wide replace, while Java rejects mixed specs.

wgtmac · 2026-07-23T07:06:00Z

+
+std::string ReplacePartitions::operation() { return DataOperation::kOverwrite; }
+
+Status ReplacePartitions::Validate(const TableMetadata& current_metadata,


Please add direct ReplacePartitions tests before merging this core class. Cover partitioned and unpartitioned/all-void replacement, empty and mixed specs, append-only validation, same- and different-partition concurrency, retry summary retention, and DV cleanup. Existing base-class tests do not exercise this orchestration.

wgtmac · 2026-07-23T07:06:00Z

+}
+
+ReplacePartitions& ReplacePartitions::ValidateAppendOnly() {
+  FailAnyDelete();


Please translate the fail-any-delete error in ReplacePartitions, as BaseReplacePartitions.apply() does. C++ currently returns "Operation would delete existing data", while Java returns "Cannot commit file that conflicts with existing partition: ...".

wgtmac · 2026-07-23T07:06:00Z

+    // partitions; without it a concurrent delete in a replaced partition would
+    // commit silently.
+    if (replace_by_row_filter_) {
+      ICEBERG_RETURN_UNEXPECTED(ValidateNoNewDeleteFiles(


Please run ValidateDeletedDataFiles before ValidateNoNewDeleteFiles in both branches, matching Java. When both conflict types exist, the current order reports a different validation failure.

wgtmac · 2026-07-23T07:06:00Z

+  ICEBERG_BUILDER_ASSIGN_OR_RETURN(auto spec, base().PartitionSpecById(spec_id));
+
+  ICEBERG_BUILDER_RETURN_IF_ERROR(AddDataFile(file));
+  // A spec is effectively unpartitioned if it has no fields, or every field


Please keep these comments focused on the rule. For example: "Specs with no non-void transforms are unpartitioned." Remove the Java comparison and implementation-history detail here and in the validation block below.

wgtmac · 2026-07-23T07:06:00Z

+  std::optional<int64_t> starting_snapshot_id_;
+  bool validate_conflicting_data_{false};
+  bool validate_conflicting_deletes_{false};
+  // True once an AddFile() call has staged a file whose partition spec is


Please describe this member directly, for example: "True when replacement uses an AlwaysTrue row filter." Keep the Java comparison and design rationale in the PR discussion.

wgtmac · 2026-07-23T07:06:01Z

+  /// \return Reference to this for method chaining
+  ReplacePartitions& ValidateNoConflictingData();
+
+  /// \brief Enable validation that no conflicting delete files were added concurrently.


Please align this contract with Java. This validation also rejects concurrent data-file removals, not only newly added delete files; mention that it prevents undeleting rows removed by a concurrent commit.

shangxinli force-pushed the feat/replace-partitions-core branch 4 times, most recently from ae973f6 to e33f433 Compare June 24, 2026 01:01

shangxinli mentioned this pull request Jun 24, 2026

feat: implement ReplacePartitions (two-PR plan) #775

Open

shangxinli force-pushed the feat/replace-partitions-core branch from e33f433 to 12dc230 Compare June 24, 2026 01:13

manuzhang reviewed Jun 24, 2026

View reviewed changes

manuzhang reviewed Jun 25, 2026

View reviewed changes

manuzhang suggested changes Jun 25, 2026

View reviewed changes

zhjwpku reviewed Jun 25, 2026

View reviewed changes

shangxinli added 2 commits June 25, 2026 09:36

wgtmac reviewed Jul 2, 2026

View reviewed changes

shangxinli added 2 commits July 11, 2026 13:22

Address review: void-transform unpartitioned check and DataSpec singl…

b03d790

…e-spec enforcement

Merge branch 'main' into feat/replace-partitions-core

52ef133

manuzhang added this to the 0.4.0 milestone Jul 21, 2026

wgtmac reviewed Jul 23, 2026

View reviewed changes


		namespace iceberg {

		/// \brief Replaces partitions in a table with new data files.


		std::string ReplacePartitions::operation() { return DataOperation::kOverwrite; }

		Status ReplacePartitions::Validate(const TableMetadata& current_metadata,

Uh oh!

Conversation

shangxinli commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in PR1

What's deferred to PR2

Design notes

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manuzhang left a comment

Choose a reason for hiding this comment

Uh oh!

zhjwpku commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shangxinli commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shangxinli commented Jun 24, 2026 •

edited

Loading