feat(commit): align functionality with java paimon commit

[lucasfang]

Purpose

Linked issue: N/A

This PR aligns the C++ commit path with Java Paimon commit semantics, with focus on correctness and conflict handling in PK/overwrite/index scenarios.

Main changes:

• Align commit conflict detection behavior with Java, including changed-entry collection flow and conflict-check triggers.
• Add and integrate dedicated commit helpers for:
  • Manifest entry change aggregation
  • Conflict detection
  • Row-id and write-column conflict checking
  • Retry waiter backoff behavior
  • Sequence snapshot property merge logic
• Align overwrite and append behavior in index-related paths:
  • Correct commit kind selection in append/overwrite edge cases
  • Proper handling of index-file commits in overwrite/filter-and-overwrite flows
• Align row-tracking commit behavior with Java:
  • Snapshot sequence assignment
  • Row-id assignment for normal/blob/vector-related files
• Include follow-up fixes from review feedback for boundary/correctness cases.

Tests

Added test cases in this PR:

• TEST(BucketModeTest, TestResolveBucketMode)
• TEST(CommitChangesProviderTest, TestProvideReturnsGivenEntries)
• TEST(CommitChangesProviderTest, TestProvideUsesCopiedInputs)
• TEST(CommitContextTest, TestDefaultValue)
• TEST(CommitContextTest, TestSetContent)
• TEST(CommitContextTest, TestSetOptionsOverridesAddedOptions)
• TEST(CommitMetricsTest, TestReportCommitFromStats)
• TEST(CommitStatsTest, TestCalcChangedPartitionsAndBuckets)
• TEST(CommitStatsTest, TestFailedAppendSnapshot)
• TEST(CommitStatsTest, TestSucceedAllSnapshot)
• TEST(CompactedChangelogPathResolverTest, IsCompactedChangelogPath)
• TEST(CompactedChangelogPathResolverTest, KeepInvalidCompactedPath)
• TEST(CompactedChangelogPathResolverTest, KeepNonCompactedPath)
• TEST(CompactedChangelogPathResolverTest, KeepRealCompactedChangelogPath)
• TEST(CompactedChangelogPathResolverTest, ResolveFakeCompactedChangelogPath)
• TEST(CompactedChangelogPathResolverTest, ResolveFileWithoutExtension)
• TEST(CompactedChangelogPathResolverTest, ResolveNonCompactedChangelogPath)
• TEST(CompactedChangelogPathResolverTest, ResolveWithDifferentFormats)
• TEST(CoreOptionsTest, TestDynamicPartitionOverwriteOption)
• TEST_F(CleanInteTest, DISABLED_TestDropPartitionAndExpireSnapshotWithIOException)
• TEST_F(CommitScannerTest, TestReadAllEntriesFromChangedPartitionsBuildsScanFilterPartitions)
• TEST_F(CommitScannerTest, TestReadAllEntriesFromChangedPartitionsEmptyFastExit)
• TEST_F(CommitScannerTest, TestReadAllEntriesFromPartitionsRequiresSupplier)
• TEST_F(ConflictDetectionTest, TestBucketKeepSame)
• TEST_F(ConflictDetectionTest, TestBucketKeepSameCacheEviction)
• TEST_F(ConflictDetectionTest, TestBucketKeepSameHelpers)
• TEST_F(ConflictDetectionTest, TestCheckLsmKeyRangeConflict)
• TEST_F(ConflictDetectionTest, TestCollectUncheckedBucketPartitionsMismatch)
• TEST_F(ConflictDetectionTest, TestDedicatedStorageRowIdRangeParityWithRelease14)
• TEST_F(ConflictDetectionTest, TestDeletionVectorsAllowedWithResolvedDynamicBucketMode)
• TEST_F(ConflictDetectionTest, TestDeletionVectorsNotSupportedWithBucketUnawareMode)
• TEST_F(ConflictDetectionTest, TestDeletionVectorsNotSupportedWithResolvedBucketUnawareModeFromMinusOne)
• TEST_F(ConflictDetectionTest, TestFileDeletionConflicts)
• TEST_F(ConflictDetectionTest, TestGlobalIndexRowIdExistenceConflicts)
• TEST_F(FileStoreCommitImplTest, FilterAndCommitSkipCompactBeforeFileCheck)
• TEST_F(FileStoreCommitImplTest, FilterAndCommitWithCompactedChangelogFakePath)
• TEST_F(FileStoreCommitImplTest, TestAppendDiscardDuplicateFiles)
• TEST_F(FileStoreCommitImplTest, TestCommitWithCompactDeletedIndexFiles)
• TEST_F(FileStoreCommitImplTest, TestCommitWithCompactIndexFiles)
• TEST_F(FileStoreCommitImplTest, TestCommitWithDeletedIndexFiles)
• TEST_F(FileStoreCommitImplTest, TestCommitWithGlobalIndexFilesChecksConflicts)
• TEST_F(FileStoreCommitImplTest, TestCommitWithIndexFiles)
• TEST_F(FileStoreCommitImplTest, TestFilterAndOverwriteWithCompactIndexFiles)
• TEST_F(FileStoreCommitImplTest, TestFilterCommittedRejectsDuplicateIdentifiers)
• TEST_F(FileStoreCommitImplTest, TestFixedBucketPKTableCommitAllowed)
• TEST_F(FileStoreCommitImplTest, TestOverwriteWithCompactIndexFiles)
• TEST_F(FileStoreCommitImplTest, TestSnapshotSequenceMaxPropertyMergedOnCommit)
• TEST_F(FileStoreCommitImplTest, TestTryOverwriteWithProperties)
• TEST(FileStoreCommitTest, TestAppendDvIndexShouldUseOverwriteCommitKind)
• TEST_F(ManifestEntryChangesTest, TestChangedPartitionsIncludesDvAndGlobalIndex)
• TEST_F(ManifestEntryChangesTest, TestCollectAndSummary)
• TEST_F(ManifestEntryChangesTest, TestCollectInvalidCommitMessageType)
• TEST_F(ManifestEntryChangesTest, TestHasGlobalIndexFileAdditions)
• TEST_F(RowIdColumnConflictCheckerTest, TestAllowsDisjointWriteColumns)
• TEST_F(RowIdColumnConflictCheckerTest, TestDetectsSameWriteColumns)
• TEST_F(RowIdColumnConflictCheckerTest, TestIgnoreUnknownNonSystemWriteColumn)
• TEST_F(RowIdColumnConflictCheckerTest, TestMergesOverlappedDeltaRangesAndWriteColumns)
• TEST_F(RowIdColumnConflictCheckerTest, TestScansAllOverlappedRangesAfterBinarySearch)
• TEST_F(RowIdColumnConflictCheckerTest, TestTreatsNullWriteColumnsAsFullSchemaWrite)
• TEST_F(RowIdColumnConflictCheckerTest, TestUsesFieldIdAcrossRename)
• TEST_F(RowTrackingCommitUtilsTest, TestAssignRowTracking)
• TEST_F(RowTrackingCommitUtilsTest, TestAssignRowTrackingDoesNotMutateInputEntries)
• TEST_F(RowTrackingCommitUtilsTest, TestAssignRowTrackingReassignsOnRetryWithAdvancedRowId)
• TEST_F(RowTrackingCommitUtilsTest, TestAssignRowTrackingStampsSequence)
• TEST_F(RowTrackingCommitUtilsTest, TestAssignRowTrackingStampsSequenceRangeStartingAtZero)
• TEST_F(RowTrackingCommitUtilsTest, TestAssignRowTrackingWithoutFileSource)
• TEST(OverwriteChangesProviderTest, TestProvidePropagatesScanErrors)
• TEST(OverwriteChangesProviderTest, TestProvideWithLatestSnapshotAddsDeletesAndAppendsAllChanges)
• TEST(OverwriteChangesProviderTest, TestProvideWithoutLatestSnapshotUsesChangesOnly)
• TEST_P(DataEvolutionTableTest, TestCommitConflictOnOverlappedRowIdAndWriteColumns)
• TEST(RetryWaiterTest, TestRetryWaitWithLargeRetryCountIsClamped)
• TEST(RetryWaiterTest, TestRetryWaitWithZeroBoundsReturnsQuickly)
• TEST(SpecialFieldsTest, TestIsSystemField)

API and Format

• No new end-user API introduced.
• Commit-related internal behavior is aligned with Java semantics.
• Snapshot/Tag metadata handling is adjusted for parity in commit path.
• No new data-file format is introduced in this PR.

Documentation

• No dedicated user-facing documentation update is required for this internal commit-path alignment change.

Generative AI tooling

Generated-by: GitHub Copilot (GPT-5.3-Codex)

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR expands commit conflict detection and index-file handling in FileStoreCommitImpl, adds row-id/write-column conflict checking for data-evolution scenarios, and updates/extends unit tests accordingly.

Changes:

• Add ConflictDetection, ManifestEntryChanges, and RowIdColumnConflictChecker to centralize commit change collection and conflict checks (including DV/global index considerations).
• Support committing index files during overwrite / filter-and-overwrite flows, and adjust commit-kind selection (e.g., DV index append => overwrite kind).
• Update commit retry behavior and add/adjust tests for index files, retries, and PK table commit behavior.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/paimon/core/operation/file_store_commit_test.cpp	Adds test validating DV index append triggers overwrite commit kind
src/paimon/core/operation/file_store_commit_impl_test.cpp	Extends commit tests for index files, retry metrics, and conflict checking signature changes
src/paimon/core/operation/file_store_commit_impl.h	Introduces new conflict/index APIs and wires in ConflictDetection
src/paimon/core/operation/file_store_commit_impl.cpp	Implements index-aware overwrite, retry commit checking, and centralized change collection
src/paimon/core/operation/file_store_commit.cpp	Removes prior PK-commit restriction logic in Create
src/paimon/core/operation/commit/row_id_column_conflict_checker.h	Adds row-id/write-column overlap conflict checker interface
src/paimon/core/operation/commit/row_id_column_conflict_checker.cpp	Implements row-id/write-column overlap conflict checker
src/paimon/core/operation/commit/manifest_entry_changes.h	Adds helper to collect manifest + index changes from commit messages
src/paimon/core/operation/commit/manifest_entry_changes.cpp	Implements change collection and “changed partitions” logic
src/paimon/core/operation/commit/manifest_entry_changes_test.cpp	Adds unit tests for ManifestEntryChanges
src/paimon/core/operation/commit/conflict_detection.h	Adds conflict detection API covering bucket/key-range/row-id/global-index checks
src/paimon/core/operation/commit/conflict_detection.cpp	Implements conflict detection logic and row-id history scan
src/paimon/core/operation/commit/conflict_detection_test.cpp	Adds unit test coverage for conflict detection behaviors
src/paimon/common/data/binary_row.h	Adds std::hash for tuple key used in conflict detection
src/paimon/CMakeLists.txt	Wires new sources + tests into the build

Comments suppressed due to low confidence (1)

src/paimon/core/operation/file_store_commit.cpp:1

• The PR description is still the unfilled template (e.g., linked issue #xxx, tests section empty), but this change alters user-visible behavior by removing the previous PK-table commit restriction in FileStoreCommit::Create. Please update the PR title/description to clearly state the behavioral change, link the real issue, and list the tests validating the new PK commit behavior.

/*

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zjw1111]

Reviewed the commit conflict-detection alignment against Java Paimon. The port is largely faithful (ManifestEntryChanges collection order, RetryWaiter backoff math, SequenceSnapshotProperties ADD-only max, row-tracking assignment all match Java). A few correctness gaps and divergences are worth fixing before merge — see inline.

Highest priority: RowIdColumnConflictChecker::ConflictsWith swallows field-resolution errors and can let a schema mismatch pass conflict detection.

Also, a few items not inlined:

• Test coverage: RowIdColumnConflictChecker has no C++ test at all (Java has 7 cases, incl. testUsesFieldIdAcrossRename / testFailsOnUnknownNonSystemWriteColumn which cover two of the issues below); CheckRowIdExistence / CheckRowIdRangeConflicts / SequenceSnapshotProperties also lack direct tests. Please add these.
• Style (docs/code-style.md): plain int in row_id_column_conflict_checker.cpp binary search (low/high/mid/index) and std::unordered_set<int> in file_store_commit_impl.cpp NumChangedBuckets — use fixed-width types.
• Intentional partial ports (please confirm and add a note): DV+UNAWARE entry enrichment, pkClusteringOverride short-circuit in CheckKeyRange, partition-expire handling in CheckDeleteInEntries, and dropStats on overwrite delete entries are all omitted.
• The PR description is still the unfilled template — please fill in the change summary, linked issue, and tests.

[Copilot]

Pull request overview

Copilot reviewed 77 out of 77 changed files in this pull request and generated 2 comments.

[zjw1111]

Cross-language review against Java Paimon found four additional correctness and boundary-condition gaps; see the inline comments.

The existing unresolved threads about overwrite ADD/DELETE deduplication and preserving the underlying commit error also still apply. I would not approve this revision until the P1 data-integrity issues are addressed.

[zjw1111]

Re-reviewed against the latest Java master (f7ec93b3c). The dedicated row-id range check, the RowRangeIndex-based existence / global-index checks, ReadIncrementalEntries, and the 3-branch sequence assignment now all align with master. No blocking data-correctness issues. Remaining items are inline.

One non-inline note: SpecialFields::IsSystemField omits _LEVEL (master's SYSTEM_FIELD_NAMES includes it). But _LEVEL only lives in the KeyValue/merge-tree schema and never reaches write_cols on the append/data-evolution path that RowIdColumnConflictChecker inspects, so it is not reachable here — optional.

[zjw1111]

+1. Row-id / RowRangeIndex alignment with master and the added test coverage look good; CI is green. The retry error-preservation comment in file_store_commit_impl.cpp is non-blocking (diagnostics only) — please follow up separately.

[zjw1111]

Post-merge follow-up (cleanup + open item):

1. Dead flag enable-pk-commit-in-inte-test: this PR removed the PK-commit gate in FileStoreCommit::Create that checked for this option, but the flag is still passed in tests and is now a no-op. Please drop it from:

   • src/paimon/testing/utils/test_helper.h (new_options["enable-pk-commit-in-inte-test"])
   • test/inte/write_inte_test.cpp, test/inte/pk_compaction_inte_test.cpp
   • src/paimon/core/operation/key_value_file_store_write_test.cpp

2. Still open (non-blocking, raised inline before merge): retry-path error preservation in file_store_commit_impl.cpp. On a non-OK CommitSnapshotImpl the underlying status is dropped and the loop retries; on timeout the message reports the configured timeout and retry count, not the root-cause error. Please preserve the last hard-error status and surface it on abort.

[zjw1111]

Additional non-blocking cleanup (from internal AI review, all verified accurate — style/perf only, no correctness impact):

1. Avoidable deep copy: GetNewFilesIncrement() / GetCompactIncrement() return const &, but the results are received by value (copies the inner vector<shared_ptr<...>>). Use const auto& at:

   • src/paimon/core/operation/commit/manifest_entry_changes.cpp:38, 57
   • src/paimon/core/operation/file_store_commit_impl.cpp:260, 268 (same pattern, not flagged by the tool)

2. src/paimon/core/operation/commit/retry_waiter.cpp: RetryWait constructs a fresh std::random_device + std::mt19937 on every call. Make the generator thread_local to reuse it across retries.

3. src/paimon/core/operation/commit/row_id_column_conflict_checker_test.cpp:143: test name TestIgnoreUnknownNonSystemWriteColumn is misleading — the assertion expects ConflictsWith to return an error, not to ignore the column (and it drops the third-person s used by sibling tests). Rename to e.g. TestFailsOnUnknownNonSystemWriteColumn.

[zjw1111]

fix the disabled case