Search before asking
Please describe the bug
Paimon Rust currently rejects ignore-delete options on primary-key tables using
merge-engine=partial-update, even though Java Paimon supports this mode.
For example, creating or opening a table with either option fails validation:
merge-engine=partial-update
ignore-delete=true
or:
merge-engine=partial-update
partial-update.ignore-delete=true
The read-side and flush-time partial-update merge functions also reject
DELETE and UPDATE_BEFORE rows unconditionally. This prevents Rust readers
from opening Java-created tables and historical files that use the supported
ignore-delete semantics.
Expected behavior:
- Both
ignore-delete and partial-update.ignore-delete are accepted.
- With an effective value of
true, DELETE and UPDATE_BEFORE are ignored.
- The default and an explicit
false still reject retract rows.
- Ignored retract rows do not consume writer buffer, sequence numbers, or enter
input changelog files.
- Once enabled, the effective value cannot be changed from
true to false,
matching Java Paimon compatibility safeguards.
- Other advanced partial-update options remain unsupported.
Proposed solution
- Narrowly allowlist the two supported option keys in partial-update validation.
- Apply ignore-delete semantics in both read-side merge and write-side filtering.
- Filter direct
_VALUE_KIND batches before writer buffering.
- Add schema, writer, reader, core integration, and DataFusion integration tests.
- Document the supported behavior and remaining limitations.
Are you willing to submit a PR?
Search before asking
Please describe the bug
Paimon Rust currently rejects
ignore-deleteoptions on primary-key tables usingmerge-engine=partial-update, even though Java Paimon supports this mode.For example, creating or opening a table with either option fails validation:
or:
The read-side and flush-time partial-update merge functions also reject
DELETEandUPDATE_BEFORErows unconditionally. This prevents Rust readersfrom opening Java-created tables and historical files that use the supported
ignore-delete semantics.
Expected behavior:
ignore-deleteandpartial-update.ignore-deleteare accepted.true,DELETEandUPDATE_BEFOREare ignored.falsestill reject retract rows.input changelog files.
truetofalse,matching Java Paimon compatibility safeguards.
Proposed solution
_VALUE_KINDbatches before writer buffering.Are you willing to submit a PR?