Skip to content

feat(table): read materialized deletion-vector PK splits#580

Open
shyjsarah wants to merge 1 commit into
apache:mainfrom
shyjsarah:fix/pk-dv-raw-read-routing
Open

feat(table): read materialized deletion-vector PK splits#580
shyjsarah wants to merge 1 commit into
apache:mainfrom
shyjsarah:fix/pk-dv-raw-read-routing

Conversation

@shyjsarah

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #579

Allow read-only access to fully materialized partial-update and aggregation primary-key plans with deletion vectors. This change is independent of #573.

Brief change log

  • Route eligible plans through DataFileReader so deletion vectors filter deleted rows.
  • Require raw-convertible compacted files with known zero delete_row_count and merge-on-read disabled.
  • Fail closed for level-0 files, ambiguous metadata, and plans that still require per-key merging.

Tests

  • Partial-update and aggregation compacted DV reads.
  • Level-0, non-raw-convertible, unknown/nonzero delete-row count, and merge-on-read rejection.
  • cargo test -p paimon --lib
  • cargo test -p paimon-datafusion --test pk_tables
  • cargo clippy -p paimon --lib --tests -- -D warnings

API and Format

No public API or storage format changes.

Documentation

Updated the SQL merge-engine documentation with the supported read-only scope and limitations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support materialized partial-update and aggregation reads with deletion vectors

1 participant