feat(transaction): RewriteFilesAction + Validation Support + Process Delete Manifests#1606
feat(transaction): RewriteFilesAction + Validation Support + Process Delete Manifests#1606CTTY wants to merge 22 commits intoapache:mainfrom
RewriteFilesAction + Validation Support + Process Delete Manifests#1606Conversation
|
|
||
| for entry in manifest.entries() { | ||
| match entry.content_type() { | ||
| DataContentType::Data => { |
There was a problem hiding this comment.
These are the exact same code blocks? So why does the ManifestEntry's DataContentType matter? Why not just do:
for entry in manifest.entries() {
if snapshot_producer
.deleted_data_files
.iter()
.any(|f| f.file_path == entry.data_file.file_path())
{
delete_entries.push(copy_with_deleted_status())
}
}There was a problem hiding this comment.
It looks very similar with one little difference: one block iterates thru deleted_data_files and another iterates thru deleted_delete_files
CTTY
force-pushed
the
ctty/pure-rewrite
branch
from
71a696f to
336c974
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Details a concrete, file-by-file plan to implement RewriteFilesAction in iceberg-rust — the compaction primitive that atomically swaps old data files for new ones in a single Replace snapshot. Key design decisions documented: - Extend SnapshotProducer with deleted_data_files + write_deleted_manifest() - New RewriteFilesOperation implementing SnapshotProduceOperation with Operation::Replace and manifest-level skip logic for deleted files - Public RewriteFilesAction API mirroring FastAppendAction patterns - Expose via Transaction::rewrite_files() - 7 unit tests covering basic flow, error cases, and catalog round-trip - Notes how this infrastructure enables OverwriteFiles and DeleteFiles next Relates to open upstream issue: apache#1606 https://claude.ai/code/session_01EkUKvhB7p4jz4qZxYaneBb
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
2 participants
Which issue does this PR close?
RewriteFilessupport to iceberg-rust #1607What changes are included in this PR?
RewriteFilesActionSnapshotValidatorand implemented it forRewriteFilesOperationSnapshotProducerand generate manifestsAre these changes tested?
Not tested so far