Add `RewriteFiles` support to iceberg-rust

[CTTY]

Is your feature request related to a problem or challenge?

Currently, iceberg-rust does not support committing operations that both add and delete files. This feature, known as RewriteFiles in iceberg-java, is crucial for our use case, particularly for compaction.

Describe the solution you'd like

Based on my initial investigation in #1606, I believe the following three components are necessary to implement RewriteFiles in iceberg-rust:

• Process delete files: We need to add logic to handle delete files when generating snapshots.
• Implement RewriteFilesAction: This will require creating a new transaction action, RewriteFilesAction, and related classes like RewriteFilesOperation.
• Support validation: We need to add validation for rewrite operations to prevent conflicts. Currently we don't have any validation mechnism in the codebase, so a new trait SnapshotValidator would be needed.

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

[debugmiller]

I was just looking to do compaction. This would be great!

[jdockerty]

I believe introducing this RewriteFiles action also means exposing the ability to mark data files as DELETED too? It doesn't seem like that is currently possible within iceberg-rust, as the only changes are additive via FastAppend.

Being able to mark data files as deleted/replaced would be very helpful for our use case too. I think doing that is part of the RewriteFile action, it is something our use case would be greatly helped by.

Is there anyway that we can help move this along?

[jdockerty]

@Xuanwo Are you able to provide any insight/guidance on ☝️?

[CTTY]

Hey @jdockerty , sorry I thought I had replied but obviously I didn't.

I believe introducing this RewriteFiles action also means exposing the ability to mark data files as DELETED too?

In this issue, I meant to support RewriteFiles as long as we can process delete files. Delete file support can be added separately

[drbothen]

Hi @CTTY, I've been working on a RewriteFiles implementation for our compaction use case and have a working version I'd like to contribute. Before opening a PR, I wanted to check with you since you own this issue:

1. Are you actively working on feat(transaction): RewriteFilesAction + Validation Support + Process Delete Manifests #1606?
2. Would you be open to reviewing a PR that implements the core RewriteFilesAction + SnapshotProducer delete file support?

The implementation covers:

• SnapshotProducer changes to support deleted_data_files (write deleted manifest, validate files exist, summary tracking)
• RewriteFilesAction that uses Operation::Replace per the spec
• Validation that files to delete exist in the current snapshot
• 6 unit tests including round-trip with catalog

Happy to coordinate however works best!

[CTTY]

Hey @drbothen sorry for the late reply, as the storage trait work is about to be wrapped up, I'm picking this back up! We want to get transaction API to a good state to enable compaction use cases