feat: support incremental scan between 2 snapshots#1470
feat: support incremental scan between 2 snapshots#1470xxchan wants to merge 4 commits intoapache:mainfrom
Conversation
| } | ||
| } | ||
|
|
||
| struct Ancestors { |
There was a problem hiding this comment.
This can be handy for validation in SnapshotProducer as well, maybe we should move this to a new module util/snapshot.rs
There was a problem hiding this comment.
btw, what kind of validation are you talking about? It seems currently there's no such place where Ancestors can be used
There was a problem hiding this comment.
Snapshot validation is not implemented yet, I have a stale PR for that. My PR also comes with the logic of scanning ancestors, but I like your version much better and would love to reuse what you have here when we restart the snapshot validation work
xxchan
force-pushed
the
xxchan/naval-heron
branch
from
a3f5068 to
d87a9d2
Compare
This refactoring addresses the PR comment suggesting that the Ancestors struct could be reused by SnapshotProducer and should be moved to a dedicated utilities module for better code organization. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
xxchan
force-pushed
the
xxchan/naval-heron
branch
from
d87a9d2 to
d5fb028
Compare
| let (delete_file_idx, delete_file_tx) = DeleteFileIndex::new(); | ||
|
|
||
| let manifest_list = plan_context.get_manifest_list().await?; | ||
| // For incremental scan, disable delete file processing |
There was a problem hiding this comment.
If we processed delete files, your PR would essentially almost be producing a changelog, like in Spark, right? Perhaps not actionable right now, but mentioning this as a possible extension in future.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
3 participants
Which issue does this PR close?
close #1469
What changes are included in this PR?
Added
to_snapshot_idandfrom_snapshot_idto the table scan API.Are these changes tested?
unit test