Incremental scan

[xanderbailey]

Which issue does this PR close?

• Closes #.

What changes are included in this PR?

Adds incremental append scan support to iceberg-rust, allowing users to read only newly added data files between two snapshots. This is the Rust equivalent of Java's BaseIncrementalAppendScan.

Core scan module (crates/iceberg/src/scan/)

• New incremental.rs module containing:
  • AppendSnapshotSet: walks the snapshot ancestry chain between from_snapshot_id and to_snapshot_id, validates connectivity, and collects only APPEND operation snapshot IDs (skipping overwrite/delete/compaction — matching Java's BaseIncrementalAppendScan behavior)
  • IncrementalAppendScanBuilder: builder with the same configuration options as TableScanBuilder (column selection, predicates, concurrency limits, row group filtering, etc.)
  • Supports both exclusive (default, like Java) and inclusive from_snapshot semantics
• Refactored shared scan-build logic into ScanConfig + build_table_scan() to eliminate duplication between TableScanBuilder and IncrementalAppendScanBuilder
• Added ManifestFileFilter and ManifestEntryFilter callback types to PlanContext, used by incremental scans to:
  • Skip delete manifests and data manifests whose added_snapshot_id is outside the scan range
  • Include only entries with status == Added and snapshot_id within the append set

Table API (crates/iceberg/src/table.rs)

• Table::incremental_append_scan(from, to) — exclusive, matches Java's newIncrementalAppendScan()
• Table::incremental_append_scan_inclusive(from, to) — inclusive variant
• Same methods on StaticTable

DataFusion integration (crates/integrations/datafusion/)

• New ScanRange enum replacing the previous Option<i64> snapshot ID, supporting Latest, PointInTime, and Incremental variants
• IcebergStaticTableProvider gains three new constructors:
  • try_new_incremental(table, from, to) — exclusive
  • try_new_incremental_inclusive(table, from, to) — inclusive
  • try_new_appends_after(table, from) — exclusive, scans to current snapshot
• get_batch_stream updated to dispatch to the appropriate scan builder based on ScanRange

Are these changes tested?

Yes

Java comparison notes:

• AppendSnapshotSet::build mirrors Java's BaseIncrementalAppendScan.snapshotsBetween() — only APPEND snapshots are collected, non-APPEND are silently skipped
• Exclusive from_snapshot semantics match Java's default IncrementalAppendScan behavior
• Inclusive mode has no direct Java equivalent but follows the same ancestry walking logic
• Manifest file filtering (skip deletes, check added_snapshot_id) and entry filtering (status == ADDED, snapshot_id in range) match BaseIncrementalAppendScan.doPlanFiles() in Java

[xanderbailey]

I think this is a nice way to inject these filters and should extend to other future scans

[xanderbailey]

ManifestFileFilter and ManifestEntryFilter can't be debug

[CTTY]

We should use https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/util/snapshot.rs#L20

[xanderbailey]

ancestors_between is used here. We need to add validation and operation-type checking on top of it, we're not re-implementing the traversal. Does that make sense?

[xanderbailey]

There is duplication here between the scan builders which I've left for now until maybe we have a 3rd variant and we can pick out the common patterns a bit better.