fix: tolerate and auto-heal FRI bitmap straddle corruption (#6623) (#6625)

## Summary

- Tables corrupted by the bug fixed in #6610 (a deferred-remap
compaction committing concurrently with `optimize_indices` on a stale
handle leaves a user index whose `fragment_bitmap` straddles an FRI
rewrite group) used to panic in `Dataset::load_indices` and were
unreadable.
- `remap_fragment_bitmap` now tolerates the straddle on read: it drops
the indexed old-frag IDs from the bitmap, logs `tracing::warn!`, and
skips the new-frag insertion. Affected rows fall through to flat scan
until the next `optimize_indices`. No data lost; no retrain.
- The fix auto-heals on the next write. `commit_transaction` already
seeds `build_manifest` from `load_indices()`, which now returns the
cleaned bitmaps, so any commit (e.g. `delete("false")`) persists the
corrected manifest. No new API, no user action required.

## Reproduction & test fixture

`rust/examples/src/fri_straddle_datagen.rs` reproduces the corrupt state
deterministically on pre-#6610 builds via the public `plan_compaction` /
`CompactionTask::execute` / `optimize_indices` / `commit_compaction`
APIs. Python can't reach this path — `Compaction.commit` hardcodes
`CompactionOptions::default()` (no `defer_index_remap`) — so the
generator is in Rust. On a fixed build the generator fails loudly on the
conflict-resolver rejection, which is the desired forward-compat
behaviour.

The output is checked in at `test_data/fri_straddle_pre_6610/` and
exercised by `index::tests::test_load_indices_tolerates_fri_straddle`
and `test_auto_heal_persists_cleaned_bitmap`. The latter opens the
fixture, performs a no-op `delete("false")`, and asserts the raw on-disk
`fragment_bitmap` (read via `read_manifest_indexes`) matches the cleaned
bitmap.

## Test plan

- [x] `cargo test -p lance --lib
index::tests::test_load_indices_tolerates_fri_straddle
index::tests::test_auto_heal_persists_cleaned_bitmap`
- [x] `cargo test -p lance --lib index::tests` — 65 passing
- [x] `cargo clippy -p lance --lib -- -D warnings`
- [x] `cargo fmt --all`

## Follow-ups (out of scope)

- Explicit `Dataset::repair()` API for users who want an auditable
trigger / report without writing.
- Better `Dataset::validate()` reporting that surfaces the corruption
explicitly rather than silently healing.

Fixes #6623

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: wjones127

rust/lance-index/src/frag_reuse.rs

@@ -323,14 +323,24 @@ impl FragReuseIndex {
 
                 if removed > 0 {
                     if removed != group.old_frags.len() {
-                        // This should never happen because we always commit a full rewrite group
-                        // and we always reindex either the entire group or nothing.
-                        // We use invalid input to be consistent with
-                        // dataset::transaction::recalculate_fragment_bitmap
-                        return Err(Error::invalid_input(format!(
-                            "The compaction plan included a rewrite group that was a split of indexed and non-indexed data: {:?}",
-                            group.old_frags
-                        )));
+                        // Straddle: the index covered only part of this rewrite
+                        // group. Caused by the bug fixed in
+                        // <https://github.com/lance-format/lance/pull/6610>.
+                        // We've already removed the indexed old_frags from the
+                        // bitmap above; deliberately do NOT insert new_frags,
+                        // since the merged fragment also contains rows that
+                        // were never indexed. Affected rows fall through to
+                        // flat scan until the next optimize_indices. The fix
+                        // is persisted on the next write via build_manifest.
+                        tracing::warn!(
+                            "Healing straddling fragment-reuse rewrite group in index bitmap: \
+                             group {:?} was only partially indexed ({} of {} old fragments). \
+                             Affected rows will use flat scan until the next optimize_indices.",
+                            group.old_frags,
+                            removed,
+                            group.old_frags.len(),
+                        );
+                        continue;
                     }
 
                     for new_frag in group.new_frags.iter() {

rust/lance/src/index.rs

@@ -7138,4 +7138,68 @@ mod tests {
         assert_eq!(desc.index_type(), "IVF_FLAT");
         assert!(!desc.field_ids().is_empty());
     }
+
+    /// FRI-straddle corruption (PR #6610) used to panic in `load_indices`.
+    /// The fixture is a pre-#6610 dataset where a user index's
+    /// `fragment_bitmap` only partially covers a rewrite group. After the
+    /// tolerant-load fix `load_indices` returns Ok; affected old-frag IDs
+    /// are dropped, no new-frag IDs are inserted, and `validate()` succeeds.
+    #[tokio::test]
+    async fn test_load_indices_tolerates_fri_straddle() {
+        let tmp = copy_test_data_to_tmp("fri_straddle_pre_6610/fri_straddle_dataset").unwrap();
+        let uri = format!("file://{}", tmp.std_path().display());
+        let dataset = Dataset::open(&uri).await.unwrap();
+
+        let indices = dataset.load_indices().await.unwrap();
+        assert!(!indices.is_empty());
+        dataset.validate().await.unwrap();
+    }
+
+    /// Any commit reseeds indices via `load_indices` → `build_manifest`,
+    /// so a single no-op write persists the cleaned bitmap to disk.
+    #[tokio::test]
+    async fn test_auto_heal_persists_cleaned_bitmap() {
+        use lance_table::io::manifest::read_manifest_indexes;
+
+        let tmp = copy_test_data_to_tmp("fri_straddle_pre_6610/fri_straddle_dataset").unwrap();
+        let uri = format!("file://{}", tmp.std_path().display());
+
+        // Sanity: the on-disk fixture has at least one straddling segment.
+        let pre = Dataset::open(&uri).await.unwrap();
+        let raw_pre =
+            read_manifest_indexes(&pre.object_store, &pre.manifest_location, &pre.manifest)
+                .await
+                .unwrap();
+        let cleaned = pre.load_indices().await.unwrap();
+        let any_changed = raw_pre
+            .iter()
+            .zip(cleaned.iter())
+            .any(|(r, c)| r.fragment_bitmap != c.fragment_bitmap);
+        assert!(
+            any_changed,
+            "fixture should have at least one segment whose bitmap is cleaned at load"
+        );
+        drop(pre);
+
+        // No-op delete commits a fresh manifest seeded from cleaned indices.
+        let mut dataset = Dataset::open(&uri).await.unwrap();
+        dataset.delete("false").await.unwrap();
+
+        // Reopen and read raw manifest indices: cleaned bitmaps now persisted.
+        let healed = Dataset::open(&uri).await.unwrap();
+        let raw_post = read_manifest_indexes(
+            &healed.object_store,
+            &healed.manifest_location,
+            &healed.manifest,
+        )
+        .await
+        .unwrap();
+        let cleaned_post = healed.load_indices().await.unwrap();
+        for (r, c) in raw_post.iter().zip(cleaned_post.iter()) {
+            assert_eq!(
+                r.fragment_bitmap, c.fragment_bitmap,
+                "after auto-heal, raw on-disk bitmap should match cleaned bitmap"
+            );
+        }
+    }
 }

test_data/fri_straddle_pre_6610/README.md

@@ -0,0 +1,37 @@
+# `fri_straddle_pre_6610`
+
+A dataset corrupted by the bug fixed in
+[lance-format/lance#6610](https://github.com/lance-format/lance/pull/6610):
+a deferred-remap compaction commits concurrently with `optimize_indices`
+on a stale handle, leaving a user index whose `fragment_bitmap` straddles
+a fragment-reuse rewrite group. `Dataset::list_indices()` panics with
+"The compaction plan included a rewrite group that was a split of indexed
+and non-indexed data".
+
+Used by `dataset::repair` Rust tests to verify detection,
+`Dataset::validate()` error reporting, and `Dataset::repair()` against a
+real on-disk corrupt manifest.
+
+## Regenerating
+
+The reproduction is deterministic in Rust because we can drive
+`plan_compaction` / `CompactionTask::execute` / `commit_compaction`
+directly with `defer_index_remap=true` and interleave `optimize_indices`
+between them. Python's `Compaction.commit` hardcodes
+`CompactionOptions::default()` (no defer-remap), so a Python datagen
+cannot reach this path without a tight commit race.
+
+The generator is a standalone crate at `datagen/` that pins `lance` to
+the last pre-#6610 release tag (`v6.0.0-beta.3`). It is deliberately not
+part of the parent Cargo workspace so it can compile against the buggy
+build. On a fixed build the conflict resolver rejects the rewrite as a
+retryable conflict and the generator fails loudly. To regenerate:
+
+```bash
+cargo run --release \
+    --manifest-path test_data/fri_straddle_pre_6610/datagen/Cargo.toml -- \
+    test_data/fri_straddle_pre_6610/fri_straddle_dataset
+```
+
+Bump the pinned tag in `datagen/Cargo.toml` only to another pre-#6610
+build.