Commit 462faf7
fix: apply fragment bitmap allow-list to index search results (#6563)
## Summary
- Restrict index search results to only fragments in the index's
`fragment_bitmap`, filtering out stale entries from fragments whose data
changed but whose index was not rewritten
- Apply the allow-list in `DatasetPreFilter` (vector/FTS queries) and
`MapIndexExec` (scalar index queries used by merge_insert)
- Add regression tests for stale index entries after `DataReplacement`
and partial-schema `merge_insert` (BTree, FTS, and vector indices)
## Test plan
- [x] `test_data_replacement_populates_invalidated_bitmap` —
DataReplacement removes fragment from bitmap
- [x] `test_fts_stale_entries_after_data_replacement` — stale FTS
entries blocked after DataReplacement
- [x] `test_vector_index_after_data_replacement` — stale vector index
entries blocked after DataReplacement
- [x] `test_partial_merge_insert_stale_index_ambiguous` — repeated
partial merge_insert doesn't cause ambiguous errors
- [x] `test_partial_merge_insert_stale_index_fragment_not_exist` — stale
btree entries don't cause fragment-not-found errors
- [x] `test_partial_merge_insert_stale_index_batch_size_mismatch` —
stale btree entries don't cause batch size mismatch
- [x] `test_partial_merge_insert_stale_vector_index_duplicates` — no
duplicate rows from stale vector index
- [x] `test_partial_merge_insert_stale_fts_index_duplicates` — no
duplicate rows from stale FTS index
- [x] `test_compaction_after_invalidated_fragment` — compaction succeeds
after fragment invalidation
- [x] All 17 existing prefilter tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent f3206e0 commit 462faf7
4 files changed
Lines changed: 1163 additions & 17 deletions
File tree
- rust/lance/src
- dataset
- tests
- write
- index
- io/exec
0 commit comments