Commit 0f08160
authored
feat: support distributed IVF_RQ segment builds (#6359)
This PR wires IVF_RQ into the distributed vector index segment build
path so workers can build and commit shard-local RQ segments through the
existing distributed indexing workflow. It also adds the missing RQ
handling in the distributed auxiliary metadata plumbing and regression
coverage for distributed RQ segment build plus auxiliary merge.
Current limitation: IVF_RQ still does not have a pre-trained/shared RQ
metadata contract. Different RQ segments can therefore carry different
rotations and centroids, so merging multiple IVF_RQ segments into a
single physical segment is not a supported workflow yet. I plan to
follow up by adding shared/pre-trained RQ metadata so segment merge can
be supported safely.1 parent 3c35f48 commit 0f08160
5 files changed
Lines changed: 472 additions & 17 deletions
File tree
- rust
- lance-index/src/vector
- distributed
- lance/src/index
- vector
- ivf
0 commit comments