Mostly get rid of FAISS fork in favor of vanilla FAISS version

[alexanderguzhva]

putting here as a backup copy, just in case if my hard drive burns away tomorrow :)

includes #1606

---

This is a broad FAISS fork-pruning pass for Knowhere. It removes large duplicated fork implementations and shifts IVF, FastScan, RaBitQ, Flat, DirectMap, and inverted-list plumbing toward baseline FAISS, while keeping Knowhere-specific behavior as focused adapters: cosine norm handling, CC segment-aware search, iterator hooks, ScaNN integration, and legacy serialization compatibility.

Faiss 1.14.1+ commit 46def46902231ed57552ddbcfd751e17720c6d32 plus merged facebookresearch/faiss#5156 plus merged facebookresearch/faiss#5172

Main pieces completed:

• Replaced fork IndexIVFFastScan with a baseline alias and deleted the fork .cpp; ordinary IndexIVFPQFastScan now aliases baseline FAISS, with only a small Knowhere cosine adapter retained.
• Preserved IwPf FastScan serialization while allowing baseline FastScan objects and HasInverseL2Norms-based cosine round-trips.
• Moved non-FastScan RaBitQ onto baseline faiss::IndexIVFRaBitQ, removed the fork RaBitQuantizer, added a thin iterator-hook shim, and exposed rbq_bits for multi-bit RaBitQ.
• Kept RaBitQ wire compatibility: legacy 1-bit indexes still write IwrQ, while baseline Iwrq and multi-bit Iwrr are readable/writable.
• Collapsed fork IVF/DirectMap/InvertedLists surfaces toward baseline types, adding InvertedListScannerStats and making IVF stats count post-filter computed distances.
• Isolated CC/qianya behavior into ConcurrentDirectMap and shared cc_search helpers.
• Slimmed IndexFlat to a baseline subclass that only preserves Knowhere’s Jaccard/Tanimoto behavior; removed IndexFlatElkan.
• Added/extended golden IVF serialization tests, RaBitQ config/search coverage, FastScan early-termination tests, and C/Python bindings for new IVF search parameters.

The commit is intentionally compatibility-heavy: the runtime implementation is pruned toward baseline FAISS, but existing Knowhere file formats, ScaNN recall-sensitive behavior, iterator paths, and cosine/refine semantics are kept as guardrails for this alpha version.

[mergify]

@alexanderguzhva 🔍 Important: PR Classification Needed!

For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:

1. If you're fixing a bug, label it as kind/bug.
2. For small tweaks (less than 20 lines without altering any functionality), please use kind/improvement.
3. Significant changes that don't modify existing functionalities should be tagged as kind/enhancement.
4. Adjusting APIs or changing functionality? Go with kind/feature.

For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”.

Thanks for your efforts and contribution to the community!.

[mergify]

@alexanderguzhva e2e jenkins job failed, comment /run-e2e-sse can trigger the job again.

[mergify]

@alexanderguzhva e2e jenkins job failed, comment /run-e2e can trigger the job again.

[mergify]

@alexanderguzhva e2e jenkins job failed, comment /run-e2e can trigger the job again.

[mergify]

@alexanderguzhva e2e jenkins job failed, comment /run-e2e-sse can trigger the job again.

[alexanderguzhva]

issue: #1501
/kind improvement

[foxspy]

/lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexanderguzhva, foxspy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [alexanderguzhva,foxspy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment