Add dense vector field

[danalvrz]

No description provided.

[davisagli]

@reekitconcept Do you see any problem with adding this?

[danalvrz]

@davisagli I removed parameters where the value matches Solr's default, keeping only the ones we're explicitly overriding.

The ones that are specified are:

• vectorDimension to 768: this vector dimensionality is fairly standard in embedding models. We could add other fields for larger/smaller vectors if needed.
• similarityFunction to dot_product: that is according to https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html an optimized version of cosine with the caveat that the vectors must be normalized. That is possible or even default for current embedding options (e.g. OpenAI, nomic), and seemed like a manageable prerequisite in order to optimize querying time.
• hnswEfConstruction to 200: this is based on examples like https://aws.amazon.com/blogs/big-data/choose-the-k-nn-algorithm-for-your-billion-scale-use-case-with-opensearch/ and https://www.pinecone.io/learn/series/faiss/hnsw/ that show that there is an improvement in the recall with a higher number of nearest neighbor candidates at indexing, specially with high-dimension vectors. Though it has an impact in indexing time, but neither in memory usage nor in query latency (unlike hnswM).

I left hnswM as default (16) because in https://arxiv.org/pdf/1603.09320 and https://opensearch.org/blog/a-practical-guide-to-selecting-hnsw-hyperparameters/ it does not seem to be a very large improvement in the recall with larger numbers of links per node, and the memory impact increases linearly. It also increases query latency and indexing time.

This seemed to me like a reasonable starting point, but probably will be tuned as we move forward.

[davisagli]

@danalvrz Thanks, that explanation is helpful, and I think having it here is enough in case someone wants to track down later why we picked the settings we did.

[davisagli]

@danalvrz Balazs told me he'll take a look when he's back tomorrow

[reekitconcept]

I don't see any problem. I have not tested it but I would say if the CI passes then we are good.