Add FlatFieldVectorsWriter#asKnnVectorValues (#16062)

FlatFieldVectorsWriter<T> exposes its accumulated vectors via List<T> getVectors().
This is both too generic and too prescriptive:

List<T> binds the return type to specific vector representation (T = float[] / byte[] heap arrays, one vector per list element); if a FlatFieldVectorsWriter implementation is backed by a different data structure (off-heap / paged / memory-mapped storage), it has to materialize a heap array even when the downstream consumer would be happy with a non-materialized view.
it does not carry enough information with it (e.g. encoding, dimensions for sub-byte packed types, etc.). In fact, current consumes have to wrap that list into a KnnVectorValues to feed the scoring / graph-building path (see e.g. Lucene99HnswVectorsWriter.FieldWriter).
This PR proposes to move this step directly to FlatFieldVectorsWriter<T>, exposing its vectors also as KnnVectorValues.

Why KnnVectorValues?

KnnVectorValues is the shape current callers (e.g. scorers) actually use. This PR updates Lucene99HnswVectorsWriter to use the new function when it calls FlatVectorsScorer.getRandomVectorScorerSupplier, which already takes KnnVectorValues.
No new Lucene type is needed; an alternative would be to introduce a new Lucene interface (e.g. FlatVectorsView). That would be very similar to  KnnVectorValues with minor variations.

Author: ldematte

lucene/CHANGES.txt

@@ -31,6 +31,9 @@ API Changes
 
 * GITHUB#15990: Add experimental api to IndexWriter for columnar indexing. (Tim Brooks)
 
+* GITHUB#16062: Add FlatFieldVectorsWriter.asKnnVectorValues to return directly a KnnVectorValues "view" over
+  vectors to be written. (Lorenzo Dematte)
+
 New Features
 ---------------------
 

lucene/core/src/java/org/apache/lucene/codecs/hnsw/FlatFieldVectorsWriter.java

@@ -20,7 +20,11 @@
 import java.io.IOException;
 import java.util.List;
 import org.apache.lucene.codecs.KnnFieldVectorsWriter;
+import org.apache.lucene.index.ByteVectorValues;
 import org.apache.lucene.index.DocsWithFieldSet;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.KnnVectorValues;
+import org.apache.lucene.index.VectorEncoding;
 
 /**
  * Vectors' writer for a field
@@ -34,6 +38,25 @@ public abstract class FlatFieldVectorsWriter<T> extends KnnFieldVectorsWriter<T>
    */
   public abstract List<T> getVectors();
 
+  /**
+   * Returns a {@link KnnVectorValues} view over the vectors to be written.
+   *
+   * <p>This default implementation uses {@link #getVectors()} and wraps them via {@link
+   * FloatVectorValues#fromFloats} / {@link ByteVectorValues#fromBytes}. Subclasses backed by
+   * alternative storage may override to return a richer view (e.g. one that exposes capability
+   * information to scorers, or avoids heap materialization).
+   *
+   * @param encoding the vector encoding/element type
+   * @param dim the field's declared vector dimension
+   */
+  @SuppressWarnings("unchecked")
+  public KnnVectorValues asKnnVectorValues(VectorEncoding encoding, int dim) throws IOException {
+    return switch (encoding) {
+      case FLOAT32 -> FloatVectorValues.fromFloats((List<float[]>) getVectors(), dim);
+      case BYTE -> ByteVectorValues.fromBytes((List<byte[]>) getVectors(), dim);
+    };
+  }
+
   /**
    * @return the docsWithFieldSet for the field writer
    */

lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java

@@ -38,10 +38,8 @@
 import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
 import org.apache.lucene.codecs.hnsw.FlatVectorsScorer;
 import org.apache.lucene.codecs.hnsw.FlatVectorsWriter;
-import org.apache.lucene.index.ByteVectorValues;
 import org.apache.lucene.index.DocsWithFieldSet;
 import org.apache.lucene.index.FieldInfo;
-import org.apache.lucene.index.FloatVectorValues;
 import org.apache.lucene.index.IndexFileNames;
 import org.apache.lucene.index.KnnVectorValues;
 import org.apache.lucene.index.MergeState;
@@ -739,20 +737,10 @@ static FieldWriter<?> create(
       this.flatFieldVectorsWriter = Objects.requireNonNull(flatFieldVectorsWriter);
       this.graphThreshold = tinySegmentsThreshold;
       this.scorerSupplier =
-          switch (fieldInfo.getVectorEncoding()) {
-            case BYTE ->
-                scorer.getRandomVectorScorerSupplier(
-                    fieldInfo.getVectorSimilarityFunction(),
-                    ByteVectorValues.fromBytes(
-                        (List<byte[]>) flatFieldVectorsWriter.getVectors(),
-                        fieldInfo.getVectorDimension()));
-            case FLOAT32 ->
-                scorer.getRandomVectorScorerSupplier(
-                    fieldInfo.getVectorSimilarityFunction(),
-                    FloatVectorValues.fromFloats(
-                        (List<float[]>) flatFieldVectorsWriter.getVectors(),
-                        fieldInfo.getVectorDimension()));
-          };
+          scorer.getRandomVectorScorerSupplier(
+              fieldInfo.getVectorSimilarityFunction(),
+              flatFieldVectorsWriter.asKnnVectorValues(
+                  fieldInfo.getVectorEncoding(), fieldInfo.getVectorDimension()));
 
       if (graphThreshold <= 0) {
         // Initialize graph builder if optimization is disabled