[Feature] Mark vectorSearch() experimental and default to efficient filtering

Mark the vectorSearch() table function as experimental in the user doc
following the repo convention (title [Experimental] suffix), and flip the
default WHERE filter placement from post-filtering to efficient
pre-filtering so a query without filter_type embeds the predicate under
knn.filter for ANN-time pruning.

Production code: filterType=null in VectorSearchIndex now resolves to
FilterType.EFFICIENT, and VectorSearchQueryBuilder's full constructor
defaults to EFFICIENT when passed null. The test-only 3-arg constructor
stays pinned to POST because it does not wire a rebuildKnnWithFilter
callback and EFFICIENT mode requires one.

Allow-list error messages are reworded to neutral wording
("vectorSearch WHERE pre-filtering does not support...") so default-path
users never see internal filter_type=efficient terminology and get a
clear "set filter_type=post" fallback hint.

Doc updates the Filtering section to describe Omitted=efficient as the
default, with post framed as the opt-in fallback for predicates outside
the efficient allow-list. Example 4 shows the default knn.filter shape;
Example 5 shows filter_type=post for arithmetic predicates.

Tests: BETWEEN / NOT IN regression guards pin filter_type=post
explicitly so they continue to assert the post-filter DSL shape.
testPostFilterReturnsOnlyMatchingDocs pins filter_type=post so the test
name still reflects what it exercises. New default-shape IT coverage
asserts knn.filter embeds the predicate and there is no outer bool
wrapping.

Signed-off-by: Eric Wei <mengwei.eric@gmail.com>

Author: mengweieric

docs/user/dql/vector-search.rst

@@ -1,7 +1,7 @@
 
-=============
-Vector Search
-=============
+==============================
+Vector Search [Experimental]
+==============================
 
 .. rubric:: Table of contents
 
@@ -12,6 +12,9 @@ Vector Search
 Introduction
 ============
 
+``vectorSearch()`` is an experimental feature. Syntax, options, and
+pushdown behavior may change in future releases based on feedback.
+
 The ``vectorSearch()`` table function runs a k-NN query against a ``knn_vector``
 field and exposes the matching documents as a relation in the ``FROM`` clause.
 It relies on the OpenSearch `k-NN plugin
@@ -170,42 +173,45 @@ A ``WHERE`` clause on non-vector fields of the ``vectorSearch()`` alias is
 pushed down to OpenSearch when it can be translated to an OpenSearch filter.
 Two placement strategies are available via the ``filter_type`` option:
 
-- ``post`` — the ``WHERE`` predicate is applied as a non-scoring
+- ``efficient`` (default): the ``WHERE`` predicate is embedded directly
+  inside the k-NN query (``knn.filter``), enabling pre-filtering during
+  the ANN search. See the `k-NN filtering guide
+  <https://docs.opensearch.org/latest/vector-search/filter-search-knn/efficient-knn-filtering/>`_
+  for engine and method requirements.
+- ``post``: the ``WHERE`` predicate is applied as a non-scoring
   ``bool.filter`` alongside the k-NN query. The k-NN query runs first and
   its results are then filtered.
-- ``efficient`` — the ``WHERE`` predicate is embedded directly inside the
-  k-NN query (``knn.filter``), enabling pre-filtering during the ANN search.
-  See the `k-NN filtering guide <https://docs.opensearch.org/latest/vector-search/filter-search-knn/efficient-knn-filtering/>`_
-  for engine and method requirements.
 
 Behavior depends on whether ``filter_type`` is specified:
 
-- **Omitted** — pushdown is attempted using the ``post`` placement.
-  Predicates that translate to native OpenSearch queries are pushed down as a
-  ``bool.filter`` alongside the k-NN query. Predicates that do not have a
-  native equivalent (for example, arithmetic or function calls on indexed
-  fields) are pushed down as an OpenSearch script query and evaluated
-  server-side. Only when predicate translation itself fails does the engine
-  fall back to evaluating the ``WHERE`` clause in memory after the k-NN
-  results are returned. A query with no ``WHERE`` clause is valid.
-- **Explicit ``post``** — a ``WHERE`` clause is required and must be
-  translatable to an OpenSearch filter query. If the ``WHERE`` clause is
-  missing or cannot be translated, the query fails with an error.
-  Specifying ``filter_type=post`` explicitly is useful when the query
-  should fail with an error instead of silently falling back to
-  in-memory filtering.
-- **Explicit ``efficient``** — a ``WHERE`` clause is required and must
-  compile to a filter shape that can be embedded under ``knn.filter``.
+- **Omitted (default, ``efficient``)**: the ``WHERE`` predicate is
+  embedded under ``knn.filter`` so the k-NN engine pre-filters candidates
+  during the ANN search. A query with no ``WHERE`` clause is valid.
   ``efficient`` supports simple native filters: ``term``, ``range``,
   ``wildcard``, ``exists``, full-text family (``match``, ``match_phrase``,
   ``match_phrase_prefix``, ``match_bool_prefix``, ``multi_match``,
   ``query_string``, ``simple_query_string``), and boolean combinations of
   those filters. Predicates that compile to script queries (arithmetic,
-  function calls, ``CASE``, date math), nested predicates, and other
-  query shapes are not supported in this mode and return an error.
+  function calls on indexed fields, ``CASE``, date math), nested
+  predicates, and other query shapes are not supported under
+  ``knn.filter`` and return an error. Set ``filter_type=post`` to apply
+  such predicates after the k-NN search.
+- **Explicit ``efficient``**: same contract as the default. Specifying
+  it is equivalent and is useful when a query should be explicit about
+  the placement strategy.
+- **Explicit ``post``**: a ``WHERE`` clause is required and must be
+  translatable to an OpenSearch filter query. Predicates that translate
+  to native OpenSearch queries are pushed down as a ``bool.filter``
+  alongside the k-NN query. Predicates that do not have a native
+  equivalent (for example, arithmetic or function calls on indexed
+  fields) are pushed down as an OpenSearch script query and evaluated
+  server-side. Only when predicate translation itself fails does the
+  engine fall back to evaluating the ``WHERE`` clause in memory after
+  the k-NN results are returned. Use ``filter_type=post`` when the
+  predicate shape is not supported by ``efficient`` pre-filtering.
 
-Example 4: Implicit pushdown (no ``filter_type``)
--------------------------------------------------
+Example 4: Default pre-filtering (no ``filter_type``)
+-----------------------------------------------------
 
 ::
 
@@ -223,10 +229,14 @@ Example 4: Implicit pushdown (no ``filter_type``)
       """
     }
 
-Example 5: Efficient (pre-)filtering
-------------------------------------
+The predicate is embedded under ``knn.filter`` so the k-NN engine
+pre-filters candidates during the ANN search.
 
-::
+Example 5: Post-filtering fallback
+----------------------------------
+
+Use ``filter_type=post`` for predicates that do not fit the ``efficient``
+allow-list, such as arithmetic or function calls on indexed fields::
 
     POST /_plugins/_sql
     {
@@ -236,9 +246,9 @@ Example 5: Efficient (pre-)filtering
           table='my_vectors',
           field='embedding',
           vector='[0.1, 0.2, 0.3]',
-          option='k=10,filter_type=efficient'
+          option='k=10,filter_type=post'
         ) AS v
-        WHERE v.category = 'books'
+        WHERE v.price * 1.1 < 100
       """
     }
 

integ-test/src/test/java/org/opensearch/sql/sql/VectorSearchExecutionIT.java

@@ -135,14 +135,16 @@ public void testTopKReturnsNearestSortedByScore() throws IOException {
   public void testPostFilterReturnsOnlyMatchingDocs() throws IOException {
     // Query from cluster B with WHERE state='TX' forces POST filtering to surface TX docs
     // (cluster A) even though the vector is closer to cluster B. k=10 covers all 6 docs so
-    // post-filtering to state='TX' deterministically yields exactly {1,2,3}.
+    // post-filtering to state='TX' deterministically yields exactly {1,2,3}. filter_type=post
+    // is specified explicitly because the default placement is EFFICIENT — this test
+    // guarantees POST continues to work when the user opts into it.
     JSONObject result =
         executeJdbcRequest(
             "SELECT v._id, v._score "
                 + "FROM vectorSearch(table='"
                 + TEST_INDEX
                 + "', field='embedding', "
-                + "vector='[9.0, 9.0]', option='k=10') AS v "
+                + "vector='[9.0, 9.0]', option='k=10,filter_type=post') AS v "
                 + "WHERE v.state = 'TX' "
                 + "LIMIT 10");
 

integ-test/src/test/java/org/opensearch/sql/sql/VectorSearchExplainIT.java

@@ -164,10 +164,10 @@ public void testExplainRadialMinScoreProducesKnnQuery() throws IOException {
         "Radial without WHERE should not embed a filter:\n" + knnJson, knnJson.contains("filter"));
   }
 
-  // ── Post-filter DSL shape ────────────────────────────────────────────
+  // ── Default (EFFICIENT) pre-filter DSL shape ────────────────────────
 
   @Test
-  public void testExplainPostFilterProducesBoolQuery() throws IOException {
+  public void testExplainDefaultFilterProducesKnnWithFilter() throws IOException {
     String explain =
         explainQuery(
             "SELECT v._id, v._score "
@@ -178,39 +178,32 @@ public void testExplainPostFilterProducesBoolQuery() throws IOException {
                 + "WHERE v.state = 'TX' "
                 + "LIMIT 10");
 
-    // Post-filter shape: outer bool.must=[knn], bool.filter=[term] — WHERE lives OUTSIDE the knn
-    // payload. Verify by decoding the wrapper and asserting the predicate field is NOT embedded.
+    // Default (EFFICIENT) shape: WHERE is embedded inside knn.filter, the knn JSON is base64-
+    // encoded inside a WrapperQueryBuilder, and there is no outer bool/must wrapping.
     String sourceBuilderJson = extractSourceBuilderJson(explain);
-    assertTrue(
-        "Explain should contain bool query:\n" + sourceBuilderJson,
+    assertFalse(
+        "Default EFFICIENT mode should not produce bool query:\n" + sourceBuilderJson,
         sourceBuilderJson.contains("\"bool\""));
-    assertTrue(
-        "Explain should contain must clause (knn in scoring context):\n" + sourceBuilderJson,
+    assertFalse(
+        "Default EFFICIENT mode should not contain must clause:\n" + sourceBuilderJson,
         sourceBuilderJson.contains("\"must\""));
-    assertTrue(
-        "Explain should contain filter clause (WHERE in non-scoring context):\n"
-            + sourceBuilderJson,
-        sourceBuilderJson.contains("\"filter\""));
-    assertTrue(
-        "Explain should contain the outer state predicate:\n" + sourceBuilderJson,
-        sourceBuilderJson.contains("\"state.keyword\""));
 
     String knnJson = decodeSoleKnnJson(explain);
     assertTrue("knn JSON should contain knn key:\n" + knnJson, knnJson.contains("\"knn\""));
     assertTrue(
         "knn JSON should target the embedding field:\n" + knnJson,
         knnJson.contains("\"embedding\""));
     assertTrue("knn JSON should contain k=10:\n" + knnJson, knnJson.contains("\"k\":10"));
-    assertFalse(
-        "Post-filter mode must not embed the WHERE predicate inside knn:\n" + knnJson,
-        knnJson.contains("state"));
-    assertFalse(
-        "Post-filter mode must not embed a filter inside knn:\n" + knnJson,
+    assertTrue(
+        "Default EFFICIENT mode must embed filter inside knn:\n" + knnJson,
         knnJson.contains("filter"));
+    assertTrue(
+        "Default EFFICIENT mode must embed the WHERE predicate inside knn:\n" + knnJson,
+        knnJson.contains("state"));
   }
 
   @Test
-  public void testExplainCompoundPredicateProducesBoolQuery() throws IOException {
+  public void testExplainDefaultCompoundPredicateProducesKnnWithFilter() throws IOException {
     String explain =
         explainQuery(
             "SELECT v._id, v._score "
@@ -221,45 +214,35 @@ public void testExplainCompoundPredicateProducesBoolQuery() throws IOException {
                 + "WHERE v.state = 'TX' AND v.age > 30 "
                 + "LIMIT 10");
 
-    // Compound post-filter still uses outer bool.must=[knn]/bool.filter=[predicates]. Both WHERE
-    // predicates must stay outside the knn payload; otherwise efficient mode could false-positive.
+    // Compound default-mode WHERE must also route through knn.filter: no outer bool/must, and
+    // both predicate fields embedded inside the knn payload.
     String sourceBuilderJson = extractSourceBuilderJson(explain);
-    assertTrue(
-        "Explain should contain bool query:\n" + sourceBuilderJson,
+    assertFalse(
+        "Default EFFICIENT mode should not produce bool query:\n" + sourceBuilderJson,
         sourceBuilderJson.contains("\"bool\""));
-    assertTrue(
-        "Explain should contain must clause (knn in scoring context):\n" + sourceBuilderJson,
+    assertFalse(
+        "Default EFFICIENT mode should not contain must clause:\n" + sourceBuilderJson,
         sourceBuilderJson.contains("\"must\""));
-    assertTrue(
-        "Explain should contain filter clause (compound WHERE in non-scoring context):\n"
-            + sourceBuilderJson,
-        sourceBuilderJson.contains("\"filter\""));
-    assertTrue(
-        "Explain should contain the outer state predicate:\n" + sourceBuilderJson,
-        sourceBuilderJson.contains("\"state.keyword\""));
-    assertTrue(
-        "Explain should contain the outer age predicate:\n" + sourceBuilderJson,
-        sourceBuilderJson.contains("\"age\""));
 
     String knnJson = decodeSoleKnnJson(explain);
     assertTrue("knn JSON should contain knn key:\n" + knnJson, knnJson.contains("\"knn\""));
     assertTrue(
         "knn JSON should target the embedding field:\n" + knnJson,
         knnJson.contains("\"embedding\""));
     assertTrue("knn JSON should contain k=10:\n" + knnJson, knnJson.contains("\"k\":10"));
-    assertFalse(
-        "Compound post-filter must not embed the state predicate inside knn:\n" + knnJson,
+    assertTrue(
+        "Compound default EFFICIENT must embed filter inside knn:\n" + knnJson,
+        knnJson.contains("filter"));
+    assertTrue(
+        "Compound default EFFICIENT must embed the state predicate inside knn:\n" + knnJson,
         knnJson.contains("state"));
-    assertFalse(
-        "Compound post-filter must not embed the age predicate inside knn:\n" + knnJson,
+    assertTrue(
+        "Compound default EFFICIENT must embed the age predicate inside knn:\n" + knnJson,
         knnJson.contains("age"));
-    assertFalse(
-        "Compound post-filter must not embed a filter inside knn:\n" + knnJson,
-        knnJson.contains("filter"));
   }
 
   @Test
-  public void testExplainRadialWithWhereProducesBoolQuery() throws IOException {
+  public void testExplainDefaultRadialWithWhereProducesKnnWithFilter() throws IOException {
     String explain =
         explainQuery(
             "SELECT v._id, v._score "
@@ -270,22 +253,15 @@ public void testExplainRadialWithWhereProducesBoolQuery() throws IOException {
                 + "WHERE v.state = 'TX' "
                 + "LIMIT 100");
 
-    // Radial + WHERE should also keep the WHERE predicate in the outer bool.filter rather than
-    // embedding it into the radial knn payload.
+    // Radial + default WHERE must also use the EFFICIENT shape: no outer bool/must, radial
+    // parameters preserved inside the knn payload, and the WHERE predicate embedded alongside.
     String sourceBuilderJson = extractSourceBuilderJson(explain);
-    assertTrue(
-        "Explain should contain bool query:\n" + sourceBuilderJson,
+    assertFalse(
+        "Default EFFICIENT mode should not produce bool query:\n" + sourceBuilderJson,
         sourceBuilderJson.contains("\"bool\""));
-    assertTrue(
-        "Explain should contain must clause (knn in scoring context):\n" + sourceBuilderJson,
+    assertFalse(
+        "Default EFFICIENT mode should not contain must clause:\n" + sourceBuilderJson,
         sourceBuilderJson.contains("\"must\""));
-    assertTrue(
-        "Explain should contain filter clause (WHERE in non-scoring context):\n"
-            + sourceBuilderJson,
-        sourceBuilderJson.contains("\"filter\""));
-    assertTrue(
-        "Explain should contain the outer state predicate:\n" + sourceBuilderJson,
-        sourceBuilderJson.contains("\"state.keyword\""));
 
     String knnJson = decodeSoleKnnJson(explain);
     assertTrue("knn JSON should contain knn key:\n" + knnJson, knnJson.contains("\"knn\""));
@@ -295,12 +271,12 @@ public void testExplainRadialWithWhereProducesBoolQuery() throws IOException {
     assertTrue(
         "knn JSON should contain max_distance=10.5:\n" + knnJson,
         knnJson.contains("\"max_distance\":10.5"));
-    assertFalse(
-        "Radial post-filter must not embed the WHERE predicate inside knn:\n" + knnJson,
-        knnJson.contains("state"));
-    assertFalse(
-        "Radial post-filter must not embed a filter inside knn:\n" + knnJson,
+    assertTrue(
+        "Radial default EFFICIENT must embed filter inside knn:\n" + knnJson,
         knnJson.contains("filter"));
+    assertTrue(
+        "Radial default EFFICIENT must embed the WHERE predicate inside knn:\n" + knnJson,
+        knnJson.contains("state"));
   }
 
   // ── Sort + LIMIT explain ─────────────────────────────────────────────
@@ -456,13 +432,16 @@ public void testEfficientFilterWithOrderByScoreDescSucceeds() throws IOException
 
   @Test
   public void testBetweenPushesAsRange() throws IOException {
+    // Pin filter_type=post to keep the regression guard aimed at the post-filter serialization
+    // path: these assertions lock in the outer bool/must/filter shape that only appears when
+    // WHERE is applied alongside knn rather than embedded under knn.filter.
     String explain =
         explainQuery(
             "SELECT v._id, v._score "
                 + "FROM vectorSearch(table='"
                 + TEST_INDEX
                 + "', field='embedding', "
-                + "vector='[1.0, 2.0, 3.0]', option='k=10') AS v "
+                + "vector='[1.0, 2.0, 3.0]', option='k=10,filter_type=post') AS v "
                 + "WHERE v.balance BETWEEN 50 AND 200 "
                 + "LIMIT 10");
 
@@ -513,13 +492,16 @@ public void testBetweenPushesAsRange() throws IOException {
 
   @Test
   public void testNotInPushesAsMustNotTerms() throws IOException {
+    // Pin filter_type=post to keep the regression guard aimed at the post-filter serialization
+    // path: these assertions lock in the outer bool/must/filter shape that only appears when
+    // WHERE is applied alongside knn rather than embedded under knn.filter.
     String explain =
         explainQuery(
             "SELECT v._id, v._score "
                 + "FROM vectorSearch(table='"
                 + TEST_INDEX
                 + "', field='embedding', "
-                + "vector='[1.0, 2.0, 3.0]', option='k=10') AS v "
+                + "vector='[1.0, 2.0, 3.0]', option='k=10,filter_type=post') AS v "
                 + "WHERE v.gender NOT IN ('M', 'F') "
                 + "LIMIT 10");
 

integ-test/src/test/java/org/opensearch/sql/sql/VectorSearchIT.java

@@ -340,7 +340,8 @@ public void testEfficientModeRejectsScriptPredicate() throws IOException {
                         + "WHERE v.age + 1 > 30 "
                         + "LIMIT 5"));
 
-    assertThat(ex.getMessage(), containsString("filter_type=efficient does not support"));
+    assertThat(
+        ex.getMessage(), containsString("vectorSearch WHERE pre-filtering does not support"));
     assertThat(ex.getMessage(), containsString("script queries"));
   }
 

opensearch/src/main/java/org/opensearch/sql/opensearch/storage/VectorSearchIndex.java

@@ -28,7 +28,7 @@ public class VectorSearchIndex extends OpenSearchIndex {
   private final String field;
   private final float[] vector;
   private final Map<String, String> options;
-  private final FilterType filterType; // null means default (POST)
+  private final FilterType filterType; // null means default (EFFICIENT)
   // Nullable for back-compat with existing tests and the non-vector-search constructor. When
   // present, the scan defers a lazy k-NN plugin probe to open() so execution fails fast with a
   // clear SQL error if the plugin is missing.
@@ -62,7 +62,10 @@ public VectorSearchIndex(
     this(client, settings, indexName, field, vector, options, filterType, null);
   }
 
-  /** Default constructor — preserves existing call sites; uses no explicit filter type. */
+  /**
+   * Default constructor — preserves existing call sites; uses no explicit filter type, so the scan
+   * falls back to the default placement ({@link FilterType#EFFICIENT}).
+   */
   public VectorSearchIndex(
       OpenSearchClient client,
       Settings settings,
@@ -97,7 +100,7 @@ public TableScanBuilder createScanBuilder() {
         whereQuery -> new WrapperQueryBuilder(buildKnnQueryJson(whereQuery.toString()));
 
     boolean filterTypeExplicit = filterType != null;
-    FilterType effectiveFilterType = filterType != null ? filterType : FilterType.POST;
+    FilterType effectiveFilterType = filterType != null ? filterType : FilterType.EFFICIENT;
 
     var queryBuilder =
         new VectorSearchQueryBuilder(