Merge remote-tracking branch 'origin/main' into cast-datetime

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Author: yuancu

docs/dev/intro-v3-engine.md

@@ -1,4 +1,4 @@
-# PPL Engine V3 (for 3.0.0-beta)
+# PPL Engine V3 (for 3.0.0)
 
 ---
 ## 1. Motivations
@@ -24,7 +24,7 @@ Find more details in [V3 Architecture](./intro-v3-architecture.md).
 ---
 ## 2. What's New
 
-In the initial release of the V3 engine (3.0.0-beta), the main new features focus on enhancing the PPL language while maintaining maximum compatibility with V2 behavior.
+In the initial release of the V3 engine (3.0.0), the main new features focus on enhancing the PPL language while maintaining maximum compatibility with V2 behavior.
 
 * **[Join](../user/ppl/cmd/join.rst) Command**
 * **[Lookup](../user/ppl/cmd/lookup.rst) Command**
@@ -35,7 +35,7 @@ In the initial release of the V3 engine (3.0.0-beta), the main new features focu
 
 ### 3.1 Breaking Changes
 
-Because of implementation changed internally, following behaviors are changed from 3.0.0-beta. (Behaviors in V3 is correct)
+Because of implementation changed internally, following behaviors are changed from 3.0.0. (Behaviors in V3 is correct)
 
 |                       Item                       |    V2     |          V3          |
 |:------------------------------------------------:|:---------:|:--------------------:|
@@ -51,7 +51,7 @@ Because of implementation changed internally, following behaviors are changed fr
 
 ### 3.2 Fallback Mechanism
 
-As v3 engine is experimental in 3.0.0-beta, not all PPL commands could work under this new engine. Those unsupported queries will be forwarded to V2 engine by fallback mechanism. To avoid impact on your side, normally you won't see any difference in a query response. If you want to check if and why your query falls back to be handled by V2 engine, please check OpenSearch log for "Fallback to V2 query engine since ...".
+As v3 engine is experimental in 3.0.0, not all PPL commands could work under this new engine. Those unsupported queries will be forwarded to V2 engine by fallback mechanism. To avoid impact on your side, normally you won't see any difference in a query response. If you want to check if and why your query falls back to be handled by V2 engine, please check OpenSearch log for "Fallback to V2 query engine since ...".
 
 ### 3.3 Limitations
 
@@ -66,24 +66,12 @@ For the following functionalities in V3 engine, the query will be forwarded to t
 
 #### Unsupported functionalities
 - All SQL queries
-- `trendline`
-- `show datasource`
-- `explain`
-- `describe`
-- `top` and `rare`
-- `fillnull`
-- `patterns`
 - `dedup` with `consecutive=true`
 - Search relevant commands
   - AD
   - ML
   - Kmeans
 - Commands with `fetch_size` parameter
-- query with metadata fields, `_id`, `_doc`, etc.
-- Json relevant functions
-  - cast to json
-  - json
-  - json_valid
 - Search relevant functions
   - match
   - match_phrase
@@ -105,6 +93,5 @@ If you're interested in the new query engine, please find more details in [V3 Ar
 The following items are on our roadmap with high priority:
 - Resolve the [V3 limitation](#33-limitations).
 - Advancing pushdown optimization and benchmarking
-- Backport to 2.19.x
 - Unified the PPL syntax between [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/opensearch-spark/blob/main/ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4)
 - Support more DSL aggregation

docs/user/limitations/limitations.rst

@@ -130,3 +130,31 @@ The response in JSON format is::
       },
       "status": 400
     }
+
+Limitations on Calcite Engine
+=============================
+
+Since 3.0.0, we introduce Apache Calcite as an experimental query engine. Please see `introduce v3 engine <../../../dev/intro-v3-engine.md>`_.
+For the following functionalities, the query will be forwarded to the V2 query engine.
+
+* All SQL queries
+
+* ``dedup`` with ``consecutive=true``
+
+* Search relevant commands
+
+    * AD
+    * ML
+    * Kmeans
+
+* Commands with ``fetch_size`` parameter
+
+* Search relevant functions
+
+    * match
+    * match_phrase
+    * match_bool_prefix
+    * match_phrase_prefix
+    * simple_query_string
+    * query_string
+    * multi_match

docs/user/ppl/limitations/limitations.rst

@@ -84,3 +84,31 @@ plugins.query.field_type_tolerance setting is enabled, the SQL/PPL plugin will h
 scalar data types, allowing basic queries (e.g., source = tbl | where condition). However, using multi-value
 fields in expressions or functions will result in exceptions. If this setting is disabled or absent, only the
 first element of an array is returned, preserving the default behavior.
+
+Unsupported Functionalities in Calcite Engine
+=============================================
+
+Since 3.0.0, we introduce Apache Calcite as an experimental query engine. Please see `introduce v3 engine <../../../dev/intro-v3-engine.md>`_.
+For the following functionalities, the query will be forwarded to the V2 query engine.
+
+* All SQL queries
+
+* ``dedup`` with ``consecutive=true``
+
+* Search relevant commands
+
+    * AD
+    * ML
+    * Kmeans
+
+* Commands with ``fetch_size`` parameter
+
+* Search relevant functions
+
+    * match
+    * match_phrase
+    * match_bool_prefix
+    * match_phrase_prefix
+    * simple_query_string
+    * query_string
+    * multi_match

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLSortIT.java

@@ -217,4 +217,22 @@ public void testSortWithNullValue() throws IOException {
         rows("Elinor", null),
         rows("Virginia", null));
   }
+
+  @Test
+  public void testSortDate() throws IOException {
+    JSONObject result =
+        executeQuery(
+            String.format(
+                "source=%s | sort birthdate | fields firstname, birthdate", TEST_INDEX_BANK));
+    verifySchema(result, schema("firstname", "string"), schema("birthdate", "timestamp"));
+    verifyDataRowsInOrder(
+        result,
+        rows("Amber JOHnny", "2017-10-23 00:00:00"),
+        rows("Hattie", "2017-11-20 00:00:00"),
+        rows("Nanette", "2018-06-23 00:00:00"),
+        rows("Elinor", "2018-06-27 00:00:00"),
+        rows("Dillard", "2018-08-11 00:00:00"),
+        rows("Virginia", "2018-08-19 00:00:00"),
+        rows("Dale", "2018-11-13 23:33:20"));
+  }
 }

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteSortCommandIT.java

@@ -5,6 +5,13 @@
 
 package org.opensearch.sql.calcite.remote;
 
+import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_BANK;
+import static org.opensearch.sql.util.MatcherUtils.rows;
+import static org.opensearch.sql.util.MatcherUtils.verifyOrder;
+
+import java.io.IOException;
+import org.json.JSONObject;
+import org.junit.Test;
 import org.opensearch.sql.ppl.SortCommandIT;
 
 public class CalciteSortCommandIT extends SortCommandIT {
@@ -14,4 +21,12 @@ public void init() throws Exception {
     enableCalcite();
     disallowCalciteFallback();
   }
+
+  // TODO: Move this test to SortCommandIT once head-then-sort is fixed in v2.
+  @Test
+  public void testHeadThenSort() throws IOException {
+    JSONObject result =
+        executeQuery(String.format("source=%s | head 2 | sort age | fields age", TEST_INDEX_BANK));
+    verifyOrder(result, rows(32), rows(36));
+  }
 }

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLDateTimeBuiltinFunctionIT.java

@@ -439,12 +439,14 @@ public void testWeekAndWeekOfYearWithFilter() {
     JSONObject actual =
         executeQuery(
             String.format(
+                Locale.ROOT,
                 "source=%s | fields  strict_date_optional_time"
                     + "| where YEAR(strict_date_optional_time) < 2000"
                     + "| where WEEK(DATE(strict_date_optional_time)) = %d"
                     + "| stats COUNT() AS CNT "
                     + "| head 1 ",
-                TEST_INDEX_DATE_FORMATS, week19840412));
+                TEST_INDEX_DATE_FORMATS,
+                week19840412));
 
     verifySchema(actual, schema("CNT", "bigint"));
 
@@ -1039,8 +1041,8 @@ public void testDateFormatAndDatetimeAndFromDays() throws IOException {
             "2017-11-02",
             expectedDatetimeAtPlus8,
             expectedDatetimeAtUTC,
-            String.format("%d 1984 %d", week19840412, week19840412),
-            String.format("%d %d 1984", week19840412Mode1, week19840412Mode1),
+            String.format(Locale.ROOT, "%d 1984 %d", week19840412, week19840412),
+            String.format(Locale.ROOT, "%d %d 1984", week19840412Mode1, week19840412Mode1),
             "09:07:42.000123"));
   }
 

integ-test/src/test/java/org/opensearch/sql/ppl/ExplainIT.java

@@ -74,7 +74,6 @@ public void testFilterAndAggPushDownExplain() throws IOException {
 
   @Test
   public void testSortPushDownExplain() throws IOException {
-    // TODO fix after https://github.com/opensearch-project/sql/issues/3380
     String expected =
         isCalciteEnabled()
             ? loadFromFile("expectedOutput/calcite/explain_sort_push.json")
@@ -89,6 +88,117 @@ public void testSortPushDownExplain() throws IOException {
                 + "| fields age"));
   }
 
+  @Test
+  public void testSortWithAggregationExplain() throws IOException {
+    // Sorts whose by fields are aggregators should not be pushed down
+    String expected =
+        isCalciteEnabled()
+            ? loadFromFile("expectedOutput/calcite/explain_sort_agg_push.json")
+            : loadFromFile("expectedOutput/ppl/explain_sort_agg_push.json");
+
+    assertJsonEqualsIgnoreId(
+        expected,
+        explainQueryToString(
+            "source=opensearch-sql_test_index_account"
+                + "| stats avg(age) AS avg_age by state, city "
+                + "| sort avg_age"));
+
+    // sorts whose by fields are not aggregators can be pushed down.
+    // This test is covered in testExplain
+  }
+
+  @Test
+  public void testMultiSortPushDownExplain() throws IOException {
+    // TODO: Fix the expected output in expectedOutput/ppl/explain_multi_sort_push.json (v2)
+    //  balance and gender should take precedence over account_number and firstname
+    String expected =
+        isCalciteEnabled()
+            ? loadFromFile("expectedOutput/calcite/explain_multi_sort_push.json")
+            : loadFromFile("expectedOutput/ppl/explain_multi_sort_push.json");
+
+    assertJsonEqualsIgnoreId(
+        expected,
+        explainQueryToString(
+            "source=opensearch-sql_test_index_account "
+                + "| sort account_number, firstname, address, balance "
+                + "| sort - balance, - gender, address "
+                + "| fields account_number, firstname, address, balance, gender"));
+  }
+
+  @Test
+  public void testSortThenAggregatePushDownExplain() throws IOException {
+    // TODO: Remove pushed-down sort in DSL in expectedOutput/ppl/explain_sort_then_agg_push.json
+    //  existing collations should be eliminated when pushing down aggregations (v2)
+    String expected =
+        isCalciteEnabled()
+            ? loadFromFile("expectedOutput/calcite/explain_sort_then_agg_push.json")
+            : loadFromFile("expectedOutput/ppl/explain_sort_then_agg_push.json");
+
+    assertJsonEqualsIgnoreId(
+        expected,
+        explainQueryToString(
+            "source=opensearch-sql_test_index_account"
+                + "| sort balance, age "
+                + "| stats avg(balance) by state"));
+  }
+
+  @Test
+  public void testSortWithRenameExplain() throws IOException {
+    String expected =
+        isCalciteEnabled()
+            ? loadFromFile("expectedOutput/calcite/explain_sort_rename_push.json")
+            : loadFromFile("expectedOutput/ppl/explain_sort_rename_push.json");
+
+    assertJsonEqualsIgnoreId(
+        expected,
+        explainQueryToString(
+            "source=opensearch-sql_test_index_account "
+                + "| rename firstname as name "
+                + "| eval alias = name "
+                + "|  sort alias "
+                + "| fields alias"));
+  }
+
+  /**
+   * Pushdown SORT and LIMIT Sort should be pushed down since DSL process sort before limit when
+   * they coexist
+   */
+  @Test
+  public void testSortThenLimitExplain() throws IOException {
+    String expected =
+        isCalciteEnabled()
+            ? loadFromFile("expectedOutput/calcite/explain_sort_then_limit_push.json")
+            : loadFromFile("expectedOutput/ppl/explain_sort_then_limit_push.json");
+    assertJsonEqualsIgnoreId(
+        expected,
+        explainQueryToString(
+            "source=opensearch-sql_test_index_account"
+                + "| sort age "
+                + "| head 5 "
+                + "| fields age"));
+  }
+
+  /**
+   * Push down LIMIT only Sort should NOT be pushed down since DSL process limit before sort when
+   * they coexist
+   */
+  @Test
+  public void testLimitThenSortExplain() throws IOException {
+    // TODO: Fix the expected output in expectedOutput/ppl/explain_limit_then_sort_push.json (v2)
+    //  limit-then-sort should not be pushed down.
+    String expected =
+        isCalciteEnabled()
+            ? loadFromFile("expectedOutput/calcite/explain_limit_then_sort_push.json")
+            : loadFromFile("expectedOutput/ppl/explain_limit_then_sort_push.json");
+    assertJsonEqualsIgnoreId(
+        expected,
+        explainQueryToString(
+            "source=opensearch-sql_test_index_account"
+                + "| head 5 "
+                + "| sort age "
+                + "| fields age"));
+  }
+
   @Test
   public void testLimitPushDownExplain() throws IOException {
     String expected =
@@ -240,6 +350,7 @@ public void testTrendlineWithSortPushDownExplain() throws IOException {
             ? loadFromFile("expectedOutput/calcite/explain_trendline_sort_push.json")
             : loadFromFile("expectedOutput/ppl/explain_trendline_sort_push.json");
 
+    // Sort will not be pushed down because there's a head before it.
     assertJsonEqualsIgnoreId(
         expected,
         explainQueryToString(

integ-test/src/test/java/org/opensearch/sql/ppl/SortCommandIT.java

@@ -154,4 +154,11 @@ public void testSortMultipleFields() throws IOException {
             String.format("source=%s | sort dog_name, age | fields dog_name, age", TEST_INDEX_DOG));
     verifyOrder(result, rows("rex", 2), rows("snoopy", 4));
   }
+
+  @Test
+  public void testSortThenHead() throws IOException {
+    JSONObject result =
+        executeQuery(String.format("source=%s | sort age | head 2 | fields age", TEST_INDEX_BANK));
+    verifyOrder(result, rows(28), rows(32));
+  }
 }

integ-test/src/test/resources/expectedOutput/calcite/explain_limit_then_sort_push.json

@@ -0,0 +1,6 @@
+{
+  "calcite": {
+    "logical": "LogicalSort(sort0=[$0], dir0=[ASC])\n  LogicalProject(age=[$8])\n    LogicalSort(sort0=[$8], dir0=[ASC])\n      LogicalSort(fetch=[5])\n        CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]])\n",
+    "physical": "EnumerableSort(sort0=[$0], dir0=[ASC])\n  CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]], PushDownContext=[[LIMIT->5, PROJECT->[age]], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"size\":5,\"timeout\":\"1m\",\"_source\":{\"includes\":[\"age\"],\"excludes\":[]}}, requestedTotalSize=5, pageSize=null, startFrom=0)])\n"
+  }
+}

integ-test/src/test/resources/expectedOutput/calcite/explain_multi_sort_push.json

@@ -0,0 +1,6 @@
+{
+  "calcite": {
+    "logical": "LogicalSort(sort0=[$3], sort1=[$4], sort2=[$2], dir0=[DESC], dir1=[DESC], dir2=[ASC])\n  LogicalProject(account_number=[$0], firstname=[$1], address=[$2], balance=[$3], gender=[$4])\n    LogicalSort(sort0=[$3], sort1=[$4], sort2=[$2], dir0=[DESC], dir1=[DESC], dir2=[ASC])\n      LogicalSort(sort0=[$0], sort1=[$1], sort2=[$2], sort3=[$3], dir0=[ASC], dir1=[ASC], dir2=[ASC], dir3=[ASC])\n        CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]])\n",
+    "physical": "CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]], PushDownContext=[[PROJECT->[account_number, firstname, address, balance, gender], SORT->[{\n  \"balance\" : {\n    \"order\" : \"desc\",\n    \"missing\" : \"_first\"\n  }\n}, {\n  \"gender\" : {\n    \"order\" : \"desc\",\n    \"missing\" : \"_first\"\n  }\n}, {\n  \"address\" : {\n    \"order\" : \"asc\",\n    \"missing\" : \"_last\"\n  }\n}]], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"timeout\":\"1m\",\"_source\":{\"includes\":[\"account_number\",\"firstname\",\"address\",\"balance\",\"gender\"],\"excludes\":[]},\"sort\":[{\"balance\":{\"order\":\"desc\",\"missing\":\"_first\"}},{\"gender\":{\"order\":\"desc\",\"missing\":\"_first\"}},{\"address\":{\"order\":\"asc\",\"missing\":\"_last\"}}]}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])\n"
+  }
+}