Narrow native lowering in visitSearch to numeric / boolean RHS values

The visitor's native RexCall lowering routes structured search predicates
through CalciteScriptEngine on the v2/Calcite-with-Lucene path. The script
engine compiles a typed comparison directly, which is fine for numeric and
boolean RHS values — but it can't reproduce Lucene's analyzer-aware string
semantics that PPL `search` has always relied on:

  - Text fields are tokenised; native `=` would compare against analyzed
    terms and never match the raw literal (doctest `severityText="INFO"`).
  - `!=` on text fields filters only documents that match the analyzed
    token, not ones containing it — doctest
    `instrumentationScope.name!="@opentelemetry/instrumentation-http"`
    expects Lucene's "matches any token" semantics.
  - `IN` lists need the same analyzer semantics for each value.
  - `@timestamp >= "..."` literals reach the script engine as
    java.lang.String vs long (date field) — Janino can't compile that
    comparison and the query fails at shard time.

Restrict native lowering to INTEGER / LONG / SHORT / FLOAT / DOUBLE /
DECIMAL / BOOLEAN values via `isNativeLowerableValue`. Everything else
(STRING, DATE/TIME/TIMESTAMP, IP, NULL) falls back to the original
`query_string` path so Lucene handles it natively — on the analytics-engine
route via the composite-parquet Lucene secondary, on the v2/Calcite path
via the Lucene primary.

The same gate applies to `SearchIn` — string IN lists fall back too.

Drops the plan-time date-math resolver introduced in the prior iteration —
it emitted TIMESTAMP literals that tripped the same script-engine compile
failure on Lucene-backed shards, and the broader value-type gate subsumes
its narrower string-shape detection.

Update `explain_search_numeric_comparison.json` (both pushdown and
no-pushdown variants) to reflect the new native shape for the
`severityNumber > 15` example — pure numeric comparisons still lower to a
native RexCall, which is the intended PR behavior.

Update `CalcitePPLSearchTest.testSearchWithFilter` for the same reason —
`DEPTNO=20` now lowers natively since the RHS is an integer literal.

Signed-off-by: Kai Huang <ahkcs@amazon.com>

Author: ahkcs

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

@@ -94,6 +94,7 @@
 import org.opensearch.sql.ast.expression.AllFieldsExcludeMeta;
 import org.opensearch.sql.ast.expression.Argument;
 import org.opensearch.sql.ast.expression.Argument.ArgumentMap;
+import org.opensearch.sql.ast.expression.DataType;
 import org.opensearch.sql.ast.expression.Field;
 import org.opensearch.sql.ast.expression.Function;
 import org.opensearch.sql.ast.expression.Let;
@@ -384,19 +385,26 @@ private Optional<UnresolvedExpression> lowerSearchExpression(
       if (!isLowerableField(comp.getField(), knownFields)) {
         return Optional.empty();
       }
-      if (isOpenSearchDateMath(comp.getValue().getLiteral())) {
+      UnresolvedExpression value = comp.getValue().getLiteral();
+      if (!isNativeLowerableValue(value)) {
+        // Only numeric / boolean RHS values lower cleanly to native RexCall.
+        //   - Strings hit Lucene's analyzer-aware semantics on text fields; native `=` would
+        //     compare against the analyzed-term storage and never match the raw literal
+        //     (`severityText="INFO"` returns 0 rows when the text field tokenises to
+        //     `[info]`). PPL `search` has always relied on Lucene's analyzer.
+        //   - Date-math values (`now`, `now-1h`, `2024-01-15||+1d`, 12-digit epoch-millis)
+        //     reach the script engine as `java.lang.String` vs `long` (date field) — Janino
+        //     can't compile that comparison and the query fails at shard time.
+        // Keep these in the original `query_string` fallback so Lucene handles them — both
+        // on the v2/Calcite path (against the Lucene primary) and on the analytics-engine
+        // path (via the composite-parquet Lucene secondary).
         return Optional.empty();
       }
-      if (containsLuceneWildcard(comp.getValue().getLiteral())) {
-        // `severityText=ERR*` / `field=foo?` / `name=*-service` — Lucene-style wildcards in
-        // the right-hand value. A native `=` lowering would compare literally and drop every
-        // matching document. Keep the query in query_string form so Lucene evaluates the
-        // wildcard.
+      if (containsLuceneWildcard(value)) {
+        // Defensive: numeric values can't contain `*`/`?`, but guard anyway.
         return Optional.empty();
       }
-      return Optional.of(
-          AstDSL.compare(
-              comp.getOperator().getSymbol(), comp.getField(), comp.getValue().getLiteral()));
+      return Optional.of(AstDSL.compare(comp.getOperator().getSymbol(), comp.getField(), value));
     }
     if (e instanceof SearchIn) {
       SearchIn in = (SearchIn) e;
@@ -405,6 +413,10 @@ private Optional<UnresolvedExpression> lowerSearchExpression(
       }
       List<UnresolvedExpression> values =
           in.getValues().stream().map(SearchLiteral::getLiteral).collect(Collectors.toList());
+      if (!values.stream().allMatch(CalciteRelNodeVisitor::isNativeLowerableValue)) {
+        // Same rationale as SearchComparison — string IN lists need Lucene analyzer semantics.
+        return Optional.empty();
+      }
       return Optional.of(AstDSL.in(in.getField(), values));
     }
     // SearchLiteral (standalone free-text / phrase) — no native equivalent.
@@ -417,42 +429,21 @@ private static boolean isLowerableField(Field field, Set<String> knownFields) {
   }
 
   /**
-   * Recognise OpenSearch date-math values produced by {@code visitTimeModifierExpression} — {@code
-   * "now"}, {@code "now-1h"}, {@code "now+1y/M-1M"}, anchored expressions like {@code
-   * "2024-01-15||+1d"}, and epoch-millis strings such as {@code "1754020060000"}. These strings
-   * have to be evaluated as Lucene-style date math; Calcite would compare them lexically. Detecting
-   * them lets the visitor keep the comparison in {@code query_string} form so the existing
-   * semantics survive.
+   * Numeric and boolean RHS values lower cleanly to native {@link org.apache.calcite.rex.RexCall}
+   * on every backend. Strings (and all date / interval shapes the parser models as strings) need
+   * Lucene's analyzer-aware {@code query_string} evaluation — both for text-field tokenisation and
+   * for date-math expressions like {@code now-1h} / anchored literals — so we keep them in the
+   * fallback path.
    */
-  private static boolean isOpenSearchDateMath(UnresolvedExpression value) {
+  private static boolean isNativeLowerableValue(UnresolvedExpression value) {
     if (!(value instanceof Literal)) {
       return false;
     }
-    Object raw = ((Literal) value).getValue();
-    if (!(raw instanceof String)) {
-      return false;
-    }
-    String s = ((String) raw).trim();
-    if (s.isEmpty()) {
-      return false;
-    }
-    if (s.regionMatches(true, 0, "now", 0, 3)) {
-      return true;
-    }
-    if (s.contains("||")) {
-      return true;
-    }
-    // Epoch-millis from `earliest=1754020060` / `latest=1754020060.123` — 12+ digits only.
-    if (s.length() >= 12) {
-      for (int i = 0; i < s.length(); i++) {
-        char c = s.charAt(i);
-        if (c < '0' || c > '9') {
-          return false;
-        }
-      }
-      return true;
-    }
-    return false;
+    DataType type = ((Literal) value).getType();
+    return switch (type) {
+      case INTEGER, LONG, SHORT, FLOAT, DOUBLE, DECIMAL, BOOLEAN -> true;
+      default -> false;
+    };
   }
 
   /**
@@ -739,9 +730,6 @@ private List<RexNode> expandProjectFields(
               .filter(addedFields::add)
               .forEach(field -> expandedFields.add(context.relBuilder.field(field)));
         }
-        case Alias alias -> {
-          expandedFields.add(rexVisitor.analyze(alias, context));
-        }
         default ->
             throw new IllegalStateException(
                 "Unexpected expression type in project list: " + expr.getClass().getSimpleName());

core/src/main/java/org/opensearch/sql/executor/analytics/AnalyticsExecutionEngine.java

@@ -170,21 +170,21 @@ private ExprType convertType(RelDataType type) {
 
   /**
    * Recovers the pre-cast datetime types for the schema. {@code DatetimeOutputCastRule} (api
-   * post-analysis pass) wraps every datetime-typed root column in {@code CAST(... AS VARCHAR)}
-   * so the analytics-engine backend can emit PPL's documented space-separator format via {@code
-   * to_char} ({@code DatetimeOutputCastRewriter} on the OpenSearch sandbox side). The downside
-   * is that {@code plan.getRowType()} then advertises those columns as VARCHAR — the response
-   * schema would report {@code "string"} for what tests (and callers) still expect as {@code
-   * "timestamp"}. The legacy / Calcite-reference path doesn't have this divergence: the wire
-   * value is a formatted string while the schema column type is the original datetime type.
+   * post-analysis pass) wraps every datetime-typed root column in {@code CAST(... AS VARCHAR)} so
+   * the analytics-engine backend can emit PPL's documented space-separator format via {@code
+   * to_char} ({@code DatetimeOutputCastRewriter} on the OpenSearch sandbox side). The downside is
+   * that {@code plan.getRowType()} then advertises those columns as VARCHAR — the response schema
+   * would report {@code "string"} for what tests (and callers) still expect as {@code "timestamp"}.
+   * The legacy / Calcite-reference path doesn't have this divergence: the wire value is a formatted
+   * string while the schema column type is the original datetime type.
    *
-   * <p>This walk inspects the OUTERMOST {@link Project} (descending through a single {@link
-   * Sort}, the {@code LogicalSystemLimit} wrapper) and, for any slot of the exact shape {@code
+   * <p>This walk inspects the OUTERMOST {@link Project} (descending through a single {@link Sort},
+   * the {@code LogicalSystemLimit} wrapper) and, for any slot of the exact shape {@code
    * CAST(<datetime> AS VARCHAR)}, swaps in the inner {@link RelDataType} for the schema. Values
    * still come back as the strings DataFusion emitted; only the schema column type is restored.
    *
-   * <p>Falls back to the slot's reported type for non-cast slots, non-datetime sources, or when
-   * the root isn't a Project (raw scan / aggregate fragment).
+   * <p>Falls back to the slot's reported type for non-cast slots, non-datetime sources, or when the
+   * root isn't a Project (raw scan / aggregate fragment).
    */
   private static List<RelDataType> recoverOriginalDatetimeTypes(RelNode plan) {
     List<RelDataTypeField> fields = plan.getRowType().getFieldList();

integ-test/src/test/resources/expectedOutput/calcite/explain_search_numeric_comparison.json

@@ -1,6 +1,6 @@
 {
   "calcite": {
-    "logical": "LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])\n  LogicalProject(spanId=[$0], traceId=[$1], @timestamp=[$2], instrumentationScope=[$3], severityText=[$7], resource=[$8], flags=[$23], attributes=[$24], droppedAttributesCount=[$162], severityNumber=[$163], time=[$164], body=[$165])\n    LogicalFilter(condition=[query_string(MAP('query', 'severityNumber:>15':VARCHAR))])\n      CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_otel_logs]])\n",
-    "physical": "CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_otel_logs]], PushDownContext=[[PROJECT->[spanId, traceId, @timestamp, instrumentationScope, severityText, resource, flags, attributes, droppedAttributesCount, severityNumber, time, body], FILTER->query_string(MAP('query', 'severityNumber:>15':VARCHAR)), LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"size\":10000,\"timeout\":\"1m\",\"query\":{\"query_string\":{\"query\":\"severityNumber:>15\",\"fields\":[],\"type\":\"best_fields\",\"default_operator\":\"or\",\"max_determinized_states\":10000,\"enable_position_increments\":true,\"fuzziness\":\"AUTO\",\"fuzzy_prefix_length\":0,\"fuzzy_max_expansions\":50,\"phrase_slop\":0,\"escape\":false,\"auto_generate_synonyms_phrase_query\":true,\"fuzzy_transpositions\":true,\"boost\":1.0}},\"_source\":{\"includes\":[\"spanId\",\"traceId\",\"@timestamp\",\"instrumentationScope\",\"severityText\",\"resource\",\"flags\",\"attributes\",\"droppedAttributesCount\",\"severityNumber\",\"time\",\"body\"]}}, requestedTotalSize=10000, pageSize=null, startFrom=0)])\n"
+    "logical": "LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])\n  LogicalProject(spanId=[$0], traceId=[$1], @timestamp=[$2], instrumentationScope=[$3], severityText=[$7], resource=[$8], flags=[$23], attributes=[$24], droppedAttributesCount=[$162], severityNumber=[$163], time=[$164], body=[$165])\n    LogicalFilter(condition=[>($163, 15)])\n      CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_otel_logs]])\n",
+    "physical": "CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_otel_logs]], PushDownContext=[[PROJECT->[spanId, traceId, @timestamp, instrumentationScope, severityText, resource, flags, attributes, droppedAttributesCount, severityNumber, time, body], FILTER->>($9, 15), LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"size\":10000,\"timeout\":\"1m\",\"query\":{\"range\":{\"severityNumber\":{\"from\":15,\"to\":null,\"include_lower\":false,\"include_upper\":true,\"boost\":1.0}}},\"_source\":{\"includes\":[\"spanId\",\"traceId\",\"@timestamp\",\"instrumentationScope\",\"severityText\",\"resource\",\"flags\",\"attributes\",\"droppedAttributesCount\",\"severityNumber\",\"time\",\"body\"]}}, requestedTotalSize=10000, pageSize=null, startFrom=0)])\n"
   }
-}
+}

integ-test/src/test/resources/expectedOutput/calcite_no_pushdown/explain_search_numeric_comparison.json

@@ -1,6 +1,6 @@
 {
   "calcite": {
-    "logical": "LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])\n  LogicalProject(spanId=[$0], traceId=[$1], @timestamp=[$2], instrumentationScope=[$3], severityText=[$7], resource=[$8], flags=[$23], attributes=[$24], droppedAttributesCount=[$162], severityNumber=[$163], time=[$164], body=[$165])\n    LogicalFilter(condition=[query_string(MAP('query', 'severityNumber:>15':VARCHAR))])\n      CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_otel_logs]])\n",
-    "physical": "EnumerableLimit(fetch=[10000])\n  EnumerableCalc(expr#0..171=[{inputs}], proj#0..3=[{exprs}], severityText=[$t7], resource=[$t8], flags=[$t23], attributes=[$t24], droppedAttributesCount=[$t162], severityNumber=[$t163], time=[$t164], body=[$t165])\n    CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_otel_logs]], PushDownContext=[[FILTER->query_string(MAP('query', 'severityNumber:>15':VARCHAR))], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"timeout\":\"1m\",\"query\":{\"query_string\":{\"query\":\"severityNumber:>15\",\"fields\":[],\"type\":\"best_fields\",\"default_operator\":\"or\",\"max_determinized_states\":10000,\"enable_position_increments\":true,\"fuzziness\":\"AUTO\",\"fuzzy_prefix_length\":0,\"fuzzy_max_expansions\":50,\"phrase_slop\":0,\"escape\":false,\"auto_generate_synonyms_phrase_query\":true,\"fuzzy_transpositions\":true,\"boost\":1.0}}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])\n"
+    "logical": "LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])\n  LogicalProject(spanId=[$0], traceId=[$1], @timestamp=[$2], instrumentationScope=[$3], severityText=[$7], resource=[$8], flags=[$23], attributes=[$24], droppedAttributesCount=[$162], severityNumber=[$163], time=[$164], body=[$165])\n    LogicalFilter(condition=[>($163, 15)])\n      CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_otel_logs]])\n",
+    "physical": "EnumerableLimit(fetch=[10000])\n  EnumerableCalc(expr#0..171=[{inputs}], expr#172=[15], expr#173=[>($t163, $t172)], proj#0..3=[{exprs}], severityText=[$t7], resource=[$t8], flags=[$t23], attributes=[$t24], droppedAttributesCount=[$t162], severityNumber=[$t163], time=[$t164], body=[$t165], $condition=[$t173])\n    CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_otel_logs]])\n"
   }
-}
+}

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLSearchTest.java

@@ -44,13 +44,13 @@ protected Frameworks.ConfigBuilder config(CalciteAssert.SchemaSpec... schemaSpec
   public void testSearchWithFilter() {
     String ppl = "search source=EMP DEPTNO=20";
     RelNode root = getRelNode(ppl);
+    // Numeric RHS values lower to a native RexCall — DataFusion and CalciteScriptEngine both
+    // compile the comparison directly, so we bypass query_string for these.
     String expectedLogical =
-        "LogicalFilter(condition=[query_string(MAP('query', 'DEPTNO:20':VARCHAR))])\n"
-            + "  LogicalTableScan(table=[[scott, EMP]])\n";
+        "LogicalFilter(condition=[=($7, 20)])\n  LogicalTableScan(table=[[scott, EMP]])\n";
     verifyLogical(root, expectedLogical);
 
-    String expectedSparkSql =
-        "SELECT *\nFROM `scott`.`EMP`\nWHERE QUERY_STRING(MAP ('query', 'DEPTNO:20'))";
+    String expectedSparkSql = "SELECT *\nFROM `scott`.`EMP`\nWHERE `DEPTNO` = 20";
     verifyPPLToSparkSQL(root, expectedSparkSql);
   }