Stabilize more PPL ITs on the analytics-engine route (sort/streamstats/IP-UDT/metadata/strip-verifier) (#5566)

Brings 16 more PPL IT classes to parity on the analytics-engine route
(-Dtests.analytics.parquet_indices=true). Route-only divergences are
gated AE-only via @RequiresCapability + a matching build.gradle
excludeTestsMatching; the v2/Calcite path runs every test unchanged.

Strip-verifier (the #5541 guardrail):
- AnalyticsUnsupportedFieldStripVerifyIT was failing because 8 datasets
  carry a multi-value JSON array for a scalar-mapped field, which the
  parquet store rejects at bulk load. That's a cardinality limitation,
  not an unsupported field *type*, so it's out of scope for the type
  strip — the same situation as the existing `join` out-of-scope skip.
  Added a curated MULTI_VALUE_DATASETS allowlist + safeToSkipForMultiValueLoad
  that skips only the exact multi-value signature on a known dataset;
  any other failure still surfaces loudly, and Legs 2-3 still type-check
  every index that loads.

Init-load contamination (not divergences — fixed, not gated):
- CalciteWhereCommandIT failed on testDoubleEqual* because init() loaded
  game_of_thrones (base) and deep_nested (subclass), both multi-value
  datasets whose bulk-load failure aborted init() and mislabeled the
  first test. Guarded both loads with isAnalyticsParquetIndicesEnabled();
  no test in the hierarchy queries them on the AE route. 32/32 now pass.

Non-deterministic sort ties stabilized (not gated — full coverage kept):
- The three CalcitePPLSortIT tie tests (testSortWithNullValue,
  testSortAgeAndFieldsNameAge, testSortWithAutoCast) sorted on a
  non-unique key, so the tied rows had no defined order and the AE route
  ordered them differently than the captured Lucene doc order. Added a
  unique secondary sort key (firstname) to each, making the order
  deterministic and engine-independent. Verified identical on BOTH
  routes: 18/18 on AE and 18/18 on v2/Calcite. No gate needed.

Engine divergences gated (new capabilities):
- INVALID_DATETIME_ERROR_SHAPE: dayname/monthname over an invalid literal
  throw a different message shape (2 tests)
- RAND_SEED_UNSUPPORTED: seeded RAND(seed) is rejected on AE
- IP_UDT_BINARY_REPRESENTATION: the IP UDT is materialized as BINARY, so
  cast(... as IP) and cidrmatch over an IP column fail (2 tests)
- TIME_TYPE_WIDENED_TO_TIMESTAMP: a TIME field reads back as TIMESTAMP,
  defeating TIMEDIFF's [TIME,TIME] signature
- BINARY_FIELD_STRIPPED: binary fields are stripped at load
- VALUES_LIMIT_NOT_HONORED: values()/list() ignore the configured limit
- INDEX_METADATA: _index metadata not exposed (sibling of ID_METADATA)
- CROSS_INDEX_OBJECT_LEAF_MERGE: an object leaf in only some wildcard
  member indices resolves to FIELD_NOT_FOUND
- TEXT_KEYWORD_PUSHDOWN_REWRITE: like() doesn't rewrite to .keyword in
  the explain plan (no Lucene term-pushdown)
- LUCENE_PUSHDOWN_EXPLAIN: a test asserting a Lucene SORT-> pushdown
  fragment can't match the DataFusion plan

Reused existing capabilities:
- WILDCARD_COLUMN_ORDER: streamstats carries all source columns through;
  AE returns them in a different column order (values and row order are
  correct, so a sort can't fix it) (4 CalciteReverseCommandIT tests)
- HEAD_WITHOUT_STABLE_SORT: the non-determinism is which rows head N
  keeps, before the trailing sort, so a sort can't recover it
  (testHeadThenSort, testAppendWithMergedColumn)
- DEDUP_NONDETERMINISTIC: consecutive dedup has no working V2 fallback on
  the AE route

Out of scope:
- FieldsCommandIT.testEnhancedFieldsWhenCalciteDisabled asserts the
  Calcite-DISABLED error; the AE route is always Calcite-enabled.
  build.gradle exclude only.

Results (this batch, on the AE route): 16 classes, 0 failures (was 24
failures), with the 3 sort tests now passing rather than skipped. V2
baseline: 0 failures, only pre-existing/by-design skips (none from these
gates).

Signed-off-by: Kai Huang <ahkcs@amazon.com>

Author: ahkcs

integ-test/build.gradle

@@ -1285,6 +1285,49 @@ task integTestRemote(type: RestIntegTestTask) {
             excludeTestsMatching '*CalciteChartCommandIT.testChartLimitTopWithUseOther'
             excludeTestsMatching '*CalciteChartCommandIT.testChartLimitBottomWithUseOther'
             excludeTestsMatching '*CalciteChartCommandIT.testChartLimitTopWithMinAgg'
+
+            // === Excludes: streamstats carries all columns through; AE reorders them ===
+            excludeTestsMatching '*CalciteReverseCommandIT.testStreamstatsWithReverse'
+            excludeTestsMatching '*CalciteReverseCommandIT.testStreamstatsByWithReverse'
+            excludeTestsMatching '*CalciteReverseCommandIT.testStreamstatsWindowWithReverse'
+            excludeTestsMatching '*CalciteReverseCommandIT.testStreamstatsWithSortThenReverse'
+
+            // === Excludes: invalid-datetime error-message shape differs on AE ===
+            excludeTestsMatching '*CalcitePPLBuiltinDatetimeFunctionInvalidIT.testDAYNAMEInvalid'
+            excludeTestsMatching '*CalcitePPLBuiltinDatetimeFunctionInvalidIT.testMONTHNAMEInvalid'
+
+            // === Excludes: seeded RAND(seed) unsupported on AE ===
+            excludeTestsMatching '*MathematicalFunctionIT.testRand'
+
+            // === Excludes: IP UDT is materialized as BINARY/byte[] on AE ===
+            excludeTestsMatching '*CastFunctionIT.testCastToIP'
+            excludeTestsMatching '*CalcitePPLAppendCommandIT.testAppendSchemaMergeWithIpUDT'
+
+            // === Excludes: append head-without-stable-sort + TIME-widened-to-TIMESTAMP signature ===
+            excludeTestsMatching '*CalcitePPLAppendCommandIT.testAppendWithMergedColumn'
+            excludeTestsMatching '*CalcitePPLBuiltinFunctionsNullIT.testTimediffNull'
+
+            // === Excludes: consecutive dedup has no working V2 fallback on AE ===
+            excludeTestsMatching '*CalciteDedupCommandIT.testConsecutiveDedup'
+
+            // === Excludes: binary field stripped at load; values() ignores configured limit ===
+            excludeTestsMatching '*CalciteMultiValueStatsIT.testListFunctionWithBinary'
+            excludeTestsMatching '*CalciteMultiValueStatsIT.testValuesFunctionRespectsConfiguredLimit'
+
+            // === Excludes: _id/_index metadata not exposed; cross-index object-leaf merge ===
+            excludeTestsMatching '*FieldsCommandIT.testMetadataFields'
+            excludeTestsMatching '*FieldsCommandIT.testMergedObjectFields'
+            // OOS: asserts the Calcite-disabled error, but the AE route is always Calcite-enabled.
+            excludeTestsMatching '*FieldsCommandIT.testEnhancedFieldsWhenCalciteDisabled'
+
+            // === Excludes: head N without a stable sort returns a non-deterministic row set ===
+            excludeTestsMatching '*SortCommandIT.testHeadThenSort'
+
+            // === Excludes: like() doesn't rewrite to .keyword in the explain plan on AE ===
+            excludeTestsMatching '*LikeQueryIT.test_convert_field_text_to_keyword'
+
+            // === Excludes: asserts a Lucene pushdown fragment absent on the AE route ===
+            excludeTestsMatching '*CalciteSortCommandIT.testPushdownSortCastToDoubleExpression'
         }
     }
 

integ-test/src/test/java/org/opensearch/sql/calcite/remote/AnalyticsUnsupportedFieldStripVerifyIT.java

@@ -70,6 +70,32 @@ public class AnalyticsUnsupportedFieldStripVerifyIT extends PPLIntegTestCase {
    */
   private static final Set<String> OUT_OF_SCOPE_TYPES = Set.of("join");
 
+  /**
+   * Datasets that carry a multi-value JSON array for a scalar-mapped field (e.g. a {@code text} or
+   * {@code long} field given several values in a doc). The parquet/composite store rejects these at
+   * bulk load with {@code Cannot accept multiple values for field} — a cardinality limitation, not
+   * an unsupported field *type*, so it is out of scope for the type strip ({@link
+   * org.opensearch.sql.legacy.TestUtils.AnalyticsIndexConfig}) the same way {@link
+   * #OUT_OF_SCOPE_TYPES} is. Tracked by the {@code MULTI_VALUE_FIELD_LOAD} capability. An index
+   * here is skipped only when its failure is the exact multi-value signature AND it is on this
+   * curated list (see {@link #safeToSkipForMultiValueLoad}); any other failure surfaces loudly.
+   * Legs 2-3 still type-check the live mapping of every index that loads, so a missed type-strip
+   * can't hide.
+   */
+  private static final Set<String> MULTI_VALUE_DATASETS =
+      Set.of(
+          "GAME_OF_THRONES",
+          "NESTED",
+          "NESTED_WITH_QUOTES",
+          "DEEP_NESTED",
+          "NESTED_WITH_NULLS",
+          "GRAPH_AIRPORTS",
+          "ARRAY",
+          "OTELLOGS");
+
+  /** Cluster's per-item bulk error when a doc supplies an array to a scalar-mapped field. */
+  private static final String MULTI_VALUE_SIGNATURE = "Cannot accept multiple values for field";
+
   @Override
   public void init() throws Exception {
     super.init();
@@ -109,6 +135,12 @@ public void everyDatasetIngestsCleanlyOnAnalyticsEngine() throws IOException {
           // unsupported type we're responsible for — not our concern. Skip.
           continue;
         }
+        if (safeToSkipForMultiValueLoad(e, index)) {
+          // Load failed with the multi-value signature on a known multi-value-array-into-scalar
+          // dataset (a cardinality limitation, not an unsupported type) — out of scope for the type
+          // strip, tracked by MULTI_VALUE_FIELD_LOAD. Skip.
+          continue;
+        }
         failures.add(
             "["
                 + index.name()
@@ -380,6 +412,23 @@ private static boolean safeToSkipForOutOfScopeType(Throwable t, String mapping)
     return !mappingContainsUnsupportedType(mapping);
   }
 
+  /**
+   * Safe to skip an index whose load failed, only when BOTH hold: (a) the error is exactly the
+   * multi-value bulk signature ({@code Cannot accept multiple values for field}), AND (b) the index
+   * is on the curated {@link #MULTI_VALUE_DATASETS} allowlist. Unlike {@link
+   * #safeToSkipForOutOfScopeType} this does NOT also require the raw mapping to be free of
+   * unsupported types: several of these datasets legitimately declare {@code nested} fields that
+   * the type strip removes at load, while the multi-value failure is on a separate scalar leaf. The
+   * curated allowlist plus the exact failure signature is the masking guard — an unanticipated
+   * failure (different message, or a dataset not on the list) is never skipped and surfaces loudly.
+   */
+  private static boolean safeToSkipForMultiValueLoad(Throwable t, Index index) {
+    String msg = rootMessage(t);
+    return msg != null
+        && msg.contains(MULTI_VALUE_SIGNATURE)
+        && MULTI_VALUE_DATASETS.contains(index.name());
+  }
+
   /** True if the raw mapping JSON declares any {@link #UNSUPPORTED} field type at any depth. */
   private static boolean mappingContainsUnsupportedType(String mapping) {
     if (mapping == null || mapping.isEmpty()) {

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteDedupCommandIT.java

@@ -5,8 +5,11 @@
 
 package org.opensearch.sql.calcite.remote;
 
+import static org.opensearch.sql.util.Capability.DEDUP_NONDETERMINISTIC;
+
 import java.io.IOException;
 import org.opensearch.sql.ppl.DedupCommandIT;
+import org.opensearch.sql.util.RequiresCapability;
 
 public class CalciteDedupCommandIT extends DedupCommandIT {
   @Override
@@ -15,6 +18,11 @@ public void init() throws Exception {
     enableCalcite();
   }
 
+  @RequiresCapability(
+      value = DEDUP_NONDETERMINISTIC,
+      note =
+          "consecutive dedup falls back to V2 on the Calcite path, but the AE route has no working"
+              + " V2 fallback (DEDUP_NONDETERMINISTIC).")
   @Override
   public void testConsecutiveDedup() throws IOException {
     withFallbackEnabled(

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultiValueStatsIT.java

@@ -10,6 +10,8 @@
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_CALCS;
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_DATATYPE_NONNUMERIC;
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_DATATYPE_NUMERIC;
+import static org.opensearch.sql.util.Capability.BINARY_FIELD_STRIPPED;
+import static org.opensearch.sql.util.Capability.VALUES_LIMIT_NOT_HONORED;
 import static org.opensearch.sql.util.MatcherUtils.rows;
 import static org.opensearch.sql.util.MatcherUtils.schema;
 import static org.opensearch.sql.util.MatcherUtils.verifyDataRows;
@@ -21,6 +23,7 @@
 import org.json.JSONObject;
 import org.junit.jupiter.api.Test;
 import org.opensearch.sql.ppl.PPLIntegTestCase;
+import org.opensearch.sql.util.RequiresCapability;
 
 public class CalciteMultiValueStatsIT extends PPLIntegTestCase {
 
@@ -169,6 +172,9 @@ public void testListFunctionWithIP() throws IOException {
   }
 
   @Test
+  @RequiresCapability(
+      value = BINARY_FIELD_STRIPPED,
+      note = "binary_value is stripped at load on the AE route (BINARY_FIELD_STRIPPED).")
   public void testListFunctionWithBinary() throws IOException {
     JSONObject response =
         executeQuery(
@@ -420,6 +426,11 @@ public void testValuesFunctionWithUnlimitedValues() throws IOException {
   }
 
   @Test
+  @RequiresCapability(
+      value = VALUES_LIMIT_NOT_HONORED,
+      note =
+          "values() ignores plugins.ppl.values.max.limit on the AE route"
+              + " (VALUES_LIMIT_NOT_HONORED).")
   public void testValuesFunctionRespectsConfiguredLimit() throws IOException, InterruptedException {
     // Test 1: Set limit to 3 and verify only 3 values are returned
     updateClusterSettings(new ClusterSetting(TRANSIENT, "plugins.ppl.values.max.limit", "3"));

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLAppendCommandIT.java

@@ -8,6 +8,8 @@
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_ACCOUNT;
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_BANK;
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_WEBLOGS;
+import static org.opensearch.sql.util.Capability.HEAD_WITHOUT_STABLE_SORT;
+import static org.opensearch.sql.util.Capability.IP_UDT_BINARY_REPRESENTATION;
 import static org.opensearch.sql.util.MatcherUtils.rows;
 import static org.opensearch.sql.util.MatcherUtils.schema;
 import static org.opensearch.sql.util.MatcherUtils.verifyDataRows;
@@ -22,6 +24,7 @@
 import org.opensearch.client.ResponseException;
 import org.opensearch.sql.common.setting.Settings;
 import org.opensearch.sql.ppl.PPLIntegTestCase;
+import org.opensearch.sql.util.RequiresCapability;
 
 public class CalcitePPLAppendCommandIT extends PPLIntegTestCase {
   @Override
@@ -195,6 +198,11 @@ public void testAppendDifferentIndex() throws IOException {
   }
 
   @Test
+  @RequiresCapability(
+      value = HEAD_WITHOUT_STABLE_SORT,
+      note =
+          "head 5 over the two-branch append has no globally-unique sort key, so the"
+              + " surviving/ordered rows diverge on the AE route (HEAD_WITHOUT_STABLE_SORT).")
   public void testAppendWithMergedColumn() throws IOException {
     JSONObject actual =
         executeQuery(
@@ -258,6 +266,11 @@ public void testAppendSchemaMergeWithTimestampUDT() throws IOException {
   }
 
   @Test
+  @RequiresCapability(
+      value = IP_UDT_BINARY_REPRESENTATION,
+      note =
+          "cidrmatch over an appended IP column hits the IP-UDT-as-byte[] gap on the AE route"
+              + " (IP_UDT_BINARY_REPRESENTATION).")
   public void testAppendSchemaMergeWithIpUDT() throws IOException {
     JSONObject actual =
         executeQuery(

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLBuiltinDatetimeFunctionInvalidIT.java

@@ -6,12 +6,14 @@
 package org.opensearch.sql.calcite.remote;
 
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_DATE_FORMATS_WITH_NULL;
+import static org.opensearch.sql.util.Capability.INVALID_DATETIME_ERROR_SHAPE;
 import static org.opensearch.sql.util.MatcherUtils.verifyErrorMessageContains;
 
 import org.junit.jupiter.api.Test;
 import org.opensearch.sql.exception.ExpressionEvaluationException;
 import org.opensearch.sql.legacy.SQLIntegTestCase;
 import org.opensearch.sql.ppl.PPLIntegTestCase;
+import org.opensearch.sql.util.RequiresCapability;
 
 public class CalcitePPLBuiltinDatetimeFunctionInvalidIT extends PPLIntegTestCase {
   @Override
@@ -216,6 +218,11 @@ public void testDAYInvalid() {
   }
 
   @Test
+  @RequiresCapability(
+      value = INVALID_DATETIME_ERROR_SHAPE,
+      note =
+          "dayname/monthname over an invalid datetime literal throws a different error-message"
+              + " shape on the AE route (INVALID_DATETIME_ERROR_SHAPE).")
   public void testDAYNAMEInvalid() {
 
     Throwable e1 =
@@ -760,6 +767,11 @@ public void testMONTH_OF_YEARInvalid() {
   }
 
   @Test
+  @RequiresCapability(
+      value = INVALID_DATETIME_ERROR_SHAPE,
+      note =
+          "dayname/monthname over an invalid datetime literal throws a different error-message"
+              + " shape on the AE route (INVALID_DATETIME_ERROR_SHAPE).")
   public void testMONTHNAMEInvalid() {
 
     Throwable e1 =

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLBuiltinFunctionsNullIT.java

@@ -8,6 +8,7 @@
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_DATE_FORMATS_WITH_NULL;
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_NULL_MISSING;
 import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_STATE_COUNTRY_WITH_NULL;
+import static org.opensearch.sql.util.Capability.TIME_TYPE_WIDENED_TO_TIMESTAMP;
 import static org.opensearch.sql.util.MatcherUtils.*;
 import static org.opensearch.sql.util.MatcherUtils.rows;
 
@@ -17,6 +18,7 @@
 import org.junit.jupiter.api.Test;
 import org.opensearch.sql.exception.ExpressionEvaluationException;
 import org.opensearch.sql.ppl.PPLIntegTestCase;
+import org.opensearch.sql.util.RequiresCapability;
 
 public class CalcitePPLBuiltinFunctionsNullIT extends PPLIntegTestCase {
   @Override
@@ -887,6 +889,11 @@ public void testTimeToSecNull() throws IOException {
   }
 
   @Test
+  @RequiresCapability(
+      value = TIME_TYPE_WIDENED_TO_TIMESTAMP,
+      note =
+          "the TIME field reads back as TIMESTAMP on the AE route, defeating TIMEDIFF's [TIME,TIME]"
+              + " signature (TIME_TYPE_WIDENED_TO_TIMESTAMP).")
   public void testTimediffNull() throws IOException {
     JSONObject actual =
         executeQuery(

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLSortIT.java

@@ -166,15 +166,18 @@ public void testSortAgeAndFieldsAge() throws IOException {
 
   @Test
   public void testSortAgeAndFieldsNameAge() throws IOException {
+    // firstname is a unique secondary key so the age=36 tie has a deterministic,
+    // engine-independent order (sort is otherwise free to order ties differently per engine).
     JSONObject actual =
         executeQuery(
-            String.format("source=%s | sort - age | fields firstname, age", TEST_INDEX_BANK));
+            String.format(
+                "source=%s | sort - age, firstname | fields firstname, age", TEST_INDEX_BANK));
     verifySchema(actual, schema("firstname", "string"), schema("age", "int"));
     verifyDataRowsInOrder(
         actual,
         rows("Virginia", 39),
-        rows("Hattie", 36),
         rows("Elinor", 36),
+        rows("Hattie", 36),
         rows("Dillard", 34),
         rows("Dale", 33),
         rows("Amber JOHnny", 32),
@@ -201,15 +204,17 @@ public void testSortAgeNameAndFieldsNameAge() throws IOException {
 
   @Test
   public void testSortWithNullValue() throws IOException {
+    // firstname is a unique secondary key so the three null-balance rows have a deterministic,
+    // engine-independent order (sort is otherwise free to order ties differently per engine).
     JSONObject result =
         executeQuery(
             String.format(
-                "source=%s | sort balance | fields firstname, balance",
+                "source=%s | sort balance, firstname | fields firstname, balance",
                 TEST_INDEX_BANK_WITH_NULL_VALUES));
     verifyDataRowsInOrder(
         result,
-        rows("Hattie", null),
         rows("Elinor", null),
+        rows("Hattie", null),
         rows("Virginia", null),
         rows("Dale", 4180),
         rows("Nanette", 32838),
@@ -316,18 +321,21 @@ public void testSortWithStrCast() throws IOException {
 
   @Test
   public void testSortWithAutoCast() throws IOException {
+    // firstname is a unique secondary key so the age=36 tie has a deterministic,
+    // engine-independent order (sort is otherwise free to order ties differently per engine).
     JSONObject result =
         executeQuery(
-            String.format("source=%s | sort AUTO(age) | fields firstname, age", TEST_INDEX_BANK));
+            String.format(
+                "source=%s | sort AUTO(age), firstname | fields firstname, age", TEST_INDEX_BANK));
     verifySchema(result, schema("firstname", "string"), schema("age", "int"));
     verifyDataRowsInOrder(
         result,
         rows("Nanette", 28),
         rows("Amber JOHnny", 32),
         rows("Dale", 33),
         rows("Dillard", 34),
-        rows("Hattie", 36),
         rows("Elinor", 36),
+        rows("Hattie", 36),
         rows("Virginia", 39));
   }
 }