[Analytics Backend / DataFusion] Onboard array_length scalar function (Part 3)

Wires Calcite's `SqlLibraryOperators.ARRAY_LENGTH` to DataFusion's native
`array_length`, completing the end-to-end story for PPL `rex` extract-mode
multi-match: queries can now size the list returned by `rex_extract_multi`
(`eval count = array_length(g)`).

  * `ScalarFunction.ARRAY_LENGTH` enum value (resolves via the `valueOf()`
    fallback on the Calcite operator name).
  * Registered in `STANDARD_PROJECT_OPS`. Returns `bigint`, so the existing
    `SUPPORTED_FIELD_TYPES` (numeric ∪ keyword ∪ date ∪ {BOOLEAN, TEXT})
    covers the capability lookup — no special-case needed.
  * `FunctionMappings.s(SqlLibraryOperators.ARRAY_LENGTH, "array_length")` in
    `DataFusionFragmentConvertor.ADDITIONAL_SCALAR_SIGS`. Library operators
    don't auto-resolve through the substrait default catalog — the same
    explicit pinning pattern used for `ILIKE`, `DATE_PART`, and the
    `REGEXP_REPLACE_*` family.
  * `array_length` extension declaration in `opensearch_scalar_functions.yaml`
    with `list<varchar<L1>>` → `i64` and `list<string>` → `i64` impls. Without
    a custom YAML extension that matches the actual list type, isthmus emits
    "Unable to convert call ARRAY_LENGTH(list<varchar<...>>)" for the
    `rex_extract_multi` output.

Lifts CalciteRexCommandIT (SQL plugin's standard rex IT class) through the
analytics-engine route from 14/18 → 17/18. The remaining failure
(testRexMaxMatchConfigurableLimit) is a unified-query architectural gap —
`UnifiedQueryContext` ignores cluster-setting overrides and uses the static
default — unrelated to rex or array_length.

Signed-off-by: Jialiang Liang <jiallian@amazon.com>

Author: RyanL1997

sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/DataFusionAnalyticsBackendPlugin.java

@@ -140,6 +140,12 @@ public class DataFusionAnalyticsBackendPlugin implements AnalyticsSearchBackendP
         ScalarFunction.REX_EXTRACT,
         ScalarFunction.REX_EXTRACT_MULTI,
         ScalarFunction.REX_OFFSET,
+        // ARRAY_LENGTH — counts elements in array<*>; needed end-to-end so PPL queries can size
+        // the list returned by `rex field=f "(?<g>...)"` extract-mode (CalciteRexCommandIT's
+        // testRexMaxMatch{Zero,Within,At}DefaultLimit and testRexMaxMatchConfigurableLimit all
+        // do `eval count = array_length(g)`). DataFusion has it natively; isthmus default
+        // catalog binds it.
+        ScalarFunction.ARRAY_LENGTH,
         ScalarFunction.PLUS,
         ScalarFunction.TIMES,
         ScalarFunction.DIVIDE,

sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/DataFusionFragmentConvertor.java

@@ -180,6 +180,7 @@ public class DataFusionFragmentConvertor implements FragmentConvertor {
         FunctionMappings.s(RexExtractAdapter.LOCAL_REX_EXTRACT_OP, "rex_extract"),
         FunctionMappings.s(RexExtractMultiAdapter.LOCAL_REX_EXTRACT_MULTI_OP, "rex_extract_multi"),
         FunctionMappings.s(RexOffsetAdapter.LOCAL_REX_OFFSET_OP, "rex_offset"),
+        FunctionMappings.s(SqlLibraryOperators.ARRAY_LENGTH, "array_length"),
         FunctionMappings.s(SqlStdOperatorTable.TRUNCATE, "trunc"),
         FunctionMappings.s(SqlStdOperatorTable.CBRT, "cbrt"),
         FunctionMappings.s(SqlStdOperatorTable.COT, "cot"),

sandbox/plugins/analytics-backend-datafusion/src/main/resources/opensearch_scalar_functions.yaml

@@ -433,6 +433,23 @@ scalar_functions:
           - value: "string"
             name: "pattern"
         return: string
+  - name: array_length
+    description: >-
+      Number of elements in a list. Lowering target for PPL's `array_length()`
+      function (Calcite `SqlLibraryOperators.ARRAY_LENGTH`). Needed end-to-end
+      so PPL queries can size the list returned by `rex` extract-mode
+      (`eval count = array_length(g)` after `rex field=f "(?<g>...)"`).
+      datafusion-substrait resolves the extension name "array_length" to
+      DataFusion's native `array_length` UDF.
+    impls:
+      - args:
+          - value: "list<varchar<L1>>"
+            name: "input"
+        return: i64
+      - args:
+          - value: "list<string>"
+            name: "input"
+        return: i64
 
   # ascii(str) — Unicode code point of the first character.
   - name: "ascii"