[Analytics Backend / DataFusion] Wire PPL rex sed-mode (Part 1) — bridge-only

Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:

  * `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
    (already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).

  * `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
    (4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
    natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.

  * `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
    New bridge in this PR. Resolves to DataFusion's `translate` UDF
    (datafusion-functions/src/unicode/translate.rs).

The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.

`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.

  * Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
    Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
    native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
    UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).

  * Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
    `REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
    support (max 4 args). Also Part 2.

  * `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
  * `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
    sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
    combined, backreferences via `$N`, transliteration `y/from/to/` and
    no-match passthrough.
  * `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.

The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.

Signed-off-by: Jialiang Liang <jiallian@amazon.com>

Author: RyanL1997

sandbox/libs/analytics-framework/src/main/java/org/opensearch/analytics/spi/ScalarFunction.java

@@ -84,6 +84,7 @@ public enum ScalarFunction {
     NUMBER_TO_STRING(Category.STRING, SqlKind.OTHER_FUNCTION), // Alias for TOSTRING
     TONUMBER(Category.STRING, SqlKind.OTHER_FUNCTION),
     STRCMP(Category.STRING, SqlKind.OTHER_FUNCTION),
+    TRANSLATE(Category.STRING, SqlKind.OTHER_FUNCTION),
 
     // ── Math ─────────────────────────────────────────────────────────
     PLUS(Category.MATH, SqlKind.PLUS),

sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/DataFusionAnalyticsBackendPlugin.java

@@ -136,6 +136,7 @@ public class DataFusionAnalyticsBackendPlugin implements AnalyticsSearchBackendP
         ScalarFunction.REGEXP_CONTAINS,
         ScalarFunction.REPLACE,
         ScalarFunction.REGEXP_REPLACE,
+        ScalarFunction.TRANSLATE,
         ScalarFunction.PLUS,
         ScalarFunction.TIMES,
         ScalarFunction.DIVIDE,

sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/DataFusionFragmentConvertor.java

@@ -113,6 +113,15 @@ public class DataFusionFragmentConvertor implements FragmentConvertor {
      *   <li>{@link SqlLibraryOperators#REGEXP_REPLACE_3} → {@code regexp_replace} (regex string
      *       replacement; lowering target for PPL `replace` command on wildcard patterns and for
      *       PPL `replace()` / `regexp_replace()` functions in `eval`).</li>
+     *   <li>{@link SqlLibraryOperators#REGEXP_REPLACE_PG_4} → {@code regexp_replace} (4-arg
+     *       PostgreSQL-style with flags string; lowering target for PPL `rex mode=sed` with
+     *       {@code g}/{@code i} flags. Reuses the same DataFusion {@code regexp_replace} UDF as
+     *       the 3-arg form.</li>
+     *   <li>{@link SqlLibraryOperators#TRANSLATE3} → {@code translate} (3-arg character
+     *       transliteration; lowering target for PPL `rex mode=sed` with {@code y/from/to/}
+     *       transliteration syntax). DataFusion's substrait consumer resolves the extension name
+     *       "translate" to its native {@code translate} UDF
+     *       (datafusion-functions/src/unicode/translate.rs).</li>
      * </ul>
      */
     private static final List<FunctionMappings.Sig> ADDITIONAL_SCALAR_SIGS = List.of(
@@ -132,8 +141,9 @@ public class DataFusionFragmentConvertor implements FragmentConvertor {
         FunctionMappings.s(SqlLibraryOperators.REGEXP_CONTAINS, "regex_match"),
         FunctionMappings.s(SqlStdOperatorTable.REPLACE, "replace"),
         FunctionMappings.s(SqlLibraryOperators.REGEXP_REPLACE_3, "regexp_replace"),
-        FunctionMappings.s(SqlLibraryOperators.REGEXP_CONTAINS, "regex_match"),
+        FunctionMappings.s(SqlLibraryOperators.REGEXP_REPLACE_PG_4, "regexp_replace"),
         FunctionMappings.s(SqlLibraryOperators.REVERSE, "reverse"),
+        FunctionMappings.s(SqlLibraryOperators.TRANSLATE3, "translate"),
         FunctionMappings.s(PositionAdapter.STRPOS, "strpos"),
         FunctionMappings.s(ToNumberFunctionAdapter.TONUMBER, "tonumber"),
         FunctionMappings.s(ToStringFunctionAdapter.TOSTRING, "tostring"),

sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/RegexpReplaceAdapter.java

@@ -43,6 +43,13 @@
  * changes. Calls without {@code \Q} in the pattern AND without bare {@code $N} in the
  * replacement pass through unchanged.
  *
+ * <p>Handles both 3-arg ({@code SqlLibraryOperators.REGEXP_REPLACE_3}) and 4-arg
+ * ({@code SqlLibraryOperators.REGEXP_REPLACE_PG_4}, with a trailing {@code flags} string)
+ * signatures. The pattern is at operand position 1 and replacement at position 2 in both —
+ * the rewrite logic is identical. The {@code flags} operand (when present) passes through
+ * verbatim. The 4-arg form is the lowering target for PPL {@code rex mode=sed} with
+ * {@code g}/{@code i} flags.
+ *
  * <p>Pattern faithful to {@link java.util.regex.Pattern} semantics: an unterminated
  * {@code \Q} (no closing {@code \E}) quotes through end-of-string. Replacement preserves
  * existing {@code ${…}} braces and the {@code $$} literal-dollar escape.
@@ -56,8 +63,12 @@ class RegexpReplaceAdapter implements ScalarFunctionAdapter {
 
     @Override
     public RexNode adapt(RexCall original, List<FieldStorageInfo> fieldStorage, RelOptCluster cluster) {
-        // REGEXP_REPLACE_3 has signature (input, pattern, replacement) — exactly 3 operands.
-        if (original.getOperands().size() != 3) {
+        // REGEXP_REPLACE has both 3-arg (input, pattern, replacement) and 4-arg
+        // (input, pattern, replacement, flags) signatures. Pattern is at position 1 and
+        // replacement at position 2 in both — the rewrite logic is identical regardless
+        // of whether a flags argument is appended. Operands beyond position 2 (e.g. the
+        // flags string) pass through unchanged.
+        if (original.getOperands().size() < 3 || original.getOperands().size() > 4) {
             return original;
         }
         RexNode patternOperand = original.getOperands().get(1);
@@ -93,10 +104,14 @@ public RexNode adapt(RexCall original, List<FieldStorageInfo> fieldStorage, RelO
         // makeLiteral(String) infers a CHAR type sized to the rewritten string. Reusing the
         // original literal's type would right-pad to the OLD length (e.g. CHAR(23) → 8 trailing
         // spaces after a 15-char rewrite), corrupting the value at runtime.
-        List<RexNode> newOperands = new ArrayList<>(3);
+        List<RexNode> newOperands = new ArrayList<>(original.getOperands().size());
         newOperands.add(original.getOperands().get(0));
         newOperands.add(rewrittenPattern != null ? rexBuilder.makeLiteral(rewrittenPattern) : patternOperand);
         newOperands.add(rewrittenReplacement != null ? rexBuilder.makeLiteral(rewrittenReplacement) : replacementOperand);
+        // Append any trailing operand (the flags string in the 4-arg form) verbatim.
+        for (int i = 3; i < original.getOperands().size(); i++) {
+            newOperands.add(original.getOperands().get(i));
+        }
         return rexBuilder.makeCall(original.getType(), original.getOperator(), newOperands);
     }
 

sandbox/plugins/analytics-backend-datafusion/src/main/resources/opensearch_scalar_functions.yaml

@@ -253,6 +253,51 @@ scalar_functions:
           - value: "string"
             name: "replacement"
         return: string
+      - args:
+          - value: "varchar<L1>"
+            name: "input"
+          - value: "varchar<L2>"
+            name: "pattern"
+          - value: "varchar<L3>"
+            name: "replacement"
+          - value: "varchar<L4>"
+            name: "flags"
+        return: "varchar<L1>"
+      - args:
+          - value: "string"
+            name: "input"
+          - value: "string"
+            name: "pattern"
+          - value: "string"
+            name: "replacement"
+          - value: "string"
+            name: "flags"
+        return: string
+  - name: translate
+    description: >-
+      Character transliteration — replace each character in `input` that appears
+      in `from` with the corresponding character in `to`. Lowering target for
+      PPL's `rex mode=sed` with {@code y/from/to/} transliteration syntax
+      (Calcite `SqlLibraryOperators.TRANSLATE3`). datafusion-substrait resolves
+      the extension name "translate" to DataFusion's native `translate` UDF
+      (datafusion-functions/src/unicode/translate.rs).
+    impls:
+      - args:
+          - value: "varchar<L1>"
+            name: "input"
+          - value: "varchar<L2>"
+            name: "from"
+          - value: "varchar<L3>"
+            name: "to"
+        return: "varchar<L1>"
+      - args:
+          - value: "string"
+            name: "input"
+          - value: "string"
+            name: "from"
+          - value: "string"
+            name: "to"
+        return: string
 
   # ascii(str) — Unicode code point of the first character.
   - name: "ascii"

sandbox/plugins/analytics-backend-datafusion/src/test/java/org/opensearch/be/datafusion/RegexpReplaceAdapterTests.java

@@ -222,4 +222,41 @@ public void testAdaptRewritesBothPatternAndReplacement() {
         assertEquals("pattern unquoted", "^(.*?) (.*?)$", ((RexLiteral) result.getOperands().get(1)).getValueAs(String.class));
         assertEquals("replacement braced", "${1}_${2}", ((RexLiteral) result.getOperands().get(2)).getValueAs(String.class));
     }
+
+    public void testAdapt4ArgRewritesPatternAndPassesFlagsThrough() {
+        // 4-arg REGEXP_REPLACE_PG_4 — emitted by PPL `rex mode=sed` with /g or /i flags.
+        // Pattern + replacement get rewritten as in the 3-arg case; the trailing flags
+        // operand passes through unchanged.
+        RexNode field = rexBuilder.makeInputRef(varcharType, 0);
+        RexNode pattern = rexBuilder.makeLiteral("^\\QFOO\\E");
+        RexNode replacement = rexBuilder.makeLiteral("$1");
+        RexNode flags = rexBuilder.makeLiteral("gi");
+        RexCall original = (RexCall) rexBuilder.makeCall(
+            SqlLibraryOperators.REGEXP_REPLACE_PG_4,
+            List.of(field, pattern, replacement, flags)
+        );
+
+        RexCall result = (RexCall) adapter.adapt(original, List.of(), cluster);
+
+        assertEquals("4-arg call preserved", 4, result.getOperands().size());
+        assertEquals("pattern unquoted", "^FOO", ((RexLiteral) result.getOperands().get(1)).getValueAs(String.class));
+        assertEquals("replacement braced", "${1}", ((RexLiteral) result.getOperands().get(2)).getValueAs(String.class));
+        assertSame("flags operand passes through verbatim", flags, result.getOperands().get(3));
+    }
+
+    public void testAdapt4ArgPassesThroughWhenNoRewriteNeeded() {
+        // 4-arg call with Rust-compatible pattern and no $N — adapter must return identity.
+        RexNode field = rexBuilder.makeInputRef(varcharType, 0);
+        RexNode pattern = rexBuilder.makeLiteral("^foo$");
+        RexNode replacement = rexBuilder.makeLiteral("bar");
+        RexNode flags = rexBuilder.makeLiteral("g");
+        RexCall original = (RexCall) rexBuilder.makeCall(
+            SqlLibraryOperators.REGEXP_REPLACE_PG_4,
+            List.of(field, pattern, replacement, flags)
+        );
+
+        RexNode adapted = adapter.adapt(original, List.of(), cluster);
+
+        assertSame("identity — 4-arg call with no \\Q and no $N passes through", original, adapted);
+    }
 }