Skip to content

Commit 67a97ad

Browse files
committed
[Analytics Backend / DataFusion] Bind PPL concat() to substrait concat
PPL's `concat(a, b, …)` registers in `PPLFuncImpTable` as `SqlLibraryOperators.CONCAT_FUNCTION` (operator name `"CONCAT"`), distinct from the `||` binary string-concat operator (`SqlStdOperatorTable.CONCAT`) that isthmus' default Calcite→Substrait Sig table already binds to substrait `concat`. Without an entry for the function form, isthmus' `RexExpressionConverter.visitCall` runs through its `callConverters` list, finds no match, and throws `Unable to convert call CONCAT(string, string)`. Concretely this surfaced on `testMvfindWithDynamicRegex`, which builds its regex via `concat('ban', '.*')` and feeds it to `mvfind`. Substrait's default `functions_string.yaml` already declares `concat` so no YAML change is needed — only the Sig bridge. # Pass-rate (CalciteArrayFunctionIT, force-routed) * Before: 42/60. * After: 43/60. Newly passing: testMvfindWithDynamicRegex. Signed-off-by: Kai Huang <ahkcs@amazon.com>
1 parent fa534e7 commit 67a97ad

1 file changed

Lines changed: 7 additions & 0 deletions

File tree

sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/DataFusionFragmentConvertor.java

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,13 @@ public class DataFusionFragmentConvertor implements FragmentConvertor {
113113
FunctionMappings.s(SqlStdOperatorTable.RAND, "random"),
114114
FunctionMappings.s(SqlLibraryOperators.LOG, "logb"),
115115
FunctionMappings.s(SignumFunction.FUNCTION, SignumFunction.NAME),
116+
// PPL `concat()` (registered in PPLFuncImpTable as SqlLibraryOperators.CONCAT_FUNCTION,
117+
// operator name "CONCAT") — distinct from `||` aka SqlStdOperatorTable.CONCAT, which
118+
// isthmus' default catalog already binds to substrait `concat`. Without this bridge,
119+
// any PPL eval using `concat(...)` (e.g. dynamic-pattern flows like
120+
// `mvfind(arr, concat('ban', '.*'))`) fails substrait conversion with
121+
// "Unable to convert call CONCAT(string, string)".
122+
FunctionMappings.s(SqlLibraryOperators.CONCAT_FUNCTION, "concat"),
116123
// Array S0 ladder — see DataFusionAnalyticsBackendPlugin.STANDARD_PROJECT_OPS /
117124
// ARRAY_RETURNING_PROJECT_OPS for the capability registration. ARRAY_LENGTH /
118125
// ARRAY_SLICE / ARRAY_DISTINCT pass through under their Calcite-stdlib names

0 commit comments

Comments
 (0)