Commit b6c4b9d
authored
[Analytics Backend / DataFusion] Substrait Plan.Root.names + CASE + untyped-NULL fixes for multisearch (opensearch-project#21528)
* [Analytics Backend / DataFusion] Fix Plan.Root.names mismatch for schema-reshaping wrappers
`DataFusionFragmentConvertor.rewire` always populated the new `Plan.Root.names`
list with the *inner* plan's names. For schema-preserving wrappers (Sort,
Filter, Fetch) those happen to coincide with the wrapper's output schema, so
the bug was hidden. For schema-reshaping wrappers (Aggregate, Project) the
wrapper's output width differs from the inner's, and DataFusion's substrait
consumer rejects the plan in `make_renamed_schema` with:
Substrait error: Names list must match exactly to nested schema,
but found {wrapper-width} uses for {inner-width} names
This shape is hit by every PPL `multisearch` query whose coordinator stage
is `Sort(Aggregate(Union(StageInputScan, StageInputScan)))` — the Aggregate
narrows the wide Union row type, and the inner-names override surfaced the
mismatch as a 500.
Fix: derive the new `Plan.Root.names` from the wrapper RelNode's row type
(`fragment.getRowType().getFieldList()`), not the inner plan. Both
`attachFragmentOnTop` and `attachPartialAggOnTop` already have the wrapper
RelNode in scope, so this is a local change with no signature ripple beyond
adding a `List<String> wrapperNames` parameter to `rewire`.
Test coverage:
- `testAttachPartialAggOnTop_PlanRootNamesMatchWrapperOutput` — the
partial-agg path with a 3-column inner scan and a 1-column wrapper
aggregate; pins names to the wrapper's output.
- `testAttachFragmentOnTop_AggregateOverMultiColumnInner_PlanRootNamesMatchWrapperOutput`
— the multisearch coordinator-stage shape (`Aggregate(Union)`).
- `testMultisearchShape_SortOverAggregateOverThreeWayUnion_PlanRootNamesMatchTopOutput`
— full chain `Sort → Aggregate → Union(Sin × 3)` modeling the
`testMultisearchWithThreeSubsearches` query plan.
- `testMultisearchShape_SystemLimitOverSortOverAggregateOverUnion_NamesMatchTopOutput`
— adds the implicit `LogicalSystemLimit` wrapper that
`QueryService.convertToCalcitePlan` injects at the top of every
analytics-engine plan, lowered to a Substrait `Fetch`.
End-to-end validation against `:integ-test:integTestRemote --tests
'org.opensearch.sql.calcite.remote.CalciteMultisearchCommandIT'` with
`-Dtests.analytics.force_routing=true -Dtests.analytics.parquet_indices=true`:
21 tests, **5 pass** (was 0/21 before this change). The remaining 16 are
blocked on orthogonal issues that are out of scope here:
- `Field [...] not found.` analyzer errors (8 tests) — pre-existing
parquet-backed-index field-resolution gap.
- TIMESTAMP / SPAN scalar functions unsupported (4 tests).
- AssertionError on error-message format (2 tests).
- One residual `Names list must match exactly to nested schema, but found
2 uses for 6 names` on `testMultisearchWithThreeSubsearches` — likely a
different code path (PARTIAL/FINAL split via `OpenSearchAggregateSplitRule`)
that the convertor unit tests don't exercise; tracked for follow-up.
- Two timeouts/long-running on the largest queries.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* [QA] Add MultisearchCommandIT for the analytics-engine REST path
Lands a self-contained QA IT covering PPL `multisearch` so the analytics-route
fix in this PR is exercised inside core without cross-plugin dependencies on
the SQL plugin. Three tests, scoped to the surface analytics already supports
end-to-end:
| Test | Shape |
|---|---|
| `testMultisearchTwoBranchesByCategory` | Basic 2-way Union over int0 buckets — `Union(Filter+Eval+Project, Filter+Eval+Project)` followed by `Aggregate(count by) | sort`. Exercises the same convertReduceFragment chain (`attachFragmentOnTop(Sort, attachFragmentOnTop(Aggregate, convertFinalAggFragment(Union)))`) that the rewire fix targets. |
| `testMultisearchThreeBranchesByStr0` | 3-way Union — the exact `Union(ER, ER, ER)` shape that surfaced the residual "2 uses for 6 names" failure I'd flagged as a follow-up in an earlier draft of the PR description; the rewire fix already covers it on a fresh cluster. |
| `testMultisearchSingleSubsearchRejected` | Arity check — pinned at the parser layer (AstBuilder.visitMultisearchCommand rejects <2 subsearches with `SyntaxCheckException`). Regression-pin against accidental relaxation of that guard. |
Each branch projects to a scalar-only field set (`fields int0, class` /
`fields str0, bucket`) so the union row type sidesteps the calcs dataset's
date/time/datetime columns — `ArrowSchemaFromCalcite.toArrowType` doesn't yet
handle TIMESTAMP, tracked separately.
Bumps `test-ppl-frontend`'s `unified-query-*` dependency from 3.6.0.0-SNAPSHOT
to 3.7.0.0-SNAPSHOT so the bundled PPL grammar exposes the `multisearch`
keyword (along with table/regex/rex/convert added since 3.6). The SQL
Snapshots repo (already declared in the build) carries the published 3.7
artifacts; for local sql-repo HEAD development, run
`./gradlew :ppl:publishUnifiedQueryPublicationToMavenLocal` from the sql repo.
The version bump is independent of the in-flight test-ppl-frontend
UnifiedQueryService.setting() change in opensearch-project#21526 — different files, no conflict.
Validates: 3/3 MultisearchCommandIT pass; full
`:sandbox:qa:analytics-engine-rest:integTest` suite still green
(110 tests across 14 ITs).
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* [Analytics Backend / DataFusion] Spotless reformat of testFinalAggInnerStageScanRowType
Single-line spotless reformat in DataFusionFragmentConvertorTests:
the OpenSearchStageInputScan constructor call now fits on one line
(it was previously broken across multiple lines).
Originally bumped sandbox/plugins/analytics-backend-datafusion's
sqlUnifiedQueryVersion 3.6 -> 3.7 to align with test-ppl-frontend, but
the entire internalClusterTest classpath block (including that pin) was
removed upstream by opensearch-project#21555 (dbe4a42, "Enable Lucene Filter delegation
from Datafusion for Correctness"). The build.gradle hunk dropped during
rebase; only the spotless reformat survives.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* [Analytics Backend / DataFusion] Register CASE in project capabilities + QA IT
Calcite emits SqlKind.CASE for any conditional expression — explicit `eval x =
case(cond, val, …)` in PPL, plus the `count(eval(predicate))` conditional-count
idiom (lowered to COUNT(CASE WHEN predicate THEN … END)) and several other
shapes. Without CASE in `STANDARD_PROJECT_OPS`, the analytics planner rejected
the operator with `No backend supports scalar function [CASE] among
[datafusion]` before substrait emission.
CASE doesn't need a backend adapter: isthmus translates SqlKind.CASE
structurally to a Substrait IfThen rel, and DataFusion's substrait consumer
handles IfThen natively. Just registering the capability is enough.
Adds `testMultisearchEvalCaseProjection` to MultisearchCommandIT to pin the
end-to-end path — multisearch + `eval bucket = case(cond, val else default)` +
stats. Uses an explicit `else` arm so isthmus doesn't have to convert an
untyped NULL literal; the implicit-else `count(eval(…))` shape that the v2-side
testMultisearchSuccessRatePattern uses still hits a separate isthmus
limitation (`Unable to convert the type NULL` from `TypeConverter` on a
SqlTypeName.NULL literal — tracked separately, out of scope here).
Validates: 4/4 MultisearchCommandIT pass; full
:sandbox:qa:analytics-engine-rest:integTest suite still green (111 tests
across 14 ITs).
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* [Analytics Backend / DataFusion] Pre-isthmus untyped-NULL rewriter for CASE arms
Calcite emits a `RexLiteral` with `SqlTypeName.NULL` for the implicit ELSE arm
of `CASE WHEN cond THEN val END` — exactly the shape PPL `count(eval(predicate))`
lowers to (`COUNT(CASE WHEN predicate THEN <projected> END)`). Isthmus'
`TypeConverter.toSubstrait` rejects `SqlTypeName.NULL` with `Unable to convert
the type NULL`, blocking the analytics path before substrait emission.
Adds `UntypedNullPreprocessor` — a `RelHomogeneousShuttle` + `RexShuttle` pass
applied in `convertToSubstrait` and `convertStandalone` *before* the
SubstraitRelVisitor sees the plan. Walks every CASE call's value operands
(THEN arms and the ELSE arm) and substitutes any `SqlTypeName.NULL` literal
with a typed null literal matching the CASE's resolved return type. Calcite
already widens the CASE's return type to the leastRestrictive of branches, so
the substituted type is correct by construction.
Scope is intentionally narrow: only CASE call operands are rewritten today.
Other untyped-NULL contexts (function arguments, comparison RHS) are rare in
PPL-generated plans and would need per-operator type inference to do safely;
defer until a concrete test surfaces one.
Test coverage:
- `UntypedNullPreprocessorTests` (4 new):
* `testCountEvalCaseRewritesElseNullToTypedNull` — the motivating shape
`COUNT(CASE WHEN cond THEN 1 ELSE null END)`.
* `testCaseWithThenNullIsAlsoRewritten` — null in the THEN arm.
* `testCaseConditionOperandUnchanged` — even-index condition operands left alone.
* `testCountOverRewrittenCaseProjectionTypechecks` — Aggregate(Project(CASE))
with the rewriter applied still type-checks end-to-end.
- New `testMultisearchCountEvalConditionalCount` in MultisearchCommandIT —
mirrors the v2-side `CalciteMultisearchCommandIT.testMultisearchSuccessRatePattern`
shape (`count(eval(predicate))`) end-to-end on the analytics-engine REST path.
Validates: 5/5 MultisearchCommandIT pass; 4/4 new + 12/12 existing
FragmentConvertor unit tests; full :sandbox:qa:analytics-engine-rest:integTest
suite still green (112 tests across 14 ITs).
Signed-off-by: Kai Huang <ahkcs@amazon.com>
* [Analytics Backend / DataFusion] Simplify testCaseConditionOperandUnchanged for spotless
The original assertion wrapped a no-op single-iteration loop around an empty-
body RexShuttle whose accept() result is just caseExpr.toString(). Spotless
flagged the empty class body (`new RexShuttle() {}`) as a formatting violation
and tried to wrap it across two lines, which read worse than the underlying
intent — comparing the input CASE expression to the rewriter's output to
prove no-op behavior when no untyped nulls are present.
Replace the loop+shuttle with a direct
`assertEquals(caseExpr.toString(), rewrittenCase.toString())` — same
semantics, cleaner code, no awkward formatting. The test still asserts the
rewriter doesn't touch CASE expressions whose operands are already typed.
Sandbox check (`./gradlew check -p sandbox -Dsandbox.enabled=true`) now
passes end-to-end.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
---------
Signed-off-by: Kai Huang <ahkcs@amazon.com>1 parent 8560342 commit b6c4b9d
7 files changed
Lines changed: 768 additions & 12 deletions
File tree
- sandbox
- plugins
- analytics-backend-datafusion/src
- main/java/org/opensearch/be/datafusion
- test/java/org/opensearch/be/datafusion
- test-ppl-frontend
- qa/analytics-engine-rest/src/test/java/org/opensearch/analytics/qa
Lines changed: 8 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
104 | 112 | | |
105 | 113 | | |
106 | 114 | | |
| |||
Lines changed: 36 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
| 73 | + | |
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| |||
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
129 | | - | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
130 | 134 | | |
131 | 135 | | |
132 | 136 | | |
| |||
150 | 154 | | |
151 | 155 | | |
152 | 156 | | |
153 | | - | |
| 157 | + | |
154 | 158 | | |
155 | 159 | | |
156 | 160 | | |
157 | 161 | | |
158 | 162 | | |
159 | | - | |
160 | | - | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
161 | 170 | | |
162 | 171 | | |
163 | 172 | | |
| |||
178 | 187 | | |
179 | 188 | | |
180 | 189 | | |
181 | | - | |
| 190 | + | |
182 | 191 | | |
183 | 192 | | |
184 | | - | |
185 | | - | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
186 | 199 | | |
187 | 200 | | |
188 | 201 | | |
| |||
191 | 204 | | |
192 | 205 | | |
193 | 206 | | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
194 | 215 | | |
195 | | - | |
| 216 | + | |
196 | 217 | | |
197 | 218 | | |
198 | 219 | | |
199 | 220 | | |
200 | 221 | | |
201 | 222 | | |
202 | | - | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
203 | 229 | | |
204 | 230 | | |
205 | 231 | | |
| |||
Lines changed: 120 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
0 commit comments