Stabilize PPL ITs on the analytics-engine route (array/map-path/datatype/basic)#5562
Merged
ahkcs merged 1 commit intoJun 18, 2026
Merged
Conversation
6a5c598 to
5646561
Compare
Swiddis
approved these changes
Jun 18, 2026
…ype/basic) Analytics-engine route parity for four PPL IT classes; test-only. Uses the @RequiresCapability annotation + Capability registry (opensearch-project#5560) plus matching excludeTestsMatching entries. CalciteArrayFunctionIT: - Skip the array-index load on the AE route (multi-value 'numbers' field the parquet store rejects); no test queries it (all build arrays inline). - 16 higher-order lambda functions (transform/mvmap, reduce, filter, exists, forall) -> new ARRAY_HIGHER_ORDER_FUNC (no DataFusion lambda execution). CalcitePPLMapPathIT: - mvcombine lowers to ARRAY_AGG, unregistered on the analytics backend -> new MVCOMBINE_ARRAY_AGG. - addtotals crashes the DataFusion backend with a join panic -> new ADDTOTALS_JOIN_PANIC. CalciteDataTypeIT (guards on base DataTypeIT; build.gradle globs broadened to '*' so they cover the Calcite subclass): - test_nonnumeric_data_types / test_alias_data_type: nested/object/geo/alias types stripped (NESTED_FIELDS). - test_numeric_data_types: scaled_float reported as bigint not double -> new SCALED_FLOAT_TYPE. - testNumericFieldFromString: empty-string -> numeric coerces to null not 0 -> new STRING_TO_NUMERIC_COERCION. - testBooleanFieldFromNumberAcrossWildcardIndices: cross-index incompatible field types rejected -> new CROSS_INDEX_INCOMPATIBLE_TYPES. - testBooleanFieldFromString: seeds+deletes a doc; DELETE unsupported (DOC_MUTATION). CalcitePPLBasicIT.testRegexpFilter: REGEXP filter throws a backend NullPointerException on the AE route -> new REGEXP_FILTER. v2/Calcite route unchanged (all run, 0 skips). Signed-off-by: Kai Huang <ahkcs@amazon.com>
5646561 to
921aa8c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Analytics-engine route (
-Dtests.analytics.parquet_indices=true) parity pass across four PPL IT classes. All changes are test-only; the v2/Calcite route is unchanged (every gated test still runs there — theassumeNotAnalytics(...)guards are no-ops off-route and the gradle excludes apply only tointegTestRemote).CalciteArrayFunctionIT (43/60 → 44/60 pass, 16 excluded)
arraydataset's multi-valuenumbersfield (scalarlongmapping) can't be bulk-loaded into the parquet store. No test queries thearrayindex (all build arrays inline viaarray(...)), so the load is skipped on the AE route.transform/mvmap,reduce,filter,exists,forall) →No backend supports scalar function [...]. NewARRAY_HIGHER_ORDER_FUNC(16 tests).CalcitePPLMapPathIT (25/27 → 25 pass, 2 excluded)
mvcombinelowers toARRAY_AGG, unregistered on the analytics backend → newMVCOMBINE_ARRAY_AGG.addtotalscrashes the DataFusion backend with a join panic → newADDTOTALS_JOIN_PANIC.CalciteDataTypeIT (6 fail → excluded; guards added to base
DataTypeIT)test_nonnumeric_data_types/test_alias_data_type: nested/object/geo/alias types stripped on the AE route (reuseNESTED_FIELDS). Also broadened the pre-existingorg.opensearch.sql.ppl.*build.gradle globs to*so they cover the Calcite subclass (a latent gap — the old globs never matchedCalciteDataTypeIT/CalciteSystemFunctionIT).test_numeric_data_types:scaled_floatreported asbigintnotdouble→ newSCALED_FLOAT_TYPE.testNumericFieldFromString: empty-string → numeric coerces tonullnot0→ newSTRING_TO_NUMERIC_COERCION.testBooleanFieldFromNumberAcrossWildcardIndices: cross-index incompatible field types rejected → newCROSS_INDEX_INCOMPATIBLE_TYPES.testBooleanFieldFromString: seeds + deletes a doc; DELETE unsupported (DOC_MUTATION).CalcitePPLBasicIT (1 fail → excluded)
testRegexpFilter: theREGEXPfilter throws a backendNullPointerException→ newREGEXP_FILTER. (The rest of this class was already brought to parity in Bring CalcitePPLBasicIT to parity on the analytics-engine route #5542.)Results (analytics route)
CalciteArrayFunctionITCalcitePPLMapPathITCalciteDataTypeITCalcitePPLBasicITCheck List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.