feat(extensions)!: deprecate std_dev and variance using function options#1019
Merged
Conversation
nielspardon
requested review from
EpsilonPrime,
cpcloud,
jacques-n,
vbarua,
westonpace and
yongchul
as code owners
March 23, 2026 18:41
nielspardon
force-pushed
the
par-depr-stddev
branch
from
March 24, 2026 08:45
d20dde3 to
f60d818
Compare
BREAKING CHANGE: deprecates the function signatures for std_dev and variance using function options in favor of the versions using enum arguments Signed-off-by: Niels Pardon <par@zurich.ibm.com>
nielspardon
force-pushed
the
par-depr-stddev
branch
from
April 7, 2026 10:40
f60d818 to
3393a10
Compare
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
Member
Author
|
rebased on latest |
jacques-n
approved these changes
Apr 7, 2026
andrew-coleman
pushed a commit
to substrait-io/substrait-java
that referenced
this pull request
Jul 15, 2026
This PR implements the bidirectional conversion between Calcite and Substrait for the statistical aggregate functions standard deviation and variance (`STDDEV_POP`, `STDDEV_SAMP`, `VAR_POP`, `VAR_SAMP`). ### Problem Substrait uses a single function name for both the population and sample variants: - `std_dev` for both `STDDEV_POP` and `STDDEV_SAMP` - `variance` for both `VAR_POP` and `VAR_SAMP` The population (n denominator) vs. sample (n-1 denominator) distinction is captured by a `distribution` value (`POPULATION` or `SAMPLE`). Previously these functions were not mapped, so the existing TPC-DS test cases using them were silently mis-mapped to `AVG` (Calcite represents these statistical functions with `SqlAvgAggFunction`). ### Solution This uses the **non-deprecated** signatures introduced in substrait-io/substrait#1011 (Substrait v0.87.0), which carry the distinction as an **enum function argument** (the leading `distribution` argument, e.g. `std_dev:req_fp64`), rather than the deprecated function-option form (substrait-io/substrait#1019). Resolves #803 Resolves #807 Since the input arguments are cast to `FP64` when necessary, the integer-based signatures proposed in substrait-io/substrait#1012 are not required. **Calcite → Substrait:** - Added function mappings in `FunctionMappings` for all four statistical operators. - `AggregateFunctionConverter` synthesizes the leading `distribution` enum operand based on the Calcite `SqlKind`, so the generic function matcher resolves the enum-arg variant and builds the `EnumArg` automatically (no bespoke option plumbing). - Statistical inputs are cast to `FP64` where necessary. **Substrait → Calcite:** - `FunctionConverter.getSqlOperatorFromSubstraitFunc` disambiguates the population/sample operator from the `distribution` enum argument. - `SubstraitRelNodeConverter` and `PreCalciteAggregateValidator` skip the non-value enum argument when building Calcite aggregate operands. **DSL & shared enum:** - A shared `StatisticalDistribution` enum (in `:core`) is the single source of truth for the `SAMPLE` / `POPULATION` values used by both the DSL builder and isthmus. - `SubstraitBuilder` gains `stddevPopulation`, `stddevSample`, `variancePopulation`, and `varianceSample` convenience methods. ### Non-floating-point inputs `std_dev` / `variance` only define `fp32` / `fp64` signatures, so a statistical aggregate over an integer (or other non-fp) column is rewritten in `SubstraitRelVisitor` to cast the argument to `fp64` (appending a cast column so other aggregates over the same column are unaffected) and cast the result back to the type Calcite inferred. The rewrite is idempotent, so the converted plan is stable under further round trips. ### Testing - `AggregationFunctionsTest` exercises full round trips (POJO ⇄ proto and Substrait ⇄ Calcite) for all four functions, with and without grouping. - `StatisticalFunctionTest` verifies the SQL round trip, asserts that each SQL operator maps to the enum-arg signature (`std_dev:req_fp64` etc.) with the correct `distribution` `EnumArg` and no function options, and covers non-fp (integer) inputs — including a column shared with a non-statistical aggregate, and with grouping. 🤖 Generated with AI --------- Signed-off-by: Niels Pardon <par@zurich.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BREAKING CHANGE: deprecates the function signatures for std_dev and variance using function options in favor of the versions using enum arguments
Follow-up to and depending on #1011 main change in commit d20dde3
This change is