fix(extensions): add signatures with distribution enum arg for std_dev and variance#1011
Merged
Merged
Conversation
nielspardon
requested review from
EpsilonPrime,
cpcloud,
jacques-n,
vbarua,
westonpace and
yongchul
as code owners
March 17, 2026 12:19
This comment was marked as resolved.
This comment was marked as resolved.
nielspardon
force-pushed
the
par-stddev-dist
branch
from
March 17, 2026 12:20
8ec63ae to
0877df5
Compare
nielspardon
changed the title
nielspardon
changed the title
Member
|
I'm in support of this change, but think we should wait until #1005 lands to make sure we all agree on the meaning of options / enums. |
yongchul
reviewed
Mar 19, 2026
nielspardon
force-pushed
the
par-stddev-dist
branch
from
March 20, 2026 19:28
5aee2ee to
bd05214
Compare
Member
Author
|
Rebased, resolved conflicts, used the new deprecation syntax. |
yongchul
approved these changes
Mar 20, 2026
yongchul
left a comment
yongchul
left a comment
Contributor
There was a problem hiding this comment.
+1 thank you for incorporating the deprecation path and making this non-breaking change! 😄
nielspardon
force-pushed
the
par-stddev-dist
branch
from
March 23, 2026 08:46
72c5cca to
4e9def0
Compare
Member
Author
|
I merged #1010 and rebased this PR |
nielspardon
changed the title
…and variance BREAKING CHANGE: changes the function signature of existing functions std_dev and variance Signed-off-by: Niels Pardon <par@zurich.ibm.com>
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
nielspardon
force-pushed
the
par-stddev-dist
branch
from
March 23, 2026 17:51
4e9def0 to
b364972
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
Member
Author
|
removed the deprecation of the old function signatures per conversation on Slack |
vbarua
reviewed
Mar 23, 2026
vbarua
reviewed
Mar 23, 2026
vbarua
reviewed
Mar 23, 2026
vbarua
reviewed
Mar 23, 2026
vbarua
reviewed
Mar 23, 2026
vbarua
left a comment
vbarua
left a comment
Member
There was a problem hiding this comment.
Left a minor suggestions about argument ordering. Want to hear what you think about it, but other than that this looks good to me.
nielspardon
force-pushed
the
par-stddev-dist
branch
from
March 24, 2026 08:39
2c6857a to
62da266
Compare
Signed-off-by: Niels Pardon <par@zurich.ibm.com>
nielspardon
force-pushed
the
par-stddev-dist
branch
from
March 24, 2026 08:40
62da266 to
8175a73
Compare
vbarua
approved these changes
Mar 24, 2026
vbarua
left a comment
vbarua
left a comment
Member
There was a problem hiding this comment.
Changes look good. Thanks for updating thee argument order.
andrew-coleman
pushed a commit
to substrait-io/substrait-java
that referenced
this pull request
Jul 15, 2026
This PR implements the bidirectional conversion between Calcite and Substrait for the statistical aggregate functions standard deviation and variance (`STDDEV_POP`, `STDDEV_SAMP`, `VAR_POP`, `VAR_SAMP`). ### Problem Substrait uses a single function name for both the population and sample variants: - `std_dev` for both `STDDEV_POP` and `STDDEV_SAMP` - `variance` for both `VAR_POP` and `VAR_SAMP` The population (n denominator) vs. sample (n-1 denominator) distinction is captured by a `distribution` value (`POPULATION` or `SAMPLE`). Previously these functions were not mapped, so the existing TPC-DS test cases using them were silently mis-mapped to `AVG` (Calcite represents these statistical functions with `SqlAvgAggFunction`). ### Solution This uses the **non-deprecated** signatures introduced in substrait-io/substrait#1011 (Substrait v0.87.0), which carry the distinction as an **enum function argument** (the leading `distribution` argument, e.g. `std_dev:req_fp64`), rather than the deprecated function-option form (substrait-io/substrait#1019). Resolves #803 Resolves #807 Since the input arguments are cast to `FP64` when necessary, the integer-based signatures proposed in substrait-io/substrait#1012 are not required. **Calcite → Substrait:** - Added function mappings in `FunctionMappings` for all four statistical operators. - `AggregateFunctionConverter` synthesizes the leading `distribution` enum operand based on the Calcite `SqlKind`, so the generic function matcher resolves the enum-arg variant and builds the `EnumArg` automatically (no bespoke option plumbing). - Statistical inputs are cast to `FP64` where necessary. **Substrait → Calcite:** - `FunctionConverter.getSqlOperatorFromSubstraitFunc` disambiguates the population/sample operator from the `distribution` enum argument. - `SubstraitRelNodeConverter` and `PreCalciteAggregateValidator` skip the non-value enum argument when building Calcite aggregate operands. **DSL & shared enum:** - A shared `StatisticalDistribution` enum (in `:core`) is the single source of truth for the `SAMPLE` / `POPULATION` values used by both the DSL builder and isthmus. - `SubstraitBuilder` gains `stddevPopulation`, `stddevSample`, `variancePopulation`, and `varianceSample` convenience methods. ### Non-floating-point inputs `std_dev` / `variance` only define `fp32` / `fp64` signatures, so a statistical aggregate over an integer (or other non-fp) column is rewritten in `SubstraitRelVisitor` to cast the argument to `fp64` (appending a cast column so other aggregates over the same column are unaffected) and cast the result back to the type Calcite inferred. The rewrite is idempotent, so the converted plan is stable under further round trips. ### Testing - `AggregationFunctionsTest` exercises full round trips (POJO ⇄ proto and Substrait ⇄ Calcite) for all four functions, with and without grouping. - `StatisticalFunctionTest` verifies the SQL round trip, asserts that each SQL operator maps to the enum-arg signature (`std_dev:req_fp64` etc.) with the correct `distribution` `EnumArg` and no function options, and covers non-fp (integer) inputs — including a column shared with a non-statistical aggregate, and with grouping. 🤖 Generated with AI --------- Signed-off-by: Niels Pardon <par@zurich.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Given the recent clarification around function options vs enum arguments I do believe the
std_devandvariancefunction definitions should change thedistributionfunction to be an enum argument.The
distributionparameter changes the semantic behavior of these functions which is why SQL systems typically model the differentdistributionvalues as distinct functions e.g.STDDEV_SAMPandSTDDEV_POP. Providing thedistributionparameter thus is not optional but required.This PR deprecates the existing function signatures for
std_devandvarianceand adds new signatures that use adistributionenum argument instead of a function option.This PR depends on #1010 whose changes are included in this PR since it uses the new enum argument syntax in the test cases I generated using AI for the two functions.
The main changes for this PR are in the second commit 8ec63ae.