Add statistical type aggregate functions, including autocorrelation, skewness, and linear regression

[CoollZzz]

This pull request adds integration tests for new statistical aggregation functions in IoTDB, specifically focusing on correlation, covariance, and related statistical measures. It introduces comprehensive test coverage for these functions, updates utility constants, and expands the list of recognized aggregation functions.

New aggregation function support and test coverage:

• Added new aggregation functions to BuiltinAggregationFunctionEnum: CORR, COVAR_POP, COVAR_SAMP, REGR_SLOPE, REGR_INTERCEPT, SKEWNESS, and KURTOSIS.
• Added corresponding string helper methods for these functions in TestConstant (e.g., corr, covarPop, covarSamp, etc.), enabling easier test query construction.

Integration tests for statistical functions:

• Added IoTDBCorrelationIT integration test class, which tests the corr (correlation) function across various scenarios: type checking, different data types, alignment by device, use in HAVING clauses, multi-device queries, group by level, and sliding window queries.
• Added IoTDBCovarianceIT integration test class, which tests covar_pop and covar_samp (covariance) functions, including type and arity validation, cross-type queries, alignment by device, HAVING clause usage, group by level, and sliding window queries.This pull request adds support for several advanced statistical aggregation functions to the IoTDB query engine, including correlation, covariance, regression, skewness, and kurtosis. It introduces new accumulator classes to implement these functions and updates the relevant factory and enum classes to register and handle them appropriately.

New statistical aggregation functions:

• Added new aggregation function types: CORR, COVAR_POP, COVAR_SAMP, REGR_SLOPE, REGR_INTERCEPT, SKEWNESS, and KURTOSIS to BuiltinAggregationFunctionEnum for recognition in the system.
• Updated AccumulatorFactory to recognize these new functions as multi-input or single-input aggregations and to instantiate the appropriate new accumulator classes when requested. [1] [2] [3]

Implementation of new accumulator classes:

• Introduced CorrelationAccumulator for correlation and covariance calculations, supporting both population and sample variants.
• Introduced RegressionAccumulator for regression slope and intercept calculations.
• Introduced CentralMomentAccumulator for skewness and kurtosis calculations.

Integration with sliding window aggregators:

• Updated SlidingWindowAggregatorFactory to support the new statistical functions within sliding window queries.This pull request introduces support for several advanced statistical aggregation functions in the IoTDB query engine, including correlation, covariance, regression, skewness, and kurtosis. It adds new accumulator implementations for these functions and integrates them into the aggregation and sliding window frameworks.

The most important changes are:

New statistical accumulator implementations:

• Added CentralMomentAccumulator for computing skewness and kurtosis. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CentralMomentAccumulator.java)
• Added CorrelationAccumulator for correlation and covariance calculations. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CorrelationAccumulator.java)
• Added RegressionAccumulator for regression slope and intercept calculations. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/RegressionAccumulator.java)

Integration with aggregation framework:

• Updated AccumulatorFactory to support the new aggregation functions, including logic for multi-input and single-input accumulators, and to instantiate the new accumulator classes as appropriate. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/AccumulatorFactory.java) [1] [2] [3]

Sliding window support:

• Modified SlidingWindowAggregatorFactory to support the new statistical functions within sliding window aggregations. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/slidingwindow/SlidingWindowAggregatorFactory.java)

[Copilot]

Pull request overview

This PR introduces several advanced statistical aggregation functions (correlation, covariance, linear regression, skewness, kurtosis) into IoTDB’s aggregation framework across both TREE and relational/table engines, and adds integration tests to validate behavior and semantics end-to-end.

Changes:

• Register new aggregation types (CORR, COVAR_POP, COVAR_SAMP, REGR_SLOPE, REGR_INTERCEPT, SKEWNESS, KURTOSIS) across thrift, SQL constants, function enums, type inference, and factories.
• Implement new accumulator/state logic for correlation/covariance/regression/central moments (including grouped + table accumulators and sliding-window support).
• Add integration tests (TREE and relational) covering correct results, edge cases, HAVING, align-by-device/group-by-level, and sliding windows.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
iotdb-protocol/thrift-commons/src/main/thrift/common.thrift	Adds new TAggregationType enum values for the statistical aggregations.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/relational/TableBuiltinAggregationFunction.java	Registers new table builtin aggregation names and intermediate-type mapping.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/TypeInferenceUtils.java	Adds result-type inference and datatype validation for the new functions (including 2-input validation helper).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/SchemaUtils.java	Wires new aggregation names/types, partial suffix handling, and scan-order consistency.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/constant/SqlConstant.java	Adds SQL function-name constants for the new aggregations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/udf/BuiltinAggregationFunction.java	Registers new functions and updates multi-phase/statistics capability flags.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/metadata/TableMetadataImpl.java	Adds relational semantic checks (arity/type) and return types for the new functions.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/parameter/AggregationDescriptor.java	Adds partial-aggregation name mapping for new TAggregationTypes.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/parser/ASTVisitor.java	Updates aggregation arity validation for new functions.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/ExpressionTypeAnalyzer.java	Adds 2-input type validation hook for multi-input statistical aggregations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/TableRegressionAccumulator.java	New table accumulator for regression slope/intercept.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/TableCovarianceAccumulator.java	New table accumulator for covariance (pop/samp).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/TableCorrelationAccumulator.java	New table accumulator for correlation.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/TableCentralMomentAccumulator.java	New table accumulator for skewness/kurtosis.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/grouped/GroupedRegressionAccumulator.java	New grouped table accumulator for regression slope/intercept.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/grouped/GroupedCovarianceAccumulator.java	New grouped table accumulator for covariance (pop/samp).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/grouped/GroupedCorrelationAccumulator.java	New grouped table accumulator for correlation.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/grouped/GroupedCentralMomentAccumulator.java	New grouped table accumulator for skewness/kurtosis.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/AccumulatorFactory.java	Registers new table/grouped accumulators and marks new multi-input aggregations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/slidingwindow/SlidingWindowAggregatorFactory.java	Enables sliding-window aggregation support for new statistical functions.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/RegressionAccumulator.java	New TREE-model accumulator for regression slope/intercept (with merge/remove).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CovarianceAccumulator.java	New TREE-model accumulator for covariance (pop/samp) (with merge/remove).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CorrelationAccumulator.java	New TREE-model accumulator for correlation (with merge/remove).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CentralMomentAccumulator.java	New TREE-model accumulator for skewness/kurtosis (with merge/remove).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/AccumulatorFactory.java	Registers new TREE-model accumulators and multi-input aggregation creation.
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/IoTDBTableAggregationIT.java	Adds relational integration tests and test data for the new statistical functions.
integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBSkewnessKurtosisIT.java	Adds TREE-model IT coverage for skewness/kurtosis across scenarios.
integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBRegressionIT.java	Adds TREE-model IT coverage for regression slope/intercept across scenarios.
integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBCovarianceIT.java	Adds TREE-model IT coverage for covariance pop/samp across scenarios.
integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBCorrelationIT.java	Adds TREE-model IT coverage for correlation across scenarios.
integration-test/src/main/java/org/apache/iotdb/itbase/constant/TestConstant.java	Adds helper string builders for new function names in tests.
integration-test/src/main/java/org/apache/iotdb/itbase/constant/BuiltinAggregationFunctionEnum.java	Extends test-side builtin aggregation enum with new functions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 3 comments.

[JackieTien97]

Thanks for the substantial work — the coverage is thorough (tree + table + grouped accumulators, sliding-window / window-frame support, type validation, and a good spread of ITs including multi-region merge).

I have one correctness issue I'd like resolved before merge, one semantics question, and a few minor cleanups (all left as inline comments):

Correctness

• long overflow in the KURTOSIS denominator (CentralMomentAccumulator, TableCentralMomentAccumulator, GroupedCentralMomentAccumulator): (count-1)*(count-2)*(count-3) is computed in long and overflows for count above ~2.1M (sign-flips near ~2.5M), producing wrong kurtosis. Million-row groups are common here. Fix by computing in double.

Semantics

• SKEWNESS (population / biased) vs KURTOSIS (sample / bias-corrected) use different estimator conventions. Please confirm intent, make them consistent, and document the exact definitions.

Minor

• Tree CentralMomentAccumulator lacks the EPSILON zero-clamp that the table variants apply after element removal — can yield spurious non-null skew/kurtosis in sliding windows over constant data.
• CorrelationType is a single-value enum with an unreachable guard branch (dead code; repeated in the table/grouped variants).
• TypeInferenceUtils: Javadoc /** */ used as an inline comment inside a switch.
• ExpressionTypeAnalyzer: a stray blank line is the only change in the file.

A couple of non-blocking notes not tied to a specific line:

• No user-facing documentation update for the 7 new SQL functions — is a docs PR planned?
• getDoubleValue(...) is duplicated nearly verbatim across ~9 new classes; it could be a shared helper.

Overall direction looks good; mainly the kurtosis overflow needs addressing.