[POC][RFC-0025] Derived column supports expression by ScrapCodes · Pull Request #27832 · prestodb/presto

ScrapCodes · 2026-05-18T17:33:44Z

Description

Following is an example of sub expression matching.

presto:perf_test> create table test2 (c1 BIGINT, c2 VARCHAR, c2_derived varchar GENERATED ALWAYS AS if(lower(c2) = 'c', 'g', 'e') PERSISTENT , c4 double);
CREATE TABLE

Query 20260518_173427_00014_c9ibg, FINISHED, 0 nodes
http://127.0.0.1:8080/ui/query.html?20260518_173427_00014_c9ibg
Splits: 0 total, 0 done (0.00%)
[Latency: client-side: 48ms, server-side: 40ms] [0 rows, 0B] [0 rows/s, 0B/s]

presto:perf_test> show create table test2;
                                           Create Table                                           
--------------------------------------------------------------------------------------------------
 CREATE TABLE iceberg.perf_test.test2 (                                                           
    "c1" bigint,                                                                                  
    "c2" varchar,                                                                                 
    "c2_derived" varchar,                                                                         
    "c4" double                                                                                   
 )                                                                                                
 WITH (                                                                                           
    "derived-columns" = ARRAY['c2_derived'],                                                      
    "derived-columns.spec.expression.json" = '{                                                   
   "expressionSpecList" : [ {                                                                     
     "derivedColumnType" : "PERSISTENT",                                                          
     "derivedColumnExpression" : "if(lower(c2)=''c'',''g'',''e'')",                               
     "derivedColumnName" : "c2_derived"                                                           
   } ]                                                                                            
 }',                                                                                              
    "format-version" = '2',                                                                       
    location = 'file:/Users/prashantsharma/work/presto_config/hive-data/iceberg/perf_test/test2', 
    "read.split.target-size" = 134217728,                                                         
    "write.delete.mode" = 'merge-on-read',                                                        
    "write.format.default" = 'PARQUET',                                                           
    "write.metadata.delete-after-commit.enabled" = false,                                         
    "write.metadata.metrics.max-inferred-column-defaults" = 100,                                  
    "write.metadata.previous-versions-max" = 100,                                                 
    "write.update.mode" = 'merge-on-read'                                                         
 )                                                                                                
(1 row)

Query 20260518_173437_00015_c9ibg, FINISHED, 1 node
http://127.0.0.1:8080/ui/query.html?20260518_173437_00015_c9ibg
Splits: 1 total, 1 done (100.00%)
[Latency: client-side: 33ms, server-side: 23ms] [0 rows, 0B] [0 rows/s, 0B/s]

presto:perf_test> set session iceberg.derived_columns_enabled=true;
SET SESSION

Query 20260518_173449_00016_c9ibg, FINISHED, 0 nodes
http://127.0.0.1:8080/ui/query.html?20260518_173449_00016_c9ibg
Splits: 0 total, 0 done (0.00%)
[Latency: client-side: 15ms, server-side: 1ms] [0 rows, 0B] [0 rows/s, 0B/s]

presto:perf_test> insert into test2 values (122, 'A', 'e', 12.2);
INSERT: 1 row

Query 20260518_173509_00017_c9ibg, FINISHED, 1 node
http://127.0.0.1:8080/ui/query.html?20260518_173509_00017_c9ibg
Splits: 35 total, 35 done (100.00%)
[Latency: client-side: 0:01, server-side: 0:01] [0 rows, 0B] [0 rows/s, 0B/s]

presto:perf_test> insert into test2 values (121, 'C', 'g', 12.3);
INSERT: 1 row

Query 20260518_173526_00018_c9ibg, FINISHED, 1 node
http://127.0.0.1:8080/ui/query.html?20260518_173526_00018_c9ibg
Splits: 35 total, 35 done (100.00%)
[Latency: client-side: 130ms, server-side: 126ms] [0 rows, 0B] [0 rows/s, 0B/s]

presto:perf_test> SELECT * from test2 WHERE (c1 > 120) AND (if(lower(c2) = 'c', 'g', 'e') = 'g');
 c1  | c2 | c2_derived |  c4  
-----+----+------------+------
 121 | C  | g          | 12.3 
(1 row)

Query 20260518_173539_00019_c9ibg, FINISHED, 1 node
http://127.0.0.1:8080/ui/query.html?20260518_173539_00019_c9ibg
Splits: 18 total, 18 done (100.00%)
[Latency: client-side: 267ms, server-side: 253ms] [2 rows, 1.25KB] [7 rows/s, 4.93KB/s]

presto:perf_test> explain SELECT * from test2 WHERE (c1 > 120) AND (if(lower(c2) = 'c', 'g', 'e') = 'g');
                                                                                                                                                           Query Plan                                                                                                                                                           
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 - Output[PlanNodeId 6][c1, c2, c2_derived, c4] => [c1:bigint, c2:varchar, c2_derived:varchar, c4:double]                                                                                                                                                                                                                       
         Estimates: {source: CostBasedSourceInfo, rows: 1 (64B), cpu: 256.00, memory: 0.00, network: 64.00}                                                                                                                                                                                                                     
     - RemoteStreamingExchange[PlanNodeId 221][GATHER - COLUMNAR] => [c1:bigint, c2:varchar, c2_derived:varchar, c4:double]                                                                                                                                                                                                     
             Estimates: {source: CostBasedSourceInfo, rows: 1 (64B), cpu: 256.00, memory: 0.00, network: 64.00}                                                                                                                                                                                                                 
         - ScanFilter[PlanNodeId 0,254][table = TableHandle {connectorId='iceberg', connectorHandle='test2$data@1546330844629902434', layout='Optional[test2$data@1546330844629902434]'}, filterPredicate = ((c1) > (BIGINT'120')) AND ((c2_derived) = (VARCHAR'g'))] => [c2:varchar, c2_derived:varchar, c4:double, c1:bigint] 
                 Estimates: {source: CostBasedSourceInfo, rows: 1 (128B), cpu: 128.00, memory: 0.00, network: 0.00}/{source: CostBasedSourceInfo, rows: 1 (64B), cpu: 256.00, memory: 0.00, network: 0.00}                                                                                                                      
                 c2 := 2:c2:varchar (1:23)                                                                                                                                                                                                                                                                                      
                 c2_derived := 3:c2_derived:varchar (1:23)                                                                                                                                                                                                                                                                      
                     :: [["g"]]                                                                                                                                                                                                                                                                                                 
                 c4 := 4:c4:double (1:23)                                                                                                                                                                                                                                                                                       
                 c1 := 1:c1:bigint (1:23)                                                                                                                                                                                                                                                                                       
                     :: [("120", <max>)]                                                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                
(1 row)

Query 20260518_173604_00022_c9ibg, FINISHED, 1 node
http://127.0.0.1:8080/ui/query.html?20260518_173604_00022_c9ibg
Splits: 1 total, 1 done (100.00%)
[Latency: client-side: 124ms, server-side: 112ms] [0 rows, 0B] [0 rows/s, 0B/s]

presto:perf_test>

Motivation and Context

Impact

Test Plan

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.
If adding new dependencies, verified they have an OpenSSF Scorecard score of 5.0 or higher (or obtained explicit TSC approval for lower scores).

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* ... 
* ... 

Hive Connector Changes
* ... 
* ...

If release note is NOT required, use:

== NO RELEASE NOTE ==

Summary by Sourcery

Add derived column support for Iceberg tables, including SQL syntax, metadata propagation, and a plan optimizer that rewrites filter predicates to use precomputed derived columns when enabled.

New Features:

Introduce derived column table and column properties in Iceberg, including JSON-based expression specifications and session/config flags to enable the feature.
Extend SQL parser and AST to support GENERATED ... AS expressions on columns and propagate derived column expressions into ColumnMetadata and Iceberg table creation.
Add an Iceberg plan optimizer that identifies filter predicates matching configured derived column expressions and rewrites them to reference the corresponding derived columns for query execution.

Enhancements:

Expose function handle return types and canonicalization to support expression equivalence when matching derived column expressions.
Propagate derived column metadata through Iceberg table metadata read/write paths so derived columns and their expressions are stored and retrieved with table properties.
Wire SqlParser and new optimizer into Iceberg modules and adjust existing utilities and helpers (e.g., table property population, metadata creation) to handle derived column properties.

Build:

Add presto-parser as a dependency of presto-iceberg and adjust the maven-dependency-plugin configuration.

Tests:

Add integration tests validating Iceberg derived column filter predicate rewrites and behavior with multiple and overlapping derived column rules.
Add configuration mapping tests for the new Iceberg derived column enable flag.

sourcery-ai · 2026-05-18T17:34:36Z

Reviewer's Guide

Introduces derived/GENERATED column support for Iceberg tables, including new SQL syntax, table and column metadata properties, a JSON-based specification for derived column expressions, and a connector-side plan optimizer that rewrites filter predicates to reuse pre-computed derived columns, plus config/session toggles and tests.

Flow diagram for CREATE TABLE with derived/GENERATED columns

flowchart LR
    subgraph Parser
        A[SQL: CREATE TABLE ...
             col GENERATED AS expr PERSISTENT]
        B[SqlBase.g4
           columnDefinition rule]
        C[AstBuilder.visitColumnDefinition]
        D[ColumnDefinition
           +DerivedColumnDefinition]
    end

    subgraph Analyzer
        E[CreateTableTask
           builds ColumnMetadata
           with DERIVED_COLUMN_EXPRESSION]
        F[ConnectorTableMetadata
           properties include
           DERIVED_COLUMN_EXPRESSION]
    end

    subgraph IcebergMetadata
        G[IcebergAbstractMetadata.createTable]
        H[DerivedColumnExpressionSpecList
           +DerivedColumnExpressionSpec]
        I[IcebergTableProperties
           getDerivedColumnExpressionSpec]
        J[IcebergUtil.populateTableProperties
           stores DERIVED_COLUMNS and
           DERIVED_COLUMN_EXPRESSION_SPEC]
    end

    A --> B --> C --> D
    D --> E --> F
    F --> G
    G --> H
    H --> I
    I --> J

File-Level Changes

Change	Details	Files
Add SQL and AST support for GENERATED/derived columns and propagate expressions into column metadata.	Extend ColumnDefinition to carry an optional DerivedColumnDefinition mutually exclusive with DEFAULT expressions and expose it to visitors/formatter. Extend SqlBase.g4, AstBuilder, and SqlFormatter to parse and render GENERATED ... AS expressions and to enforce validation rules around default vs generated columns. Update CreateTableTask to capture derived column definitions and store the expression string into ColumnMetadata properties under a dedicated key.	`presto-parser/src/main/antlr4/com/facebook/presto/sql/parser/SqlBase.g4` `presto-parser/src/main/java/com/facebook/presto/sql/tree/ColumnDefinition.java` `presto-parser/src/main/java/com/facebook/presto/sql/tree/DerivedColumnDefinition.java` `presto-parser/src/main/java/com/facebook/presto/sql/parser/AstBuilder.java` `presto-parser/src/main/java/com/facebook/presto/sql/SqlFormatter.java` `presto-main-base/src/main/java/com/facebook/presto/execution/CreateTableTask.java` `presto-spi/src/main/java/com/facebook/presto/spi/ColumnMetadata.java`
Introduce Iceberg table/column properties and config wiring for derived columns, including JSON spec storage and enabling flags.	Add table properties for derived-columns and derived column expression spec, with JSON codec mapping to DerivedColumnExpressionSpecList and accessors for both property maps and Iceberg table properties. Wire IcebergConfig and IcebergSessionProperties with an iceberg.derived_columns.enable config and session property toggle plus corresponding tests. Populate derived column properties on createTable (from ColumnMetadata) and expose them via createMetadataProperties when reading existing tables.	`presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergTableProperties.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergConfig.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergSessionProperties.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergAbstractMetadata.java` `presto-iceberg/src/test/java/com/facebook/presto/iceberg/TestIcebergConfig.java`
Add data model and JSON serialization classes for derived column expression specifications.	Introduce DerivedColumnExpressionSpec, DerivedColumnExpressionSpecList, DerivedColumnType, DerivedColumnRef, and DerivedColumnArgumentSpec to describe derived column expressions and their arguments. Register a JsonCodec in IcebergUtil for serializing/deserializing DerivedColumnExpressionSpecList to/from table properties. Define enums and helper structures to categorize derived column types and references for future matching/extension.	`presto-iceberg/src/main/java/com/facebook/presto/iceberg/derivedColumn/DerivedColumnExpressionSpec.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/derivedColumn/DerivedColumnExpressionSpecList.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/derivedColumn/DerivedColumnType.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/derivedColumn/DerivedColumnRef.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/derivedColumn/DerivedColumnArgumentSpec.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java`
Implement an Iceberg connector plan optimizer that rewrites filter predicates to reuse precomputed derived columns based on expression matching.	Add AstExpressionToRowExpression to convert parsed SQL expressions to RowExpression using StandardFunctionResolution and TypeManager. Implement EquivalentRowExpression, RowExpressionComparator, RewriteCommonSubExpression, and RewrittenExpressionMetadata to detect and replace equivalent row expressions with variable references to derived columns. Introduce FilterPredicateCSERewriter and IcebergDerivedColumnRewriter as a ConnectorPlanOptimizer that, when enabled, inspects Filter+TableScan, builds expression→derived-column mappings from table properties, rewrites predicates, and extends requested columns/layout accordingly. Wire the optimizer into IcebergPlanOptimizerProvider and register SqlParser in IcebergCommonModule to support parsing stored expressions.	`presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/IcebergPlanOptimizerProvider.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/derivedColumns/IcebergDerivedColumnRewriter.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/derivedColumns/FilterPredicateCSERewriter.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/derivedColumns/AstExpressionToRowExpression.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/derivedColumns/EquivalentRowExpression.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/derivedColumns/RowExpressionComparator.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/derivedColumns/RewriteCommonSubExpression.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/derivedColumns/RewrittenExpressionMetadata.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergCommonModule.java`
Extend function handle and signature infrastructure to support expression equivalence via canonicalization.	Add getReturnType and canonicalize defaults to FunctionHandle, and implement them in BuiltInFunctionHandle so that type-parameterized functions compare equal when base types match. Add SignatureBuilder.from(Signature) helper to simplify rebuilding signatures with modified argument/return types. Use canonicalization and base-type comparison inside EquivalentRowExpression and related comparators to treat semantically identical expressions as equal.	`presto-spi/src/main/java/com/facebook/presto/spi/function/FunctionHandle.java` `presto-main-base/src/main/java/com/facebook/presto/metadata/BuiltInFunctionHandle.java` `presto-main-base/src/main/java/com/facebook/presto/metadata/SignatureBuilder.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/optimizer/derivedColumns/EquivalentRowExpression.java`
Add tests and minor wiring/formatting changes to support the new feature.	Add TestIcebergDerivedColumnOptimizer exercising predicate rewrite behavior with various derived column specs and ensure predicate formatting matches expectations. Add TestEquivalentRowExpressions (currently mostly harness) and adjust TestIcebergConfig for the new config option. Update IcebergNativeMetadata/HiveMetadata to pass derived column spec into table property population and simplify TableMetadataParser usage. Update pom to add presto-parser dependency to presto-iceberg and skip maven-dependency-plugin in that module, relax ReservedIdentifiers validation, and perform minor formatting/indentation cleanup in existing methods.	`presto-iceberg/src/test/java/com/facebook/presto/iceberg/TestIcebergDerivedColumnOptimizer.java` `presto-iceberg/src/test/java/com/facebook/presto/iceberg/optimizer/derivedColumns/TestEquivalentRowExpressions.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergNativeMetadata.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergHiveMetadata.java` `presto-iceberg/pom.xml` `presto-parser/src/main/java/com/facebook/presto/sql/ReservedIdentifiers.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/HiveTableOperations.java` `presto-main-base/src/main/java/com/facebook/presto/sql/rewrite/ShowQueriesRewrite.java` `presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

aaneja · 2026-05-20T06:41:44Z

+
+import java.util.Comparator;
+
+public class RowExpressionComparator


Can you try using this implementation -

presto/presto-main-base/src/main/java/com/facebook/presto/sql/planner/optimizations/ExpressionEquivalence.java

Lines 101 to 107 in 47a7080

public boolean areExpressionsEquivalent(RowExpression leftExpression, RowExpression rightExpression)

{

RowExpression canonicalizedLeft = leftExpression.accept(canonicalizationVisitor, null);

RowExpression canonicalizedRight = rightExpression.accept(canonicalizationVisitor, null);

return canonicalizedLeft.equals(canonicalizedRight);

}

We can move it presto-common if it suffices

prestodb-ci added the from:IBM PR from IBM label May 18, 2026

ScrapCodes changed the title ~~[POC][RFC-0025]Derived column support expression~~ [POC][RFC-0025] Derived column supports expression May 18, 2026

ScrapCodes force-pushed the derived-column-support-expression branch from f2ef2f1 to 9aff20b Compare May 18, 2026 17:46

aaneja reviewed May 20, 2026

View reviewed changes

ScrapCodes force-pushed the derived-column-support-expression branch 4 times, most recently from 4d3344a to 3e04e2b Compare May 27, 2026 09:59

feat(plugin-iceberg): Add support for Derived columns RFC-0025.

0a7f4a8

ScrapCodes force-pushed the derived-column-support-expression branch from 3e04e2b to 0a7f4a8 Compare May 27, 2026 12:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[POC][RFC-0025] Derived column supports expression#27832

[POC][RFC-0025] Derived column supports expression#27832
ScrapCodes wants to merge 1 commit into
prestodb:masterfrom
ScrapCodes:derived-column-support-expression

ScrapCodes commented May 18, 2026 •

edited

Loading

Uh oh!

sourcery-ai Bot commented May 18, 2026

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

aaneja May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		import java.util.Comparator;

		public class RowExpressionComparator

	public boolean areExpressionsEquivalent(RowExpression leftExpression, RowExpression rightExpression)
	{
	RowExpression canonicalizedLeft = leftExpression.accept(canonicalizationVisitor, null);
	RowExpression canonicalizedRight = rightExpression.accept(canonicalizationVisitor, null);

	return canonicalizedLeft.equals(canonicalizedRight);
	}

Conversation

ScrapCodes commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Summary by Sourcery

Uh oh!

sourcery-ai Bot commented May 18, 2026

Reviewer's Guide

Flow diagram for CREATE TABLE with derived/GENERATED columns

File-Level Changes

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

aaneja May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ScrapCodes commented May 18, 2026 •

edited

Loading