Skip to content

fix: Fix regression with plan stability tests in CI #2492

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
andygrove wants to merge 30 commits into from
Closed

fix: Fix regression with plan stability tests in CI #2492

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
e1096b2
revert
andygrove Sep 29, 2025
e72c4d7
Revert
andygrove Sep 29, 2025
e62ba22
add native_comet plans
andygrove Sep 29, 2025
fcbeee3
update native_datafusion plans
andygrove Sep 29, 2025
6dd7c6c
update native_iceberg_compat plans
andygrove Sep 29, 2025
a1a748c
Spark 3.4 plans
andygrove Sep 29, 2025
cef89aa
spark 4
andygrove Sep 29, 2025
588b576
test all the scan impls
andygrove Sep 29, 2025
20ef9f3
revert
andygrove Sep 29, 2025
5456897
just test auto and native_iceberg_compat
andygrove Sep 29, 2025
d26c8c1
format
andygrove Sep 29, 2025
096f2b4
Merge remote-tracking branch 'apache/main' into plan-stability-regres…
andygrove Sep 30, 2025
34d49bd
move tpc tests to separate suite
andygrove Sep 30, 2025
1ed8f28
update 3.5 plans
andygrove Sep 30, 2025
227ddae
update 3.4 golden files
andygrove Sep 30, 2025
10af6d7
fix verbose logging
andygrove Sep 30, 2025
6166b4e
4.0 golden files
andygrove Sep 30, 2025
e768142
format
andygrove Sep 30, 2025
f5f73bb
improve test
andygrove Sep 30, 2025
41970a6
debug
andygrove Sep 30, 2025
e611b20
simplify code
andygrove Sep 30, 2025
31e1a71
save
andygrove Sep 30, 2025
94838a3
save
andygrove Sep 30, 2025
378b5ad
fix explain plan for q9
andygrove Oct 1, 2025
867696c
revert
andygrove Oct 1, 2025
5d23c41
address feedback
andygrove Oct 1, 2025
d458cab
Merge remote-tracking branch 'apache/main' into plan-stability-regres…
andygrove Oct 1, 2025
1c8fe04
prep for review
andygrove Oct 1, 2025
d2580d0
skip q9
andygrove Oct 1, 2025
ee7102e
Merge remote-tracking branch 'apache/main' into plan-stability-regres…
andygrove Oct 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
212 changes: 0 additions & 212 deletions ...ources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1.native_datafusion/explain.txt
View file
Open in desktop

This file was deleted.

42 changes: 0 additions & 42 deletions ...ces/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1.native_datafusion/simplified.txt
View file
Open in desktop

This file was deleted.

65 changes: 35 additions & 30 deletions ...es/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1.native_iceberg_compat/explain.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
== Physical Plan ==
* ColumnarToRow (40)
+- CometTakeOrderedAndProject (39)
+- CometProject (38)
+- CometBroadcastHashJoin (37)
* CometColumnarToRow (41)
+- CometTakeOrderedAndProject (40)
+- CometProject (39)
+- CometBroadcastHashJoin (38)
:- CometProject (33)
: +- CometBroadcastHashJoin (32)
: :- CometProject (27)
Expand All @@ -14,11 +14,11 @@
: : : +- CometProject (8)
: : : +- CometBroadcastHashJoin (7)
: : : :- CometFilter (2)
: : : : +- CometScan parquet spark_catalog.default.store_returns (1)
: : : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns (1)
: : : +- CometBroadcastExchange (6)
: : : +- CometProject (5)
: : : +- CometFilter (4)
: : : +- CometScan parquet spark_catalog.default.date_dim (3)
: : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim (3)
: : +- CometBroadcastExchange (25)
: : +- CometFilter (24)
: : +- CometHashAggregate (23)
Expand All @@ -30,18 +30,19 @@
: : +- CometProject (17)
: : +- CometBroadcastHashJoin (16)
: : :- CometFilter (14)
: : : +- CometScan parquet spark_catalog.default.store_returns (13)
: : : +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns (13)
: : +- ReusedExchange (15)
: +- CometBroadcastExchange (31)
: +- CometProject (30)
: +- CometFilter (29)
: +- CometScan parquet spark_catalog.default.store (28)
+- CometBroadcastExchange (36)
+- CometFilter (35)
+- CometScan parquet spark_catalog.default.customer (34)
: +- CometScan [native_iceberg_compat] parquet spark_catalog.default.store (28)
+- CometBroadcastExchange (37)
+- CometProject (36)
+- CometFilter (35)
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.customer (34)


(1) CometScan parquet spark_catalog.default.store_returns
(1) CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns
Output [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]
Batched: true
Location: InMemoryFileIndex []
Expand All @@ -53,7 +54,7 @@ ReadSchema: struct<sr_customer_sk:int,sr_store_sk:int,sr_return_amt:decimal(7,2)
Input [4]: [sr_customer_sk#1, sr_store_sk#2, sr_return_amt#3, sr_returned_date_sk#4]
Condition : (isnotnull(sr_store_sk#2) AND isnotnull(sr_customer_sk#1))

(3) CometScan parquet spark_catalog.default.date_dim
(3) CometScan [native_iceberg_compat] parquet spark_catalog.default.date_dim
Output [2]: [d_date_sk#5, d_year#6]
Batched: true
Location [not included in comparison]/{warehouse_dir}/date_dim]
Expand Down Expand Up @@ -99,7 +100,7 @@ Functions [1]: [sum(UnscaledValue(sr_return_amt#3))]
Input [3]: [ctr_customer_sk#8, ctr_store_sk#9, ctr_total_return#10]
Condition : isnotnull(ctr_total_return#10)

(13) CometScan parquet spark_catalog.default.store_returns
(13) CometScan [native_iceberg_compat] parquet spark_catalog.default.store_returns
Output [4]: [sr_customer_sk#11, sr_store_sk#12, sr_return_amt#13, sr_returned_date_sk#14]
Batched: true
Location: InMemoryFileIndex []
Expand Down Expand Up @@ -168,16 +169,16 @@ Arguments: [ctr_store_sk#9], [ctr_store_sk#17], Inner, (cast(ctr_total_return#10
Input [5]: [ctr_customer_sk#8, ctr_store_sk#9, ctr_total_return#10, (avg(ctr_total_return) * 1.2)#21, ctr_store_sk#17]
Arguments: [ctr_customer_sk#8, ctr_store_sk#9], [ctr_customer_sk#8, ctr_store_sk#9]

(28) CometScan parquet spark_catalog.default.store
(28) CometScan [native_iceberg_compat] parquet spark_catalog.default.store
Output [2]: [s_store_sk#22, s_state#23]
Batched: true
Location [not included in comparison]/{warehouse_dir}/store]
PushedFilters: [IsNotNull(s_state), EqualTo(s_state,TN), IsNotNull(s_store_sk)]
PushedFilters: [IsNotNull(s_store_sk)]
ReadSchema: struct<s_store_sk:int,s_state:string>

(29) CometFilter
Input [2]: [s_store_sk#22, s_state#23]
Condition : ((isnotnull(s_state#23) AND (s_state#23 = TN)) AND isnotnull(s_store_sk#22))
Condition : ((staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, s_state#23, 2, true, false, true) = TN) AND isnotnull(s_store_sk#22))

(30) CometProject
Input [2]: [s_store_sk#22, s_state#23]
Expand All @@ -196,7 +197,7 @@ Arguments: [ctr_store_sk#9], [s_store_sk#22], Inner, BuildRight
Input [3]: [ctr_customer_sk#8, ctr_store_sk#9, s_store_sk#22]
Arguments: [ctr_customer_sk#8], [ctr_customer_sk#8]

(34) CometScan parquet spark_catalog.default.customer
(34) CometScan [native_iceberg_compat] parquet spark_catalog.default.customer
Output [2]: [c_customer_sk#24, c_customer_id#25]
Batched: true
Location [not included in comparison]/{warehouse_dir}/customer]
Expand All @@ -207,23 +208,27 @@ ReadSchema: struct<c_customer_sk:int,c_customer_id:string>
Input [2]: [c_customer_sk#24, c_customer_id#25]
Condition : isnotnull(c_customer_sk#24)

(36) CometBroadcastExchange
(36) CometProject
Input [2]: [c_customer_sk#24, c_customer_id#25]
Arguments: [c_customer_sk#24, c_customer_id#25]
Arguments: [c_customer_sk#24, c_customer_id#26], [c_customer_sk#24, staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, c_customer_id#25, 16, true, false, true) AS c_customer_id#26]

(37) CometBroadcastHashJoin
(37) CometBroadcastExchange
Input [2]: [c_customer_sk#24, c_customer_id#26]
Arguments: [c_customer_sk#24, c_customer_id#26]

(38) CometBroadcastHashJoin
Left output [1]: [ctr_customer_sk#8]
Right output [2]: [c_customer_sk#24, c_customer_id#25]
Right output [2]: [c_customer_sk#24, c_customer_id#26]
Arguments: [ctr_customer_sk#8], [c_customer_sk#24], Inner, BuildRight

(38) CometProject
Input [3]: [ctr_customer_sk#8, c_customer_sk#24, c_customer_id#25]
Arguments: [c_customer_id#25], [c_customer_id#25]
(39) CometProject
Input [3]: [ctr_customer_sk#8, c_customer_sk#24, c_customer_id#26]
Arguments: [c_customer_id#26], [c_customer_id#26]

(39) CometTakeOrderedAndProject
Input [1]: [c_customer_id#25]
Arguments: TakeOrderedAndProject(limit=100, orderBy=[c_customer_id#25 ASC NULLS FIRST], output=[c_customer_id#25]), [c_customer_id#25], 100, [c_customer_id#25 ASC NULLS FIRST], [c_customer_id#25]
(40) CometTakeOrderedAndProject
Input [1]: [c_customer_id#26]
Arguments: TakeOrderedAndProject(limit=100, orderBy=[c_customer_id#26 ASC NULLS FIRST], output=[c_customer_id#26]), [c_customer_id#26], 100, 0, [c_customer_id#26 ASC NULLS FIRST], [c_customer_id#26]

(40) ColumnarToRow [codegen id : 1]
Input [1]: [c_customer_id#25]
(41) CometColumnarToRow [codegen id : 1]
Input [1]: [c_customer_id#26]

Loading
Loading