Skip to content

Commit b7cc66f

Browse files
andygroveandygrove
committed
test: regenerate TPCDS plan stability golden files
The CometHashAggregateExec and CometBroadcastExchangeExec canonicalization fixes change how the optimizer reuses subqueries and exchanges, altering the physical plans for several TPCDS queries. Regenerate the approved golden plans so the plan stability suites pass.
1 parent 06f8b59 commit b7cc66f

18 files changed

Lines changed: 796 additions & 172 deletions

File tree

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
CometNativeColumnarToRow
2+
+- CometTakeOrderedAndProject
3+
+- CometProject
4+
+- CometBroadcastHashJoin
5+
:- CometProject
6+
: +- CometBroadcastHashJoin
7+
: :- CometHashAggregate
8+
: : +- CometExchange
9+
: : +- CometHashAggregate
10+
: : +- CometProject
11+
: : +- CometBroadcastHashJoin
12+
: : :- CometProject
13+
: : : +- CometBroadcastHashJoin
14+
: : : :- CometFilter
15+
: : : : +- CometNativeScan parquet spark_catalog.default.store_returns
16+
: : : : +- CometSubqueryBroadcast
17+
: : : : +- CometBroadcastExchange
18+
: : : : +- CometProject
19+
: : : : +- CometBroadcastHashJoin
20+
: : : : :- CometFilter
21+
: : : : : +- CometNativeScan parquet spark_catalog.default.date_dim
22+
: : : : +- CometBroadcastExchange
23+
: : : : +- CometProject
24+
: : : : +- CometBroadcastHashJoin
25+
: : : : :- CometNativeScan parquet spark_catalog.default.date_dim
26+
: : : : +- CometBroadcastExchange
27+
: : : : +- CometProject
28+
: : : : +- CometFilter
29+
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
30+
: : : +- CometBroadcastExchange
31+
: : : +- CometProject
32+
: : : +- CometFilter
33+
: : : +- CometNativeScan parquet spark_catalog.default.item
34+
: : +- CometBroadcastExchange
35+
: : +- CometProject
36+
: : +- CometBroadcastHashJoin
37+
: : :- CometFilter
38+
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
39+
: : +- CometBroadcastExchange
40+
: : +- CometProject
41+
: : +- CometBroadcastHashJoin
42+
: : :- CometNativeScan parquet spark_catalog.default.date_dim
43+
: : +- CometBroadcastExchange
44+
: : +- CometProject
45+
: : +- CometFilter
46+
: : +- CometNativeScan parquet spark_catalog.default.date_dim
47+
: +- CometBroadcastExchange
48+
: +- CometHashAggregate
49+
: +- CometExchange
50+
: +- CometHashAggregate
51+
: +- CometProject
52+
: +- CometBroadcastHashJoin
53+
: :- CometProject
54+
: : +- CometBroadcastHashJoin
55+
: : :- CometFilter
56+
: : : +- CometNativeScan parquet spark_catalog.default.catalog_returns
57+
: : : +- ReusedSubquery
58+
: : +- CometBroadcastExchange
59+
: : +- CometProject
60+
: : +- CometFilter
61+
: : +- CometNativeScan parquet spark_catalog.default.item
62+
: +- CometBroadcastExchange
63+
: +- CometProject
64+
: +- CometBroadcastHashJoin
65+
: :- CometFilter
66+
: : +- CometNativeScan parquet spark_catalog.default.date_dim
67+
: +- CometBroadcastExchange
68+
: +- CometProject
69+
: +- CometBroadcastHashJoin
70+
: :- CometNativeScan parquet spark_catalog.default.date_dim
71+
: +- CometBroadcastExchange
72+
: +- CometProject
73+
: +- CometFilter
74+
: +- CometNativeScan parquet spark_catalog.default.date_dim
75+
+- CometBroadcastExchange
76+
+- CometHashAggregate
77+
+- CometExchange
78+
+- CometHashAggregate
79+
+- CometProject
80+
+- CometBroadcastHashJoin
81+
:- CometProject
82+
: +- CometBroadcastHashJoin
83+
: :- CometFilter
84+
: : +- CometNativeScan parquet spark_catalog.default.web_returns
85+
: : +- ReusedSubquery
86+
: +- CometBroadcastExchange
87+
: +- CometProject
88+
: +- CometFilter
89+
: +- CometNativeScan parquet spark_catalog.default.item
90+
+- CometBroadcastExchange
91+
+- CometProject
92+
+- CometBroadcastHashJoin
93+
:- CometFilter
94+
: +- CometNativeScan parquet spark_catalog.default.date_dim
95+
+- CometBroadcastExchange
96+
+- CometProject
97+
+- CometBroadcastHashJoin
98+
:- CometNativeScan parquet spark_catalog.default.date_dim
99+
+- CometBroadcastExchange
100+
+- CometProject
101+
+- CometFilter
102+
+- CometNativeScan parquet spark_catalog.default.date_dim
103+
104+
Comet accelerated 98 out of 101 eligible operators (97%). Final plan contains 1 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q14b/extended.txt‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,11 @@ CometNativeColumnarToRow
132132
: : : : +- CometBroadcastHashJoin
133133
: : : : :- CometFilter
134134
: : : : : +- CometNativeScan parquet spark_catalog.default.store_sales
135-
: : : : : +- ReusedSubquery
135+
: : : : : +- CometSubqueryBroadcast
136+
: : : : : +- CometBroadcastExchange
137+
: : : : : +- CometProject
138+
: : : : : +- CometFilter
139+
: : : : : +- CometNativeScan parquet spark_catalog.default.date_dim
136140
: : : : +- CometBroadcastExchange
137141
: : : : +- CometBroadcastHashJoin
138142
: : : : :- CometFilter
@@ -176,7 +180,11 @@ CometNativeColumnarToRow
176180
: +- CometFilter
177181
: : +- ReusedSubquery
178182
: +- CometNativeScan parquet spark_catalog.default.date_dim
179-
: +- ReusedSubquery
183+
: +- Subquery
184+
: +- CometNativeColumnarToRow
185+
: +- CometProject
186+
: +- CometFilter
187+
: +- CometNativeScan parquet spark_catalog.default.date_dim
180188
+- CometBroadcastExchange
181189
+- CometFilter
182190
: +- ReusedSubquery
@@ -217,7 +225,11 @@ CometNativeColumnarToRow
217225
: : : : +- CometBroadcastHashJoin
218226
: : : : :- CometFilter
219227
: : : : : +- CometNativeScan parquet spark_catalog.default.store_sales
220-
: : : : : +- ReusedSubquery
228+
: : : : : +- CometSubqueryBroadcast
229+
: : : : : +- CometBroadcastExchange
230+
: : : : : +- CometProject
231+
: : : : : +- CometFilter
232+
: : : : : +- CometNativeScan parquet spark_catalog.default.date_dim
221233
: : : : +- CometBroadcastExchange
222234
: : : : +- CometBroadcastHashJoin
223235
: : : : :- CometFilter
@@ -276,7 +288,11 @@ CometNativeColumnarToRow
276288
: : : +- CometBroadcastHashJoin
277289
: : : :- CometFilter
278290
: : : : +- CometNativeScan parquet spark_catalog.default.store_sales
279-
: : : : +- ReusedSubquery
291+
: : : : +- CometSubqueryBroadcast
292+
: : : : +- CometBroadcastExchange
293+
: : : : +- CometProject
294+
: : : : +- CometFilter
295+
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
280296
: : : +- CometBroadcastExchange
281297
: : : +- CometBroadcastHashJoin
282298
: : : :- CometFilter
@@ -320,6 +336,10 @@ CometNativeColumnarToRow
320336
+- CometFilter
321337
: +- ReusedSubquery
322338
+- CometNativeScan parquet spark_catalog.default.date_dim
323-
+- ReusedSubquery
339+
+- Subquery
340+
+- CometNativeColumnarToRow
341+
+- CometProject
342+
+- CometFilter
343+
+- CometNativeScan parquet spark_catalog.default.date_dim
324344

325-
Comet accelerated 292 out of 319 eligible operators (91%). Final plan contains 4 transitions between Spark and Comet.
345+
Comet accelerated 310 out of 337 eligible operators (91%). Final plan contains 6 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q54/extended.txt‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,21 @@ CometNativeColumnarToRow
8787
: :- ReusedSubquery
8888
: +- ReusedSubquery
8989
+- CometNativeScan parquet spark_catalog.default.date_dim
90-
:- ReusedSubquery
91-
+- ReusedSubquery
90+
:- Subquery
91+
: +- CometNativeColumnarToRow
92+
: +- CometHashAggregate
93+
: +- CometExchange
94+
: +- CometHashAggregate
95+
: +- CometProject
96+
: +- CometFilter
97+
: +- CometNativeScan parquet spark_catalog.default.date_dim
98+
+- Subquery
99+
+- CometNativeColumnarToRow
100+
+- CometHashAggregate
101+
+- CometExchange
102+
+- CometHashAggregate
103+
+- CometProject
104+
+- CometFilter
105+
+- CometNativeScan parquet spark_catalog.default.date_dim
92106

93-
Comet accelerated 77 out of 88 eligible operators (87%). Final plan contains 3 transitions between Spark and Comet.
107+
Comet accelerated 89 out of 100 eligible operators (89%). Final plan contains 5 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q6/extended.txt‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,14 @@ CometNativeColumnarToRow
4040
: +- CometFilter
4141
: : +- ReusedSubquery
4242
: +- CometNativeScan parquet spark_catalog.default.date_dim
43-
: +- ReusedSubquery
43+
: +- Subquery
44+
: +- CometNativeColumnarToRow
45+
: +- CometHashAggregate
46+
: +- CometExchange
47+
: +- CometHashAggregate
48+
: +- CometProject
49+
: +- CometFilter
50+
: +- CometNativeScan parquet spark_catalog.default.date_dim
4451
+- CometBroadcastExchange
4552
+- CometProject
4653
+- CometBroadcastHashJoin
@@ -55,4 +62,4 @@ CometNativeColumnarToRow
5562
+- CometFilter
5663
+- CometNativeScan parquet spark_catalog.default.item
5764

58-
Comet accelerated 49 out of 54 eligible operators (90%). Final plan contains 2 transitions between Spark and Comet.
65+
Comet accelerated 55 out of 60 eligible operators (91%). Final plan contains 3 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q83.ansi/extended.txt‎

Lines changed: 3 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -54,20 +54,7 @@ CometNativeColumnarToRow
5454
: : +- CometBroadcastHashJoin
5555
: : :- CometFilter
5656
: : : +- CometNativeScan parquet spark_catalog.default.catalog_returns
57-
: : : +- CometSubqueryBroadcast
58-
: : : +- CometBroadcastExchange
59-
: : : +- CometProject
60-
: : : +- CometBroadcastHashJoin
61-
: : : :- CometFilter
62-
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
63-
: : : +- CometBroadcastExchange
64-
: : : +- CometProject
65-
: : : +- CometBroadcastHashJoin
66-
: : : :- CometNativeScan parquet spark_catalog.default.date_dim
67-
: : : +- CometBroadcastExchange
68-
: : : +- CometProject
69-
: : : +- CometFilter
70-
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
57+
: : : +- ReusedSubquery
7158
: : +- CometBroadcastExchange
7259
: : +- CometProject
7360
: : +- CometFilter
@@ -95,20 +82,7 @@ CometNativeColumnarToRow
9582
: +- CometBroadcastHashJoin
9683
: :- CometFilter
9784
: : +- CometNativeScan parquet spark_catalog.default.web_returns
98-
: : +- CometSubqueryBroadcast
99-
: : +- CometBroadcastExchange
100-
: : +- CometProject
101-
: : +- CometBroadcastHashJoin
102-
: : :- CometFilter
103-
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
104-
: : +- CometBroadcastExchange
105-
: : +- CometProject
106-
: : +- CometBroadcastHashJoin
107-
: : :- CometNativeScan parquet spark_catalog.default.date_dim
108-
: : +- CometBroadcastExchange
109-
: : +- CometProject
110-
: : +- CometFilter
111-
: : +- CometNativeScan parquet spark_catalog.default.date_dim
85+
: : +- ReusedSubquery
11286
: +- CometBroadcastExchange
11387
: +- CometProject
11488
: +- CometFilter
@@ -127,4 +101,4 @@ CometNativeColumnarToRow
127101
+- CometFilter
128102
+- CometNativeScan parquet spark_catalog.default.date_dim
129103

130-
Comet accelerated 124 out of 127 eligible operators (97%). Final plan contains 1 transitions between Spark and Comet.
104+
Comet accelerated 98 out of 101 eligible operators (97%). Final plan contains 1 transitions between Spark and Comet.

0 commit comments

Comments
 (0)