@@ -119,41 +119,63 @@ case class CometExecRule(session: SparkSession) extends Rule[SparkPlan] {
119119 /**
120120 * Tries to transform a Spark physical plan into a Comet plan.
121121 *
122- * This rule traverses bottom-up from the original Spark plan and for each plan node, there are
123- * a few cases to consider:
122+ * This rule traverses bottom-up from the original Spark plan and for each plan node, there
123+ * are a few cases to consider:
124124 *
125- * 1. The child(ren) of the current node `p` cannot be converted to native In this case, we'll
126- * simply return the original Spark plan, since Comet native execution cannot start from an
127- * arbitrary Spark operator (unless it is special node such as scan or sink such as shuffle
128- * exchange, union etc., which are wrapped by `CometScanWrapper` and `CometSinkPlaceHolder`
129- * respectively).
125+ * 1. The child(ren) of the current node `p` cannot be converted to native
126+ * In this case, we'll simply return the original Spark plan, since Comet native
127+ * execution cannot start from an arbitrary Spark operator (unless it is special node
128+ * such as scan or sink such as shuffle exchange, union etc., which are wrapped by
129+ * `CometScanWrapper` and `CometSinkPlaceHolder` respectively).
130130 *
131- * 2. The child(ren) of the current node `p` can be converted to native There are two sub-cases
132- * for this scenario: 1) This node `p` can also be converted to native. In this case, we'll
133- * create a new native Comet operator for `p` and connect it with its previously converted
134- * child(ren); 2) This node `p` cannot be converted to native. In this case, similar to 1)
135- * above, we simply return `p` as it is. Its child(ren) would still be native Comet operators.
131+ * 2. The child(ren) of the current node `p` can be converted to native
132+ * There are two sub-cases for this scenario: 1) This node `p` can also be converted to
133+ * native. In this case, we'll create a new native Comet operator for `p` and connect it with
134+ * its previously converted child(ren); 2) This node `p` cannot be converted to native. In
135+ * this case, similar to 1) above, we simply return `p` as it is. Its child(ren) would still
136+ * be native Comet operators.
136137 *
137138 * After this rule finishes, we'll do another pass on the final plan to convert all adjacent
138- * Comet native operators into a single native execution block. Please see where `convertBlock`
139- * is called below.
139+ * Comet native operators into a single native execution block. Please see where
140+ * `convertBlock` is called below.
140141 *
141142 * Here are a few examples:
142143 *
143- * Scan ======> CometScan \| | Filter CometFilter \| | HashAggregate CometHashAggregate \| |
144- * Exchange CometExchange \| | HashAggregate CometHashAggregate \| | UnsupportedOperator
145- * UnsupportedOperator
144+ * Scan ======> CometScan
145+ * | |
146+ * Filter CometFilter
147+ * | |
148+ * HashAggregate CometHashAggregate
149+ * | |
150+ * Exchange CometExchange
151+ * | |
152+ * HashAggregate CometHashAggregate
153+ * | |
154+ * UnsupportedOperator UnsupportedOperator
146155 *
147156 * Native execution doesn't necessarily have to start from `CometScan`:
148157 *
149- * Scan =======> CometScan \| | UnsupportedOperator UnsupportedOperator \| | HashAggregate
150- * HashAggregate \| | Exchange CometExchange \| | HashAggregate CometHashAggregate \| |
151- * UnsupportedOperator UnsupportedOperator
158+ * Scan =======> CometScan
159+ * | |
160+ * UnsupportedOperator UnsupportedOperator
161+ * | |
162+ * HashAggregate HashAggregate
163+ * | |
164+ * Exchange CometExchange
165+ * | |
166+ * HashAggregate CometHashAggregate
167+ * | |
168+ * UnsupportedOperator UnsupportedOperator
152169 *
153170 * A sink can also be Comet operators other than `CometExchange`, for instance `CometUnion`:
154171 *
155- * Scan Scan =======> CometScan CometScan \| | | | Filter Filter CometFilter CometFilter \| | |
156- * \| Union CometUnion \| | Project CometProject
172+ * Scan Scan =======> CometScan CometScan
173+ * | | | |
174+ * Filter Filter CometFilter CometFilter
175+ * | | | |
176+ * Union CometUnion
177+ * | |
178+ * Project CometProject
157179 */
158180 // spotless:on
159181 private def transform (plan : SparkPlan ): SparkPlan = {
0 commit comments