Skip to content

Commit 136ec45

Browse files
committed
Revert "[spark] Move row-level write rules into Resolution batch for Spark 4.1"
This reverts commit cc1d93c.
1 parent c2ac1bb commit 136ec45

1 file changed

Lines changed: 3 additions & 12 deletions

File tree

paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/extensions/PaimonSparkSessionExtensions.scala

Lines changed: 3 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -47,18 +47,9 @@ class PaimonSparkSessionExtensions extends (SparkSessionExtensions => Unit) {
4747
extensions.injectPostHocResolutionRule(spark => ReplacePaimonFunctions(spark))
4848
extensions.injectPostHocResolutionRule(spark => PaimonPostHocResolutionRules(spark))
4949

50-
// Row-level write rewrites must fire during the Resolution batch, not as postHoc rules. Spark
51-
// 4.1 moved `RewriteUpdateTable` / `RewriteDeleteFromTable` / `RewriteMergeIntoTable` into the
52-
// Resolution batch and marks the plan as `analyzed=true` by the time postHoc rules run;
53-
// `AnalysisHelper.resolveOperators*` then short-circuits and Spark 4.1 treats the postHoc
54-
// rule's return value as a no-op, leaving the original UpdateTable node for the physical
55-
// planner to reject with `UNSUPPORTED_FEATURE.TABLE_OPERATION`. Registering as resolution
56-
// rules lets them transform before the analyzed flag is set. See
57-
// AnalysisHelper.resolveOperatorsUpWithPruning: it returns `self` unchanged when
58-
// `analyzed=true`, which is the root cause of the Spark 4.1 append-only UPDATE/DELETE failures.
59-
extensions.injectResolutionRule(_ => PaimonUpdateTable)
60-
extensions.injectResolutionRule(_ => PaimonDeleteTable)
61-
extensions.injectResolutionRule(spark => PaimonMergeInto(spark))
50+
extensions.injectPostHocResolutionRule(_ => PaimonUpdateTable)
51+
extensions.injectPostHocResolutionRule(_ => PaimonDeleteTable)
52+
extensions.injectPostHocResolutionRule(spark => PaimonMergeInto(spark))
6253

6354
// table function extensions
6455
PaimonTableValuedFunctions.supportedFnNames.foreach {

0 commit comments

Comments
 (0)