Skip to content

Commit aca2904

Browse files
brijrajkclaude
andcommitted
[GLUTEN-12013][VL] Guard optimizer rule against DPP-injected bloom filters
In Spark 4.1, the optimizer processes BloomFilterMightContainJointRewriteRule after InjectRuntimeFilter, so the rule sees DPP-injected bloom filters (which use xxhash64(col, seed) as their value expression). Rewriting those to VeloxBloomFilterMightContain caused FilterExecTransformer to fail Velox validation and fall back to JVM, changing the physical plan structure and breaking 19 TPC-DS plan-stability golden files across v1.4, v2.7, and modified suites (q2, q10, q16, q24a, q24b, q32, q37, q40, q59, q69, q80, q82, q85, q92, q94, q95, q10a, q64, q80a). Fix: guard the first arm of transformAllExpressions with `v: Attribute`. DPP bloom filters always hash the join key — xxhash64(col, seed) — so their value is never a bare Attribute and the guard skips them cleanly. User-facing bloom filters (might_contain(subquery, col)) pass a plain column reference and continue to be rewritten correctly. Also guard the catch-all arm against ScalarSubquery to be consistent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 2ce021e commit aca2904

1 file changed

Lines changed: 19 additions & 6 deletions

File tree

backends-velox/src/main/scala/org/apache/gluten/extension/BloomFilterMightContainJointRewriteRule.scala

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ import org.apache.gluten.expression.VeloxBloomFilterMightContain
2121
import org.apache.gluten.expression.aggregate.VeloxBloomFilterAggregate
2222

2323
import org.apache.spark.sql.SparkSession
24-
import org.apache.spark.sql.catalyst.expressions.{BloomFilterMightContain, ScalarSubquery}
24+
import org.apache.spark.sql.catalyst.expressions.{Attribute, BloomFilterMightContain, ScalarSubquery}
2525
import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, BloomFilterAggregate}
2626
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
2727
import org.apache.spark.sql.catalyst.rules.Rule
@@ -37,9 +37,17 @@ import org.apache.spark.sql.catalyst.rules.Rule
3737
* stages fall back to JVM execution after AQE re-planning.
3838
*
3939
* `BloomFilterAggregate` is only rewritten when it appears inside the [[ScalarSubquery]] of a
40-
* [[BloomFilterMightContain]]. Standalone usages (in particular `DataFrame.stat.bloomFilter()`,
41-
* which collects bloom filter bytes and passes them to `BloomFilter.readFrom()`) are intentionally
42-
* left untouched so that the returned bytes remain in Spark-native format.
40+
* [[BloomFilterMightContain]] whose value expression is a plain column reference ([[Attribute]]).
41+
* This deliberately excludes:
42+
* - Standalone `BloomFilterAggregate` (e.g., `DataFrame.stat.bloomFilter()`) — so that the
43+
* returned bytes stay in Spark-native format.
44+
* - DPP-injected bloom filters from
45+
* [[org.apache.spark.sql.execution.dynamicpruning.PlanDynamicPruningFilters]] and Spark's
46+
* `InjectRuntimeFilter` — those always hash the join key with `xxhash64(col, seed)` before
47+
* passing it to `BloomFilterMightContain`, so their value expression is never a bare
48+
* [[Attribute]]. Rewriting them would cause `VeloxBloomFilterMightContain` to appear in a
49+
* `FilterExecTransformer`, which fails Velox expression validation and falls back to JVM,
50+
* changing the physical plan structure and breaking plan-stability golden files.
4351
*/
4452
case class BloomFilterMightContainJointRewriteRule(spark: SparkSession)
4553
extends Rule[LogicalPlan] {
@@ -49,7 +57,11 @@ case class BloomFilterMightContainJointRewriteRule(spark: SparkSession)
4957
return plan
5058
}
5159
plan.transformAllExpressions {
52-
case BloomFilterMightContain(subq: ScalarSubquery, v) =>
60+
case BloomFilterMightContain(subq: ScalarSubquery, v: Attribute) =>
61+
// Only rewrite when the tested value is a plain column reference (user-facing
62+
// bloom filter). DPP-injected bloom filters always hash the join key —
63+
// xxhash64(col, seed) — so their value expression is never a bare Attribute;
64+
// they are intentionally left as vanilla BloomFilterMightContain.
5365
val rewrittenPlan = subq.plan.transformAllExpressions {
5466
case ae @ AggregateExpression(b: BloomFilterAggregate, _, _, _, _) =>
5567
ae.copy(aggregateFunction = VeloxBloomFilterAggregate(
@@ -60,7 +72,8 @@ case class BloomFilterMightContainJointRewriteRule(spark: SparkSession)
6072
b.inputAggBufferOffset))
6173
}
6274
VeloxBloomFilterMightContain(subq.withNewPlan(rewrittenPlan), v)
63-
case BloomFilterMightContain(bf, v) =>
75+
case BloomFilterMightContain(bf, v) if !bf.isInstanceOf[ScalarSubquery] =>
76+
// Pre-computed literal bloom filter bytes — rewrite to consume Velox-format bytes.
6477
VeloxBloomFilterMightContain(bf, v)
6578
}
6679
}

0 commit comments

Comments
 (0)