Commit 8639584
committed
[GLUTEN-12013][VL] Fix bloom-filter bytes corruption on whole-stage AQE fallback
`BloomFilterMightContainJointRewriteRule` previously rewrote every
`BloomFilterAggregate` it encountered, including standalone usages such
as `DataFrame.stat.bloomFilter()`. That API collects the aggregate
output bytes and passes them directly to `BloomFilter.readFrom()`, which
expects Spark-native format; receiving Velox-format bytes caused
`java.io.IOException: Unexpected Bloom filter version number` (surfaced
as a CI failure in `GlutenDataFrameStatSuite - Bloom filter`).
Fix: only rewrite `BloomFilterAggregate` when it appears inside the
`ScalarSubquery` of a `BloomFilterMightContain`. Standalone aggregates
are left untouched so that collected bytes remain in Spark-native format.
Add a regression test (`GlutenBloomFilterFallbackSuite`) to guard
against reintroducing this regression.
Local test results (Spark 4.0, Velox backend):
- GlutenDataFrameStatSuite : 25/25 passed (was failing)
- GlutenBloomFilterFallbackSuite : 4/4 passed
- GlutenBloomFilterAggregateQuerySuite: 14/14 passed
- GlutenInjectRuntimeFilterSuite : 13/13 passed1 parent 7cc6faf commit 8639584
2 files changed
Lines changed: 34 additions & 8 deletions
File tree
- backends-velox/src/main/scala/org/apache/gluten/extension
- gluten-ut/test/src/test/scala/org/apache/gluten/sql
Lines changed: 17 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
38 | 43 | | |
39 | 44 | | |
40 | 45 | | |
| |||
44 | 49 | | |
45 | 50 | | |
46 | 51 | | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
54 | 63 | | |
55 | 64 | | |
56 | 65 | | |
| |||
Lines changed: 17 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
155 | 155 | | |
156 | 156 | | |
157 | 157 | | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
158 | 175 | | |
159 | 176 | | |
160 | 177 | | |
| |||
0 commit comments