Add configurable UNION DISTINCT to FILTER rewrite optimization by xiedeyantu · Pull Request #21075 · apache/datafusion

xiedeyantu · 2026-03-20T10:23:59Z

Which issue does this PR close?
- Closes Add configurable UNION DISTINCT support to FILTER rewrite optimization #21310.
Rationale for this change

This PR adds a configurable optimizer rewrite for UNION DISTINCT queries. The goal is to allow the optimizer to collapse eligible union branches into a single filtered scan when the branches read from the same source and differ only by filter predicates.

This optimization can reduce duplicated work and avoid scanning the same input multiple times. Keeping it behind a configuration flag makes the behavior explicit and safe to enable only when desired.

What changes are included in this PR?
- Adds a new optimizer configuration option: datafusion.optimizer.enable_unions_to_filter, which is disabled by default.
- Enables the UnionsToFilter optimization rule in the logical optimizer pipeline.
- Adds documentation for the new configuration option, including plan-shape examples.
- Extends sqllogictest coverage in datafusion/sqllogictest/test_files/union.slt to cover both the disabled and enabled cases.
- Verifies that the rewrite only applies to eligible UNION DISTINCT queries.
Example rewrite

When the rule is enabled, a query such as:

SQL
```
SELECT id, name FROM t1 WHERE id = 1
UNION
SELECT id, name FROM t1 WHERE id = 2
```
may be rewritten into an equivalent plan that scans t1 once and applies a combined filter such as:

SQL
```
SELECT id, name FROM t1 WHERE id = 1 OR id = 2
```
This keeps the results unchanged while avoiding repeated reads from the same source.

Are these changes tested?

Yes.

The new behavior is covered by sqllogictest cases that validate both plan variants:
- the original UNION DISTINCT execution path when the option is disabled
- the rewritten single-scan plan when the option is enabled
Are there any user-facing changes?

Yes.

A new configuration option is introduced:
- datafusion.optimizer.enable_unions_to_filter
When enabled, some UNION DISTINCT queries may be optimized into a different plan shape. Query results remain the same, but the execution plan may change.

xiedeyantu · 2026-03-21T03:54:30Z

Hi @alamb , here's another PR related to plan optimization. Do we need it? I'd also like to know what aspects of optimization we accept.

xiedeyantu · 2026-03-21T13:55:42Z

Hi @Dandandan , I noticed that you’re very knowledgeable about SQL optimization. It would be great if you could help me review this!

alamb · 2026-03-22T13:26:15Z

@xiedeyantu -- can you please file a ticket explaining what usecase you are targeting with this optimization?

Your explanation in

Rationale for this change

Mostly focuses on "what" is changed, not the "why"

In terms of usecase I think what is important:

Example SQL queries that show the pattern you are optimizing for

Then for this optimization it would be great to have some benchmark numbers showing that the query of interest iindeed gets faster with this optimization compared than without it

xiedeyantu · 2026-03-22T13:57:19Z

+
+query TT
+EXPLAIN SELECT id, name FROM t1 WHERE id = 1 UNION SELECT id, name FROM t1 WHERE id = 2
+----


@alamb Here is an example: There is a rule for eliminating UNION under specific conditions. It applies when the branches of the UNION come from the same table and only differ in their WHERE conditions. This rule allows us to avoid an extra table scan — we only need to perform a single combined conditional filter.

I haven't looked at the proposed implementation but the rewrite can surely help in case of repeated costly union branches (especially when coming from possibly complex data sources powered via TableProvider).

It seems also particularly relevant until #8777 gets addressed (broader scope, CTE materialization), as currently there is no other way to mutualize repeated reads AFAIK.

Thank you @asolimando , for connecting me to such a valuable discussion. I think the idea in #8777 is excellent, but it seems we might need to perform a union operation once more. It looks like a final conclusion hasn't been reached yet? However, this approach can support UNION ALL. It seems the two might complement each other?

xiedeyantu · 2026-03-29T05:29:45Z

I tested the execution times, and in reality, the difference between the two was not significant. My understanding is that the multiple branches within a UNION operation can be processed in parallel; therefore, one would not expect to see a substantial reduction in overall execution time (and if a significant improvement were observed, it would likely indicate an issue elsewhere). Consequently, I had to rely on EXPLAIN ANALYZE to inspect the execution plan, thereby demonstrating that—prior to optimization—the data required two separate scans, whereas after optimization, only a single scan was necessary.

The test SQL script is as follows:

# test_data.csv
id,category,amount,created_at
1,A,10,2026-03-29 00:00:01
2,B,20,2026-03-29 00:00:02
3,C,30,2026-03-29 00:00:03
......
9999,D,99990,2026-03-29 02:46:39
10000,E,100000,2026-03-29 02:46:40

# create table
CREATE EXTERNAL TABLE t (
  id INT,
  category STRING,
  amount INT,
  created_at TIMESTAMP
)
STORED AS CSV
LOCATION 'test_data.csv'
OPTIONS (
  has_header 'true'
);

set datafusion.optimizer.enable_unions_to_filter=false;
EXPLAIN ANALYZE 
SELECT category FROM t WHERE id > 5
UNION
SELECT category FROM t WHERE id < 10;
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type         | plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | AggregateExec: mode=FinalPartitioned, gby=[category@0 as category], aggr=[], metrics=[output_rows=5, elapsed_compute=512.96µs, output_bytes=128.0 B, output_batches=2, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, peak_mem_used=3.49 K, aggregate_arguments_time=8ns, aggregation_time=8ns, emitting_time=5.51µs, time_calculating_group_ids=21.34µs]                                                                                                                                |
|                   |   RepartitionExec: partitioning=Hash([category@0], 8), input_partitions=16, metrics=[output_rows=15, elapsed_compute=129.16µs, output_bytes=256.0 KB, output_batches=2, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, fetch_time=311.11ms, repartition_time=52.80µs, send_time=27.66µs]                                                                                                                                                                                                 |
|                   |     AggregateExec: mode=Partial, gby=[category@0 as category], aggr=[], metrics=[output_rows=15, elapsed_compute=2.50ms, output_bytes=384.0 B, output_batches=3, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, skipped_aggregation_rows=0, peak_mem_used=167.2 K, aggregate_arguments_time=16ns, aggregation_time=16ns, emitting_time=16.47µs, time_calculating_group_ids=2.06ms, reduction_factor=0.15% (15/10.00 K)]                                                                  |
|                   |       UnionExec, metrics=[output_rows=10.00 K, elapsed_compute=298.29µs, output_bytes=383.9 KB, output_batches=3]                                                                                                                                                                                                                                                                                                                                                                             |
|                   |         FilterExec: id@0 > 5, projection=[category@1], metrics=[output_rows=9.99 K, elapsed_compute=418.21µs, output_bytes=255.9 KB, output_batches=2, selectivity=100% (9.99 K/10.00 K)]                                                                                                                                                                                                                                                                                                     |
|                   |           RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1, metrics=[output_rows=10.00 K, elapsed_compute=35.17µs, output_bytes=320.0 KB, output_batches=2, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, fetch_time=19.31ms, repartition_time=1ns, send_time=16.59µs]                                                                                                                                                                                              |
|                   |             DataSourceExec: file_groups={1 group: [[Users/jensen/test/test_data.csv]]}, projection=[id, category], file_type=csv, has_header=true, metrics=[output_rows=10.00 K, elapsed_compute=18.83ms, output_bytes=196.2 KB, output_batches=2, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, time_elapsed_opening=359.75µs, time_elapsed_processing=19.09ms, time_elapsed_scanning_total=18.89ms, time_elapsed_scanning_until_data=15.39ms] |
|                   |         FilterExec: id@0 < 10, projection=[category@1], metrics=[output_rows=9, elapsed_compute=310.84µs, output_bytes=128.0 KB, output_batches=1, selectivity=0.09% (9/10.00 K)]                                                                                                                                                                                                                                                                                                             |
|                   |           RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1, metrics=[output_rows=10.00 K, elapsed_compute=62.12µs, output_bytes=320.0 KB, output_batches=2, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, fetch_time=18.82ms, repartition_time=1ns, send_time=47.76µs]                                                                                                                                                                                              |
|                   |             DataSourceExec: file_groups={1 group: [[Users/jensen/test/test_data.csv]]}, projection=[id, category], file_type=csv, has_header=true, metrics=[output_rows=10.00 K, elapsed_compute=18.35ms, output_bytes=196.2 KB, output_batches=2, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, time_elapsed_opening=364.33µs, time_elapsed_processing=18.63ms, time_elapsed_scanning_total=18.48ms, time_elapsed_scanning_until_data=15.05ms] |
|                   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched. 
Elapsed 0.038 seconds.

set datafusion.optimizer.enable_unions_to_filter=true;
EXPLAIN ANALYZE 
SELECT category FROM t WHERE id > 5
UNION
SELECT category FROM t WHERE id < 10;
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type         | plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | AggregateExec: mode=FinalPartitioned, gby=[category@0 as category], aggr=[], metrics=[output_rows=5, elapsed_compute=374.54µs, output_bytes=128.0 B, output_batches=2, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, peak_mem_used=3.36 K, aggregate_arguments_time=8ns, aggregation_time=8ns, emitting_time=5.50µs, time_calculating_group_ids=19.13µs]                                                                                                                              |
|                   |   RepartitionExec: partitioning=Hash([category@0], 8), input_partitions=8, metrics=[output_rows=10, elapsed_compute=97.63µs, output_bytes=256.0 KB, output_batches=2, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, fetch_time=180.39ms, repartition_time=48.84µs, send_time=30.44µs]                                                                                                                                                                                                 |
|                   |     AggregateExec: mode=Partial, gby=[category@0 as category], aggr=[], metrics=[output_rows=10, elapsed_compute=2.00ms, output_bytes=256.0 B, output_batches=2, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, skipped_aggregation_rows=0, peak_mem_used=164.5 K, aggregate_arguments_time=8ns, aggregation_time=8ns, emitting_time=14.63µs, time_calculating_group_ids=1.81ms, reduction_factor=0.1% (10/10.00 K)]                                                                   |
|                   |       FilterExec: id@0 > 5 OR id@0 < 10, projection=[category@1], metrics=[output_rows=10.00 K, elapsed_compute=526.17µs, output_bytes=256.0 KB, output_batches=2, selectivity=100% (10.00 K/10.00 K)]                                                                                                                                                                                                                                                                                      |
|                   |         RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1, metrics=[output_rows=10.00 K, elapsed_compute=58.50µs, output_bytes=320.0 KB, output_batches=2, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, fetch_time=22.05ms, repartition_time=1ns, send_time=25.67µs]                                                                                                                                                                                              |
|                   |           DataSourceExec: file_groups={1 group: [[Users/jensen/test/test_data.csv]]}, projection=[id, category], file_type=csv, has_header=true, metrics=[output_rows=10.00 K, elapsed_compute=21.49ms, output_bytes=196.2 KB, output_batches=2, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, time_elapsed_opening=468.54µs, time_elapsed_processing=21.82ms, time_elapsed_scanning_total=21.57ms, time_elapsed_scanning_until_data=18.30ms] |
|                   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched. 
Elapsed 0.040 seconds.

xiedeyantu · 2026-03-29T05:33:31Z

@alamb I have refined the PR description and added a test scenario; could you please take a look and let me know if you find it valuable?

xiedeyantu · 2026-03-31T22:47:07Z

@alamb If you have a moment, could you please take another look? Thank you!

xiedeyantu · 2026-04-01T22:40:56Z

@comphead I'm not sure if you can help me review this PR?

comphead · 2026-04-02T00:17:50Z

I would agree with @alamb to create initial ticket stating the problem. PR description is nice but it is a solution whereas ticket is a problem statement and some people could also participate in problem discussion

xiedeyantu · 2026-04-02T00:55:29Z

I would agree with @alamb to create initial ticket stating the problem. PR description is nice but it is a solution whereas ticket is a problem statement and some people could also participate in problem discussion

@comphead Apologies—I may not have fully understood the process earlier; I thought simply describing the issue within the PR itself (including both the problem and the proposed solution) would suffice. I have now created a new issue #21310 for everyone to discuss. Please take a look and let me know if it looks appropriate.

comphead · 2026-04-02T15:10:34Z

Thanks @xiedeyantu I'll take a look this week, would be super useful for users and also for regression to have internal microbenchmarks, similar to datafusion/core/benches/push_down_filter.rs

comphead

Thanks @xiedeyantu for this PR, I think we need to run current tests with this optimizer rule on. The easiest way to proceed if you enable the rule by default in this PR so we can run CI and also benchmarks to get some data points and then disable the rule

xiedeyantu · 2026-04-07T15:30:16Z

Thanks @xiedeyantu for this PR, I think we need to run current tests with this optimizer rule on. The easiest way to proceed if you enable the rule by default in this PR so we can run CI and also benchmarks to get some data points and then disable the rule

@comphead Thank you for your detailed review. I will immediately address these issues based on your comments and temporarily set the parameter to true.

xiedeyantu · 2026-04-07T15:48:15Z

@comphead Could you tell me how to run benchmark? I have changed enable_unions_to_filter to default true.

comphead · 2026-04-07T16:12:36Z

run benchmark

adriangbot · 2026-04-07T16:12:38Z

Hi @comphead, run benchmark requires benchmark names (#21075 (comment)).

Supported benchmarks:

Standard: clickbench_1, clickbench_extended, clickbench_partitioned, clickbench_pushdown, external_aggr, smj, sort_pushdown, sort_pushdown_sorted, tpcds, tpch, tpch10, tpch_mem, tpch_mem10
Criterion: (any)

Usage:

run benchmark <name>           # run specific benchmark(s)
run benchmarks                 # run default suite
run benchmarks <name1> <name2> # run specific benchmarks

Per-side configuration (run benchmark tpch followed by):

env:
SHARED_SETTING: enabled
baseline:
ref: v45.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G

File an issue against this benchmark runner

comphead · 2026-04-07T16:13:06Z

run benchmark tpch tpcds

adriangbot · 2026-04-07T16:15:45Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4200500846-927-96l2g 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing union-filter (056e734) to 9885f4b (merge-base) diff using: tpcds
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-07T16:15:48Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4200500846-926-kfkwv 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing union-filter (056e734) to 9885f4b (merge-base) diff using: tpch
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-07T16:30:19Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and union-filter
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query     ┃                           HEAD ┃                   union-filter ┃    Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1  │ 45.98 / 46.74 ±0.65 / 47.88 ms │ 45.80 / 46.29 ±0.59 / 47.43 ms │ no change │
│ QQuery 2  │ 25.75 / 25.98 ±0.22 / 26.39 ms │ 25.02 / 25.42 ±0.29 / 25.78 ms │ no change │
│ QQuery 3  │ 33.74 / 34.92 ±0.96 / 36.37 ms │ 33.67 / 34.08 ±0.35 / 34.65 ms │ no change │
│ QQuery 4  │ 21.87 / 22.58 ±0.49 / 23.23 ms │ 21.57 / 22.63 ±0.88 / 23.78 ms │ no change │
│ QQuery 5  │ 50.41 / 52.46 ±1.49 / 54.08 ms │ 49.79 / 52.58 ±2.32 / 56.58 ms │ no change │
│ QQuery 6  │ 17.57 / 17.70 ±0.16 / 18.01 ms │ 17.73 / 18.04 ±0.26 / 18.37 ms │ no change │
│ QQuery 7  │ 57.33 / 58.85 ±1.82 / 62.30 ms │ 58.66 / 59.54 ±0.71 / 60.70 ms │ no change │
│ QQuery 8  │ 51.15 / 51.43 ±0.30 / 52.00 ms │ 52.02 / 52.50 ±0.31 / 53.00 ms │ no change │
│ QQuery 9  │ 55.63 / 57.25 ±1.26 / 59.35 ms │ 56.48 / 57.05 ±0.46 / 57.53 ms │ no change │
│ QQuery 10 │ 74.33 / 75.51 ±1.51 / 78.47 ms │ 73.75 / 74.63 ±1.07 / 76.61 ms │ no change │
│ QQuery 11 │ 17.25 / 17.66 ±0.36 / 18.33 ms │ 17.30 / 17.81 ±0.43 / 18.37 ms │ no change │
│ QQuery 12 │ 28.33 / 28.94 ±0.53 / 29.91 ms │ 29.10 / 29.45 ±0.52 / 30.44 ms │ no change │
│ QQuery 13 │ 40.36 / 40.84 ±0.33 / 41.36 ms │ 40.63 / 40.97 ±0.37 / 41.63 ms │ no change │
│ QQuery 14 │ 29.05 / 29.70 ±0.48 / 30.38 ms │ 28.78 / 29.10 ±0.25 / 29.54 ms │ no change │
│ QQuery 15 │ 36.55 / 36.79 ±0.37 / 37.52 ms │ 36.26 / 36.67 ±0.41 / 37.39 ms │ no change │
│ QQuery 16 │ 17.60 / 18.22 ±0.38 / 18.67 ms │ 18.19 / 18.79 ±0.68 / 20.07 ms │ no change │
│ QQuery 17 │ 74.98 / 75.51 ±0.40 / 76.09 ms │ 74.38 / 75.15 ±0.62 / 75.91 ms │ no change │
│ QQuery 18 │ 79.45 / 82.30 ±1.67 / 84.50 ms │ 79.51 / 81.80 ±1.22 / 83.13 ms │ no change │
│ QQuery 19 │ 38.46 / 39.15 ±0.42 / 39.56 ms │ 39.25 / 40.02 ±0.92 / 41.82 ms │ no change │
│ QQuery 20 │ 43.63 / 44.63 ±0.85 / 45.73 ms │ 43.63 / 44.45 ±0.70 / 45.52 ms │ no change │
│ QQuery 21 │ 69.05 / 70.36 ±0.89 / 71.59 ms │ 67.84 / 70.33 ±1.93 / 72.80 ms │ no change │
│ QQuery 22 │ 20.05 / 21.43 ±1.02 / 22.74 ms │ 20.35 / 20.95 ±0.63 / 22.11 ms │ no change │
└───────────┴────────────────────────────────┴────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary           ┃          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD)           │ 948.95ms │
│ Total Time (union-filter)   │ 948.25ms │
│ Average Time (HEAD)         │  43.13ms │
│ Average Time (union-filter) │  43.10ms │
│ Queries Faster              │        0 │
│ Queries Slower              │        0 │
│ Queries with No Change      │       22 │
│ Queries with Failure        │        0 │
└─────────────────────────────┴──────────┘

Resource Usage

tpch — base (merge-base)

Metric	Value
Wall time	5.0s
Peak memory	4.1 GiB
Avg memory	3.6 GiB
CPU user	33.6s
CPU sys	2.9s
Peak spill	0 B

tpch — branch

Metric	Value
Wall time	5.0s
Peak memory	4.1 GiB
Avg memory	3.6 GiB
CPU user	33.7s
CPU sys	2.9s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-04-07T16:35:47Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and union-filter
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query     ┃                                     HEAD ┃                             union-filter ┃       Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1  │           46.69 / 47.44 ±0.69 / 48.73 ms │           46.95 / 47.94 ±0.77 / 48.93 ms │    no change │
│ QQuery 2  │        150.39 / 151.11 ±0.51 / 151.59 ms │        151.35 / 151.76 ±0.28 / 152.12 ms │    no change │
│ QQuery 3  │        115.88 / 117.15 ±0.94 / 118.46 ms │        115.59 / 117.11 ±0.97 / 118.42 ms │    no change │
│ QQuery 4  │    1285.89 / 1308.50 ±14.97 / 1326.79 ms │    1308.37 / 1397.69 ±49.78 / 1455.87 ms │ 1.07x slower │
│ QQuery 5  │        182.67 / 184.35 ±1.39 / 186.67 ms │        181.61 / 183.10 ±0.95 / 184.17 ms │    no change │
│ QQuery 6  │     963.56 / 1011.85 ±36.27 / 1066.32 ms │    1000.68 / 1013.38 ±17.17 / 1046.71 ms │    no change │
│ QQuery 7  │        358.02 / 359.52 ±1.31 / 361.22 ms │        357.81 / 363.78 ±5.24 / 373.21 ms │    no change │
│ QQuery 8  │        119.21 / 120.78 ±0.97 / 122.17 ms │        120.38 / 121.11 ±0.92 / 122.67 ms │    no change │
│ QQuery 9  │        111.09 / 117.84 ±5.69 / 128.37 ms │        118.49 / 118.89 ±0.64 / 120.16 ms │    no change │
│ QQuery 10 │        111.79 / 113.45 ±0.96 / 114.67 ms │        114.42 / 116.05 ±1.14 / 117.90 ms │    no change │
│ QQuery 11 │       880.19 / 916.86 ±34.19 / 970.54 ms │     917.50 / 1021.59 ±92.95 / 1186.48 ms │ 1.11x slower │
│ QQuery 12 │           45.81 / 47.36 ±1.15 / 48.99 ms │           47.72 / 49.46 ±0.96 / 50.45 ms │    no change │
│ QQuery 13 │        403.32 / 407.42 ±3.93 / 414.49 ms │        410.96 / 413.11 ±1.12 / 414.18 ms │    no change │
│ QQuery 14 │    1065.33 / 1081.24 ±13.57 / 1106.35 ms │    1051.09 / 1076.44 ±17.49 / 1098.09 ms │    no change │
│ QQuery 15 │           17.77 / 19.43 ±1.28 / 21.60 ms │           18.20 / 19.95 ±1.77 / 23.03 ms │    no change │
│ QQuery 16 │           43.66 / 44.93 ±0.85 / 45.93 ms │           43.68 / 44.54 ±0.98 / 46.24 ms │    no change │
│ QQuery 17 │        246.28 / 248.30 ±1.85 / 251.31 ms │        241.88 / 242.95 ±0.86 / 244.04 ms │    no change │
│ QQuery 18 │        132.82 / 134.21 ±1.27 / 135.79 ms │        133.51 / 134.69 ±1.25 / 137.09 ms │    no change │
│ QQuery 19 │        156.64 / 158.80 ±1.26 / 160.12 ms │        157.55 / 159.38 ±1.17 / 160.99 ms │    no change │
│ QQuery 20 │           15.78 / 16.23 ±0.60 / 17.35 ms │           16.16 / 16.39 ±0.23 / 16.80 ms │    no change │
│ QQuery 21 │           23.61 / 24.06 ±0.37 / 24.73 ms │           23.77 / 24.16 ±0.28 / 24.56 ms │    no change │
│ QQuery 22 │        492.00 / 494.80 ±1.99 / 497.79 ms │        487.80 / 490.71 ±2.14 / 493.95 ms │    no change │
│ QQuery 23 │        925.50 / 937.87 ±7.73 / 949.61 ms │       932.52 / 947.32 ±10.14 / 958.92 ms │    no change │
│ QQuery 24 │        427.30 / 428.46 ±0.65 / 429.08 ms │        427.74 / 430.42 ±1.97 / 432.65 ms │    no change │
│ QQuery 25 │        359.14 / 360.58 ±1.08 / 362.27 ms │        358.62 / 359.11 ±0.50 / 359.91 ms │    no change │
│ QQuery 26 │           85.29 / 87.51 ±1.79 / 90.51 ms │           85.82 / 88.50 ±1.70 / 90.52 ms │    no change │
│ QQuery 27 │        352.98 / 355.56 ±2.38 / 359.25 ms │        352.03 / 355.78 ±2.46 / 358.71 ms │    no change │
│ QQuery 28 │        153.10 / 156.04 ±2.14 / 159.67 ms │        153.22 / 154.09 ±0.52 / 154.67 ms │    no change │
│ QQuery 29 │        298.75 / 304.26 ±2.99 / 307.04 ms │        299.17 / 302.41 ±2.15 / 304.92 ms │    no change │
│ QQuery 30 │           46.95 / 48.65 ±1.16 / 50.45 ms │           47.42 / 49.25 ±1.30 / 50.77 ms │    no change │
│ QQuery 31 │        177.77 / 180.17 ±1.51 / 181.54 ms │        178.41 / 179.90 ±1.54 / 182.36 ms │    no change │
│ QQuery 32 │           59.62 / 60.26 ±0.68 / 61.26 ms │           59.28 / 61.22 ±1.28 / 62.54 ms │    no change │
│ QQuery 33 │        147.48 / 149.06 ±0.96 / 150.26 ms │        148.66 / 150.64 ±1.79 / 152.79 ms │    no change │
│ QQuery 34 │        108.88 / 110.36 ±1.32 / 112.59 ms │        109.71 / 110.35 ±0.57 / 111.23 ms │    no change │
│ QQuery 35 │        114.35 / 115.23 ±1.11 / 117.33 ms │        114.06 / 115.63 ±0.84 / 116.50 ms │    no change │
│ QQuery 36 │        218.37 / 222.90 ±3.39 / 228.48 ms │        216.15 / 224.27 ±5.06 / 229.64 ms │    no change │
│ QQuery 37 │        178.38 / 179.70 ±1.12 / 181.33 ms │        181.46 / 182.58 ±0.96 / 184.31 ms │    no change │
│ QQuery 38 │           90.27 / 93.06 ±1.84 / 95.20 ms │           88.81 / 92.30 ±1.92 / 94.30 ms │    no change │
│ QQuery 39 │        139.16 / 141.36 ±2.42 / 145.86 ms │        138.02 / 139.19 ±0.87 / 140.66 ms │    no change │
│ QQuery 40 │        116.18 / 121.09 ±5.28 / 131.19 ms │        117.47 / 121.72 ±4.50 / 130.20 ms │    no change │
│ QQuery 41 │           17.45 / 18.35 ±0.52 / 18.99 ms │           17.29 / 18.57 ±1.34 / 21.11 ms │    no change │
│ QQuery 42 │        107.32 / 109.77 ±1.55 / 111.35 ms │        108.42 / 111.60 ±1.68 / 113.13 ms │    no change │
│ QQuery 43 │           85.38 / 86.30 ±0.84 / 87.87 ms │           85.55 / 86.09 ±0.45 / 86.79 ms │    no change │
│ QQuery 44 │           16.50 / 17.14 ±0.75 / 18.58 ms │           17.01 / 17.50 ±0.30 / 17.93 ms │    no change │
│ QQuery 45 │           54.88 / 56.02 ±0.90 / 57.28 ms │           56.15 / 57.47 ±0.89 / 58.53 ms │    no change │
│ QQuery 46 │        233.78 / 236.16 ±2.07 / 239.65 ms │        237.53 / 242.47 ±3.34 / 246.58 ms │    no change │
│ QQuery 47 │        709.44 / 714.53 ±3.52 / 718.74 ms │       755.59 / 769.23 ±15.51 / 798.98 ms │ 1.08x slower │
│ QQuery 48 │        291.11 / 293.80 ±2.37 / 298.17 ms │        294.72 / 297.84 ±3.05 / 303.19 ms │    no change │
│ QQuery 49 │        261.83 / 266.70 ±2.81 / 270.35 ms │        264.06 / 266.92 ±1.92 / 268.97 ms │    no change │
│ QQuery 50 │        231.58 / 236.73 ±3.44 / 241.44 ms │        241.36 / 243.74 ±1.98 / 245.85 ms │    no change │
│ QQuery 51 │        188.69 / 189.98 ±0.91 / 191.42 ms │        189.57 / 192.68 ±2.10 / 195.73 ms │    no change │
│ QQuery 52 │        109.08 / 109.60 ±0.44 / 110.37 ms │        109.52 / 111.13 ±1.32 / 113.22 ms │    no change │
│ QQuery 53 │        105.18 / 106.35 ±0.78 / 107.63 ms │        107.13 / 108.00 ±1.15 / 110.27 ms │    no change │
│ QQuery 54 │        151.77 / 153.37 ±0.90 / 154.47 ms │        153.86 / 155.65 ±1.74 / 158.48 ms │    no change │
│ QQuery 55 │        108.03 / 108.97 ±1.16 / 111.14 ms │        108.33 / 109.21 ±0.72 / 110.30 ms │    no change │
│ QQuery 56 │        147.85 / 150.05 ±1.38 / 151.47 ms │        149.47 / 150.12 ±0.57 / 150.92 ms │    no change │
│ QQuery 57 │        185.05 / 187.02 ±1.40 / 188.87 ms │        187.32 / 188.72 ±1.13 / 190.05 ms │    no change │
│ QQuery 58 │        296.87 / 306.75 ±6.64 / 317.64 ms │        301.09 / 315.10 ±8.63 / 328.21 ms │    no change │
│ QQuery 59 │        204.08 / 206.81 ±2.12 / 209.18 ms │        203.09 / 205.48 ±1.70 / 207.82 ms │    no change │
│ QQuery 60 │        150.07 / 152.32 ±1.42 / 153.81 ms │        152.13 / 153.15 ±0.73 / 153.97 ms │    no change │
│ QQuery 61 │        174.86 / 176.94 ±1.46 / 178.64 ms │        177.37 / 178.59 ±1.12 / 180.31 ms │    no change │
│ QQuery 62 │       869.23 / 930.63 ±38.11 / 980.14 ms │       921.35 / 947.71 ±18.45 / 976.35 ms │    no change │
│ QQuery 63 │        105.32 / 108.61 ±2.27 / 112.09 ms │        107.91 / 109.98 ±1.88 / 113.37 ms │    no change │
│ QQuery 64 │        706.60 / 713.84 ±4.53 / 720.24 ms │        727.43 / 734.84 ±6.84 / 747.33 ms │    no change │
│ QQuery 65 │        255.09 / 258.02 ±2.80 / 262.86 ms │        271.96 / 275.98 ±3.41 / 280.63 ms │ 1.07x slower │
│ QQuery 66 │        250.68 / 260.28 ±8.23 / 272.32 ms │        264.39 / 268.91 ±3.89 / 275.78 ms │    no change │
│ QQuery 67 │        312.00 / 320.07 ±6.56 / 331.33 ms │        321.20 / 333.69 ±8.99 / 344.92 ms │    no change │
│ QQuery 68 │        281.69 / 285.40 ±2.30 / 288.67 ms │        289.87 / 294.84 ±3.23 / 299.69 ms │    no change │
│ QQuery 69 │        108.39 / 109.87 ±0.89 / 110.78 ms │        109.91 / 112.51 ±1.90 / 115.19 ms │    no change │
│ QQuery 70 │        326.76 / 343.84 ±9.49 / 354.55 ms │        347.63 / 356.26 ±8.21 / 371.56 ms │    no change │
│ QQuery 71 │        137.22 / 139.35 ±1.88 / 142.49 ms │        137.36 / 140.35 ±2.23 / 142.91 ms │    no change │
│ QQuery 72 │        722.58 / 737.21 ±9.25 / 747.90 ms │       748.42 / 762.58 ±11.27 / 776.99 ms │    no change │
│ QQuery 73 │        105.62 / 107.04 ±1.65 / 110.24 ms │        106.04 / 108.77 ±2.98 / 114.23 ms │    no change │
│ QQuery 74 │        550.12 / 554.15 ±2.67 / 556.57 ms │        621.71 / 632.03 ±7.39 / 641.47 ms │ 1.14x slower │
│ QQuery 75 │        286.51 / 288.98 ±1.84 / 291.73 ms │        289.85 / 291.92 ±1.25 / 293.64 ms │    no change │
│ QQuery 76 │        136.81 / 139.40 ±2.25 / 142.40 ms │        139.22 / 140.56 ±1.23 / 142.35 ms │    no change │
│ QQuery 77 │        199.13 / 200.98 ±1.10 / 202.53 ms │        197.28 / 201.94 ±2.65 / 204.92 ms │    no change │
│ QQuery 78 │        359.66 / 363.90 ±2.82 / 367.07 ms │        363.54 / 367.25 ±2.15 / 370.10 ms │    no change │
│ QQuery 79 │        241.48 / 243.99 ±2.45 / 248.25 ms │        245.66 / 250.43 ±3.22 / 255.11 ms │    no change │
│ QQuery 80 │        338.52 / 341.91 ±4.03 / 349.62 ms │        336.09 / 339.62 ±2.03 / 341.74 ms │    no change │
│ QQuery 81 │           31.31 / 32.68 ±0.83 / 33.78 ms │           31.61 / 32.91 ±1.34 / 35.00 ms │    no change │
│ QQuery 82 │        201.76 / 205.66 ±2.37 / 208.24 ms │        206.73 / 209.10 ±1.74 / 211.02 ms │    no change │
│ QQuery 83 │           47.04 / 47.68 ±0.82 / 49.24 ms │           47.34 / 48.33 ±1.07 / 50.35 ms │    no change │
│ QQuery 84 │           51.15 / 51.69 ±0.38 / 52.31 ms │           50.90 / 51.58 ±0.62 / 52.41 ms │    no change │
│ QQuery 85 │        151.08 / 154.14 ±2.07 / 156.96 ms │        153.41 / 155.25 ±0.99 / 156.26 ms │    no change │
│ QQuery 86 │           42.41 / 43.00 ±0.52 / 43.67 ms │           41.50 / 42.37 ±0.90 / 43.91 ms │    no change │
│ QQuery 87 │           92.00 / 94.67 ±2.57 / 98.41 ms │          92.12 / 96.32 ±4.11 / 102.94 ms │    no change │
│ QQuery 88 │        111.57 / 113.11 ±1.04 / 114.34 ms │        110.62 / 111.52 ±0.93 / 112.96 ms │    no change │
│ QQuery 89 │        122.45 / 124.13 ±1.54 / 126.52 ms │        122.00 / 122.91 ±0.86 / 124.52 ms │    no change │
│ QQuery 90 │           28.78 / 29.26 ±0.39 / 29.74 ms │           27.55 / 28.36 ±0.87 / 29.93 ms │    no change │
│ QQuery 91 │           66.88 / 69.49 ±1.56 / 71.56 ms │           67.55 / 68.75 ±0.78 / 69.94 ms │    no change │
│ QQuery 92 │           62.16 / 63.36 ±0.98 / 64.57 ms │           60.44 / 60.90 ±0.36 / 61.34 ms │    no change │
│ QQuery 93 │        194.56 / 196.82 ±1.44 / 198.48 ms │        192.86 / 195.26 ±1.70 / 197.58 ms │    no change │
│ QQuery 94 │           65.58 / 66.29 ±0.40 / 66.82 ms │           65.16 / 66.05 ±0.57 / 66.87 ms │    no change │
│ QQuery 95 │        140.37 / 143.07 ±1.89 / 146.05 ms │        139.59 / 141.14 ±1.22 / 142.70 ms │    no change │
│ QQuery 96 │           74.55 / 77.89 ±1.69 / 79.06 ms │           72.73 / 75.35 ±2.12 / 77.36 ms │    no change │
│ QQuery 97 │        132.21 / 136.45 ±2.68 / 140.07 ms │        132.45 / 133.72 ±1.10 / 135.00 ms │    no change │
│ QQuery 98 │        158.97 / 161.06 ±1.33 / 162.49 ms │        153.78 / 157.26 ±1.99 / 159.23 ms │    no change │
│ QQuery 99 │ 10734.61 / 10809.07 ±59.52 / 10898.02 ms │ 10808.41 / 10867.26 ±51.69 / 10938.00 ms │    no change │
└───────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary           ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)           │ 34128.39ms │
│ Total Time (union-filter)   │ 34702.63ms │
│ Average Time (HEAD)         │   344.73ms │
│ Average Time (union-filter) │   350.53ms │
│ Queries Faster              │          0 │
│ Queries Slower              │          5 │
│ Queries with No Change      │         94 │
│ Queries with Failure        │          0 │
└─────────────────────────────┴────────────┘

Resource Usage

tpcds — base (merge-base)

Metric	Value
Wall time	171.0s
Peak memory	5.8 GiB
Avg memory	4.7 GiB
CPU user	271.0s
CPU sys	20.9s
Peak spill	0 B

tpcds — branch

Metric	Value
Wall time	173.8s
Peak memory	5.9 GiB
Avg memory	4.9 GiB
CPU user	276.0s
CPU sys	20.3s
Peak spill	0 B

File an issue against this benchmark runner

xiedeyantu · 2026-04-22T14:21:58Z

The security_audit failed appears unrelated to my PR; the main branch has the same issue.

xiedeyantu · 2026-04-22T23:47:47Z

The security_audit failed appears unrelated to my PR; the main branch has the same issue.

The main branch has been fixed via #21785.

xiedeyantu · 2026-04-23T02:02:14Z

@comphead I have made the changes based on the comments. Could you please take another look and let me know if it's ready?

comphead · 2026-04-23T15:25:53Z

+            Ok(None)
+        }
+        other => Ok(Some(UnionBranch {
+            source: strip_passthrough_nodes(other.clone()),


this might be expensive and redundant clone

comphead · 2026-04-23T15:39:50Z

I used Claude to make another round of review and there are some findings worth to address

  Performance Issues                                                                                                                                                                                       
                                                                                                                                                                                                           
  P1: plan.exists() pre-check — tradeoff, not a clear win                                                                                                                                                  
                                                                                                                                                                                                           
  File: unions_to_filter.rs:62-68                                                                                                                                                                          
                                                                                                                                                                                                           
  The pre-check walks the entire tree to look for Distinct::All, then rewrite_with_subqueries walks it again. For plans with Distinct::All (the rewrite target), this doubles traversal cost. For plans    
  without it, the pre-check is cheaper than spinning up the full rewriter. Net value depends on workload mix. Worth benchmarking with/without.
                                                                                                                                                                                                           
  P2: HashMap hashing entire LogicalPlan subtrees                                                                                                                                                          
   
  File: unions_to_filter.rs:109-128                                                                                                                                                                        
                  
  GroupKey contains a full LogicalPlan + Vec<Wrapper>. Every HashMap insert/lookup recursively hashes the entire plan tree. For 128 union branches (as in the benchmarks), this is O(N * tree_size) hash   
  work. For typical 2-10 branches, a Vec with linear scan + PartialEq would be faster and avoids the hashing cost entirely.
                                                                                                                                                                                                           
  P3: Possible double-projection from coerce + align                                                                                                                                                       
   
  File: unions_to_filter.rs:147-148                                                                                                                                                                        
                  
  coerce_plan_expr_for_schema may insert a Projection, then align_plan_to_schema may insert another one on top. In practice, align_plan_to_schema short-circuits if the schema already matches (line 299), 
  so this is likely benign. But it's worth verifying that coercion always produces a matching schema, or merging the two into one pass.
                                                                                                                                                                                                           
  P4: Recursive strip_passthrough_nodes (line 283)                                                                                                                                                         
   
  Should be a loop to avoid stack overflow on deeply nested plans and to reduce frame overhead. Simple mechanical fix.                                                                                     
                  
  P5: inner.clone() (line 83) and other.clone() (line 209)                                                                                                                                                 
                  
  Already discussed — both are unnecessary clones.                                                                                                                                                         
                  
  ---                                                                                                                                                                                                      
  Design / Correctness Concerns
                                                                                                                                                                                                           
  D1: Early bail-out abandons entire UNION on any failed branch
                                                                                                                                                                                                           
  File: unions_to_filter.rs:114-116                                                                                                                                                                        
                                                                                                                                                                                                           
  If one branch can't be extracted (has LIMIT, SORT, volatile), the entire UNION is left untouched — even if other branches could be merged. Example: 10 branches from same table with filters + 1 branch  
  with LIMIT = no optimization at all. A partial-merge strategy would be more effective, though more complex.
                                                                                                                                                                                                           
  D2: wrapper_projections_are_safe may miss window functions                                                                                                                                               
   
  File: unions_to_filter.rs:332-339                                                                                                                                                                        
                  
  Checks for volatility and subqueries but not window functions. However, in DataFusion's plan representation, window functions get their own Window node — they don't appear as expressions inside        
  Projection. So this is likely not a real issue in practice, but worth confirming with a test.
                                                                                                                                                                                                           
  D3: SubqueryAlias schema recomputed in wrap_branch                                                                                                                                                       
   
  File: unions_to_filter.rs:275-277                                                                                                                                                                        
                  
  wrap_branch uses SubqueryAlias::try_new() which recomputes the schema from the new input, discarding the stored schema from peel_wrappers. Since the source table is the same and only the filter        
  changed, the recomputed schema should match. Likely benign but a subtle coupling worth documenting.
                                                                                                                                                                                                           
  D4: GroupKey fragility with LogicalPlan as HashMap key                                                                                                                                                   
   
  If any upstream optimizer rule mutates plan internals (e.g., pushdown flags) between branches, two "same" sources could hash differently. Currently safe since the rule runs early in the pipeline, but  
  this is a latent risk.

ghost · 2026-05-05T13:43:19Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v53.1.0 (current)
       Built [  94.317s] (current)
     Parsing datafusion v53.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion v53.1.0 (baseline)
       Built [  94.132s] (baseline)
     Parsing datafusion v53.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.591s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 190.719s] datafusion
    Building datafusion-common v53.1.0 (current)
       Built [  32.115s] (current)
     Parsing datafusion-common v53.1.0 (current)
      Parsed [   0.057s] (current)
    Building datafusion-common v53.1.0 (baseline)
       Built [  31.977s] (baseline)
     Parsing datafusion-common v53.1.0 (baseline)
      Parsed [   0.058s] (baseline)
    Checking datafusion-common v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.718s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field OptimizerOptions.enable_unions_to_filter in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1075

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  66.042s] datafusion-common
    Building datafusion-optimizer v53.1.0 (current)
       Built [  26.013s] (current)
     Parsing datafusion-optimizer v53.1.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-optimizer v53.1.0 (baseline)
       Built [  25.487s] (baseline)
     Parsing datafusion-optimizer v53.1.0 (baseline)
      Parsed [   0.028s] (baseline)
    Checking datafusion-optimizer v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.151s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [  52.643s] datafusion-optimizer
    Building datafusion-sqllogictest v53.1.0 (current)
       Built [ 161.303s] (current)
     Parsing datafusion-sqllogictest v53.1.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v53.1.0 (baseline)
       Built [ 163.359s] (baseline)
     Parsing datafusion-sqllogictest v53.1.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.085s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 327.694s] datafusion-sqllogictest

xiedeyantu · 2026-05-14T12:58:39Z

Hi @comphead , should we still consider merging this PR?

comphead · 2026-05-14T15:08:25Z

Hi @xiedeyantu checking it

comphead

Thanks @xiedeyantu lets give a try! epic work!

comphead · 2026-05-14T15:09:56Z

Please resolve the conflicts

xiedeyantu · 2026-05-14T23:10:35Z

Please resolve the conflicts

@comphead Thanks for your review! I have resolved this conflicts.

xiedeyantu · 2026-05-18T23:06:13Z

@comphead This merge queue may have timed out.

Co-authored-by: Oleks V <comphead@users.noreply.github.com>

Co-authored-by: Copilot <copilot@github.com>

xiedeyantu · 2026-05-18T23:25:43Z

I resolved the conflict in the upgrade documentation.

xiedeyantu marked this pull request as draft March 20, 2026 10:24

xiedeyantu changed the title ~~Add configurable UNION DISTINCT to filter rewrite optimization~~ Add configurable UNION DISTINCT to FILTER rewrite optimization Mar 20, 2026

xiedeyantu force-pushed the union-filter branch 2 times, most recently from 4b113f8 to 3b11b5d Compare March 20, 2026 13:43

xiedeyantu marked this pull request as ready for review March 20, 2026 14:12

xiedeyantu commented Mar 22, 2026

View reviewed changes

comphead reviewed Apr 7, 2026

View reviewed changes

Comment thread datafusion/optimizer/src/unions_to_filter.rs

comphead reviewed Apr 7, 2026

View reviewed changes

Comment thread datafusion/optimizer/src/unions_to_filter.rs Outdated

comphead reviewed Apr 7, 2026

View reviewed changes

Comment thread datafusion/optimizer/src/unions_to_filter.rs Outdated

comphead reviewed Apr 7, 2026

View reviewed changes

xiedeyantu force-pushed the union-filter branch from 056e734 to 593b519 Compare April 10, 2026 16:07

xiedeyantu force-pushed the union-filter branch from 5626d11 to bf17571 Compare April 22, 2026 15:16

comphead reviewed Apr 23, 2026

View reviewed changes

Comment thread datafusion/optimizer/src/unions_to_filter.rs

xiedeyantu force-pushed the union-filter branch from 85a5029 to f0bb2ee Compare May 5, 2026 14:03

xiedeyantu requested a review from comphead May 5, 2026 14:18

comphead approved these changes May 14, 2026

View reviewed changes

xiedeyantu force-pushed the union-filter branch from f0bb2ee to dd7e3c2 Compare May 14, 2026 23:09

comphead added this pull request to the merge queue May 18, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks May 18, 2026

xiedeyantu and others added 4 commits May 19, 2026 07:19

Add configurable UNION DISTINCT to FILTER rewrite optimization

8aac147

Update datafusion/optimizer/src/unions_to_filter.rs

4cf91eb

Co-authored-by: Oleks V <comphead@users.noreply.github.com>

Addressed

46bcc10

Co-authored-by: Copilot <copilot@github.com>

fix doc

5b40c53

xiedeyantu force-pushed the union-filter branch from dd7e3c2 to 5b40c53 Compare May 18, 2026 23:20

Prettier

df4b243

xiedeyantu mentioned this pull request May 19, 2026

Release DataFusion 54.0.0 (Apr 2026 / May 2026) #21080

Open

47 tasks

comphead added this pull request to the merge queue May 19, 2026

Merged via the queue into apache:main with commit 6e58d74 May 19, 2026
40 checks passed

Conversation

xiedeyantu commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Example rewrite

Are these changes tested?

Are there any user-facing changes?

Uh oh!

xiedeyantu commented Mar 21, 2026

Uh oh!

xiedeyantu commented Mar 21, 2026

Uh oh!

alamb commented Mar 22, 2026

Uh oh!

xiedeyantu Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asolimando Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

xiedeyantu Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

xiedeyantu commented Mar 29, 2026

Uh oh!

xiedeyantu commented Mar 29, 2026

Uh oh!

xiedeyantu commented Mar 31, 2026

Uh oh!

xiedeyantu commented Apr 1, 2026

Uh oh!

comphead commented Apr 2, 2026

Uh oh!

xiedeyantu commented Apr 2, 2026

Uh oh!

comphead commented Apr 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

xiedeyantu commented Apr 7, 2026

Uh oh!

xiedeyantu commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

comphead commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

adriangbot commented Apr 7, 2026

Uh oh!

xiedeyantu commented Apr 22, 2026

Uh oh!

xiedeyantu commented Apr 22, 2026

Uh oh!

xiedeyantu commented Apr 23, 2026

Uh oh!

comphead Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

comphead commented Apr 23, 2026

Uh oh!

ghost commented May 5, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiedeyantu commented Mar 20, 2026 •

edited

Loading

xiedeyantu Mar 22, 2026 •

edited

Loading

xiedeyantu commented Apr 7, 2026 •

edited

Loading

ghost commented May 5, 2026 •

edited by github-actions Bot

Loading

xiedeyantu commented May 14, 2026 •

edited

Loading