Skip to content

Commit 0f8a121

Browse files
adriangbclaude
adriangb
and
claude
authored
bench: add predicate_eval SQL micro-benchmark suite for conjunctive filter evaluation (apache#22704)
## Which issue does this PR close? <!-- No tracking issue; this is a standalone benchmark contribution. --> This PR does not close an issue. It adds a benchmark suite to support work and discussion around predicate ordering in filter evaluation (e.g. the static reordering in apache#22343 and the runtime/statistics-based reordering explored in apache#22698). It deliberately benchmarks *no specific implementation* — see below. ## Rationale for this change Conjunctive (`AND`) filter evaluation in `FilterExec` is a left-deep `BinaryExpr(And)` chain, and the order conjuncts are evaluated in can change runtime by large factors: once a leading conjunct passes few enough rows the batch is physically compacted before the rest, so a cheap-and-selective predicate evaluated early saves later predicates work. Predicate ordering is therefore an active area (static heuristics, runtime/adaptive schemes, cost models). There is currently no benchmark suite that isolates the dimensions that drive this. Existing macro-benchmarks (TPC-H/DS, ClickBench) only incidentally exercise filter ordering, so they can't show *why* a change to ordering helped or hurt, or guard the order-insensitive case against regressions. ## What changes are included in this PR? A new SQL benchmark suite, `benchmarks/sql_benchmarks/predicate_eval`, built on the existing `.benchmark` template framework (no engine code, no new Rust). It sets no engine config of its own and measures DataFusion's built-in short-circuit by default; a system under test is toggled purely via its native `DATAFUSION_EXECUTION_*` env var (the bench harness builds its `SessionContext` with `SessionConfig::from_env`), so the same scenarios can characterise the baseline, a static heuristic, an adaptive scheme, or a cost model and be compared apples-to-apples. It is organised into 10 subgroups (select with `BENCH_SUBGROUP`), each varying one property of conjunctive filter evaluation while holding the others fixed: | Subgroup | What it varies (others held fixed) | |---|---| | `costsel` | cost and selectivity point in different directions (expensive predicate is the selective one) | | `cost` | per-predicate cost, at equal selectivity | | `selectivity` | per-predicate selectivity, at equal cost | | `cardinality` | conjunct count `k = 2/4/8/16` | | `width` | string-column width (`PRED_FILL` = 2 / 30 / 170 chars) | | `scale` | row count `5k / 100k / 5M / 50M` | | `neutral` | predicates are interchangeable (equal cost, none selective) — an order-insensitive control | | `correlation` | conditional vs marginal selectivity (independent / positively / anti-correlated) | | `drift` | selectivity that changes across the scan | | `nulls` | null density (two- vs three-valued predicate results) | Each query's comment notes the per-predicate cost/selectivity that the data generation hides from the SQL. Data is synthetic and generated inline by each subgroup's load SQL (no external files); `PRED_ROWS` sizes it and `PRED_FILL` sets string width. Wired into `bench.sh` (`./bench.sh run predicate_eval`) and documented in `benchmarks/sql_benchmarks/README.md`. The design was informed by surveying how Velox drives the analogous decision (it ranks by cycles-per-row-eliminated, `time / (rows_in - rows_out)`). > Note: the `scale` subgroup's `q52`/`q53` build 5M / 50M-row tables (the latter > ~9 GB); run a single point with `BENCH_QUERY` if that is too heavy. ## Are these changes tested? These are benchmark definitions, not engine code. Each `.benchmark` includes an `assert` that the generated table is non-empty, and every subgroup was run locally at small `PRED_ROWS` to confirm the suite parses, loads, asserts, and executes end-to-end. The queries are order-invariant (`SELECT count(*) ...`), so any predicate-ordering system can also be checked for correctness by diffing counts with the optimization on vs. off. ## Are there any user-facing changes? No. This only adds an opt-in benchmark suite and its documentation; no public API, engine behavior, or default configuration changes. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent a630994 commit 0f8a121

60 files changed

Lines changed: 543 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎benchmarks/bench.sh‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,10 @@ push_down_topk: Benchmark of ORDER BY ... LIMIT over outer joins on TPC-
103103
external_aggr: External aggregation benchmark on TPC-H dataset (SF=1)
104104
wide_schema: Small-projection queries on a wide synthetic dataset (1024 cols × 256 files) — measures per-file metadata overhead
105105
(runs both 'wide' and 'narrow' subgroups: narrow is an internal baseline; the wide-vs-narrow ratio is the signal)
106+
predicate_eval: Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an
107+
adaptive predicate-ordering system behaves across them (see https://github.com/apache/datafusion/issues/11262)
108+
(subgroups via BENCH_SUBGROUP: costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift)
109+
(toggle a system under test with its native DATAFUSION_* env var; size data with PRED_ROWS, string width with PRED_FILL)
106110
107111
# ClickBench Benchmarks
108112
clickbench_1: ClickBench queries against a single parquet file
@@ -246,6 +250,10 @@ main() {
246250
wide_schema)
247251
data_wide_schema
248252
;;
253+
predicate_eval)
254+
# Data is generated inline by the suite's load SQL.
255+
echo "predicate_eval: no external data to generate"
256+
;;
249257
tpcds)
250258
data_tpcds
251259
;;
@@ -463,6 +471,9 @@ main() {
463471
wide_schema)
464472
run_wide_schema
465473
;;
474+
predicate_eval)
475+
run_predicate_eval
476+
;;
466477
tpcds)
467478
run_tpcds
468479
;;
@@ -800,6 +811,33 @@ run_push_down_topk() {
800811
bash -c "$SQL_CARGO_COMMAND"
801812
}
802813

814+
# Runs the predicate_eval benchmark suite: conjunctive (AND) filter-evaluation
815+
# micro-benchmarks where each subgroup is a different predicate pattern, used to
816+
# test how an adaptive predicate-ordering system behaves across them (see
817+
# https://github.com/apache/datafusion/issues/11262). Data is generated inline
818+
# by the suite's load SQL, so there is no data step.
819+
#
820+
# By default the suite measures DataFusion's built-in left-deep AND short-circuit
821+
# and sets no engine config of its own. To evaluate a system under test, export
822+
# its native DATAFUSION_* config before invoking bench.sh -- the harness reads
823+
# SessionConfig::from_env, and that environment is inherited here, e.g.
824+
# DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING=true ./bench.sh run predicate_eval
825+
# Suite-specific knobs (string-substituted into the load SQL, not engine config):
826+
# BENCH_SUBGROUP run one subgroup (costsel, cost, selectivity, cardinality,
827+
# width, scale, neutral, correlation, drift)
828+
# PRED_ROWS synthetic row count (default 1_000_000; the scale subgroup
829+
# overrides this per query)
830+
# PRED_FILL filler chars per marker = string-column width knob
831+
run_predicate_eval() {
832+
echo "Running predicate_eval benchmark (subgroup=${BENCH_SUBGROUP:-all}, rows=${PRED_ROWS:-1000000})..."
833+
debug_run env BENCH_NAME=predicate_eval \
834+
${BENCH_SUBGROUP:+BENCH_SUBGROUP="${BENCH_SUBGROUP}"} \
835+
PRED_ROWS="${PRED_ROWS:-1000000}" \
836+
${PRED_FILL:+PRED_FILL="${PRED_FILL}"} \
837+
${QUERY:+BENCH_QUERY="${QUERY}"} \
838+
bash -c "$SQL_CARGO_COMMAND"
839+
}
840+
803841
# Runs the tpch in memory (needs tpch parquet data)
804842
run_tpch_mem() {
805843
SCALE_FACTOR=$1

‎benchmarks/sql_benchmarks/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ in the community:
4343
| `tpcds` | TPC‑DS queries |
4444
| `tpch` | TPC‑H queries |
4545
| `wide_schema` | Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
46+
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`BENCH_SUBGROUP`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`. Toggle a system under test with its native `DATAFUSION_*` env var |
4647

4748
# Running Benchmarks
4849

@@ -95,6 +96,8 @@ Some benchmarks use custom environment variables as outlined below:
9596
| BENCH_SORTED | Used in the sort_tpch benchmark to indicate whether the lineitem table should be sorted. | false |
9697
| SORTED_BY | Used in the clickbench_sorted benchmark to indicate the column to sort by. | `EventTime` |
9798
| SORTED_ORDER | Used in the clickbench_sorted benchmark to indicate the sort order of the column. | `ASC` |
99+
| PRED_ROWS | Used in the predicate_eval benchmark to size the synthetic table (the `scale` subgroup overrides this per query). | `1000000` |
100+
| PRED_FILL | Used in the predicate_eval benchmark as the string-column width knob (filler chars per marker). | `30` |
98101

99102
## How it works
100103

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q30.benchmark‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
subgroup cardinality
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
SUBGROUP=cardinality
5+
QPAD=30
6+
DATASET=ints
7+
NAME=cardinality_q30_k2

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q31.benchmark‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
subgroup cardinality
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
SUBGROUP=cardinality
5+
QPAD=31
6+
DATASET=ints
7+
NAME=cardinality_q31_k4

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q32.benchmark‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
subgroup cardinality
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
SUBGROUP=cardinality
5+
QPAD=32
6+
DATASET=ints
7+
NAME=cardinality_q32_k8

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q33.benchmark‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
subgroup cardinality
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
SUBGROUP=cardinality
5+
QPAD=33
6+
DATASET=ints
7+
NAME=cardinality_q33_k16

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q70.benchmark‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
subgroup correlation
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
SUBGROUP=correlation
5+
QPAD=70
6+
DATASET=corr
7+
NAME=correlation_q70_independent

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q71.benchmark‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
subgroup correlation
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
SUBGROUP=correlation
5+
QPAD=71
6+
DATASET=corr
7+
NAME=correlation_q71_positive

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q72.benchmark‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
subgroup correlation
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
SUBGROUP=correlation
5+
QPAD=72
6+
DATASET=corr
7+
NAME=correlation_q72_anti

‎benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q10.benchmark‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
subgroup cost
2+
3+
template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template
4+
SUBGROUP=cost
5+
QPAD=10
6+
DATASET=mixed
7+
NAME=cost_q10_expensive_first

0 commit comments

Comments
 (0)