Skip to content

Commit 1dd43a5

Browse files
adriangbclaude
andcommitted
feat(pruning): PruningConjunction with per-leaf observation
Adds `PruningConjunction` — an AND of tagged `PruningPredicate` leaves — plus a `PruningObserver` trait fired once per leaf actually evaluated. The AND short-circuits once every container is pruned; short-circuited leaves are not observed, so per-conjunct stats are never biased by predicates that did not run. Construction is via a small builder: let conj = PruningConjunction::builder(schema) .push(filter_id, expr) // runs try_new, skips always-true / errors .build(); // Option<PruningConjunction> `PruningConjunction::single(predicate)` wraps one already-built predicate for the untagged path, which uses `prune()` / `NoopObserver` at zero overhead. `ConjunctStatsObserver` accumulates `PerConjunctPruneStats { tag, containers_seen, containers_pruned }`. OR and NOT are intentionally *not* structural nodes: they are handled inside a leaf via `PruningPredicate::try_new`, which both prunes better (cross-branch reasoning) and keeps the per-leaf "containers pruned" count sound (it is only monotonic under AND). Per-branch OR short-circuit, where it is useful, belongs to the row-filter decode path over exact masks — not here. Replaces the placeholder-`true`-literal `sub_predicates` design and its double traversal from the earlier draft (#22235). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 50d74a7 commit 1dd43a5

2 files changed

Lines changed: 449 additions & 0 deletions

File tree

datafusion/pruning/src/lib.rs

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,14 @@
1818
#![cfg_attr(test, allow(clippy::needless_pass_by_value))]
1919

2020
mod file_pruner;
21+
mod pruning_conjunction;
2122
mod pruning_predicate;
2223

2324
pub use file_pruner::FilePruner;
25+
pub use pruning_conjunction::{
26+
ConjunctStatsObserver, NoopObserver, PerConjunctPruneStats, PruningConjunction,
27+
PruningConjunctionBuilder, PruningObserver, Tag,
28+
};
2429
pub use pruning_predicate::{
2530
PredicateRewriter, PruningPredicate, PruningStatistics, RequiredColumns,
2631
UnhandledPredicateHook, build_pruning_predicate,

0 commit comments

Comments
 (0)