[DISCUSSION] Extending Partitioning to Support More Variants

[gene-bordegaray]

I’d like to restart the partitioning side of this discussion separately from the dynamic-filter PRs and threads (see #21207 for more background)

The main issue seems to be that DataFusion cannot always represent the physical partitioning that some data sources actually have. In our case, range-partitioned data has had to claim Partitioning::Hash, which lets us avoid repartitioning but makes later optimizer and dynamic-filter decisions brittle due to this false advertisement.

Today we are in this scenario:

Data is actually range-partitioned:
- partition 0: key < 100
- partition 1: 100 <= key < 200
- partition 2: 200 <= key < 300

To mimic this we tell DataFusion our partitioning properties are:
- Partitioning::Hash([key], 3)

Those are not the same partitioning scheme.

The general goal should be that DataFusion should be able to describe physical partitioning truthfully, be able to compare two partitionings for compatibility, and use that information only when compatibility is proven.

This is well shown in dynamic filtering:

Build side partitioning:
  p0: key < 100
  p1: 100 <= key < 200
  p2: 200 <= key < 300

Probe side partitioning:
  p0: key < 100
  p1: 100 <= key < 200 
  p2: 200 <= key < 300

These are compatible as every partition X on the build side is compatible with partition X on the probe side. With this information, optimizations can safely use partition-local behavior:

if partitioning is compatible:
  use partition-local filter (more selective)
else:
  use global/safe fallback

A possible direction is to evolve Partitioning toward an extensible abstraction, such as a PhysicalPartitioning trait, and add range partitioning as the first use case. Longer term, built-ins like hash partitioning could move behind the same abstraction.

This would give us a cleaner foundation for:

• range/file partitioned sources
• partition-local dynamic filters when both sides share the same partition map Gene.bordegaray/2026/04/partition routed dynamic filters #21832

Does this direction seem interesting to others in the community nd any thoughts on this proposal?

cc: @NGA-TRAN @alamb @jayshrivastava @gabotechs @adriangb

[NGA-TRAN]

Great description. The exact use case at DataDog. We plan to use this a lot.

[gene-bordegaray]

I have added a draft PR #22002 which shows my vision for what partitioning can look like going forward and how this will apply to an optimization like dynamic filter routing. The API design was done by me but implementation details by AI and is not meant to be a mergeable PR. I provided a description of the API, would love any suggestions/ideas on this general approach.

I separated it into 4 commits to show the logical progression of the implementation:

1. Adding Range partitioning variant that models what a custom partitioning trait would look like
2. Introduce the general partitioning trait using this model and have range partitioning implement it
3. Have file preserved partitioning use range partitioning rather than false advertising hash
4. Route dynamic filters using partitioning compatibility and create partitioning-local filters when range partitioned

[alamb]

I left some comments on #22002

[alamb]

I think it would also help to decide sooner rather than later if we want a general partitioning trait, or if we are just going to hard code Range Partitioning. I think either could work.

If we are going to go with a trait, it might be good to declare that we eventually want to shoot to remove all special cases for hash partitioning 🤔

[NGA-TRAN]

I think it would also help to decide sooner rather than later if we want a general partitioning trait, or if we are just going to hard code Range Partitioning. I think either could work.

I vote for a general partitioning trait

[adriangb]

I'm not sure if it should be a trait or an eum. I generally prefer an explicit enum if the trait doesn't generalize well. E.g. if we can't point to an obvious 3rd use case for the trait that an external system could conceivably implement and where everything would work for them, I fear a trait is the wrong choice.

@NGA-TRAN could you share why you prefer a trait to an enum here? Maybe you know some additional partitioning strategies that are commonly used in the database world that I don't know of.

[NGA-TRAN]

@adriangb : Can you provide an example of enum? If it covers our use cases, we are happy to go with it.

Currently, our range come from 2 columns (or 2 expressions). For example (time, int_val). So we can have these ranges:

• ( [2026.05.01 - 2026.05.07] , [1 - 100] )
• ( [2026.05.01 - 2026.05.07] , [101 - 200] )
• ...
• ( [2026.05.08 - 2026.05.15] , [1 - 500] )
• ( [2026.05.08 - 2026.05.15] , [501 - 1000] )
• ...

As long as they do not overlaps, they are considered as ranges. We can definitely map those into simple ranges or indexes and use those simple ones in DF instead.

[adriangb]

I don't know that the enum is any different than the trait in terms of representation. I.e. if we can represent it as a trait we can represent it as an enum. The question is more if we want to hard code all of the handled cases or leave it open for users to add more down the road.

I'd defer to @gene-bordegaray as to what an enum version would look like but I think something like this:

  pub enum Partitioning {
      RoundRobinBatch(usize),
      Hash {
          exprs: Vec<Arc<dyn PhysicalExpr>>,
          partition_count: usize,
      },
      Range {
          exprs: Vec<Arc<dyn PhysicalExpr>>,
          partitions: Vec<RangePartition>,
      },
  }

  pub struct RangePartition {
      /// One interval per `Range.exprs` entry.
      ///
      /// For exprs = [time, int_val], this can represent:
      /// time in [2026-05-01, 2026-05-07]
      /// AND int_val in [1, 100]
      pub bounds: Vec<RangeInterval>,
  }

  pub struct RangeInterval {
      pub lower: Option<RangeBound>,
      pub upper: Option<RangeBound>,
  }

  pub struct RangeBound {
      pub value: ScalarValue,
      pub inclusive: bool,
  }

[NGA-TRAN]

The enum Gene proposed looks good. As long as there are PhysicalExpr, it will work

Range {
          exprs: Vec<Arc<dyn PhysicalExpr>>,
          partitions: Vec<RangePartition>,
      },