Is your feature request related to a problem or challenge?
Related:
Partitioning::Range can provide the key-colocation guarantee required by Distribution::KeyPartitioned: rows with the same required key are placed in the same partition when the range key is compatible with the required key.
Partitioning::Range does not generally satisfy Distribution::KeyPartitioned through Partitioning::satisfaction. This is done on a operator-to-operator basis. Now we are able to generally satisfy this and consolidate the logic here.
Describe the solution you'd like
Allow Partitioning::Range to generally satisfy Distribution::KeyPartitioned in Partitioning::satisfaction.
The prvate unary implementations are here:
This issue should:
- Add general
Range satisfaction for Distribution::KeyPartitioned
- Support exact satisfaction, such as
Range([a]) satisfying KeyPartitioned([a])
- Support equivalence-aware satisfaction, such as
a = b allowing Range([a]) to satisfy KeyPartitioned([b])
- Support subset satisfaction, such as
Range([a]) satisfying KeyPartitioned([a, b]), when subset satisfaction is enabled
- Reject incompatible keys, such as
Range([a]) with KeyPartitioned([b]) when a != b
- Remove the temporary private range satisfaction helpers and TODOs added during unary rollout work
Multi-input safety
This change must not allow multi-input operators to skip repartitioning based only on each child's independent KeyPartitioned satisfaction result. There is discussion about this in #23184
For unary operators we ask:
Does this input satisfy this operator's key distribution requirement?
For multi-input operators, such as joins, this satisfaction is not enough. We must also prove the inputs are compatible by partition index:
left satisfies KeyPartitioned(left_key)
right satisfies KeyPartitioned(right_key)
left partition i is compatible with right partition i
That compatibility should be enforced through Partitioning::compatible_with or an equivalent co-partitioning check before avoiding repartitioning for multi-input execution. This will be a separate effort.
Describe alternatives you've considered
Another alternative is to enable general Range satisfaction immediately. That is risky because joins and other multi-input operators need an additional co-partitioning check beyond each child's independent satisfaction.
Additional context
The goal is to use private unary rollouts to prove the semantics first, then move the behavior into Partitioning::satisfaction and delete the temporary helpers.
Is your feature request related to a problem or challenge?
Related:
Distribution::HashPartitionedtoDistribution::KeyPartitionedmigration: Replace / rename HashPartitioned distribution as KeyPartitioned #23236Partitioning::Range#23239Partitioning::Rangecan provide the key-colocation guarantee required byDistribution::KeyPartitioned: rows with the same required key are placed in the same partition when the range key is compatible with the required key.Partitioning::Rangedoes not generally satisfyDistribution::KeyPartitionedthroughPartitioning::satisfaction. This is done on a operator-to-operator basis. Now we are able to generally satisfy this and consolidate the logic here.Describe the solution you'd like
Allow
Partitioning::Rangeto generally satisfyDistribution::KeyPartitionedinPartitioning::satisfaction.The prvate unary implementations are here:
Partitioning::Range#23239WindowAggExecandBoundedWindowAggExecPartitionedTopKExecThis issue should:
Rangesatisfaction forDistribution::KeyPartitionedRange([a])satisfyingKeyPartitioned([a])a = ballowingRange([a])to satisfyKeyPartitioned([b])Range([a])satisfyingKeyPartitioned([a, b]), when subset satisfaction is enabledRange([a])withKeyPartitioned([b])whena != bMulti-input safety
This change must not allow multi-input operators to skip repartitioning based only on each child's independent
KeyPartitionedsatisfaction result. There is discussion about this in #23184For unary operators we ask:
For multi-input operators, such as joins, this satisfaction is not enough. We must also prove the inputs are compatible by partition index:
That compatibility should be enforced through
Partitioning::compatible_withor an equivalent co-partitioning check before avoiding repartitioning for multi-input execution. This will be a separate effort.Describe alternatives you've considered
Another alternative is to enable general
Rangesatisfaction immediately. That is risky because joins and other multi-input operators need an additional co-partitioning check beyond each child's independent satisfaction.Additional context
The goal is to use private unary rollouts to prove the semantics first, then move the behavior into
Partitioning::satisfactionand delete the temporary helpers.