Return None for cardinality overflow#22309
Conversation
|
Thank you @jx2lee for working on this. Can you add the original test from the issue to make sure the change is working. |
4711b0b to
98ce8fc
Compare
| 01)ProjectionExec: expr=[c1@0 >= 2 AND c1@0 <= 3 as select_between_data.c1 BETWEEN Int64(2) AND Int64(3)] | ||
| 02)--DataSourceExec: partitions=1, partition_sizes=[1] | ||
|
|
||
| # regression test: full i64 BETWEEN bounds should not overflow |
There was a problem hiding this comment.
@jx2lee
Thanks for the fix here. I verified the core invariant around Interval::cardinality() and the new checked_add at the final inclusive + 1 layer correctly returns None instead of panicking or wrapping when the inclusive cardinality exceeds u64::MAX. This covers the full i64 range while preserving existing max-but-representable ranges.
I just had one small non-blocking suggestion.
| } | ||
|
|
||
| #[test] | ||
| fn test_cardinality_full_i64_range_does_not_overflow() -> Result<()> { |
There was a problem hiding this comment.
Nice catch on the overflow edge case. One additional regression test that could be helpful would be a sibling case for ScalarValue::UInt64(Some(0))..=ScalarValue::UInt64(Some(u64::MAX)) returning None.
The fix applies more generally to any integer or temporal range where distance == u64::MAX, so adding the unsigned boundary case would help document that broader invariant and guard against future regressions there as well.
There was a problem hiding this comment.
@kosiew Thanks for reviewing ! I added testcase for u64
dbd8bbc to
615665c
Compare
Which issue does this PR close?
Closes #22232
(cc @Dandandan )
Rationale for this change
Prevent
Interval::cardinality()from overflowing on the fulli64range.What changes are included in this PR?
checked_add(1)instead of+ 1i64rangeAre these changes tested?
cargo fmt --all --checkcargo test -p datafusion-expr-common test_cardinality_full_i64_range_does_not_overflow --libcargo clippy --all-targets --all-features -- -D warningscould not run locally becausecmakeis not installedAre there any user-facing changes?