Note: DataFusion 55.0.0 has not been released yet. The information provided
in this section pertains to features and changes that have already been merged
to the main branch and are awaiting release in this version.
Previously, any context that needed a common type for a Decimal and a
floating-point value (Float16, Float32, or Float64) chose the decimal
type. This produced errors for legitimate floating-point inputs that have
no decimal representation:
-- Before: errored with "Cast error: Cannot cast to Decimal128(...). Overflowing on inf"
SELECT '1'::decimal(10,0) = arrow_cast('inf', 'Float64');DataFusion now coerces the decimal side to the floating-point type instead.
Decimal types cannot represent NaN, ±Infinity, or values outside their
precision/scale range, so the float is the only choice that is always
representable. This also matches the behavior of PostgreSQL, DuckDB, and the
existing rule for arithmetic operators in DataFusion.
Migration guide:
Most queries become more correct with no source change required — previously
errored queries (against NaN, ±Inf, or out-of-range values) now succeed,
and previously-coerced expressions that went through Decimal128(30, 15)
are now done in the natural float type.
The one behavior to be aware of: a decimal value with more than ~15–17
significant digits will lose precision when cast to Float64. If your query
needs to preserve full decimal precision, cast the float operand to the
decimal type explicitly, or use a decimal literal instead of a float-typed
value:
-- If high-precision decimal comparison matters, force the float into the
-- decimal domain explicitly:
SELECT * FROM t WHERE big_decimal_col > CAST(my_float AS DECIMAL(38, 10));See #14272 for the original report and discussion.