fix: surface SparkArithmeticException(DIVIDE_BY_ZERO) for divide-by-zero in dispatched ScalaUDF path#4653
fix: surface SparkArithmeticException(DIVIDE_BY_ZERO) for divide-by-zero in dispatched ScalaUDF path#46530lai0 wants to merge 5 commits into
Conversation
|
Thanks @andygrove. Just to confirm, should I revert both the #4517 workaround (SQLQueryTestSuite DIVIDE_BY_ZERO) and the #4516 workaround (SQLQuerySuite UDF call-count) from dev/diffs? The #4517 one is clearly no longer needed with this fix, but wanted to check on #4516 since that issue is still open. |
|
I've reverted the #4514 diff changes. After the revert,
This PR fixes the error type for the project/aggregate path (e.g. Should I extend this PR to cover the filter path too, or file a separate issue and keep this test excluded for now? |
Which issue does this PR close?
Closes #4517 .
Rationale for this change
DataFusion 53+ wraps errors with
.context(...), producingDataFusionError::Context(_, Box<inner>).Previous handling mostly matched top-level
DataFusionError::External(...), soContext-wrapped Spark errors could bypass typed conversion and leak asCometNativeExceptioninstead of Spark’sSparkArithmeticException(DIVIDE_BY_ZERO).What changes are included in this PR?
native/core/src/execution/expressions/arithmetic.rsCheckedBinaryExpr::evaluatenow matches genericErr(e)(when query context exists), then recursively extractsSparkErrorviaextract_spark_error.DataFusionError::Contextand nestedExternal.Err(e)as-is (instead of re-wrapping asExternal).native/jni-bridge/src/errors.rsSparkPayloadandextract_spark_payloadto recursively unwrapContext/ nestedExternal.throw_exceptionnow handles allCometError::DataFusion { source }throughextract_spark_payload, covering:JavaExceptionpassthroughSparkErrorWithContextJSON throw pathSparkErrorJSON throw pathspark/src/test/scala/org/apache/comet/CometCodegenSuite.scaladivide-by-zero through dispatched ScalaUDF surfaces SparkArithmeticException (#4517).SELECT 1 / identity_int(a)over data containing zero, and validates exception propagation semantics.How are these changes tested?
./mvnw test -Dtest=none -Dsuites=org.apache.comet.CometCodegenSuite