Implement getErrorClass explicitly (CI validation on fork)#1
Open
dev-ankit wants to merge 2 commits into
Open
Conversation
…ions
PulsarIllegalStateException and PulsarIllegalArgumentException implement
SparkThrowable but override only getCondition, relying on the deprecated
SparkThrowable.getErrorClass default (which delegates to getCondition).
Some Spark distributions — notably Databricks Runtime 18 (Spark 4.1) — leave
getErrorClass abstract rather than providing that default. When one of these
exceptions escapes a task, Spark's TaskResultGetter virtual-calls
getErrorClass(), finds no implementation, and raises:
java.lang.AbstractMethodError:
'java.lang.String org.apache.spark.SparkThrowable.getErrorClass()'
at org.apache.spark.sql.pulsar.PulsarIllegalStateException.getErrorClass
This kills the result-getter thread, so the original failure (e.g. an
incompatible-schema produce) is never reported and the Spark job hangs
indefinitely instead of failing.
Implement getErrorClass explicitly (delegating to the error class, same as
getCondition) so the body lives on the class itself. This is harmless on OSS
Spark, where it simply overrides the deprecated default, and restores correct
error propagation on distributions that drop it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PulsarExceptions.pulsarSinkInvalidSchema references the error class "PULSAR_SINK_INVALID_SCHEMA", which is not present in error/pulsar-error-classes.json (only "PULSAR_SINK_INVALID_SCHEMA_TYPE" is), so constructing it raises INTERNAL_ERROR. Use pulsarProviderInvalidSaveMode, whose error class is defined, to exercise PulsarIllegalArgumentException. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
CI validation on the fork for the same fix proposed upstream in streamnative#198.
PulsarIllegalStateException/PulsarIllegalArgumentExceptionimplementSparkThrowablebut override onlygetCondition, relying on the deprecatedSparkThrowable.getErrorClassdefault. Spark distributions that leavegetErrorClassabstract (e.g. Databricks Runtime 18 / Spark 4.1) then raiseAbstractMethodErrorwhen one of these exceptions escapes a task, which hangs the job. This implementsgetErrorClassexplicitly on both classes.Modifications
getErrorClassoverride on both connectorSparkThrowablesubclasses.PulsarExceptionsSuiteregression test.Verifying this change
PulsarExceptionsSuite; full./mvnw clean verifypasses locally (123 tests + 2 ignored).Documentation
doc-requiredno-need-docdoc