Skip to content

Commit 81101bc

Browse files
committed
[SPARK-55757][CORE] Improve spark.task.cpus validation
### What changes were proposed in this pull request? This PR aim to improve `spark.task.cpus` validation. ### Why are the changes needed? Currently, Apache Spark throws `java.lang.ArithmeticException` for the invalid `spark.task.cpus`. We had better provide actionable direction. **BEFORE** ``` $ bin/spark-shell -c spark.task.cpus=0 26/02/28 09:00:56 ERROR SparkContext: Error initializing SparkContext. java.lang.ArithmeticException: / by zero at org.apache.spark.resource.ResourceUtils$.warnOnWastedResources(ResourceUtils.scala:444) ``` **AFTER** ``` $ bin/spark-shell -c spark.task.cpus=0 ... 26/02/28 09:02:15 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkIllegalArgumentException: [INVALID_CONF_VALUE.REQUIREMENT] The value '0' in the config "spark.task.cpus" is invalid. The number of task CPUs must be positive. SQLSTATE: 22022 ``` ### Does this PR introduce _any_ user-facing change? This only changes the error message before starting Spark job. ### How was this patch tested? Pass the CIs with newly added test case. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: `Gemini 3.1 Pro (High)` on `Antigravity`. Closes #54559 from dongjoon-hyun/SPARK-55757. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 3f54ac5 commit 81101bc

2 files changed

Lines changed: 13 additions & 1 deletion

File tree

core/src/main/scala/org/apache/spark/internal/config/package.scala

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -711,7 +711,11 @@ package object config {
711711
.createWithDefault(false)
712712

713713
private[spark] val CPUS_PER_TASK =
714-
ConfigBuilder("spark.task.cpus").version("0.5.0").intConf.createWithDefault(1)
714+
ConfigBuilder("spark.task.cpus")
715+
.version("0.5.0")
716+
.intConf
717+
.checkValue(_ > 0, "Number of cores to allocate for each task should be positive.")
718+
.createWithDefault(1)
715719

716720
private[spark] val DYN_ALLOCATION_ENABLED =
717721
ConfigBuilder("spark.dynamicAllocation.enabled")

core/src/test/scala/org/apache/spark/SparkContextSuite.scala

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1475,6 +1475,14 @@ class SparkContextSuite extends SparkFunSuite with LocalSparkContext with Eventu
14751475
assert(!sc.conf.get(SparkLauncher.EXECUTOR_EXTRA_JAVA_OPTIONS).contains("-Dfoo=bar"))
14761476
sc.stop()
14771477
}
1478+
1479+
test("SPARK-55757: Improve `spark.task.cpus` validation") {
1480+
val conf = new SparkConf().setAppName("test").setMaster("local").set(CPUS_PER_TASK, 0)
1481+
val m = intercept[SparkIllegalArgumentException] {
1482+
sc = new SparkContext(conf)
1483+
}.getMessage
1484+
assert(m.contains("Number of cores to allocate for each task should be positive."))
1485+
}
14781486
}
14791487

14801488
object SparkContextSuite {

0 commit comments

Comments
 (0)