Commit 81101bc
committed
[SPARK-55757][CORE] Improve
### What changes were proposed in this pull request?
This PR aim to improve `spark.task.cpus` validation.
### Why are the changes needed?
Currently, Apache Spark throws `java.lang.ArithmeticException` for the invalid `spark.task.cpus`. We had better provide actionable direction.
**BEFORE**
```
$ bin/spark-shell -c spark.task.cpus=0
26/02/28 09:00:56 ERROR SparkContext: Error initializing SparkContext.
java.lang.ArithmeticException: / by zero
at org.apache.spark.resource.ResourceUtils$.warnOnWastedResources(ResourceUtils.scala:444)
```
**AFTER**
```
$ bin/spark-shell -c spark.task.cpus=0
...
26/02/28 09:02:15 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkIllegalArgumentException: [INVALID_CONF_VALUE.REQUIREMENT] The value '0' in the config "spark.task.cpus" is invalid. The number of task CPUs must be positive. SQLSTATE: 22022
```
### Does this PR introduce _any_ user-facing change?
This only changes the error message before starting Spark job.
### How was this patch tested?
Pass the CIs with newly added test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: `Gemini 3.1 Pro (High)` on `Antigravity`.
Closes #54559 from dongjoon-hyun/SPARK-55757.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>spark.task.cpus validation1 parent 3f54ac5 commit 81101bc
2 files changed
Lines changed: 13 additions & 1 deletion
File tree
- core/src
- main/scala/org/apache/spark/internal/config
- test/scala/org/apache/spark
Lines changed: 5 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
711 | 711 | | |
712 | 712 | | |
713 | 713 | | |
714 | | - | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
715 | 719 | | |
716 | 720 | | |
717 | 721 | | |
| |||
Lines changed: 8 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1475 | 1475 | | |
1476 | 1476 | | |
1477 | 1477 | | |
| 1478 | + | |
| 1479 | + | |
| 1480 | + | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
1478 | 1486 | | |
1479 | 1487 | | |
1480 | 1488 | | |
| |||
0 commit comments