Skip to content

Commit aa702a7

Browse files
committed
[SPARK-57798] Exclude net.sf.py4j dependency
### What changes were proposed in this pull request? This PR excludes the transitive `net.sf.py4j:py4j` dependency from the operator shadow jar. ### Why are the changes needed? `py4j` is PySpark's Python-to-JVM gateway, pulled in transitively by `spark-core`. It only runs inside the Spark driver pod, never in the operator JVM, yet all 71 `py4j/` classes were bundled into the operator fat jar as dead weight. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 Closes #724 from dongjoon-hyun/SPARK-57798. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 63fd088 commit aa702a7

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

spark-submission-worker/build.gradle

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ dependencies {
3030
exclude group: "commons-codec"
3131
exclude group: "commons-collections", module: "commons-collections"
3232
exclude group: "io.fabric8"
33+
exclude group: "net.sf.py4j"
3334
exclude group: "org.apache.avro"
3435
exclude group: "org.apache.commons", module: "commons-compress"
3536
exclude group: "org.apache.logging.log4j"

0 commit comments

Comments
 (0)