Skip to content

Commit 8aa8515

Browse files
authored
infra: spark 4 already comes with spark connect jar (apache#2887)
<!-- Thanks for opening a pull request! --> <!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. --> <!-- Closes #${GITHUB_ISSUE_ID} --> # Rationale for this change Check against the `apache/spark:4.0.1` image. Save a couple MBs of download :D ``` ➜ iceberg-python git:(main) docker run --rm apache/spark:4.0.1 ls -la /opt/spark/jars | grep -i c onnect Unable to find image 'apache/spark:4.0.1' locally 4.0.1: Pulling from apache/spark fdf67ba0bcdc: Already exists 33c2d1bf8782: Already exists 9a562fdfee59: Already exists b6953a5f4a05: Already exists c00fd4799efc: Already exists bf65a9815f44: Already exists df261ba86031: Already exists 986913f619c8: Already exists 9c58c89601af: Already exists 4f4fb700ef54: Already exists 9da5037c6883: Already exists Digest: sha256:acfe7e06e95dd13aa32ee03c464766900d52fba15f58e421f8e1213ec041bb94 Status: Downloaded newer image for apache/spark:4.0.1 drwxr-xr-x 1 spark spark 2330 Sep 2 03:39 connect-repl -rw-r--r-- 1 spark spark 16577649 Sep 2 03:39 spark-connect_2.13-4.0.1.jar ``` ## Are these changes tested? ## Are there any user-facing changes? <!-- In the case of user-facing changes, please add the changelog label. -->
1 parent 70c63be commit 8aa8515

1 file changed

Lines changed: 0 additions & 2 deletions

File tree

dev/spark/Dockerfile

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ FROM apache/spark:${BASE_IMAGE_SPARK_VERSION}
2020
# Dependency versions - keep these compatible
2121
ARG ICEBERG_VERSION=1.10.1
2222
ARG ICEBERG_SPARK_RUNTIME_VERSION=4.0_2.13
23-
ARG SPARK_VERSION=4.0.1
2423
ARG HADOOP_VERSION=3.4.1
2524
ARG SCALA_VERSION=2.13
2625
ARG AWS_SDK_VERSION=2.24.6
@@ -43,7 +42,6 @@ RUN mkdir -p /home/iceberg/spark-events && \
4342

4443
# Required JAR dependencies
4544
ENV JARS_TO_DOWNLOAD="\
46-
org/apache/spark/spark-connect_${SCALA_VERSION}/${SPARK_VERSION}/spark-connect_${SCALA_VERSION}-${SPARK_VERSION}.jar \
4745
org/apache/iceberg/iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}/${ICEBERG_VERSION}/iceberg-spark-runtime-${ICEBERG_SPARK_RUNTIME_VERSION}-${ICEBERG_VERSION}.jar \
4846
org/apache/iceberg/iceberg-aws-bundle/${ICEBERG_VERSION}/iceberg-aws-bundle-${ICEBERG_VERSION}.jar \
4947
org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar \

0 commit comments

Comments
 (0)