Skip to content

Commit 29caede

Browse files
committed
[SPARK-57771][INFRA] Use Python 3.11 in spark-rm release image
### What changes were proposed in this pull request? This PR uses `Python 3.11` instead of 3.10 in `dev/create-release/spark-rm/Dockerfile` for Apache Spark 4.3.0+. ### Why are the changes needed? Since `Python 3.10` reaches end-of-life in October 2026 before Apache Spark 4.3.0 release, Apache Spark 3.4.0 dropped it. - #55914 So, ideally, we need to use Python 3.11+ in the release script. However, recently, on 2026-01-23, we share `Dockerfile.base` for all maintenance releases and `ubuntu:jammy` has only `Python 3.11.0rc1`. Since `Python 3.11.0rc1` is better than the EOL `Python 3.10`, this PR upgrades it only for Spark release. - #53905 https://github.com/apache/spark/blob/e28edaf0480e19abca83cd2b215450b85fdaf438/dev/create-release/spark-rm/Dockerfile.base#L30 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 Closes #56885 from dongjoon-hyun/SPARK-57771. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit a57c537) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 0bdb10c commit 29caede

1 file changed

Lines changed: 17 additions & 17 deletions

File tree

dev/create-release/spark-rm/Dockerfile

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
# Spark master (4.2-SNAPSHOT) release image
1919
# Extends the base image with:
2020
# - Java 17
21-
# - Python 3.10 with required packages
21+
# - Python 3.11 with required packages
2222

2323
FROM spark-rm-base:latest
2424

@@ -27,16 +27,16 @@ RUN apt-get update && apt-get install -y \
2727
openjdk-17-jdk-headless \
2828
&& rm -rf /var/lib/apt/lists/*
2929

30-
# Install Python 3.10
30+
# Install Python 3.11
3131
RUN apt-get update && apt-get install -y \
32-
python3.10 \
33-
python3.10-dev \
32+
python3.11 \
33+
python3.11-dev \
3434
python3-psutil \
3535
libpython3-dev \
3636
&& rm -rf /var/lib/apt/lists/*
3737

38-
# Install pip for Python 3.10
39-
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
38+
# Install pip for Python 3.11
39+
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
4040

4141
# Basic Python packages for master
4242
ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 pandas==2.3.3 scipy plotly<6.0.0 \
@@ -46,18 +46,18 @@ ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 pandas==2.3.3 scipy plotly
4646
ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0 protobuf==6.33.5 \
4747
googleapis-common-protos==1.71.0 graphviz==0.20.3"
4848

49-
# Install Python 3.10 packages
50-
RUN python3.10 -m pip install --ignore-installed 'blinker>=1.6.2' && \
51-
python3.10 -m pip install --ignore-installed 'six==1.16.0' && \
52-
python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting $CONNECT_PIP_PKGS && \
53-
python3.10 -m pip install 'torch<2.6.0' torchvision --index-url https://download.pytorch.org/whl/cpu && \
54-
python3.10 -m pip install deepspeed torcheval && \
55-
python3.10 -m pip cache purge
49+
# Install Python 3.11 packages
50+
RUN python3.11 -m pip install --ignore-installed 'blinker>=1.6.2' && \
51+
python3.11 -m pip install --ignore-installed 'six==1.16.0' && \
52+
python3.11 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting $CONNECT_PIP_PKGS && \
53+
python3.11 -m pip install 'torch<2.6.0' torchvision --index-url https://download.pytorch.org/whl/cpu && \
54+
python3.11 -m pip install deepspeed torcheval && \
55+
python3.11 -m pip cache purge
5656

5757
# Sphinx and documentation packages
5858
# Should unpin 'sphinxcontrib-*' after upgrading sphinx>5
5959
# See 'ipython_genutils' in SPARK-38517, 'docutils<0.18.0' in SPARK-39421
60-
RUN python3.10 -m pip install 'sphinx==4.5.0' mkdocs 'pydata_sphinx_theme>=0.13' \
60+
RUN python3.11 -m pip install 'sphinx==4.5.0' mkdocs 'pydata_sphinx_theme>=0.13' \
6161
sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0' \
6262
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.23.2' pyarrow pandas \
6363
'plotly>=4.8' 'docutils<0.18.0' 'flake8==3.9.0' 'mypy==1.19.1' 'pytest==7.1.3' \
@@ -67,9 +67,9 @@ RUN python3.10 -m pip install 'sphinx==4.5.0' mkdocs 'pydata_sphinx_theme>=0.13'
6767
'sphinxcontrib-htmlhelp==2.0.1' 'sphinxcontrib-qthelp==1.0.3' \
6868
'sphinxcontrib-serializinghtml==1.1.5'
6969

70-
# Set Python 3.10 as the default
71-
RUN ln -sf "$(which python3.10)" "/usr/local/bin/python" && \
72-
ln -sf "$(which python3.10)" "/usr/local/bin/python3"
70+
# Set Python 3.11 as the default
71+
RUN ln -sf "$(which python3.11)" "/usr/local/bin/python" && \
72+
ln -sf "$(which python3.11)" "/usr/local/bin/python3"
7373

7474
# Create user for release manager
7575
ARG UID

0 commit comments

Comments
 (0)