Skip to content

Commit 4230fac

Browse files
sarutakzhengruifeng
authored andcommitted
[SPARK-56513][INFRA] Pin meson<1.11.0 and lxml==4.9.4 in Dockerfile for PyPy to recover CI
### What changes were proposed in this pull request? This PR pin `meson<1.11.0` and `lxml==4.9.4` to recover CI. ### Why are the changes needed? Recently, CI fails due to docker image building failure. https://github.com/apache/spark/actions/runs/24489111484/job/71571576157 The root causes are: * meson `1.11.0` was recently released and pandas 2.3.3 can not be built with it (pandas-dev/pandas#65252) * Newer lxml was also released (https://pypi.org/project/lxml/#history). This PR pin to `4.9.4` to consistent with the version specified in `.github/workflows/build_and_test.yml`. Without pinning `lxml`, `docker build` will fail with following error. ``` 538.3 Collecting lxml 538.3 Downloading lxml-6.0.4.tar.gz (4.2 MB) 538.4 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 91.5 MB/s 0:00:00 538.7 Installing build dependencies: started 540.6 Installing build dependencies: finished with status 'done' 540.6 Getting requirements to build wheel: started 541.3 Getting requirements to build wheel: finished with status 'error' 541.3 error: subprocess-exited-with-error 541.3 541.3 × Getting requirements to build wheel did not run successfully. 541.3 │ exit code: 1 541.3 ╰─> [3 lines of output] 541.3 Building lxml version 6.0.4. 541.3 Building without Cython. 541.3 Error: Please make sure the libxml2 and libxslt development packages are installed. 541.3 [end of output] 541.3 541.3 note: This error originates from a subprocess, and is likely not a problem with pip. 541.3 ERROR: Failed to build 'lxml' when getting requirements to build wheel ------ Dockerfile:67 -------------------- 66 | RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3 67 | >>> RUN echo 'meson<1.11.0' > /tmp/constraints.txt && \ 68 | >>> PIP_CONSTRAINT=/tmp/constraints.txt pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.3.3' scipy coverage matplotlib lxml && \ 69 | >>> rm -f /tmp/constraints.txt 70 | -------------------- ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed that `docker build` works for both `dev/infra/Dockerfile` and `dev/spark-test-image/pypy-310/Dockerfile` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55369 from sarutak/pin-meson-lxml. Authored-by: Kousuke Saruta <sarutak@amazon.co.jp> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>
1 parent cd64def commit 4230fac

2 files changed

Lines changed: 6 additions & 2 deletions

File tree

dev/infra/Dockerfile

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,9 @@ RUN mkdir -p /usr/local/pypy/pypy3.10 && \
9292
ln -sf /usr/local/pypy/pypy3.10/bin/pypy /usr/local/bin/pypy3.10 && \
9393
ln -sf /usr/local/pypy/pypy3.10/bin/pypy /usr/local/bin/pypy3
9494
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
95-
RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.3.3' scipy coverage matplotlib lxml
95+
RUN echo 'meson<1.11.0' > /tmp/constraints.txt && \
96+
PIP_CONSTRAINT=/tmp/constraints.txt pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.3.3' scipy coverage matplotlib 'lxml==4.9.4' && \
97+
rm -f /tmp/constraints.txt
9698

9799

98100
ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 pandas==2.3.3 scipy plotly>=4.8 mlflow>=2.8.1 coverage matplotlib openpyxl memory-profiler>=0.61.0 scikit-learn>=1.3.2"

dev/spark-test-image/pypy-310/Dockerfile

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,4 +64,6 @@ ENV PATH="$VIRTUAL_ENV/bin:$PATH"
6464

6565
# Install PyPy 3.10 packages
6666
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
67-
RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.3.3' scipy coverage matplotlib lxml
67+
RUN echo 'meson<1.11.0' > /tmp/constraints.txt && \
68+
PIP_CONSTRAINT=/tmp/constraints.txt pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.3.3' scipy coverage matplotlib 'lxml==4.9.4' && \
69+
rm -f /tmp/constraints.txt

0 commit comments

Comments
 (0)