Skip to content

Commit c7cd352

Browse files
esafakesafak
authored
fix(ci): Build python wheels using interpreters in manylinux2014 (#2486)
## What does this PR do? The existing workflow did not build wheels correctly because the python interpreter installed in the workflow was not used in the container where the actual building was taking place. The manylinux images already have every python version pre-installed. We now use them instead. Key changes include: - **New Workflows:** - `build-containerized-pr.yml`: Handles building wheels for pull requests using containerized environments for Linux. - `build-containerized-release.yml`: Handles building wheels for releases using containerized environments for Linux. - `build-native-pr.yml`: Handles building wheels for pull requests on macOS and Windows. - `build-native-release.yml`: Handles building wheels for releases on macOS and Windows. - **Removed Workflows:** - `.github/workflows/build-wheels.yaml`: This generic workflow has been superseded by the new, more specific workflows. - `.github/workflows/build-wheels-for-pr.yaml`: Merged into `build-containerized-pr.yml` and `build-native-pr.yml`. - `.github/workflows/build-wheels-for-release.yaml`: Merged into `build-containerized-release.yml` and `build-native-release.yml`. - `.github/workflows/build-native-pr.yml`: This was a duplicate and is now handled by the new `build-native-pr.yml`. - **Script Improvements:** - `ci/build_linux_wheels.py`: Introduced a new script to manage Linux wheel building within Docker containers, replacing `build_manylinux_wheel.sh`. This script now uses explicit Docker image definitions and a simplified execution model. - `ci/deploy.sh`: Updated to use `PYTHON_PATH` and `PIP_CMD` for better Python environment management, and adjusted `pyarrow` installation versions for different Python versions. - `ci/run_ci.sh`: Updated to correctly handle Bazel installation paths (avoiding sudo) and use `curl` for downloading Bazel, since wget is not in the container. - **Workflow Triggers:** - Workflows are now triggered based on their intended purpose (e.g., `push` to `main` or tags for releases, `pull_request` events for PR builds). - Path filters have been adjusted to ensure workflows trigger only when relevant files are changed. ## Related issues Closes #2480 ## Notes I also attempted to create musl builds but they failed because the bazel binaries are incompatible with it. We would need to rebuild bazel for musllinux.
1 parent b84255f commit c7cd352

12 files changed

Lines changed: 334 additions & 320 deletions

.github/workflows/build-wheels-for-release.yaml renamed to .github/workflows/build-containerized-pr.yml

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -15,19 +15,21 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18-
name: build wheels for release
18+
name: Build Containerized PR Wheels
1919
on:
20+
pull_request:
21+
paths: [ci/**, python/**, .github/workflows/**]
2022
push:
21-
tags: ["v*"]
23+
branches: [main]
24+
paths: [ci/**, python/**, .github/workflows/**]
2225

2326
jobs:
24-
build-wheels:
25-
uses: ./.github/workflows/build-wheels.yaml
27+
build:
28+
runs-on: ${{ matrix.os }}
2629
strategy:
2730
matrix:
28-
os: [ubuntu-latest, ubuntu-24.04-arm, macos-13, macos-14, macos-latest, windows-latest]
29-
python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
30-
with:
31-
os: ${{ matrix.os }}
32-
python-version: ${{ matrix.python-version }}
33-
bump-version: true
31+
os: [ubuntu-latest, ubuntu-24.04-arm]
32+
steps:
33+
- uses: actions/checkout@v5
34+
- name: Build and test wheels
35+
run: ./ci/build_linux_wheels.py --arch ${{ runner.arch }}

.github/workflows/build-wheels-for-pr.yaml renamed to .github/workflows/build-containerized-release.yml

Lines changed: 19 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -15,33 +15,27 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18-
name: build wheels for pull request
18+
name: Build Containerized Release Wheels
1919
on:
2020
push:
21-
branches:
22-
- main
23-
paths:
24-
- 'python/**'
25-
- 'cpp/**'
26-
- 'bazel/**'
27-
- 'BUILD'
28-
- 'WORKSPACE'
29-
- '.github/workflows/build-wheels*.yml'
30-
pull_request:
31-
paths:
32-
- 'python/**'
33-
- 'cpp/**'
34-
- 'bazel/**'
35-
- 'BUILD'
36-
- 'WORKSPACE'
37-
- '.github/workflows/build-wheels*.yml'
21+
tags: ['v*'] # NO PATH FILTER - critical for releases
22+
3823
jobs:
39-
build-wheels:
40-
uses: ./.github/workflows/build-wheels.yaml
24+
build:
25+
runs-on: ${{ matrix.os }}
4126
strategy:
4227
matrix:
43-
os: [ubuntu-latest, ubuntu-24.04-arm, macos-latest, windows-latest]
44-
python-version: ['3.8', '3.13']
45-
with:
46-
os: ${{ matrix.os }}
47-
python-version: ${{ matrix.python-version }}
28+
os: [ubuntu-latest, ubuntu-24.04-arm]
29+
steps:
30+
- uses: actions/checkout@v5
31+
- name: Bump version
32+
run: ./ci/deploy.sh bump_py_version
33+
- name: Install bazel
34+
run: ./ci/run_ci.sh install_bazel
35+
- name: Build and test wheels
36+
run: ./ci/build_linux_wheels.py --arch ${{ runner.arch }} --release
37+
- name: Upload artifacts
38+
uses: actions/upload-artifact@v4
39+
with:
40+
name: pyfory-wheels-${{ matrix.os }}-${{ runner.arch }}-${{ github.ref_name }}
41+
path: dist/*.whl

‎.github/workflows/build-native-pr.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Build Native PR Wheels
19+
on:
20+
pull_request:
21+
paths: [ci/**, python/**, .github/workflows/**]
22+
push:
23+
branches: [main]
24+
paths: [ci/**, python/**, .github/workflows/**]
25+
26+
jobs:
27+
build:
28+
runs-on: ${{ matrix.os }}
29+
strategy:
30+
matrix:
31+
os: [macos-latest, windows-latest]
32+
python-version: ['3.8', '3.13']
33+
steps:
34+
- uses: actions/checkout@v5
35+
- uses: actions/setup-python@v5
36+
with:
37+
python-version: ${{ matrix.python-version }}
38+
- name: Install bazel
39+
if: runner.os != 'Windows'
40+
run: ./ci/run_ci.sh install_bazel
41+
- name: Install bazel (Windows)
42+
if: runner.os == 'Windows'
43+
run: ./ci/run_ci.sh install_bazel_windows
44+
shell: bash
45+
- name: Build wheel
46+
run: ./ci/deploy.sh build_pyfory
47+
shell: bash
48+
- name: Install and verify wheel
49+
shell: bash
50+
run: |
51+
python -m pip install --upgrade pip
52+
pip install dist/*.whl
53+
python -c "import pyfory; print(pyfory.__version__)"

‎.github/workflows/build-native-release.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Build Native Release Wheels
19+
on:
20+
push:
21+
tags: ['v*'] # NO PATH FILTER - critical for releases
22+
23+
jobs:
24+
build:
25+
runs-on: ${{ matrix.os }}
26+
strategy:
27+
matrix:
28+
os: [macos-latest, windows-latest]
29+
python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
30+
steps:
31+
- uses: actions/checkout@v5
32+
- name: Bump version
33+
run: ./ci/deploy.sh bump_py_version
34+
- uses: actions/setup-python@v5
35+
with:
36+
python-version: ${{ matrix.python-version }}
37+
- name: Install bazel
38+
if: runner.os != 'Windows'
39+
run: ./ci/run_ci.sh install_bazel
40+
- name: Install bazel (Windows)
41+
if: runner.os == 'Windows'
42+
run: ./ci/run_ci.sh install_bazel_windows
43+
shell: bash
44+
- name: Build wheel
45+
run: ./ci/deploy.sh build_pyfory
46+
- name: Install and verify wheel
47+
shell: bash
48+
run: |
49+
python -m pip install --upgrade pip
50+
pip install dist/*.whl
51+
python -c "import pyfory; print(pyfory.__version__)"
52+
- name: Upload artifacts
53+
uses: actions/upload-artifact@v4
54+
with:
55+
name: pyfory-wheels-${{ matrix.os }}-${{ matrix.python-version }}-${{ github.ref_name }}
56+
path: dist/*.whl

‎.github/workflows/build-wheels.yaml‎

Lines changed: 0 additions & 94 deletions
This file was deleted.

‎.github/workflows/release-python.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ name: Publish Python
1919

2020
on:
2121
workflow_run:
22-
workflows: ["build wheels for release"]
22+
workflows: ["Build Containerized Release Wheels", "Build Native Release Wheels"]
2323
types: [completed]
2424

2525
permissions:

0 commit comments

Comments
 (0)