Skip to content

Commit bef364a

Browse files
esafakgoogle-labs-jules[bot]chaokunyang
authored
feat: Chain wheel test/build and release workflows (#2483)
## What does this PR do? This commit refactors the Python wheel CI and release process to use a chained workflow model, ensuring that the exact same test and build process is run for both CI checks and releases. A new reusable workflow, `.github/workflows/build-and-test-core.yml`, is introduced. This workflow is triggered on `push` and `pull_request` for CI purposes, and can also be called by other workflows via `workflow_call`. It contains the full logic for building, testing, and packaging the Python wheel across a matrix of operating systems and Python versions. The `release.yaml` workflow is refactored to be an orchestrator. On a new tag, it now calls the `build-and-test-core.yml` workflow to run all tests. If the tests pass, it proceeds to a separate job to download the wheel artifacts produced by the test run and publish them to PyPI. This architecture ensures that every release is automatically and thoroughly tested in the exact same manner as pull requests, just before publication. ## Related issues #2472 #2480 ## Does this PR introduce any user-facing change? No --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: Shawn Yang <chaokunyang@apache.org>
1 parent 68b9f08 commit bef364a

9 files changed

Lines changed: 409 additions & 152 deletions

‎.github/workflows/build-wheels-for-pr.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: build wheels for pull request
19+
on:
20+
push:
21+
branches:
22+
- main
23+
paths:
24+
- 'python/**'
25+
- 'cpp/**'
26+
- 'bazel/**'
27+
- 'BUILD'
28+
- 'WORKSPACE'
29+
- '.github/workflows/build-wheels*.yml'
30+
pull_request:
31+
paths:
32+
- 'python/**'
33+
- 'cpp/**'
34+
- 'bazel/**'
35+
- 'BUILD'
36+
- 'WORKSPACE'
37+
- '.github/workflows/build-wheels*.yml'
38+
jobs:
39+
build-wheels:
40+
uses: ./.github/workflows/build-wheels.yaml
41+
strategy:
42+
matrix:
43+
os: [ubuntu-latest, ubuntu-24.04-arm, macos-latest, windows-latest]
44+
python-version: ['3.8', '3.13']
45+
with:
46+
os: ${{ matrix.os }}
47+
python-version: ${{ matrix.python-version }}

‎.github/workflows/build-wheels-for-release.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: build wheels for release
19+
on:
20+
push:
21+
tags: ["v*"]
22+
23+
jobs:
24+
build-wheels:
25+
uses: ./.github/workflows/build-wheels.yaml
26+
strategy:
27+
matrix:
28+
os: [ubuntu-latest, ubuntu-24.04-arm, macos-13, macos-14, macos-latest, windows-latest]
29+
python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
30+
with:
31+
os: ${{ matrix.os }}
32+
python-version: ${{ matrix.python-version }}
33+
bump-version: true

‎.github/workflows/build-wheels.yaml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Build Wheels
19+
20+
on:
21+
workflow_call:
22+
inputs:
23+
os:
24+
required: true
25+
type: string
26+
python-version:
27+
required: true
28+
type: string
29+
bump-version:
30+
description: 'Whether to bump the version in setup.py'
31+
required: false
32+
type: boolean
33+
default: false
34+
35+
permissions:
36+
contents: read
37+
actions: write
38+
39+
jobs:
40+
build_and_test:
41+
name: Build and Test
42+
runs-on: ${{ inputs.os }}
43+
44+
steps:
45+
- uses: actions/checkout@v5
46+
47+
- name: Set up Python ${{ inputs.python-version }}
48+
uses: actions/setup-python@v5
49+
with:
50+
python-version: ${{ inputs.python-version }}
51+
52+
- name: Install bazel
53+
if: "runner.os != 'Windows'"
54+
run: ./ci/run_ci.sh install_bazel
55+
56+
- name: Install bazel
57+
if: "runner.os == 'Windows'"
58+
run: ./ci/run_ci.sh install_bazel_windows
59+
shell: bash
60+
61+
- name: Update version in setup.py
62+
if: "inputs.bump-version"
63+
run: ./ci/deploy.sh bump_py_version
64+
65+
- name: Build a binary wheel (Linux, manylinux)
66+
if: "runner.os == 'Linux'"
67+
env:
68+
manylinux_x86_64_image: ${{ env.manylinux_x86_64_image }}
69+
manylinux_aarch64_image: ${{ env.manylinux_aarch64_image }}
70+
GITHUB_WORKSPACE: ${{ github.workspace }}
71+
run: |
72+
./ci/build_manylinux_wheel.sh --os "${{ runner.os }}" \
73+
--arch "${{ runner.arch }}" \
74+
--python "${{ inputs.python-version }}" \
75+
--workspace "${GITHUB_WORKSPACE}"
76+
77+
- name: Build a binary wheel (native)
78+
if: "runner.os != 'Linux'"
79+
run: ./ci/deploy.sh build_pyfory
80+
shell: bash
81+
82+
- name: Install and verify wheel
83+
shell: bash
84+
run: |
85+
python -m pip install --upgrade pip
86+
pip install dist/*.whl
87+
python -c "import pyfory; print(pyfory.__version__)"
88+
89+
- name: Upload wheel
90+
# if: ${{ inputs.bump-version }}
91+
uses: actions/upload-artifact@v4
92+
with:
93+
name: pyfory-wheels-${{ inputs.os }}-${{ inputs.python-version }}${{ inputs.bump-version && '-tagged' || github.sha }}
94+
path: dist/*.whl

‎.github/workflows/ci.yml‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ jobs:
4848
matrix:
4949
java-version: ["8", "11", "17", "21", "24"]
5050
steps:
51-
- uses: actions/checkout@v4
51+
- uses: actions/checkout@v5
5252
- name: Set up JDK ${{ matrix.java-version }}
5353
uses: actions/setup-java@v4
5454
with:
@@ -81,7 +81,7 @@ jobs:
8181
# String in openj9 1.8 share byte array by offset, fory doesn't allow it.
8282
java-version: ["21"]
8383
steps:
84-
- uses: actions/checkout@v4
84+
- uses: actions/checkout@v5
8585
- name: Set up JDK ${{ matrix.java-version }}
8686
uses: actions/setup-java@v4
8787
with:
@@ -107,7 +107,7 @@ jobs:
107107
matrix:
108108
java-version: ["21"]
109109
steps:
110-
- uses: actions/checkout@v4
110+
- uses: actions/checkout@v5
111111
- name: Set up JDK ${{ matrix.java-version }}
112112
uses: actions/setup-java@v4
113113
with:
@@ -128,7 +128,7 @@ jobs:
128128
matrix:
129129
java-version: ["17", "21", "23"]
130130
steps:
131-
- uses: actions/checkout@v4
131+
- uses: actions/checkout@v5
132132
- uses: graalvm/setup-graalvm@v1
133133
with:
134134
java-version: ${{ matrix.java-version }}
@@ -152,7 +152,7 @@ jobs:
152152
matrix:
153153
java-version: ["8", "11", "17", "21"]
154154
steps:
155-
- uses: actions/checkout@v4
155+
- uses: actions/checkout@v5
156156
- name: Set up JDK ${{ matrix.java-version }}
157157
uses: actions/setup-java@v4
158158
with:
@@ -171,7 +171,7 @@ jobs:
171171
name: Scala CI
172172
runs-on: ubuntu-latest
173173
steps:
174-
- uses: actions/checkout@v4
174+
- uses: actions/checkout@v5
175175
- name: Set up JDK8
176176
uses: actions/setup-java@v4
177177
with:
@@ -189,7 +189,7 @@ jobs:
189189
name: Integration Tests
190190
runs-on: ubuntu-latest
191191
steps:
192-
- uses: actions/checkout@v4
192+
- uses: actions/checkout@v5
193193
- name: Set up JDK8
194194
uses: actions/setup-java@v4
195195
with:
@@ -210,7 +210,7 @@ jobs:
210210
os: [ubuntu-latest, macos-13, windows-2022]
211211
runs-on: ${{ matrix.os }}
212212
steps:
213-
- uses: actions/checkout@v4
213+
- uses: actions/checkout@v5
214214
- name: Use Node.js ${{ matrix.node-version }}
215215
uses: actions/setup-node@v4
216216
with:
@@ -237,7 +237,7 @@ jobs:
237237
runs-on: ${{ matrix.os }}
238238
timeout-minutes: 45
239239
steps:
240-
- uses: actions/checkout@v4
240+
- uses: actions/checkout@v5
241241
- name: Set up Python 3.11
242242
uses: actions/setup-python@v5
243243
with:
@@ -252,7 +252,7 @@ jobs:
252252
os: [ubuntu-latest, macos-13, macos-14, windows-2022] # macos-13: x86, macos-14: arm64
253253
runs-on: ${{ matrix.os }}
254254
steps:
255-
- uses: actions/checkout@v4
255+
- uses: actions/checkout@v5
256256
- name: Set up Python 3.11
257257
uses: actions/setup-python@v5
258258
with:
@@ -267,7 +267,7 @@ jobs:
267267
python-version: [3.8, 3.12, 3.13.3]
268268
os: [ubuntu-latest, ubuntu-24.04-arm, macos-13, macos-14, windows-2022]
269269
steps:
270-
- uses: actions/checkout@v4
270+
- uses: actions/checkout@v5
271271
- name: Set up Python ${{ matrix.python-version }}
272272
uses: actions/setup-python@v5
273273
with:
@@ -286,7 +286,7 @@ jobs:
286286
matrix:
287287
go-version: ["1.13", "1.18"]
288288
steps:
289-
- uses: actions/checkout@v4
289+
- uses: actions/checkout@v5
290290
- name: Setup Go ${{ matrix.go-version }}
291291
uses: actions/setup-go@v4
292292
with:
@@ -308,7 +308,7 @@ jobs:
308308
name: Code Style Check
309309
runs-on: ubuntu-latest
310310
steps:
311-
- uses: actions/checkout@v4
311+
- uses: actions/checkout@v5
312312
- name: Set up JDK ${{ matrix.java-version }}
313313
uses: actions/setup-java@v4
314314
with:

‎.github/workflows/release-java-snapshot.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ jobs:
2828
runs-on: ubuntu-latest
2929
if: github.repository == 'apache/fory'
3030
steps:
31-
- uses: actions/checkout@v4
31+
- uses: actions/checkout@v5
3232
- name: Set up Maven Central Repository
3333
uses: actions/setup-java@v4
3434
with:

‎.github/workflows/release-python.yaml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Publish Python
19+
20+
on:
21+
workflow_run:
22+
workflows: ["build wheels for release"]
23+
types: [completed]
24+
25+
permissions:
26+
contents: read
27+
id-token: write
28+
29+
jobs:
30+
publish-wheels:
31+
name: Publish Wheels
32+
if: ${{ github.event.workflow_run.conclusion == 'success' }}
33+
runs-on: ubuntu-latest
34+
steps:
35+
- name: Download all wheel artifacts
36+
uses: actions/download-artifact@v5
37+
with:
38+
path: downloaded_wheels
39+
40+
- name: Move wheels to a single directory
41+
shell: bash
42+
run: |
43+
mkdir dist
44+
find downloaded_wheels -type f -name "*.whl" -exec mv {} dist/ \;
45+
ls -R dist
46+
47+
- name: Publish to TestPyPI
48+
uses: pypa/gh-action-pypi-publish@release/v1
49+
if: startsWith(github.ref, 'refs/tags/') && contains(github.ref, '-')
50+
with:
51+
repository-url: https://test.pypi.org/legacy/
52+
skip-existing: true
53+
verbose: true
54+
verify-metadata: false
55+
packages-dir: dist
56+
57+
- name: Publish to PyPI
58+
uses: pypa/gh-action-pypi-publish@release/v1
59+
if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-')
60+
with:
61+
skip-existing: true
62+
verify-metadata: false
63+
packages-dir: dist

0 commit comments

Comments
 (0)