Skip to content

Commit 1c45d16

Browse files
threexcthreexc
authored
Support publishing GPL sources in workflows (#172)
* actions: collect-gpl-sources: add Add a custom action for pulling GPL sources which we need to provide access to for certain package builds. The canonical example is NumPy, which we redistribute the gcc source used to build the wheels (and is therefore the default). Since we expect this to be used with manylinux container images (based on RockyLinux), use dnf-based tooling in the action's steps. * actions: publish-wheels: support publishing GPL sources Publish GPL sources alongside our wheels if they're present so we can be license compliant. * workflows: build-numpy: add gpl_sources job We need to ensure that the GPL sources used in the NumPy builds are provided to be license compliant. Add a new job to get them and pass them along to the publish step. Use a new MANYLINUX_RISCV64_IMAGE variable at the environment level so that the collect-gpl-sources action pulls the sources from the same container image we used for building. * ci_scripts: update_doc.py: support GPL sources linking Add some extra logic to automate linking to any GPL source artifacts we provide along with our builds. * docs: development.md: Add a 'Publishing GPL Sources' Section * docs: development.md: fix heading level --------- AI-Generated: Uses Claude Sonnet 5 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
1 parent fd5e1cc commit 1c45d16

5 files changed

Lines changed: 248 additions & 5 deletions

File tree

‎.github/workflows/build-numpy.yml‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ env:
2424
UV_EXTRA_INDEX_URL: https://pypi.riseproject.dev/simple/
2525
UV_INDEX_STRATEGY: unsafe-best-match
2626
UV_ONLY_BINARY: ':all:'
27+
# Pinned explicitly (rather than left to cibuildwheel's default resolution)
28+
# so the gpl_sources job below can pull the exact same image the wheels
29+
# were built in.
30+
MANYLINUX_RISCV64_IMAGE: quay.io/pypa/manylinux_2_39_riscv64
2731

2832
jobs:
2933
build_wheels:
@@ -79,6 +83,7 @@ jobs:
7983
CCACHE_BASEDIR
8084
CCACHE_COMPILERCHECK
8185
CIBW_CONTAINER_ENGINE: "docker; create_args: --volume ${{ env.CCACHE_DIR }}:/root/.ccache"
86+
CIBW_MANYLINUX_RISCV64_IMAGE: ${{ env.MANYLINUX_RISCV64_IMAGE }}
8287

8388
- name: Save compilation cache
8489
if: always() && github.ref == 'refs/heads/main'
@@ -93,9 +98,26 @@ jobs:
9398
path: ./wheelhouse/*.whl
9499
if-no-files-found: error
95100

101+
gpl_sources:
102+
name: Collect GPL sources (gcc) for numpy ${{ inputs.version || '2.5.1' }}
103+
runs-on: ubuntu-24.04-riscv
104+
steps:
105+
- name: Collect gcc source RPM from manylinux_riscv64
106+
uses: riseproject-dev/python-wheels/actions/collect-gpl-sources@main
107+
with:
108+
image: ${{ env.MANYLINUX_RISCV64_IMAGE }}
109+
packages: gcc
110+
output: gpl-sources.tar
111+
112+
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
113+
with:
114+
name: numpy-${{ env.NUMPY_VERSION }}-gpl-sources
115+
path: gpl-sources.tar
116+
if-no-files-found: error
117+
96118
publish:
97119
name: Publish numpy ${{ inputs.version || '2.5.1' }} to GitLab
98-
needs: build_wheels
120+
needs: [build_wheels, gpl_sources]
99121
# Only publish when the workflow was triggered from main with a specific
100122
# version. Manual trigger is the only entry point, so checking the ref is
101123
# enough to gate uploads.
@@ -114,3 +136,6 @@ jobs:
114136
gitlab-token: ${{ secrets.GITLAB_DEPLOY_TOKEN }}
115137
gitlab-project-id: ${{ vars.GITLAB_PROJECT_ID }}
116138
gh-token: ${{ secrets.GITHUB_TOKEN }}
139+
gpl-sources-artifact: numpy-${{ env.NUMPY_VERSION }}-gpl-sources
140+
gpl-sources-release-tag: numpy-v${{ env.NUMPY_VERSION }}
141+
gpl-sources-description: gcc

‎actions/collect-gpl-sources/action.yml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
name: 'Collect GPL Source RPMs from a manylinux Container'
2+
description: >
3+
Pulls the manylinux/musllinux container image used to build a package's
4+
riscv64 wheels and downloads the source RPM(s) for one or more of its
5+
installed packages (e.g. gcc), satisfying GPL source-distribution
6+
requirements for the toolchain baked into the container. The collected
7+
.src.rpm files are bundled into a single tar archive.
8+
9+
inputs:
10+
11+
# ── Required ────────────────────────────────────────────────────────────────
12+
13+
image:
14+
description: >
15+
Container image reference to pull GPL sources from, e.g.
16+
quay.io/pypa/manylinux_2_39_riscv64. Should match the image actually
17+
used to build the wheels (e.g. via CIBW_MANYLINUX_RISCV64_IMAGE) so the
18+
published sources correspond to the toolchain that produced them.
19+
required: true
20+
21+
# ── Optional ────────────────────────────────────────────────────────────────
22+
23+
packages:
24+
description: Space-separated list of installed RPM package names to fetch source RPMs for.
25+
required: false
26+
default: 'gcc'
27+
28+
output:
29+
description: Path of the tar archive to produce.
30+
required: false
31+
default: 'gpl-sources.tar'
32+
33+
runs:
34+
using: 'composite'
35+
steps:
36+
37+
- name: Pull container image
38+
shell: bash
39+
run: docker pull "${{ inputs.image }}"
40+
41+
- name: Download source RPMs from the container's package repos
42+
shell: bash
43+
run: |
44+
set -euxo pipefail
45+
46+
workdir="$(mktemp -d)"
47+
48+
# Package name and config-manager syntax for enabling the *-source
49+
# repos differ between dnf4 (dnf-plugins-core, `--set-enabled`) and
50+
# dnf5 (dnf5-plugins, `setopt <repo>.enabled=1`). Rocky/RHEL 10 based
51+
# manylinux images ship dnf5, but both forms are attempted so this
52+
# keeps working if the base image changes.
53+
docker run --rm -v "${workdir}:/out" "${{ inputs.image }}" bash -c '
54+
set -euxo pipefail
55+
dnf -y install dnf5-plugins || dnf -y install dnf-plugins-core
56+
for repo in baseos-source appstream-source crb-source; do
57+
dnf -y config-manager setopt "${repo}.enabled=1" \
58+
|| dnf -y config-manager --set-enabled "$repo" \
59+
|| true
60+
done
61+
dnf -y download --source --destdir /out ${{ inputs.packages }}
62+
'
63+
64+
found=$(find "${workdir}" -maxdepth 1 -name "*.src.rpm" -printf '%f\n')
65+
if [[ -z "${found}" ]]; then
66+
echo "::error::No .src.rpm files were collected from ${{ inputs.image }} for packages: ${{ inputs.packages }}"
67+
exit 1
68+
fi
69+
printf '%s\n' "${found}"
70+
71+
tar -cf "${{ inputs.output }}" -C "${workdir}" .
72+
rm -rf "${workdir}"
73+
74+
- name: Print summary
75+
shell: bash
76+
run: |
77+
echo "### GPL Sources" >> "$GITHUB_STEP_SUMMARY"
78+
echo "" >> "$GITHUB_STEP_SUMMARY"
79+
echo "Collected from \`${{ inputs.image }}\`:" >> "$GITHUB_STEP_SUMMARY"
80+
echo "" >> "$GITHUB_STEP_SUMMARY"
81+
tar -tf "${{ inputs.output }}" | sed 's/^/- `/;s/$/`/' >> "$GITHUB_STEP_SUMMARY"

‎actions/publish-wheels/action.yml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,35 @@ inputs:
5757
required: false
5858
default: ''
5959

60+
gpl-sources-artifact:
61+
description: >
62+
Name of an artifact (produced by an earlier job in the caller workflow)
63+
containing a gpl-sources.tar file with bundled GPL source packages
64+
(e.g. a manylinux toolchain's src.rpm, collected via
65+
collect-gpl-sources). When set, it is downloaded, published
66+
as an asset on the GitHub Release named by gpl-sources-release-tag,
67+
and linked in the docs PR opened by update_doc.py. Leave empty to skip
68+
GPL sources publishing entirely.
69+
required: false
70+
default: ''
71+
72+
gpl-sources-release-tag:
73+
description: >
74+
Git tag for the GitHub Release that hosts the gpl-sources-artifact
75+
asset, e.g. "numpy-v2.5.1". Required when gpl-sources-artifact is set;
76+
the release is created if it doesn't already exist.
77+
required: false
78+
default: ''
79+
80+
gpl-sources-description:
81+
description: >
82+
Short parenthetical describing what's bundled in the GPL sources
83+
artifact, e.g. "gcc". Rendered in the docs comment as
84+
"... bundled GPL libraries (gcc)". Only used when gpl-sources-artifact
85+
is set.
86+
required: false
87+
default: ''
88+
6089
runs:
6190
using: 'composite'
6291
steps:
@@ -83,6 +112,40 @@ runs:
83112
skip-existing: ${{ inputs.skip-existing }}
84113
twine-version: ${{ inputs.twine-version }}
85114

115+
- name: Download GPL sources artifact
116+
if: inputs.gpl-sources-artifact != ''
117+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
118+
with:
119+
name: ${{ inputs.gpl-sources-artifact }}
120+
path: gpl-sources
121+
122+
- name: Publish GPL sources as a GitHub Release asset
123+
if: inputs.gpl-sources-artifact != ''
124+
shell: bash
125+
env:
126+
GH_TOKEN: ${{ inputs.gh-token }}
127+
run: |
128+
set -euo pipefail
129+
130+
tag="${{ inputs.gpl-sources-release-tag }}"
131+
if [[ -z "$tag" ]]; then
132+
echo "::error::gpl-sources-release-tag must be set when gpl-sources-artifact is used."
133+
exit 1
134+
fi
135+
136+
asset="gpl-sources/gpl-sources.tar"
137+
138+
if gh release view "$tag" --repo "${{ github.repository }}" >/dev/null 2>&1; then
139+
gh release upload "$tag" "$asset" --repo "${{ github.repository }}" --clobber
140+
else
141+
gh release create "$tag" "$asset" \
142+
--repo "${{ github.repository }}" \
143+
--title "$tag" \
144+
--notes "Bundled GPL source packages for $tag."
145+
fi
146+
147+
echo "GPL_SOURCES_URL=https://github.com/${{ github.repository }}/releases/download/$tag/gpl-sources.tar" >> "$GITHUB_ENV"
148+
86149
- uses: actions/setup-python@v5
87150
with:
88151
python-version: '3'
@@ -96,4 +159,6 @@ runs:
96159
env:
97160
GH_TOKEN: ${{ inputs.gh-token }}
98161
ARTIFACTS_PATH: ${{ inputs.artifact-path }}
162+
GPL_SOURCES_URL: ${{ env.GPL_SOURCES_URL }}
163+
GPL_SOURCES_DESCRIPTION: ${{ inputs.gpl-sources-description }}
99164
run: python3 ci_scripts/update_doc.py

‎ci_scripts/update_doc.py‎

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@
2828
DOCS_DIR = Path("docs/packages")
2929
PACKAGES_FILE = Path("ci_scripts/packages.txt")
3030
ARTIFACTS_PATH = os.environ.get("ARTIFACTS_PATH", "dist")
31+
GPL_SOURCES_URL = os.environ.get("GPL_SOURCES_URL")
32+
GPL_SOURCES_DESCRIPTION = os.environ.get("GPL_SOURCES_DESCRIPTION", "").strip()
3133

3234

3335
def find_wheel_file(path):
@@ -102,6 +104,18 @@ def extract_metadata_from_whl(whl_path):
102104
}
103105

104106

107+
def render_gpl_sources_comment():
108+
"""
109+
Render a doc comment linking to a permanently-hosted GPL sources
110+
artifact (e.g. a manylinux toolchain's src.rpm), if the calling workflow
111+
published one for this build.
112+
"""
113+
if not GPL_SOURCES_URL:
114+
return None
115+
suffix = f" ({GPL_SOURCES_DESCRIPTION})" if GPL_SOURCES_DESCRIPTION else ""
116+
return f"`Link <{GPL_SOURCES_URL}>`__ to sources of bundled GPL libraries{suffix}"
117+
118+
105119
def find_patch_dir(slug, version):
106120
"""
107121
Look for a `patches/<slug>/<version_tag>` directory as described in
@@ -121,7 +135,7 @@ def yaml_line(key, value):
121135
).rstrip("\n")
122136

123137

124-
def render_new_yaml(slug, source_code, license, version, patch_dir):
138+
def render_new_yaml(slug, source_code, license, version, patch_dir, comment=None):
125139
"""Render a brand-new docs/packages/<slug>.yaml for a package's first version."""
126140
lines = [yaml_line("package-name", slug)]
127141
if source_code:
@@ -131,10 +145,12 @@ def render_new_yaml(slug, source_code, license, version, patch_dir):
131145
lines.append(f" - {yaml_line('version', version)}")
132146
if patch_dir is not None:
133147
lines.append(" patched:")
148+
if comment:
149+
lines.append(f" {yaml_line('comment', comment)}")
134150
return "\n".join(lines) + "\n"
135151

136152

137-
def append_version(content, package_data, version, license, patch_dir):
153+
def append_version(content, package_data, version, license, patch_dir, comment=None):
138154
"""
139155
Append a new version entry to the end of an existing package YAML file's
140156
`versions:` list, preserving the rest of the file byte-for-byte.
@@ -153,6 +169,8 @@ def append_version(content, package_data, version, license, patch_dir):
153169
lines.append(" patched:")
154170
if license and license != top_level_license:
155171
lines.append(f" {yaml_line('license', license)}")
172+
if comment:
173+
lines.append(f" {yaml_line('comment', comment)}")
156174

157175
return content.rstrip("\n") + "\n" + "\n".join(lines) + "\n"
158176

@@ -200,17 +218,20 @@ def main():
200218

201219
slug = normalize_name(display_name)
202220
patch_dir = find_patch_dir(slug, version)
221+
comment = render_gpl_sources_comment()
203222
yaml_path = DOCS_DIR / f"{slug}.yaml"
204223
is_new = not yaml_path.exists()
205224

206225
if is_new:
207226
yaml_path.write_text(
208-
render_new_yaml(slug, source_code, license, version, patch_dir)
227+
render_new_yaml(slug, source_code, license, version, patch_dir, comment)
209228
)
210229
else:
211230
content = yaml_path.read_text()
212231
package_data = yaml.safe_load(content) or {}
213-
updated = append_version(content, package_data, version, license, patch_dir)
232+
updated = append_version(
233+
content, package_data, version, license, patch_dir, comment
234+
)
214235
if updated is None:
215236
print(f"{slug} {version} is already documented; nothing to do")
216237
return

‎docs/development.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,57 @@ If either point is not met, we should follow the [Patching a
256256
Project](#patching-a-project) process for patching our build, and submit an
257257
issue and/or PR upstream to help them comply with license requirements as well.
258258

259+
### Publishing GPL Sources
260+
261+
Some builds statically or dynamically link against GPL-licensed components
262+
that aren't part of the upstream project itself, but come from our build
263+
environment - most commonly the toolchain baked into the manylinux_riscv64
264+
container (e.g. `gcc`). As the distributor of the resulting wheel, we need to
265+
make the corresponding source available permanently, not just for as long as
266+
a CI job's artifacts happen to be retained.
267+
268+
Use the `collect-gpl-sources` action in a job alongside
269+
`build_wheels` to pull the same container image and bundle the source RPM(s)
270+
for one or more installed packages into a `gpl-sources.tar` artifact:
271+
272+
```
273+
gpl_sources:
274+
name: Collect GPL sources (gcc) for <package> ${{ inputs.version }}
275+
runs-on: ubuntu-24.04-riscv
276+
steps:
277+
- name: Collect gcc source RPM from manylinux_riscv64
278+
uses: riseproject-dev/python-wheels/actions/collect-gpl-sources@main
279+
with:
280+
image: ${{ env.MANYLINUX_RISCV64_IMAGE }}
281+
packages: gcc
282+
output: gpl-sources.tar
283+
284+
- uses: actions/upload-artifact@v7
285+
with:
286+
name: <package>-${{ inputs.version }}-gpl-sources
287+
path: gpl-sources.tar
288+
if-no-files-found: error
289+
```
290+
291+
Pin `MANYLINUX_RISCV64_IMAGE` as a workflow-level `env` and pass it to both
292+
`CIBW_MANYLINUX_RISCV64_IMAGE` in the build job and this job, so the sources
293+
collected actually correspond to the toolchain that produced the wheels (see
294+
`build-numpy.yml` for a complete example).
295+
296+
Then add `gpl_sources` to the `publish` job's `needs:`, and pass the artifact
297+
through to `publish-wheels`:
298+
299+
```
300+
gpl-sources-artifact: <package>-${{ inputs.version }}-gpl-sources
301+
gpl-sources-release-tag: <package>-v${{ inputs.version }}
302+
gpl-sources-description: gcc
303+
```
304+
305+
`publish-wheels` publishes the tar as a permanent asset on a GitHub Release
306+
(created if it doesn't already exist) and passes its download URL to
307+
`ci_scripts/update_doc.py`, which renders it as a `comment:` on the new
308+
version entry automatically - no manual doc edit needed.
309+
259310
### Adding Builds for Rust Packages
260311

261312
Modules which are cross-compiled from Rust to Python typically use

0 commit comments

Comments
 (0)