Skip to content

Commit 70207ba

Browse files
threexcthreexc
committed
docs: development.md: add
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
1 parent f5778eb commit 70207ba

1 file changed

Lines changed: 240 additions & 0 deletions

File tree

‎docs/development.md‎

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
---
2+
title: Development Guide
3+
layout: default
4+
nav_order: 6
5+
---
6+
7+
# Development Guide
8+
9+
Structurally, the [python-wheels](https://github.com/riseproject-dev/python-wheels) repository has three goals:
10+
11+
1. Provide a simple interface for users to install Python wheels from.
12+
2. Create GitHub Actions workflows for building binary Python wheels that
13+
closely match upstream projects' existing CI/CD, but which build and test
14+
only for riscv64.
15+
3. Add supplemental workflows and tooling to track upstream releases, automate
16+
version upgrades, and simplify deprecation once upstream projects incorporate
17+
riscv64 builds, allowing developers to focus on broader package support.
18+
19+
## Workflow Creation Process
20+
21+
`python-wheels` workflows should closely match those for the upstream project to
22+
ensure that our build process for riscv64 wheels is consistent and can be
23+
submitted to the upstream maintainers as evidence of feasibility. Unless
24+
otherwise noted, this guide will reference the existing `build-numpy.yml`
25+
workflow for example code.
26+
27+
The general process:
28+
29+
1. Review the upstream project's build and test workflows (they may be called
30+
`python.yml`, `wheel.yml`, `build.yml`, `release.yml`, or something else
31+
entirely), identifying the section(s) which build for Linux with glibc and
32+
musl.
33+
2. Create a copy of the upstream build workflow in the `python-wheels` repo at
34+
`.github/workflows/build-<package>.yml`, where `<package>` matches the
35+
project name (e.g. `build-numpy.yml` for NumPy`).
36+
3. Strip out any logic not related to the Linux glibc and musl (if present)
37+
build processes.
38+
4. Repeat steps #2 and #3 for the corresponding test workflow, if it is separate
39+
from the upstream build file.
40+
5. Strip out any build logic which is not relevant to riscv64 for Linux. This
41+
includes all other architectures, along with builds for Windows, Mac OS, and
42+
so on. Also remove the sdist and publish steps.
43+
44+
From this point, some customizations are required to enable builds targeting
45+
riscv64.
46+
47+
## Workflow Customizations for riscv64
48+
49+
### riscv64 Runners
50+
51+
We make use of the official [RISE RISC-V
52+
Runners](https://riscv-runners.riseproject.dev/) for any jobs which should run
53+
on a riscv64 platform, particularly build and test jobs. The `python-wheels`
54+
repository is already configured to access them. The `runs-on` directives in any
55+
new workflows should be changed like so:
56+
57+
```
58+
jobs:
59+
build:
60+
runs-on: ubuntu-24.04-riscv
61+
```
62+
63+
### Target Python Versions
64+
65+
By default, riscv64 wheel should be built for a matrix covering the four latest
66+
released Python versions. As of July 14th, 2026, this includes Pythons 3.11,
67+
3.12, 3.13, and 3.14. There is also a freethreaded version of Python 3.14t which
68+
we include. Some wheels have previously been built for 3.13t, but since this was
69+
an experimental version with limited support we avoid it now.
70+
71+
It is worth noting that NumPy releases follow a minimum supported version
72+
pattern that implies a narrower matrix - for example, as of NumPy 2.5.0, only
73+
Python 3.12 and newer are supported. If we follow this precedent, then our
74+
entire riscv64 build matrix consists of `3.12`, `3.13`, `3.14`, and `3.14t`
75+
until `3.15` releases.
76+
77+
### uv
78+
79+
The official `actions/setup-python` Action does not yet support riscv64 builds,
80+
so riscv64 workflows using it will generally fail. A simple workaround is to
81+
replace any usage of `actions/setup-python` in the upstream workflow with
82+
`astral-sh/setup-uv` like so:
83+
84+
```
85+
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
86+
name: Install Python
87+
with:
88+
python-version: '3.12'
89+
activate-environment: true
90+
enable-cache: false
91+
```
92+
93+
Note the `python-version` 'activate-environment', and 'enable-cache' options.
94+
The first two allow us to select the environment Python and have it pre-enabled
95+
(matching `actions/setup-python` behaviour for our purposes). The `enable-cache`
96+
option is disabled for now, as it has caused failures in previous build
97+
attempts.
98+
99+
### Upstream Project Checkouts
100+
101+
We use the `actions/checkout` action to checkout the upstream repository at the
102+
desired tag:
103+
104+
```
105+
- name: Checkout numpy v${{ env.NUMPY_VERSION }}
106+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
107+
with:
108+
repository: numpy/numpy
109+
ref: v${{ env.NUMPY_VERSION }}
110+
submodules: true
111+
persist-credentials: false
112+
```
113+
114+
This effectively overwrites the default project layout for the workflow, which
115+
would otherwise be a copy of `python-wheels`. It allows our workflows to operate
116+
as if they are part of the upstream project without having to include them in a
117+
fork. More importantly, it is critical for uncomplicated usage of tools like
118+
cibuildwheel, which assumes that the root directory is the project to be built
119+
when invoked.
120+
121+
### python-wheels Checkouts
122+
123+
The `python-wheels` repository contains some custom Actions we require, and
124+
patch files to apply for certain projects. The most critical example is the
125+
`publish-to-gitlab` Action. With it in place, the `build-numpy.yml` script's
126+
`publish` job looks like this:
127+
128+
```
129+
publish:
130+
name: Publish numpy ${{ inputs.version || '2.5.0' }} to GitLab
131+
needs: build_wheels
132+
# Only publish when the workflow was triggered from main with a specific
133+
# version. Manual trigger is the only entry point, so checking the ref is
134+
# enough to gate uploads.
135+
if: github.ref == 'refs/heads/main'
136+
runs-on: ubuntu-24.04-riscv
137+
permissions:
138+
contents: read
139+
140+
steps:
141+
- name: Checkout python-wheels
142+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
143+
with:
144+
path: python-wheels-repo
145+
persist-credentials: false
146+
147+
- name: Download wheels
148+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
149+
with:
150+
pattern: numpy-${{ env.NUMPY_VERSION }}-*-manylinux_riscv64
151+
path: dist
152+
merge-multiple: true
153+
154+
- name: Publish to GitLab PyPI registry
155+
uses: ./python-wheels-repo/actions/publish-to-gitlab
156+
with:
157+
gitlab-username: ${{ vars.GITLAB_DEPLOY_USER }}
158+
gitlab-token: ${{ secrets.GITLAB_DEPLOY_TOKEN }}
159+
gitlab-project-id: ${{ vars.GITLAB_PROJECT_ID }}
160+
files: |
161+
dist/*.whl
162+
```
163+
164+
Other workflows need to follow a similar process - checkout the `python-wheels`
165+
repo, and run the `publish-to-gitlab` action to upload built wheels to the RISE
166+
Python registry.
167+
168+
## Testing a New Workflow
169+
170+
Open a new draft PR with the workflow(s) included, and include a `Trigger:` line
171+
with a version for each package version you want to build, like so:
172+
173+
`Trigger: numpy:v2.5.0`
174+
`Trigger: numpy:v2.5.1`
175+
176+
The repository's automation logic will pick up on and trigger the appropriate
177+
build workflows for each version. Achieving a passing (green) build may require
178+
several attempts including rework and possible patches, depending on the nature
179+
of the failure.
180+
181+
### Skipping musl Builds
182+
183+
While building for both glibc and musl (in cibuildwheel terms, `manylinux` and
184+
`musllinux`) is desirable, some of the projects we target do not build for
185+
musllinux (or they do, but run into various issues on riscv64 specifically), and
186+
so dependent packages cannot rely on musl versions of the packages either. If
187+
the musl builds fail without an obvious solution, strip those jobs from the
188+
workflow and retry, while opening an issue to track the musl incompatibility.
189+
190+
### Patching a Project
191+
192+
If a workflow fails consistently when building or testing a module, consider
193+
whether the failure meets one of the following three critieria:
194+
195+
1. The failure exercises a narrow part of the module's functionality, or relies
196+
on external resources (e.g. large downloads over the network)
197+
2. The failure is due to reliance on some other software unavailable on riscv64
198+
3. The failure is a consequence of an artificial test limitation, e.g. a maximum
199+
timeout
200+
201+
In these cases, it may be justified to add one or more patch files to remove
202+
these cases from the workflow. In this scenario, follow these steps:
203+
204+
1. Any such patches should be placed in a `patches/<package_name>/<version_tag>`
205+
path inside `python-wheels`.
206+
2. an extra step should be added to the build/test workflows before execution to
207+
use `git apply` to make necessary modifications to the project source.
208+
3. The change should be documented for the package, so that users are aware of
209+
modifications made.
210+
211+
**Patching should be performed and reviewed on a case-by-case basis - as much
212+
functionality as possible should be tested by our system to ensure a smooth user
213+
experience when consuming wheels from RISE's package registry.**
214+
215+
## Releasing a Wheel
216+
217+
The `publish-to-gitlab` action does not run unless the workflow is triggered
218+
from main. This is intentional, and is meant to ensure that only those workflows
219+
which have been fully tested, reviewed, and merged are used to build and push
220+
packages. Following the merge of a PR, the workflow(s) must be re-triggered from
221+
the `main` branch in order to release the wheels to the package registry.
222+
223+
## Other Workflow Tips and Tricks
224+
225+
### Licensing
226+
227+
The wheels built by the `python-wheels` project use a variety of open-source
228+
licenses. Since RISE is the distributor of riscv64 wheels in the corresponding
229+
package registry, ensuring that the wheels adhere to each project's licensing
230+
requirements. More specifically, check:
231+
232+
1. The built wheel contains one or more `LICENSE` files corresponding to those
233+
contained in the upstream project source.
234+
2. If the wheel ships any statically- or dynamically-linked libraries from other
235+
projects, the licensing requirements for those projects are also correctly
236+
addressed.
237+
238+
If either point is not met, we should follow the [Patching a
239+
Project](#patching-a-project) process for patching our build, and submit an
240+
issue and/or PR upstream to help them comply with license requirements as well.

0 commit comments

Comments
 (0)