Skip to content

Commit 06a789a

Browse files
committed
workflows, ci_scripts: add automation logic from wheel_builder (#100)
* LICENSE-Apache-2.0: add Some of the scripts we have ported from the wheel_builder repository are unchanged, and the originals were licensed under Apache-2.0 terms. Add a copy of that LICENSE file as LICENSE-Apache-2.0 for compliance purposes. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: add audit.sh Migrate the script from wheel_builder, with the following changes: - Add new RISE copyright - Include a short description - Make it output a json report for further analysis Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: add check_versions.py Migrate the check_versions.py script from wheel_builder. Much of the script has been refactored with the following goals: 1. Use GitHub conventions (instead of GitLab) 2. Create GitHub-specific replacement functions where necessary 3. Simplify other functions where possible 4. Use more Pythonic styling 5. Remove comments where the code is self-documenting More specifically: - Replace usage of `glab` with the corresponding `gh` CLI - Create a REPO variable pointing to "riseproject-dev/python-wheels", and use it wherever the wheel_builder was referenced (in calls to `glab`) - Change configure_git_identity() to use a "github-actions[bot]" user and corresponding noreply email - Change generated branch names' prefix from "gitlab-ci" to "github-actions" - Remove some auth/login-related logic using tokens to generate automated PRs, since the GitHub project allows us to enable this in the repository settings (not programatically) - Add dispatch_workflow() and create_upgrade_pr() functions for triggering a desired workflow (e.g. for a "matplotlib" upgrade, the script should trigger the "build-matplotlib.yml" and "test-matplotlib.yml" workflows for the target version(s) if the workflow scripts exist, or indicate otherwise if not possible - Allow updating of default versions in workflow scripts when auto-generating upgrade PRs. - Add "Trigger" labels to auto-generated PRs for the new versions detected. - Importantly, limit the maximum number of versions to be targeted by an auto-generated PR to three, so as not to create a runaway build queue. - Remove most comments, since the code objectives are obvious after the refactor AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: check_patch.py: add Migrate check_patch.py from the wheel_builder repository, adding a new RISE copyright and updating the description. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: audit_report: add Create a new script for reviewing reports created by ci_scripts/audit.sh, and opening/commenting on issues in GitHub to track vulnerabilities identified by nightly pip-audit runs. The issues are opened on a per-vulnerability basis with affected versions, but only if an existing issue matching the same vulnerability and with the "pip-audit" label isn't found. Previously-closed issues are ignored when determining if a new one should be opened. Also use GHA warning notations so that the nightly CI pipeline can report a pass (instead of a failure) while still reporting a vulnerability in the issue list. AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: migrate deprecated.txt Migrate the deprecated packages file from wheel_builder, adding a new RISE copyright. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: migrate packages.txt Migrate the supported packages file from wheel_builder, adding a new RISE copyright in the process. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: add list-packages.sh Migrate the script from wheel_builder, adding a new RISE copyright and a short description in the process. Also make sure that it only tries to install binary wheels, so that we don't try and build them from scratch during simple checks. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * workflows: pr-trigger: add Add a workflow for triggering workflows named "build-<package>.yml" (and if present, "test-<package>.yml") with desired release tags for a given package. For automated upgrade checks and manual changes, this means that the tags only need to be specified in the PR description when opened. This is based on the process in wheel_builder where a child pipeline would be generated automatically when a merge request was submitted with one or more package version tags listed as e.g. "Trigger: polars:py-1.42.1" in the MR description. AI-Generated: Uses Claude Code Sonnet 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * workflows: pr-checks: add Provide a workflow for verifying that content added/modified in commits meets basic requirements, i.e. no debug patches with "revertme", "revert me", or "DO NOT MERGE" at the beginning. Also ensure that an "Upstream-Status" line is provided in any carried patches (by running check_patch.py), so that developers understand the patches' purpose. This is based on a combination of the "check_commit_messages" and "check_patches" jobs in wheel_builder's .gitlab-ci.yml script. AI-Generated: Uses Claude Code Sonnet 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * workflows: nightly: add Provide a nightly CI workflow for checking: 1. Latest upstream package versions versus those available in the RISE registry 2. Packages which have upstream riscv64 wheels (ready for deprecation in our CI) 3. Check that packages install correctly in a riscv64 container 4. Run pip-audit to scan for vulnerabilities This is based on the following jobs from wheel_builder's .gitlab-ci.yml script: - check_deprecated_packages - check_installation - check_versions - pip_audit Two notable differences compared to the original logic (in addition to the GitLab CI -> GHA rework): 1. We use RISE's RISC-V Runners for the steps which would otherwise run the setup-qemu action. 2. A step is added using the new audit_report.py script to file issues when vulnerabilities are detected. AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> --------- Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
1 parent 42107e0 commit 06a789a

11 files changed

Lines changed: 1593 additions & 0 deletions

File tree

.github/workflows/nightly.yml

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
4+
name: Run nightly package version and status checks
5+
6+
on:
7+
schedule:
8+
- cron: '0 2 * * *'
9+
workflow_dispatch:
10+
11+
permissions:
12+
contents: write
13+
pull-requests: write
14+
actions: write
15+
issues: read
16+
17+
env:
18+
PIP_EXTRA_INDEX_URL: https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple
19+
20+
jobs:
21+
check_versions:
22+
runs-on: ubuntu-latest
23+
steps:
24+
- uses: actions/checkout@v4
25+
with:
26+
fetch-depth: 0
27+
28+
- uses: actions/setup-python@v5
29+
with:
30+
python-version: '3'
31+
32+
- name: Install Python dependencies
33+
run: pip install requests packaging
34+
35+
- name: Check versions and open PRs
36+
env:
37+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
38+
run: python3 ci_scripts/check_versions.py --summary --create-prs
39+
40+
check_deprecated_packages:
41+
runs-on: ubuntu-latest
42+
steps:
43+
- uses: actions/checkout@v4
44+
45+
- name: Install curl and jq
46+
run: sudo apt-get update -qq && sudo apt-get install -qq -y curl jq
47+
48+
- name: Verify riscv64 wheels still published
49+
run: |
50+
ret=0
51+
while IFS= read -r pkg; do
52+
[[ -z "$pkg" || "$pkg" =~ ^[[:space:]]*# ]] && continue
53+
54+
if ! output=$(curl -sf "https://pypi.org/pypi/${pkg}/json"); then
55+
echo "Failed to fetch $pkg, skipping..."
56+
continue
57+
fi
58+
59+
if ! filenames=$(echo "$output" | jq -r '.urls[].filename'); then
60+
echo "Failed to parse JSON for $pkg, skipping..."
61+
continue
62+
fi
63+
64+
if ! echo "$filenames" | grep -qi riscv; then
65+
echo "$pkg does not distribute riscv64 wheel anymore!"
66+
ret=1
67+
fi
68+
done < ci_scripts/deprecated.txt
69+
exit $ret
70+
71+
check_installation:
72+
runs-on: ubuntu-24.04-riscv
73+
steps:
74+
- uses: actions/checkout@v4
75+
76+
- name: Install packages inside riscv64 container
77+
working-directory: ci_scripts
78+
run: |
79+
sed -Ei 's/^scipy.*$//' packages.txt
80+
sed -Ei 's/^patchelf.*$//' packages.txt
81+
./list-packages.sh python-wheels
82+
83+
pip_audit:
84+
runs-on: ubuntu-24.04-riscv
85+
continue-on-error: true
86+
permissions:
87+
contents: read
88+
issues: write
89+
steps:
90+
- uses: actions/checkout@v4
91+
92+
- name: Run pip-audit
93+
working-directory: ci_scripts
94+
run: ./audit.sh python-wheels
95+
96+
- name: Report vulnerabilities and open issues
97+
env:
98+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
99+
run: python3 ci_scripts/audit_report.py ci_scripts/audit-report.json

.github/workflows/pr-checks.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
4+
name: PR verification checks
5+
6+
on:
7+
pull_request:
8+
9+
jobs:
10+
check_commit_messages:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
with:
15+
fetch-depth: 0
16+
17+
- name: Reject revertme / DO NOT MERGE commits
18+
env:
19+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
20+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
21+
run: |
22+
set -euo pipefail
23+
range="$BASE_SHA..$HEAD_SHA"
24+
if git log --pretty=format:%s "$range" | grep -i -e '^revertme' -e '^revert me'; then
25+
echo "::error::Merge request contains at least a commit starting with a 'revert me' tag. Fix it before merging."
26+
exit 1
27+
fi
28+
if git log --pretty=format:%s "$range" | grep -i -e '^DO NOT MERGE'; then
29+
echo "::error::Merge request contains at least a commit starting with a 'DO NOT MERGE' tag. Review it."
30+
exit 1
31+
fi
32+
33+
check_patches:
34+
runs-on: ubuntu-latest
35+
steps:
36+
- uses: actions/checkout@v4
37+
with:
38+
fetch-depth: 0
39+
40+
- uses: actions/setup-python@v5
41+
with:
42+
python-version: '3'
43+
44+
- name: Validate Upstream-Status in added/modified patches
45+
env:
46+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
47+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
48+
run: python ci_scripts/check_patch.py "$BASE_SHA" "$HEAD_SHA"

.github/workflows/pr-trigger.yml

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
4+
name: PR package build trigger
5+
6+
on:
7+
pull_request:
8+
types: [opened, edited, synchronize, reopened]
9+
10+
permissions:
11+
contents: read
12+
actions: write
13+
pull-requests: read
14+
15+
jobs:
16+
dispatch-triggers:
17+
runs-on: ubuntu-latest
18+
if: contains(github.event.pull_request.body, 'Trigger:')
19+
steps:
20+
- uses: actions/checkout@v4
21+
with:
22+
package_version: ${{ github.event.pull_request.head.sha }}
23+
24+
- name: Dispatch build/test workflows for Trigger directives
25+
env:
26+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
27+
PR_BODY: ${{ github.event.pull_request.body }}
28+
HEAD_REF: ${{ github.event.pull_request.head.ref }}
29+
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
30+
BASE_REPO: ${{ github.repository }}
31+
run: |
32+
set -euo pipefail
33+
34+
if [ "$HEAD_REPO" != "$BASE_REPO" ]; then
35+
echo "PR from fork ($HEAD_REPO); cannot dispatch workflows against fork ref. Skipping."
36+
exit 0
37+
fi
38+
39+
triggers=$(printf '%s\n' "$PR_BODY" | grep -oE '^Trigger:[[:space:]]*[^[:space:]]+' || true)
40+
41+
if [ -z "$triggers" ]; then
42+
echo "No Trigger: directives found."
43+
exit 0
44+
fi
45+
46+
printf '%s\n' "$triggers" | while IFS= read -r line; do
47+
directive=${line#Trigger:}
48+
directive=${directive# }
49+
package_name=${directive%:*}
50+
package_version=${directive#*:}
51+
52+
if [ -z "$package_name" ] || [ -z "$package_version" ] || [ "$package_name" = "$package_version" ]; then
53+
echo "Skipping malformed directive: $line"
54+
continue
55+
fi
56+
57+
# Normalize: build workflows prepend `v` to the version themselves
58+
# (e.g. `ref: v${{ env.NUMPY_VERSION }}`). Strip any leading v/V
59+
# in the directive so `Trigger: numpy:v2.5.1` and
60+
# `Trigger: numpy:2.5.1` behave identically.
61+
package_version=${package_version#[vV]}
62+
63+
build_wf=".github/workflows/build-${package_name}.yml"
64+
test_wf=".github/workflows/test-${package_name}.yml"
65+
66+
if [ ! -f "$build_wf" ]; then
67+
echo "No build workflow for $package_name at $build_wf; skipping."
68+
continue
69+
fi
70+
71+
echo "Dispatching build-${package_name}.yml on $HEAD_REF with version=$package_version"
72+
gh workflow run "build-${package_name}.yml" --ref "$HEAD_REF" -f "version=$package_version"
73+
74+
if [ -f "$test_wf" ]; then
75+
echo "Dispatching test-${package_name}.yml on $HEAD_REF with version=$package_version"
76+
gh workflow run "test-${package_name}.yml" --ref "$HEAD_REF" -f "version=$package_version"
77+
fi
78+
done

0 commit comments

Comments
 (0)