Skip to content

Commit e0a1c19

Browse files
rtibblesrtibbles
authored
Merge pull request #14945 from rtibblesbot/issue-14663-13d55f
Set up Python monorepo with uv workspaces; migrate kolibri-sync-extras-plugin
2 parents c5fba5e + 0c43fbc commit e0a1c19

39 files changed

Lines changed: 1532 additions & 20 deletions

‎.github/workflows/npm_publish.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ jobs:
7979
run_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
8080
{
8181
echo "text<<SLACKEOF"
82-
echo ":package: *npm packages published* (<${run_url}|workflow run>)"
82+
echo ":package: *npm package published* (<${run_url}|workflow run>)"
8383
sed -n 's#^| *\[\([^]]*\)\](\([^)]*\)) *| *\([^ |]*\) *|$#• <\2|\1> v\3#p' publish_summary.md
8484
echo "SLACKEOF"
8585
} >> "$GITHUB_OUTPUT"

‎.github/workflows/pypi_packages_publish.yml‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
name: Publish packages to PyPI
2+
on:
3+
push:
4+
branches:
5+
- develop
6+
paths:
7+
- 'python_packages/kolibri-sync-extras-plugin/pyproject.toml'
8+
workflow_dispatch:
9+
inputs:
10+
pypi_package:
11+
description: 'Package to publish (or "all" to auto-detect)'
12+
required: true
13+
type: choice
14+
default: 'all'
15+
options:
16+
- 'all'
17+
- 'kolibri-sync-extras-plugin'
18+
target:
19+
description: 'Index to publish to'
20+
required: true
21+
type: choice
22+
default: 'pypi'
23+
options:
24+
- 'pypi'
25+
- 'testpypi'
26+
permissions:
27+
contents: read
28+
id-token: write
29+
jobs:
30+
publish:
31+
runs-on: ubuntu-latest
32+
env:
33+
PYPI_TARGET: ${{ github.event.inputs.target }}
34+
permissions:
35+
contents: read
36+
id-token: write # Required for PyPI OIDC trusted publishing
37+
steps:
38+
- uses: actions/checkout@v7.0.0
39+
- name: Set up uv
40+
uses: astral-sh/setup-uv@v8.2.0
41+
with:
42+
enable-cache: true
43+
cache-python: true
44+
- name: Install Python
45+
run: uv python install 3.10
46+
- name: Publish
47+
id: publish
48+
run: |
49+
if [ "${{ github.event_name }}" = "workflow_dispatch" ] && [ "${{ github.event.inputs.pypi_package }}" != "all" ]; then
50+
./scripts/pypi_publish.sh "${{ github.event.inputs.pypi_package }}"
51+
else
52+
./scripts/pypi_publish.sh
53+
fi
54+
if [ -f publish_summary.md ]; then
55+
cat publish_summary.md >> "$GITHUB_STEP_SUMMARY"
56+
echo "published=true" >> "$GITHUB_OUTPUT"
57+
else
58+
echo "No packages needed publishing — all versions match the target index." >> "$GITHUB_STEP_SUMMARY"
59+
fi
60+
- name: Build Slack message
61+
if: steps.publish.outputs.published == 'true' && env.PYPI_TARGET != 'testpypi'
62+
id: slack_message
63+
run: |
64+
run_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
65+
{
66+
echo "text<<SLACKEOF"
67+
echo ":package: *PyPI package published* (<${run_url}|workflow run>)"
68+
sed -n 's#^| *\[\([^]]*\)\](\([^)]*\)) *| *\([^ |]*\) *|$#• <\2|\1> v\3#p' publish_summary.md
69+
echo "SLACKEOF"
70+
} >> "$GITHUB_OUTPUT"
71+
- name: Notify Slack
72+
if: steps.publish.outputs.published == 'true' && env.PYPI_TARGET != 'testpypi'
73+
uses: slackapi/slack-github-action@v3
74+
with:
75+
# Intentionally shared with npm_publish.yml — one Slack channel for
76+
# all package-publish notices; the message text disambiguates them.
77+
webhook: ${{ secrets.SLACK_NPM_PACKAGES_WEBHOOK }}
78+
webhook-type: incoming-webhook
79+
payload: |
80+
{
81+
"text": ${{ toJSON(steps.slack_message.outputs.text) }}
82+
}

‎.github/workflows/tox.yml‎

Lines changed: 89 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -26,16 +26,42 @@ jobs:
2626
uses: fkirc/skip-duplicate-actions@master
2727
with:
2828
github_token: ${{ github.token }}
29-
paths: '["**.py", ".github/workflows/tox.yml", "pyproject.toml", "uv.lock"]'
30-
unit_test:
31-
name: Python unit tests for uv-supported Python versions
29+
paths: '["**.py", ".github/workflows/tox.yml", "pyproject.toml", "uv.lock", "python_packages/**"]'
30+
unit_test_stage1:
31+
name: Python unit tests (Python 3.10)
3232
needs: pre_job
3333
if: ${{ needs.pre_job.outputs.should_skip != 'true' }}
3434
runs-on: ubuntu-latest
35+
steps:
36+
- uses: actions/checkout@v7.0.0
37+
with:
38+
fetch-depth: 0
39+
- name: Set up uv
40+
uses: astral-sh/setup-uv@v8.2.0
41+
with:
42+
enable-cache: true
43+
cache-python: true
44+
- name: Install Python 3.10
45+
run: uv python install 3.10
46+
- name: Install dependencies
47+
run: uv sync --group test --python 3.10
48+
- name: Check migrations
49+
run: uv run kolibri manage makemigrations --check
50+
env:
51+
KOLIBRI_HOME: ${{ runner.temp }}/.kolibri
52+
- name: Run tests
53+
run: |
54+
uv run python -O -m pytest kolibri --color=no
55+
uv run python -O -m pytest --color=no -p no:django test
56+
unit_test_stage2:
57+
name: Python unit tests for uv-supported Python versions
58+
needs: [pre_job, stage1_required_checks]
59+
if: ${{ needs.pre_job.outputs.should_skip != 'true' && needs.stage1_required_checks.result == 'success' }}
60+
runs-on: ubuntu-latest
3561
strategy:
3662
max-parallel: 5
3763
matrix:
38-
python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13', '3.14']
64+
python-version: ['3.8', '3.9', '3.11', '3.12', '3.13', '3.14']
3965
steps:
4066
- uses: actions/checkout@v7.0.0
4167
with:
@@ -59,8 +85,8 @@ jobs:
5985
uv run python -O -m pytest --color=no -p no:django test
6086
unit_test_no_uv:
6187
name: Python unit tests for Python versions without uv support
62-
needs: pre_job
63-
if: ${{ needs.pre_job.outputs.should_skip != 'true' }}
88+
needs: [pre_job, stage1_required_checks]
89+
if: ${{ needs.pre_job.outputs.should_skip != 'true' && needs.stage1_required_checks.result == 'success' }}
6490
runs-on: ubuntu-latest
6591
strategy:
6692
max-parallel: 5
@@ -153,8 +179,8 @@ jobs:
153179
KOLIBRI_DATABASE_PORT: "5432"
154180
postgres_ssl:
155181
name: Python postgres SSL smoke tests
156-
needs: pre_job
157-
if: ${{ needs.pre_job.outputs.should_skip != 'true' }}
182+
needs: [pre_job, stage1_required_checks]
183+
if: ${{ needs.pre_job.outputs.should_skip != 'true' && needs.stage1_required_checks.result == 'success' }}
158184
runs-on: ubuntu-latest
159185
strategy:
160186
matrix:
@@ -197,8 +223,8 @@ jobs:
197223
KOLIBRI_DATABASE_SSL_MODE: require
198224
macos:
199225
name: Python unit tests on Mac OS
200-
needs: pre_job
201-
if: ${{ needs.pre_job.outputs.should_skip != 'true' }}
226+
needs: [pre_job, stage1_required_checks]
227+
if: ${{ needs.pre_job.outputs.should_skip != 'true' && needs.stage1_required_checks.result == 'success' }}
202228
runs-on: macos-14
203229
strategy:
204230
max-parallel: 5
@@ -223,8 +249,8 @@ jobs:
223249
uv run python -O -m pytest --color=no -p no:django test
224250
windows:
225251
name: Python unit tests on Windows Server
226-
needs: pre_job
227-
if: ${{ needs.pre_job.outputs.should_skip != 'true' }}
252+
needs: [pre_job, stage1_required_checks]
253+
if: ${{ needs.pre_job.outputs.should_skip != 'true' && needs.stage1_required_checks.result == 'success' }}
228254
runs-on: windows-latest
229255
strategy:
230256
max-parallel: 5
@@ -249,19 +275,65 @@ jobs:
249275
run: |
250276
uv run python -O -m pytest kolibri --color=no
251277
uv run python -O -m pytest --color=no -p no:django test
278+
sync_extras_plugin_tests:
279+
name: kolibri-sync-extras-plugin tests
280+
needs: [pre_job, stage1_required_checks]
281+
if: ${{ needs.pre_job.outputs.should_skip != 'true' && needs.stage1_required_checks.result == 'success' }}
282+
runs-on: ubuntu-latest
283+
steps:
284+
- uses: actions/checkout@v7.0.0
285+
with:
286+
fetch-depth: 0
287+
- name: Set up uv
288+
uses: astral-sh/setup-uv@v8.2.0
289+
with:
290+
enable-cache: true
291+
cache-python: true
292+
- name: Install Python 3.10
293+
run: uv python install 3.10
294+
- name: Install dependencies
295+
# --all-packages is required: a plain `uv sync` scopes to this
296+
# subproject alone and drops root kolibri's own runtime deps
297+
# (Django, Click, etc.) from the shared venv.
298+
working-directory: python_packages/kolibri-sync-extras-plugin
299+
run: uv sync --group test --python 3.10 --all-packages
300+
- name: Enable kolibri_sync_extras_plugin
301+
working-directory: python_packages/kolibri-sync-extras-plugin
302+
env:
303+
KOLIBRI_HOME: .kolibri
304+
run: uv run kolibri plugin enable kolibri_sync_extras_plugin
305+
- name: Run tests
306+
working-directory: python_packages/kolibri-sync-extras-plugin
307+
env:
308+
KOLIBRI_HOME: .kolibri
309+
run: uv run python -O -m pytest --reuse-db
252310
# Single stable check for branch protection. Skipped matrix jobs don't
253311
# produce per-version check runs, so matrix-suffixed required checks hang
254-
# as "Expected" on PRs that skip the tests; require this job instead.
255-
required_checks:
256-
name: Python tests
312+
# as "Expected" on PRs that skip the tests; require these jobs instead.
313+
# Stage 1: fast, blocking feedback.
314+
stage1_required_checks:
315+
name: Python tests (stage 1)
257316
needs:
258317
- pre_job
259-
- unit_test
260-
- unit_test_no_uv
318+
- unit_test_stage1
261319
- postgres
320+
if: always()
321+
runs-on: ubuntu-latest
322+
steps:
323+
- name: Fail if any needed job failed or was cancelled
324+
if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')
325+
run: exit 1
326+
# Stage 2: rest of the Python version matrix plus per-member-package tests.
327+
stage2_required_checks:
328+
name: Python tests
329+
needs:
330+
- stage1_required_checks
331+
- unit_test_stage2
332+
- unit_test_no_uv
262333
- postgres_ssl
263334
- macos
264335
- windows
336+
- sync_extras_plugin_tests
265337
if: always()
266338
runs-on: ubuntu-latest
267339
steps:

‎docs/development_workflow.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,9 @@ First, all automated checks need to pass before merging. Then...
9595
* The reviewer might approve the PR, but also request minor changes such as a typo fix or variable name update. The submitter can then make the change and merge it themselves, with the understanding that the new changes will be limited in scope
9696
* Stale reviews should be dismissed by the PR submitter when the feedback has been addressed
9797

98+
.. note::
99+
Python tests run in a two-stage cascade (Stage 1: Python 3.10 + Postgres required before Stage 2). See :doc:`/howtos/python_monorepo`.
100+
98101

99102
Copyright and licensing
100103
~~~~~~~~~~~~~~~~~~~~~~~

‎docs/howtos/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,4 @@ These guides are step by step guides for common tasks in getting started and wor
2020
working_with_urls_and_api_endpoints
2121
dev_data_setup
2222
multi_agent_setup
23+
python_monorepo

‎docs/howtos/python_monorepo.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Adding a Python member package
2+
3+
Kolibri's Python code is organized as a [uv workspace](https://docs.astral.sh/uv/concepts/projects/workspaces/). The main `kolibri` package is the workspace root; additional packages live under `python_packages/`.
4+
5+
## Adding a new package
6+
7+
1. Create `python_packages/<package-name>/` with its own `pyproject.toml` — a normal, independently-buildable Python package (own `[build-system]`, own static `version`, own dependencies).
8+
2. If it depends on `kolibri` itself, resolve that dependency against this workspace instead of PyPI:
9+
```toml
10+
[tool.uv.sources]
11+
kolibri = { workspace = true }
12+
```
13+
3. No change is needed to the root `pyproject.toml` — its `[tool.uv.workspace]` `members` list already includes the glob `python_packages/*`, so any new package directory under it is picked up automatically.
14+
4. Run `uv sync --group dev` at the repo root to update the shared lockfile.
15+
5. Run `uv sync --group dev --all-packages` to install the new package into the shared workspace venv. `--all-packages` is required — a plain `uv sync` (or `uv sync --package <package-name>`) scopes the sync to a single project and drops root Kolibri's own runtime dependencies (Django, Click, etc.) from the shared venv.
16+
6. Add a test job for it in `.github/workflows/tox.yml`, in the Stage 2 section (see "CI cascade" below) — model it on the `sync_extras_plugin_tests` job, and add the new job's id to `stage2_required_checks`'s `needs:` list in the same file. A job left out of that list can fail without blocking merge, since branch protection only requires `stage2_required_checks` itself to pass.
17+
7. Add the package's import name to `known-first-party` in root `pyproject.toml`'s `[tool.ruff.lint.isort]` table, so ruff sorts its own imports as first-party rather than third-party.
18+
19+
Member package versions are independent of each other and of the main `kolibri` package — there's no enforcement linking them. Use a static `version = "x.y.z"` field, not `setuptools-scm`-derived dynamic versioning: this repo's git tags are Kolibri's own release tags, so dynamic versioning inside the workspace would report Kolibri's version instead of the package's own.
20+
21+
## Marking a package as publishable
22+
23+
By default, a package under `python_packages/` is workspace-only — nothing publishes it. To publish it to PyPI:
24+
25+
1. Add its `pyproject.toml` path to the `paths:` filter in `.github/workflows/pypi_packages_publish.yml`'s `push` trigger.
26+
2. Add its name to the `workflow_dispatch.inputs.pypi_package.options` list in the same file.
27+
3. Register a pending trusted publisher on PyPI and TestPyPI (see the "Python packages" section of [the release process docs](../release_process.rst)) before merging.
28+
29+
## CI cascade
30+
31+
Python tests run in two stages (`.github/workflows/tox.yml`):
32+
33+
- **Stage 1** (blocking, fast feedback): core Kolibri tests on Python 3.10, and Postgres tests. Runs in parallel.
34+
- **Stage 2** (gated on Stage 1 via `needs:`): the rest of the Python version matrix, plus a test job per publishable member package. Acts as a broader safety net — it rarely fails once Stage 1 passes, but still gates merge.
35+
36+
Both stages are required checks in branch protection. Lint, wheel build, and JS tests are unaffected — they run in their own workflows, in parallel with Stage 1.

‎docs/release_process.rst‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,32 @@ Every publish includes an SLSA provenance attestation linking the npm version to
5757
.. code-block:: bash
5858
5959
./scripts/npm_provenance.sh <package-name> [version]
60+
61+
62+
Python packages
63+
===============
64+
65+
Packages in ``python_packages/`` are published to PyPI independently of Kolibri releases and independently of each other. Only publishable packages are published; everything else is workspace-only. A package is publishable if it's listed in ``pypi_packages_publish.yml``'s ``paths:`` filter and ``workflow_dispatch`` options.
66+
67+
Automatic publishing
68+
--------------------
69+
70+
When code merging to ``develop`` changes a listed package's ``pyproject.toml``, the ``pypi_packages_publish.yml`` workflow compares that package's version against PyPI and publishes it if newer.
71+
72+
Authentication uses PyPI OIDC trusted publishing (no API tokens).
73+
74+
The workflow can also be triggered manually from the Actions tab — either for a specific package or for all packages — and can target TestPyPI instead of PyPI via the ``target`` input.
75+
76+
Bumping a version
77+
-----------------
78+
79+
.. code-block:: bash
80+
81+
uv version --package <package-name> --bump patch
82+
83+
Or set an exact version: ``uv version --package <package-name> <new-version>``. Commit the resulting ``pyproject.toml``/``uv.lock`` change and merge to ``develop``.
84+
85+
First publish of a new package
86+
------------------------------
87+
88+
PyPI and TestPyPI support registering a *pending* trusted publisher for a project that doesn't exist yet. Before merging the PR that adds the package, register a pending publisher on both `pypi.org <https://pypi.org/manage/account/publishing/>`__ and `test.pypi.org <https://test.pypi.org/manage/account/publishing/>`__ for the new project name, with owner ``learningequality``, repository ``kolibri``, and workflow ``pypi_packages_publish.yml``.

‎pyproject.toml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ exclude-newer = "7 days"
148148
exclude-newer-package = {le-utils = "0 seconds", morango = "0 seconds"}
149149

150150
[tool.uv.workspace]
151-
members = ["."]
151+
members = [".", "python_packages/*"]
152152

153153
[tool.setuptools]
154154
packages = ["kolibri"]
@@ -182,7 +182,7 @@ max-line-length = 160
182182
max-complexity = 10
183183

184184
[tool.ruff.lint.isort]
185-
known-first-party = ["kolibri"]
185+
known-first-party = ["kolibri", "kolibri_sync_extras_plugin"]
186186
# Match the prior reorder-python-imports style: one import per line,
187187
# sorted case-insensitively rather than grouped by type.
188188
force-single-line = true

‎python_packages/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# python_packages/
2+
3+
Member packages of the Kolibri Python monorepo, managed as a uv workspace.
4+
5+
Each subdirectory is an independently-versioned, independently-publishable Python
6+
package with its own `pyproject.toml`. See
7+
[docs/howtos/python_monorepo.md](../docs/howtos/python_monorepo.md) for how to add
8+
one.

‎python_packages/kolibri-sync-extras-plugin/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Kolibri's root .gitignore already covers standard Python/IDE artifacts;
2+
# this file only needs entries specific to this package.
3+
.kolibri/*
4+
!.kolibri/options.ini

0 commit comments

Comments
 (0)