Set up Python monorepo with uv workspaces; migrate kolibri-sync-extras-plugin - #14945
Merged
rtibbles merged 52 commits intoJul 2, 2026
Merged
Conversation
Complete initial working plugin setup
…s-plugin into main
Fix issue with Manifest excluding sources
Integration fixes with KDP
Resolve issue with deserializing_operations not returning the correct operations
…o_0.16 upgrade queue mechanisms to kolibri 0.16
…comma Missing comma
Remove extranous command option
…tion Migrate to uv, setuptools-scm, prek, ruff plus some pre-commit add-ons
…c-extras-plugin's pyproject.toml onto it Registers python_packages/* as a uv workspace member glob alongside the root kolibri package, and adapts the merged-in kolibri-sync-extras-plugin so it resolves as a working member of that workspace: - Static version (0.2.2, its last published PyPI release) instead of setuptools-scm, which would otherwise read Kolibri's own git tags now that the plugin shares this repo's tag namespace. - tool.uv.sources so its kolibri dependency resolves against this workspace's in-tree package instead of PyPI. - requires-python lowered to match root's >=3.6 floor: uv computes a workspace's effective requires-python as an intersection across all members, so the plugin's previous >=3.9 floor broke any `uv sync --python 3.8` at the repo root, even though root-level syncs never actually install the plugin. The plugin's own tested range is still 3.9+ (see its classifiers). - A python_version marker on its prek dev dependency, which has no build for Python <3.8 and would otherwise break lock resolution across the workspace's now-wider requires-python range.
The plugin previously lived in its own repo with its own CI, publish, and pre-commit setup. All of it is now superseded by this monorepo's own tooling: - .github/workflows/python.yml is replaced by a new per-member-package test job (added in a later commit). - .github/workflows/python-publish.yml published via a long-lived PYPI_API_TOKEN — exactly the pattern this task's OIDC-based publishing workflow replaces. - .pre-commit-config.yaml and .yamlfmt are replaced by the root's own prek/pre-commit setup, which already covers python_packages/**. - Its standalone uv.lock is dead weight now that the workspace uses a single lockfile at the repo root. - Its own [tool.ruff] table is removed: ruff uses the closest pyproject.toml with a [tool.ruff] table rather than merging parent and child configs, so leaving it in place would let it silently diverge from root's. - Root's known-first-party list gains kolibri_sync_extras_plugin so its imports still group as first-party without the plugin's own isort config. Losing the plugin's own overrides means root's stricter rules now apply: 88-char formatting and newly-enforced complexity/print-statement checks. The BackgroundJobOperation subclasses' duplicated job-status handling is extracted into a shared _handle_job_state to resolve the resulting complexity finding.
Adds scripts/pypi_publish.sh and .github/workflows/pypi_packages_publish.yml: - Compares each listed package's version against the target index and publishes only newer ones. - Runs on push to develop and via workflow_dispatch. - workflow_dispatch can target TestPyPI instead of PyPI for testing. - Authenticates with PyPI OIDC trusted publishing — no long-lived API tokens. Successful publishes notify Slack via SLACK_NPM_PACKAGES_WEBHOOK, reused for both workflows. Updates npm_publish.yml's message to say "npm package published" (singular) to match the new workflow's phrasing.
Adding member-package Python tests on top of the existing matrix would slow PR feedback unacceptably, so tox.yml's Python tests now run in two stages instead of one flat job set: - Stage 1 (blocking, parallel): core Kolibri tests on Python 3.10, and Postgres tests. - Stage 2 (gated on Stage 1 via needs:): the rest of the Python version matrix, unit_test_no_uv, postgres_ssl, macos, windows, and a new sync_extras_plugin_tests job for kolibri-sync-extras-plugin. Acts as a safety net for issues Stage 1 is unlikely to catch. Each Stage 2 job's if: explicitly checks needs.stage1_required_checks.result == 'success', since a custom if: replaces rather than ANDs with the implicit all-needs-succeeded check. The pre-existing required_checks job is replaced by two aggregate jobs: stage1_required_checks (new, needs adding to branch protection) and stage2_required_checks (keeps the old job's name "Python tests" so branch protection doesn't need to be re-pointed at a renamed context). Also widens pre_job's path filter to python_packages/** so member-package changes trigger these jobs.
Adds docs/howtos/python_monorepo.md: how to add a member package, how to mark it publishable, and how the Stage 1 / Stage 2 CI cascade works. Linked from the howtos toctree. Adds a "Python packages" section to docs/release_process.rst mirroring the existing "npm packages" section: automatic publishing, bumping a version, and registering a pending trusted publisher for a new package's first release. Notes the two-stage CI cascade in development_workflow.rst's "Merging PRs" section.
uv sync only installs the current project's own dependencies by default; workspace members not declared as a root dependency (e.g. kolibri-sync-extras-plugin) still resolve into the lockfile but need --all-packages or --package to actually install into the venv. After a plain `uv sync --group dev`, `import kolibri_sync_extras_plugin` fails.
github-actions
Bot
added
DEV: dev-ops
…ed venv uv scopes a plain `uv sync` to the "current" project (the plugin, via working-directory), so it was resyncing the shared workspace venv down to just the plugin's own test group — dropping root kolibri's runtime deps (Django, Click, etc., declared under the root's dependency-groups rather than project.dependencies). Reproduced locally: `kolibri plugin enable` failed with `ModuleNotFoundError: No module named 'click'`. --all-packages keeps every workspace member's dependencies installed together. Verified locally: plugin's 28 tests pass from a clean venv.
The plugin's own README still told contributors to run plain `uv sync` from its directory. In the monorepo's shared workspace venv, that scopes the sync down to just the plugin and drops root Kolibri's own runtime deps (Django, Click, etc.) — the exact bug already found and fixed for the CI job. Reproduced locally: `kolibri plugin enable` failed with `ModuleNotFoundError: No module named 'click'` after a plain `uv sync`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Build Artifacts
Smoke test screenshot |
rtibblesbot
marked this pull request as ready for review
July 2, 2026 20:22
Reproduced: `uv sync --group dev --package kolibri-sync-extras-plugin` scopes the sync to that single project just like a plain `uv sync`, dropping root Kolibri's runtime deps (Django, Click, etc.) from the shared venv the same way already fixed for the CI job and the plugin's own README. Only `--all-packages` keeps every workspace member's deps installed together, so the howto's alternative `--package` suggestion was itself wrong and is removed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The "adding a package" steps in the python_monorepo howto were missing two steps a contributor needs: - Add the new member package's test job id to stage2_required_checks's needs: list, since a job left out of that list can fail without blocking merge. - Add the package's import name to root pyproject.toml's ruff isort known-first-party list.
…acts - .gitignore: drop entries already covered by root's own .gitignore, keeping only the .kolibri/ test-fixture entries specific to this package. - CONTRIBUTING.rst: point bug-report and contribution links at the kolibri repo instead of the archived standalone plugin repo. - Makefile: remove it. Its clean/dist/release targets are superseded by uv build and scripts/pypi_publish.sh at the workspace level.
bjester
approved these changes
Jul 2, 2026
bjester
left a comment
bjester
left a comment
Member
There was a problem hiding this comment.
Everything looks good to me, but I do have a comment whether some pure bash for the version wrangling would be simpler instead of in-and-out of python. My opinion is yes but only from a brief review
rtibbles
reviewed
Jul 2, 2026
rtibbles
left a comment
rtibbles
left a comment
Member
There was a problem hiding this comment.
Some questions before we merge.
The old code interpolated published_version — fetched from PyPI's JSON API, so not under our control — into a second python -c string via bash substitution. A version string containing a quote or backslash would break out of the embedded snippet. Doing the fetch and comparison in one python3 invocation drops that interpolation entirely.
rtibblesbot
force-pushed
the
issue-14663-13d55f
branch
from
July 2, 2026 22:46
0b5dedd to
0c43fbc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns the Kolibri repo into a Python monorepo, as a prerequisite for migrating external Kolibri plugins (per #13720) into this repository. Follows the existing
packages/-and-npm_publish.ymlpattern already used for JS packages, adapted for Python:python_packages/directory holds independently-versioned uv workspace members, declared via[tool.uv.workspace].kolibri-sync-extras-pluginis migrated in as the first member, with its git history preserved (git filter-repo --to-subdirectory-filter+git merge --allow-unrelated-histories).pypi_packages_publish.yml) publishes explicitly-listed member packages, mirroringnpm_publish.ymlbut with no long-lived API tokens.tox.yml) is split into a fast, blocking Stage 1 (Python 3.10 + Postgres) and a gated Stage 2 (rest of the version matrix, plus per-member-package tests) so member-package tests don't slow down core PR feedback.Root Kolibri's Python 3.8 support (required for Windows CI) conflicts with
kolibri-sync-extras-plugin's ownrequires-pythonfloor, since uv computes a workspace's effective Python requirement as the intersection of every member's declared range. Per maintainer discussion on the issue, resolved by lowering the plugin's declared floor to match root's (>=3.6) — safe because root-level syncs never actually build the plugin, so the check is metadata-only — while leaving Python 3.8 out of the plugin's own CI test job. See issue comments for the full trail.References
Closes #14663
.github/workflows/npm_publish.yml/scripts/npm_publish.shReviewer guidance
python_packages/kolibri-sync-extras-plugin/pyproject.toml:requires-pythonis lowered to>=3.6to satisfy uv's workspace-wide intersection check, but the plugin's own CI job (sync_extras_plugin_testsintox.yml) only runs on 3.10 — worth confirming this reads clearly, since the declared floor no longer reflects what's actually tested.tox.yml: every Stage 2 job'sif:explicitly re-checksneeds.stage1_required_checks.result == 'success', because a job's customif:replaces (rather than ANDs with) the implicit "allneedssucceeded" check. Worth double-checking each Stage 2 job has this.docs/howtos/python_monorepo.mddocuments thatuv sync --group dev --all-packages(or--package kolibri-sync-extras-plugin) is needed to actually install a workspace member locally — see AI usage below for why this needed fixing.a23998569c) is not a normal diff —git blame python_packages/kolibri-sync-extras-plugin/kolibri_sync_extras_plugin/kolibri_plugin.pyshows original authors/dates predating the merge, confirming history preservation.prek run --all-files.uv sync --group devanduv sync --group dev --all-packages.uv run make docs.cd python_packages/kolibri-sync-extras-plugin && KOLIBRI_HOME=.kolibri uv run python -O -m pytest --reuse-db.AI usage
Used Claude Code end-to-end. It drafted the issue iteratively from a brief prompt (see the issue's own AI-usage note for what I drove vs. what it proposed). It then wrote an implementation plan from the accepted issue and executed it task-by-task. I reviewed each phase's diff during execution. While preparing this PR, I independently reproduced a gap Claude's plan had missed: plain
uv sync --group devresolves but doesn't install non-root workspace members, contradicting both the plan's own verification claim and the howto doc it wrote. I had Claude correctdocs/howtos/python_monorepo.md's guidance accordingly.@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly
How was this generated?
Status: 🟡 Waiting for feedback · updated 2026-07-02 23:24 UTC