|
| 1 | +.. _Use uv for Python dependency management: |
| 2 | + |
| 3 | +Use uv for Python dependency management |
| 4 | +######################################## |
| 5 | + |
| 6 | +Status |
| 7 | +****** |
| 8 | + |
| 9 | +Accepted |
| 10 | + |
| 11 | +Context |
| 12 | +******* |
| 13 | + |
| 14 | +Since 2018, :ref:`OEP-18 Python Dependency Management` has defined the standard |
| 15 | +workflow for Python dependencies in Open edX repositories: |
| 16 | + |
| 17 | +* Direct dependencies declared in ``requirements/*.in`` files |
| 18 | + |
| 19 | +* ``pip-compile`` (from `pip-tools`_) generates pinned ``requirements/*.txt`` |
| 20 | + lockfiles |
| 21 | + |
| 22 | +* ``pip-sync`` installs exactly those pinned versions into a virtualenv |
| 23 | + |
| 24 | +* ``make upgrade`` orchestrates the whole process |
| 25 | + |
| 26 | +This workflow has served the project well, but the Python packaging ecosystem |
| 27 | +has evolved significantly: |
| 28 | + |
| 29 | +* `PEP 621`_ (2021) standardized project metadata in ``pyproject.toml``, |
| 30 | + making ``setup.py`` and ``setup.cfg`` legacy. |
| 31 | + |
| 32 | +* `PEP 735`_ (2024) added dependency groups to ``pyproject.toml``, providing a |
| 33 | + standard way to declare context-specific dependencies (test, docs, CI, etc.) |
| 34 | + — the same problem OEP-18 solved with multiple ``.in`` files. |
| 35 | + |
| 36 | +* `uv`_ emerged as a fast, unified tool that handles dependency resolution, |
| 37 | + locking, virtual environment management, and tool execution in a single |
| 38 | + binary. It natively understands ``pyproject.toml`` and PEP 735 dependency |
| 39 | + groups. |
| 40 | + |
| 41 | +Meanwhile, ``pip-tools`` has not added support for PEP 735 dependency groups, |
| 42 | +and continuing to use it means maintaining a parallel dependency declaration |
| 43 | +system (``requirements/*.in``) alongside ``pyproject.toml``. Additionally, |
| 44 | +`Jazzband`_ — the organization that maintains ``pip-tools`` — announced in |
| 45 | +March 2026 that it is `sunsetting`_, leaving the long-term governance and |
| 46 | +maintenance of ``pip-tools`` uncertain. |
| 47 | + |
| 48 | +.. _Jazzband: https://jazzband.co/ |
| 49 | +.. _sunsetting: https://jazzband.co/news/2026/03/14/sunsetting-jazzband |
| 50 | + |
| 51 | +.. _pip-tools: https://github.com/jazzband/pip-tools |
| 52 | +.. _PEP 621: https://peps.python.org/pep-0621/ |
| 53 | +.. _PEP 735: https://peps.python.org/pep-0735/ |
| 54 | +.. _uv: https://docs.astral.sh/uv/ |
| 55 | + |
| 56 | +Decision/Consequence |
| 57 | +******************** |
| 58 | + |
| 59 | +Open edX Python repositories should use **uv** for dependency management: |
| 60 | + |
| 61 | +* **Declare dependencies in** ``pyproject.toml`` using PEP 621 metadata and |
| 62 | + PEP 735 dependency groups. This replaces ``requirements/*.in`` files. |
| 63 | + |
| 64 | +* **Lock dependencies with** ``uv lock``, producing a single ``uv.lock`` file. |
| 65 | + This replaces multiple ``requirements/*.txt`` files generated by |
| 66 | + ``pip-compile``. |
| 67 | + |
| 68 | +* **Install dependencies with** ``uv sync``, which creates and manages the |
| 69 | + virtual environment. This replaces ``pip-sync`` and manual ``python -m venv`` |
| 70 | + steps. |
| 71 | + |
| 72 | +* **Run tools with** ``uv run`` when a tool is installed in the project's |
| 73 | + virtualenv (e.g., ``uv run tox``, ``uv run pytest``). |
| 74 | + |
| 75 | +* **Preserve the** ``make upgrade`` **pattern** — the target now runs |
| 76 | + ``uv lock --upgrade`` instead of ``pip-compile --upgrade``. |
| 77 | + |
| 78 | +For test matrices that need multiple versions of a dependency (e.g., Django), |
| 79 | +use ``[tool.uv].conflicts`` with separate dependency groups and |
| 80 | +``uv-venv-lock-runner`` in tox. This ensures fully reproducible, locked |
| 81 | +resolutions for each matrix combination. |
| 82 | + |
| 83 | +Shared constraints (e.g., the global constraints file from ``edx-lint``) are |
| 84 | +written into ``[tool.uv].constraint-dependencies`` in ``pyproject.toml`` via |
| 85 | +the ``edx_lint write_uv_constraints`` command, since ``uv lock`` does not |
| 86 | +accept a ``--constraint`` CLI flag. |
| 87 | + |
| 88 | +Rejected Alternatives |
| 89 | +********************* |
| 90 | + |
| 91 | +1. **pip-tools (status quo)** |
| 92 | + |
| 93 | + pip-tools is mature and well-understood in the Open edX community. However, |
| 94 | + it does not support PEP 735 dependency groups, requires a separate tool for |
| 95 | + virtual environment management, and is significantly slower at dependency |
| 96 | + resolution than uv. Continuing to use it means maintaining |
| 97 | + ``requirements/*.in`` files alongside ``pyproject.toml``, duplicating |
| 98 | + dependency declarations. The `sunsetting of Jazzband <sunsetting_>`_ also |
| 99 | + raises questions about long-term maintenance. |
| 100 | + |
| 101 | +2. **Poetry** |
| 102 | + |
| 103 | + `Poetry <https://python-poetry.org/>`_ is a popular all-in-one tool, but it |
| 104 | + is opinionated about project structure, uses a non-standard lock format |
| 105 | + (``poetry.lock``), and has limited support for the kind of dependency groups |
| 106 | + needed for Open edX's test matrices (e.g., multiple Django versions). Its |
| 107 | + resolver has historically been slower than uv's. |
| 108 | + |
| 109 | +3. **PDM** |
| 110 | + |
| 111 | + `PDM <https://pdm-project.org/>`_ supports PEP 735 and produces a standard |
| 112 | + lock format, but has a smaller community and less ecosystem adoption. uv has |
| 113 | + gained broader momentum in the Python packaging space and is backed by the |
| 114 | + same team (Astral) that maintains ``ruff``, which is already widely adopted |
| 115 | + in the broader Python community. |
| 116 | + |
| 117 | +Examples |
| 118 | +******** |
| 119 | + |
| 120 | +The following examples are drawn from `openedx/sample-plugin`_ and illustrate |
| 121 | +the key patterns. See that repository for a complete working reference. |
| 122 | + |
| 123 | +.. _openedx/sample-plugin: https://github.com/openedx/sample-plugin |
| 124 | + |
| 125 | +Example pyproject.toml |
| 126 | +====================== |
| 127 | + |
| 128 | +.. code-block:: toml |
| 129 | +
|
| 130 | + [build-system] |
| 131 | + requires = ["setuptools", "setuptools-scm>8.1"] |
| 132 | + build-backend = "setuptools.build_meta" |
| 133 | +
|
| 134 | + [project] |
| 135 | + name = "openedx-sample-plugin" |
| 136 | + description = "A sample backend plugin for the Open edX Platform" |
| 137 | + requires-python = ">=3.12" |
| 138 | + license = "Apache-2.0" |
| 139 | +
|
| 140 | + dependencies = [ |
| 141 | + "Django", |
| 142 | + "djangorestframework", |
| 143 | + "openedx-events", |
| 144 | + ] |
| 145 | +
|
| 146 | + # -- Dependency groups (PEP 735) ------------------------------------------ |
| 147 | + # |
| 148 | + # These replace the old requirements/*.in files. Each group maps to a usage |
| 149 | + # context (testing, quality checks, docs, CI tooling, local dev). |
| 150 | +
|
| 151 | + [dependency-groups] |
| 152 | + # Framework-agnostic test deps. Not used directly — included by the |
| 153 | + # version-specific groups below. |
| 154 | + test-base = [ |
| 155 | + "pytest-cov", |
| 156 | + "pytest-django", |
| 157 | + ] |
| 158 | +
|
| 159 | + # Current default Django version. Used by quality, docs, and as the default |
| 160 | + # test matrix entry. |
| 161 | + test = [ |
| 162 | + {include-group = "test-base"}, |
| 163 | + "Django>=5.0,<6.0", |
| 164 | + ] |
| 165 | +
|
| 166 | + # Additional Django versions under test. Each gets its own group so uv can |
| 167 | + # produce a separate locked resolution via [tool.uv].conflicts. |
| 168 | + django60 = [ |
| 169 | + {include-group = "test-base"}, |
| 170 | + "Django>=6.0,<7.0", |
| 171 | + ] |
| 172 | +
|
| 173 | + quality = [ |
| 174 | + {include-group = "test"}, |
| 175 | + "edx-lint", |
| 176 | + "pycodestyle", |
| 177 | + ] |
| 178 | +
|
| 179 | + doc = [ |
| 180 | + {include-group = "test"}, |
| 181 | + "Sphinx", |
| 182 | + "doc8", |
| 183 | + ] |
| 184 | +
|
| 185 | + ci = [ |
| 186 | + "tox", |
| 187 | + "tox-uv", |
| 188 | + ] |
| 189 | +
|
| 190 | + dev = [ |
| 191 | + {include-group = "quality"}, |
| 192 | + {include-group = "ci"}, |
| 193 | + ] |
| 194 | +
|
| 195 | + # -- uv configuration ----------------------------------------------------- |
| 196 | +
|
| 197 | + [tool.uv] |
| 198 | + # Mutually exclusive groups: uv produces one locked resolution per entry. |
| 199 | + conflicts = [ |
| 200 | + [{group = "test"}, {group = "django60"}], |
| 201 | + ] |
| 202 | +
|
| 203 | + # DO NOT EDIT constraint-dependencies DIRECTLY. |
| 204 | + # This list is managed by `edx_lint write_uv_constraints` |
| 205 | + # and will be overwritten the next time `make upgrade` is run. |
| 206 | + # - GLOBAL constraints: edit edx_lint/files/common_constraints.txt |
| 207 | + # - REPO-SPECIFIC constraints: edit constraints.txt next to pyproject.toml |
| 208 | + constraint-dependencies = [ |
| 209 | + "Django<7.0", |
| 210 | + "elasticsearch<7.14.0", |
| 211 | + ] |
| 212 | +
|
| 213 | +Example tox.ini |
| 214 | +=============== |
| 215 | + |
| 216 | +.. code-block:: ini |
| 217 | +
|
| 218 | + [tox] |
| 219 | + envlist = py312-django{52,60},quality,docs |
| 220 | + requires = |
| 221 | + tox-uv>=1 |
| 222 | +
|
| 223 | + [testenv] |
| 224 | + runner = uv-venv-lock-runner |
| 225 | + dependency_groups = |
| 226 | + django52: test |
| 227 | + django60: django60 |
| 228 | + commands = |
| 229 | + pytest {posargs} |
| 230 | +
|
| 231 | + [testenv:quality] |
| 232 | + runner = uv-venv-lock-runner |
| 233 | + dependency_groups = |
| 234 | + quality |
| 235 | + commands = |
| 236 | + pylint my_package tests |
| 237 | +
|
| 238 | + [testenv:docs] |
| 239 | + runner = uv-venv-lock-runner |
| 240 | + dependency_groups = |
| 241 | + doc |
| 242 | + commands = |
| 243 | + doc8 docs |
| 244 | + make -e -C docs html |
| 245 | +
|
| 246 | +Common commands |
| 247 | +=============== |
| 248 | + |
| 249 | +.. code-block:: bash |
| 250 | +
|
| 251 | + # Create a virtual environment |
| 252 | + uv venv .venv --seed --python 3.12 |
| 253 | +
|
| 254 | + # Install all dev dependencies (creates .venv if needed) |
| 255 | + uv sync --group dev |
| 256 | +
|
| 257 | + # Run tests via tox |
| 258 | + uv run tox |
| 259 | +
|
| 260 | + # Run a single tox environment |
| 261 | + uv run tox -e py312-django60 |
| 262 | +
|
| 263 | + # Upgrade all locked dependencies (the new "make upgrade") |
| 264 | + uv lock --upgrade |
| 265 | +
|
| 266 | + # Sync shared constraints from edx-lint into pyproject.toml |
| 267 | + uv run --with edx-lint edx_lint write_uv_constraints |
| 268 | +
|
| 269 | + # Full upgrade sequence (typically wrapped in a Makefile target) |
| 270 | + uv run --with edx-lint edx_lint write_uv_constraints |
| 271 | + uv lock --upgrade |
| 272 | +
|
| 273 | +References |
| 274 | +********** |
| 275 | + |
| 276 | +* `PEP 621 — Storing project metadata in pyproject.toml <https://peps.python.org/pep-0621/>`_ |
| 277 | + |
| 278 | +* `PEP 735 — Dependency Groups in pyproject.toml <https://peps.python.org/pep-0735/>`_ |
| 279 | + |
| 280 | +* `uv documentation <https://docs.astral.sh/uv/>`_ |
| 281 | + |
| 282 | +* `pip-tools <https://github.com/jazzband/pip-tools>`_ |
| 283 | + |
| 284 | +* `Jazzband sunsetting announcement <https://jazzband.co/news/2026/03/14/sunsetting-jazzband>`_ |
| 285 | + |
| 286 | +* `openedx/sample-plugin <https://github.com/openedx/sample-plugin>`_ — reference implementation |
| 287 | + |
| 288 | +* :ref:`OEP-18 Python Dependency Management` — the predecessor workflow this decision replaces |
0 commit comments