Skip to content

tools, build system: introduce PYTHON3_WITH_PEP723 #22223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
maribu wants to merge 3 commits into
base: master
Choose a base branch
from
+39 −4
Open

tools, build system: introduce PYTHON3_WITH_PEP723 #22223

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4faf1d8
dist/tools/usb-serial: use `uv run` for as runner
maribu Apr 28, 2026
e3604b8
tools, build system: introduce PYTHON3_WITH_PEP723
maribu Jun 4, 2026
184586a
Update CODING_CONVENTIONS.md
maribu Jun 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .murdock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ export CFLAGS_DBG=""
export DLCACHE_DIR=${DLCACHE_DIR:-~/.dlcache}
export ENABLE_TEST_CACHE=${ENABLE_TEST_CACHE:-1}
export MURDOCK_REDIS_HOST=${MURDOCK_REDIS_HOST:-127.0.0.1}
export PYTHON3_WITH_PEP723=python3

NIGHTLY=${NIGHTLY:-0}

Expand Down
21 changes: 19 additions & 2 deletions CODING_CONVENTIONS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -701,15 +701,32 @@ not a string literal`.
[McCabe project](https://pypi.python.org/pypi/mccabe)
* A line length of maximum of 120 is allowed instead of 79 as per PEP 8. This
increases tests readability as they can expects long line of output.
* Only runnable scripts shall start with `#!/usr/bin/env python3`
* All runnable scripts and only runnable scripts **MUST** have a shebang
* Runnable scripts **SHOULD** use `#!/usr/bin/env -S uv run` as shebang
* [PEP 723 Inline script metadata](https://peps.python.org/pep-0723/) **SHOULD**
be used for runnable scripts to declare dependencies
* Calls to the python executables from the build system **MUST** be done with
the wrapper `PYTHON3_WITH_PEP723` if they have PEP 723 annotations.
* E.g. to run `dist/tools/fancy/fancy.py` use the following snippet:
`$(Q)$(PYTHON3_WITH_PEP723) $(RIOTTOOLS)/fancy/fancy.py`
* In the CI, `PYTHON3_WITH_PEP723` is overwritten as `python3` and the
CI container **MUST** ship all required python modules to avoid duplicate
downloads and slower CI time due to venv setup time
* Runnable scripts shall use the following scheme:

```python
#!/usr/bin/env python3
#!/usr/bin/env -S uv run
Comment on lines -708 to +718
Copy link
Copy Markdown
Contributor

@kfessel kfessel Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use python to exec python scripts


# SPDX-FileCopyrightText: 2026 <your name/company>
# SPDX-License-Identifier: LGPL-2.1-only

# /// script
# requires-python = ">=3.10"
# dependencies = [
# "pyudev"
# ]
# ///

# put the module imports first
# see https://www.python.org/dev/peps/pep-0008/#imports
# for more details
Expand Down
8 changes: 7 additions & 1 deletion Makefile.include
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,12 @@ OS_ARCH = $(word 2, $(UNAME))
# set python path, e.g. for tests
PYTHONPATH := $(RIOTBASE)/dist/pythonlibs/:$(PYTHONPATH)

# Set python3 frontend capable of PEP 723 inline script metadata. Users may
# choose `pipx run` or just `python3` instead. In case of just `python3`, there
# will be no PEP 723 support and users will just have to install the right
# python modules beforehand.
PYTHON3_WITH_PEP723 ?= uv run
Copy link
Copy Markdown
Contributor

@kfessel kfessel Jun 5, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
PYTHON3_WITH_PEP723 ?= uv run
# your python runner we conveniently chose a default that supports PEP 723 but any other might do
PYTHON ?= uv run

as python without PEP 723 is a valid choice ( as illustrated by this PR and the documentation above)

further more it reduces the inconvenience if one does not want to use uv
to

PYTHON=python make ...


# Basic utilities included before anything else
include $(RIOTMAKE)/utils/checks.mk

Expand Down Expand Up @@ -877,7 +883,7 @@ cleanterm: $(TERMDEPS)
$(TERMPROG) $(TERMFLAGS) $(TERMTEE)

list-ttys:
$(Q)$(RIOTTOOLS)/usb-serial/ttys.py
$(Q)$(PYTHON3_WITH_PEP723) $(RIOTTOOLS)/usb-serial/ttys.py

doc doc-man doc-latex:
$(MAKE) -C $(RIOTBASE) $@
Expand Down
13 changes: 12 additions & 1 deletion dist/tools/usb-serial/ttys.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
#!/usr/bin/env python3
#!/usr/bin/env -S uv run
Copy link
Copy Markdown
Contributor

@kfessel kfessel Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't the entire point of PEP723 to not introduce a dependancy on a specific tool

Copy link
Copy Markdown
Member

@chrysn chrysn Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes But.

Python does not ship any out of the box that qualifies for the shebang.

All our invocations of any of our Python scripts happen through the PYTHON3_WITH_PEP723 executor and thus ignore the shebang.

The use that remains for the shebang (or, really, the executable bit set on the file) is for developers while they work on that script, or when they invoke it directly for some other reason. Nothing we set in there will work for everyone, but that path is not taken by RIOT itself, so just placing something in there is no new dependency, and anyone who invokes it and runs into trouble can just prefix it with the tool of their choosing.

Would it be better if python3 just did this? Yes. But that's not in there now, not sure if planned, and even if, some time down the road until users have it. Until then, we can make the best possible guess there, or leave it to the respective script's maintainers to pick any that works for them.

Copy link
Copy Markdown
Member Author

@maribu maribu Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please read the cover letter of the PR first before commenting.

Copy link
Copy Markdown
Member Author

@maribu maribu Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⬆️ I was referring to Karl. @chrysn clearly did.

Copy link
Copy Markdown
Contributor

@kfessel kfessel Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but uv is just a random choice
and a user might have their package manager mange their python dependencies

why should the shebang point to uv and not just python which with the right dependencies installed is capable of executing a python-script

since our automatic exec happens through a PEP runner (selectable) anyway there is no reason to put that into the shebang

Copy link
Copy Markdown
Member Author

@maribu maribu Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optimization goal for the she-bang is to pick something that is simple to maintain and works well for the majority of users.

Asking users to install uv is much easier than asking users to install a long list of python modules. Especially when the documentation containing the long list of python modules becomes out of date or contains modules that easily conflict with each other.

Copy link
Copy Markdown
Contributor

@kfessel kfessel Jun 5, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why uv and not npm or pip or pipx or poetry or nix or ...

the python packagemanger should be a user choice a python script should not care

not all packemanger are able to read pep but if they install a python and the dependancies the *system python is able to exec that script

Copy link
Copy Markdown
Member Author

@maribu maribu Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the build system we can control how scripts are executed. The proposal here is to just use PYTHON3_WITH_PEP723 as knob that allows users to pick what they please, while picking a sane default that is going to work for most users.

The shebang format however expects a single interpreter. So we have to make a choice here.

Here are the options for the list that I consider as serious suggestions, and what pros/cons I see for them as shebang:

  1. Use python3 directly:
    • Pros:
      • Karl is happy
    • Cons:
      • Not one dependency to care about, but many
      • Not running into a missing tool once, but for each and every script
      • Documentation may get stale, needs to manually be kept in sync with the requirements in code
  2. Use pipx run:
    • Pros:
      • Only a recent version of pipx needs to be installed
    • Cons:
      • Footgun: Ubuntu 24.04 ships a version of pipx that is not compatible with PEP 723 and fails with cryptic errors
      • Needs network access the first time you execute a script
      • Relatively slow the first to you run a script
  3. Use uv run:
    • Pros:
      • Only a recent version of uv needs to be installed
      • Very portable and easy to install (e.g. even on a raspberry pi without root permissions installing it was a one-liner)
      • Orders of magnitude faster than pipx
    • Cons:
      • Needs network access the first time you execute a script

Copy link
Copy Markdown
Contributor

@kfessel kfessel Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this

Image

a valid way of installing something (at least they added a you might want to take a look first box and do not default to we want to be placed in /bin aka sudo needed)

Its is not even like clicking on every executable (something people get made fun of after security incidents) it is injecting a download directly into your system no rubber-ducky needed.

maybe we should add the download ahead of the shebang just to be sure.

Copy link
Copy Markdown
Contributor

@kfessel kfessel Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to add to suggestion 1
default to python3 directly and add documentation that one might want to run python scripts with their favourite python runner

uv run dist/tools/usb-serial/ttys.py
or
pipx run dist/tools/usb-serial/ttys.py
or
check the dependency documented in PEP.. form at the beginning of the python file


# SPDX-FileCopyrightText: 2023 Otto-von-Guericke-Universität Magdeburg
# SPDX-FileCopyrightText: 2026 ML!PA Consulting GmbH
# SPDX-License-Identifier: LGPL-2.1-only

# /// script
# requires-python = ">=3.10"
# dependencies = [
# "pyudev"
# ]
# ///
"""
Command line utility to list and filter TTYs
"""
Expand Down
Loading