Skip to content

packaging_compiled.md

Latest commit

 

History

History
456 lines (387 loc) · 12.4 KB

packaging_compiled.md

File metadata and controls

456 lines (387 loc) · 12.4 KB
Raw

Packaging Compiled Projects

:::{tip} Quick start Once you've done this at least once, feel free to use our cookiecutter/copier template, or uv init to get started quickly on new packages! :::

There are a variety of ways to package compiled projects. In the past, the only way to do it was to use setuptools/distutils, which required using lots of fragile internals - distutils was intended primarily to compile CPython, and setuptools tried to stay away from changing the compile portions except as required. Now, however, we have several very nice options for compiled packages!

The most exciting developments have been new native build backends:

  • scikit-build-core: Builds C/C++/Fortran using CMake.
  • meson-python: Builds C/C++/Fortran using Meson.
  • maturin: Builds Rust using Cargo. Written entirely in Rust!
  • [enscons][]: Builds C/C++ using SCONs. (Aging now, but this was the first native backend!)

:::{note} You should be familiar with packing a pure Python project - the metadata configuration is the same. ::: There are also classic setuptools plugins:

:::{tip} Selecting a backend Selecting a backend: If you are using Rust, use maturin. If you are using CUDA, use scikit-build-core. If you are using a classic language (C, C++, Fortran), then you can use either scikit-build-core or meson-python, depending on whether you prefer writing CMake or Meson. Meson is a lot more opinionated; it requires you use version control, it requires a README.md and LICENSE file. It requires your compiler be properly set up. Etc. While CMake can be as elegant as Meson, there are a lot of historical examples of poorly written CMake. :::

pyproject.toml: build-system

{rr}PY001 Packages must have a pyproject.toml file {rr}PP001 that selects the backend:

::::{tab-set} :::{tab-item} Scikit-build-core :sync: scikit-build-core

[build-system]
requires = ["pybind11", "scikit-build-core>=0.12"]
build-backend = "scikit_build_core.build"

::: :::{tab-item} Meson-python :sync: meson-python

[build-system]
requires = ["meson-python>=0.18", "pybind11"]
build-backend = "mesonpy"

::: :::{tab-item} Maturin :sync: maturin

[build-system]
requires = ["maturin>=1.9,<2"]
build-backend = "maturin"

::: ::::

Tool section in pyproject.toml

These tools all read the project table. They also have extra configuration options in tool.* settings.

::::{tab-set} :::{tab-item} Scikit-build-core :sync: scikit-build-core

[tool.scikit-build]
minimum-version = "build-system.requires"
build-dir = "build/{wheel_tag}"

These options are not required, but can improve your experience. ::: :::{tab-item} Meson-python :sync: meson-python No tool.meson-python configuration required for this example. ::: :::{tab-item} Maturin :sync: maturin

[tool.maturin]
module-name = "package._core"
python-source = "src"
sdist-generator = "git"  # default is cargo

Maturin assumes you follow Rust's package structure, so we need a little bit of configuration here to follow the convention of the other tools here. ::: ::::

Backend specific files

::::{tab-set} :::{tab-item} Scikit-build-core :sync: scikit-build-core Example CMakeLists.txt file (using pybind11, so include pybind11 in build-system.requires too):

cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)

set(PYBIND11_FINDPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(_core MODULE src/main.cpp)
install(TARGETS _core DESTINATION ${SKBUILD_PROJECT_NAME})

Scikit-build-core will use your .gitignore to help it avoid adding ignored files to your distributions; it also has a default ignore for common cache files, so you can get started without one, but it's recommended. ::: :::{tab-item} Meson-python :sync: meson-python Example meson.build file (using pybind11, so include pybind11 in build-system.requires too):

project(
    'package',
    'cpp',
    version: '0.1.0',
    license: 'BSD',
    meson_version: '>= 1.1.0',
    default_options: [
        'cpp_std=c++11',
    ],
)

py = import('python').find_installation(pure: false)
pybind11_dep = dependency('pybind11')

py.extension_module('_core',
    'src/main.cpp',
    subdir: 'package',
    install: true,
    dependencies : [pybind11_dep],
)

install_subdir('src/package', install_dir: py.get_install_dir() / 'package', strip_directory: true)

Meson requires that your source be tracked by version control. In a real project, you will likely be doing this, but when trying out a build backend you might not think to set up a git repo to build it. ::: :::{tab-item} Maturin :sync: maturin Example Cargo.toml file:

[package]
name = "package"
version = "0.1.0"
edition = "2021"

[lib]
name = "_core"
# "cdylib" is necessary to produce a shared library for Python to import from.
crate-type = ["cdylib"]

[dependencies]
rand = "0.9.2"

[dependencies.pyo3]
version = "0.27.2"
# "extension-module" tells pyo3 we want to build an extension module (skips linking against libpython.so)
# "abi3-py310" tells pyo3 (and maturin) to build using the stable ABI with minimum Python version 3.10
features = ["extension-module", "abi3-py310"]

::: ::::

Example compiled file

This example will make a _core extension inside your package; this pattern allows you to easily provide both Python files and compiled extensions, and keeping the details of your compiled extension private. You can select whatever name you wish, though, or even make your compiled extension a top level module.

::::{tab-set} :::{tab-item} Scikit-build-core :sync: scikit-build-core Example src/main.cpp file:

#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
  m.doc() = R"pbdoc(
      Pybind11 example plugin
      -----------------------
      .. currentmodule:: python_example
      .. autosummary::
         :toctree: _generate
         add
         subtract
  )pbdoc";

  m.def("add", &add, R"pbdoc(
      Add two numbers
      Some other explanation about the add function.
  )pbdoc");

  m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
      Subtract two numbers
      Some other explanation about the subtract function.
  )pbdoc");
}

::: :::{tab-item} Meson-python :sync: meson-python Example src/main.cpp file:

#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
  m.doc() = R"pbdoc(
      Pybind11 example plugin
      -----------------------
      .. currentmodule:: python_example
      .. autosummary::
         :toctree: _generate
         add
         subtract
  )pbdoc";

  m.def("add", &add, R"pbdoc(
      Add two numbers
      Some other explanation about the add function.
  )pbdoc");

  m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
      Subtract two numbers
      Some other explanation about the subtract function.
  )pbdoc");
}

::: :::{tab-item} Maturin :sync: maturin Example src/lib.rs file:

use pyo3::prelude::*;

/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
mod _core {
    use super::*;

    #[pyfunction]
    fn add(x: i64, y: i64) -> i64 {
        x + y
    }

    #[pyfunction]
    fn subtract(x: i64, y: i64) -> i64 {
        x - y
    }


    #[pymodule_init]
    fn pymodule_init(m: &Bound<'_, PyModule>) -> PyResult<()> {
        m.add("__version__", env!("CARGO_PKG_VERSION"))?;
        Ok(())
    }
}

::: ::::

Package structure

The recommendation (followed above) is to have source code in /src, and the Python package files in /src/<package>. The compiled files also can go in /src.

Versioning

Check the documentation for the tools above to see what forms of dynamic versioning the tool supports.

Including/excluding files in the SDist

Each tool uses a different mechanism to include or remove files from the SDist, though the defaults are reasonable.

Distributing

Unlike pure Python, you'll need to build redistributable wheels for each platform and supported Python version if you want to avoid compilation on the user's system using cibuildwheel. See the CI page on wheels for a suggested workflow.

Special considerations

NumPy

Modern versions of NumPy (1.25+) allow you to target older versions when building, which is highly recommended, and this became required in NumPy 2.0. Now you add:

#define NPY_TARGET_VERSION NPY_1_22_API_VERSION

(Where that number is whatever version you support as a minimum) then make sure you build with NumPy 1.25+ (or 2.0+). Before 1.25, it was necessary to actually pin the oldest NumPy you supported (the oldest-supported-numpy package is the easiest method). If you support Python < 3.9, you'll have to use the old method for those versions.

If using pybind11, you don't need NumPy at build-time in the first place.

:::{important} Python 3.13.4 is broken on Windows for compiling code - it always reports that it is free-threaded. 3.13.5 was rushed out to fix it. :::