[WIP] Cython language

[LecrisUT]

So I am playing around with creating a Cython language to try and simplify the way to make Cython files. The basic files that need to be created are documented here and I am looking at other implemented Fortran, CSharp, etc. languages for reference.

The end-goal is to make the CMakeLists.txt as:

cmake_minimum_required(VERSION 3.25...3.29)
project(simple LANGUAGES C Cython)

python_add_library(simple MODULE WITH_SOABI)
target_sources(simple PRIVATE simple.pyx)

install(TARGETS simple DESTINATION .)

Approach 1: set(CMAKE_Cython_OUTPUT_EXTENSION ".c")

The hope here is that we could define only the bare minimum to create the necessary .c files and have the C compiler part take over from there. Unfortunately CMake does not seem to chain the parts accordingly.

Maybe if we can manipulate the dependencies in .dep this approach would work.

Approach 2: chain CMAKE_Cython_COMPILE_OBJECT with CMAKE_C_COMPILE_OBJECT

This should probably work with the current implementation, and at first glance it seems to do what it needs to do, but the link-library part doesn't seem to be configured correctly yet

[LecrisUT]

With a6ad971 it somewhat works:

• Figure out why it appends .so twice. If you manually edit that, the installed files seem to import correctly
• Dependencies how?
• Could python_add_library also be simplified to add_library?

[henryiii]

🤖 AI text below 🤖

Rebased onto current main and resolved the linking issue (force-pushed; squashed the two [WIP] commits into one feat: commit).

The .so.so linking bug

The build produced simple.cpython-XYZ-darwin.so.so — a double extension, so the module was not importable. The old test only checked the wheel file count, so it passed despite shipping a broken wheel.

Root cause: CMakeDetermineCythonCompiler.cmake called find_package(Python ... Development.Module) during language determination, when CMAKE_SHARED_LIBRARY_SUFFIX is still empty. FindPython's SOABI-stripping regex depends on that variable, so it cached a malformed Python_SOABI (cpython-XYZ-darwin.so, with a trailing .so). python_add_library(... WITH_SOABI) then appended another .so.

Fix: drop find_package(Python) from the determine step (it was unused there — only find_program(cython) is needed) and let the consuming CMakeLists.txt find Python after C is enabled. The module now builds as simple.cpython-XYZ-darwin.so and imports correctly (square(3.0) == 9.0).

Other cleanups

• Removed debug noise: message(WARNING "Entering...") in every module, the dummy2/3/4 placeholder compiler IDs, and the dead "Approach 1/2" comment blocks.
• CMakeTestCythonCompiler.cmake now reports a real FATAL_ERROR on a failed test compile instead of silently swallowing it.
• Test package CMakeLists.txt: enabled the real find_package(Python), dropped the redundant target_link_libraries(... Python::Module) and the CMAKE_VERBOSE_MAKEFILE toggle.
• test_simple_language now asserts the extension carries exactly one EXT_SUFFIX (regression guard against .so.so) and that the Cython→C transpile output exists.

Full suite (14 tests) and pre-commit pass.

Still experimental / possible follow-ups

The linking blocker is resolved, but the language does not yet have feature parity with cython_transpile (no CXX selection, CYTHON_ARGS, or dependency tracking wired in), and the generated C lands at CMakeFiles/<target>.dir/<source>.pyx.o.c.

[henryiii]

🤖 AI text below 🤖

Pushed a follow-up commit (6ea2c64) moving the Cython language closer to cython_transpile parity. CI is green (the language tests run on modern CMake and skip on Windows / CMake < 3.25).

Now supported in the language

• Include directories — <INCLUDES> is forwarded to the cython step, so target_include_directories() resolves cimported .pxd files (the same dirs still reach the C compiler for headers).
• CYTHON_ARGS — baked into the compile rule when the language is enabled, so it must be set before project()/enable_language(Cython). It is deliberately kept off <FLAGS>: CMake fills <FLAGS> with C-only options (-fPIC, build-type flags) that cython rejects (passing --annotate through <FLAGS> makes the C compiler error). Multi-value lists are space-joined into one token, since CMAKE_Cython_COMPILE_OBJECT is itself a ;-list of commands.
• Dependency tracking — cython -M writes a gcc-style depfile that is copied onto CMake's <DEP_FILE> (CMAKE_Cython_DEPFILE_FORMAT=gcc), so editing a cimported .pxd triggers a rebuild.

A new language_features test package exercises all three: a cimport resolved via an include dir, multi-value CYTHON_ARGS (asserting the --annotate HTML report), and the .pxd depfile.

Remaining gaps vs cython_transpile

• Per-source C/CXX selection — the language has a single compile rule, so the per-file LANGUAGE C|CXX / # distutils: language=c++ behaviour is not achievable in this design. This is the main remaining gap; cython_transpile stays the full-featured path for C++.
• Genex inside CYTHON_ARGS — supported by the function, not by the baked language rule.
• Windows and CMake < 3.25 — still unsupported (tests skipped there).

[henryiii]

🤖 AI text below 🤖

C++ support via a source property — feasibility (prototyped, works)

Short answer: yes, a source property is the right mechanism — but it has to be the LANGUAGE property pointed at a parallel CythonCXX language, not a flag on the existing Cython language.

Why a plain property on the single Cython language can't work

• The only per-source channel into the compile rule is <FLAGS>. In this design <FLAGS> is owned by the C-compile stage and carries -fPIC / build-type optimization flags that the cython CLI rejects, so it can't be used to slip --cplus into the cython invocation.
• Bare cython does not honor # distutils: language=c++ (it warns Using 'cppclass' while Cython is not in c++ mode and emits broken output), so --cplus genuinely must be passed per source.
• The compiler and output extension are global to a CMake language, so one rule can't switch between cc/.c and c++/.cpp per file.

What does work: a sibling CythonCXX language selected per source

project(mixed LANGUAGES C CXX Cython CythonCXX)
find_package(Python COMPONENTS Interpreter Development.Module REQUIRED)

# .pyx defaults to the C `Cython` language; opt individual files into C++:
set_source_files_properties(cppmod.pyx PROPERTIES LANGUAGE CythonCXX)

python_add_library(cppmod MODULE WITH_SOABI)
target_sources(cppmod PRIVATE cppmod.pyx)

CythonCXX runs cython --cplus → .cpp → the CXX compiler → CXX link. I prototyped it and verified end to end:

• A module using from libcpp.vector cimport vector builds, imports, and runs (total([1,2,3,4]) == 10).
• C and C++ Cython modules coexist in one project: cmod builds as Cython, cppmod as CythonCXX; both import and run.

.pyx stays mapped to the C Cython language by extension (two languages can't claim the same extension), so C++ is opt-in per source via the property — which is the behaviour we want.

To productionize

1. Add the four CythonCXX language files (CMakeDetermineCythonCXXCompiler.cmake, CMakeCythonCXXCompiler.cmake.in, CMakeTestCythonCXXCompiler.cmake, CMakeCythonCXXInformation.cmake) — ideally factoring the shared compile-rule logic into a common include to avoid duplicating the C variants.
2. A mixed C/C++ test package + test.
3. Optionally a convenience helper that auto-applies LANGUAGE CythonCXX when a file contains # distutils: language=c++ (mirroring _cython_compute_language in UseCython.cmake), so the language path gets the same auto-detection as cython_transpile.

[henryiii]

I think we should probably split this out into a separate package, as it doesn't seem like this approach has an on/off switch. If there ever is a Cython language, I wouldn't want to interfere with it, and there are some drawbacks (like C++ vs C selection). We could add a new cython-lang-cmake package.

[LecrisUT]

I think we should probably split this out into a separate package, as it doesn't seem like this approach has an on/off switch. If there ever is a Cython language, I wouldn't want to interfere with it, and there are some drawbacks (like C++ vs C selection). We could add a new cython-lang-cmake package.

Ah as in another package also defining the language or worse CMake itself? I think it is fine since the user would probably not have a use for the other parts of this if that were the case?

Splitting it on the spec side is trivial, and for pypi, probably a sub project?