Cannot map shared library installed by CMake subproject to wheel install location

[tobiasdiez]

In a pyproject-based project, an external dependency is built using cmake (and another one using autotools) as fallback. This works well using plain meson, however pip install --editable . fails with

meson-python: error: Could not map installation path to an equivalent wheel directory: '{prefix}/lib/.so'

(see https://github.com/sagemath/sage/actions/runs/8222294612/job/22483617409?pr=37077#step:5:54174)

I can workaround this error message by providing the skip-subprojects argument for install. Then the editable wheel is built successful, but on runtime it does not find the <external>.so file just built.

I've seen

• Best practice for optionally bundling a library dependency #556
• How can I bundle a shared library in a wheel with mesonpy? #410
• Building Windows wheels with libraries fails #525

which discuss similar issues. However, my main problem (at least for the moment) is to create an editable installation for (local) development, and not build a wheel for pypi. So cibuildwheel etc are not really a solution.

[dnicolodi]

Does it work if you build a regular wheel? I don't think there is a way for meson-python to support cmake or autotools subprojects: there is no way to map the files installed by these subprojects to wheel install locations.

[dnicolodi]

By the way, what's the purpose of building an editable wheel for running the tests in CI? The only advantage of editable build is to have the package automatic rebuilt on edit, but I think your CI jobs are not editing the source code.

[dnicolodi]

To clarify: meson-python assembles a Python wheel from the files installed by the Meson project. To do so it uses introspection data exported by Meson and maps installation locations into the host filesystem to wheel install locations. The wheel format is designed to install Python modules, Python extension modules, and little else. meson-python somehow extends this to also support shared libraries (but not on Windows), which are installed in a non-standard location. Consequently, meson-python is able to map files installed to a few locations:

meson-python/mesonpy/__init__.py

Lines 83 to 93 in 2daa991

	_INSTALLATION_PATH_MAP = {
	'{bindir}': 'scripts',
	'{py_purelib}': 'purelib',
	'{py_platlib}': 'platlib',
	'{moduledir_shared}': 'platlib',
	'{includedir}': 'headers',
	'{datadir}': 'data',
	# custom location
	'{libdir}': 'mesonpy-libs',
	'{libdir_shared}': 'mesonpy-libs',
	}

These locations are enough to cover all files that can be mapped into a wheel. However, when using the cmake subproject support, Meson does not map the files installed by the cmake project to these installation prefixes. I think it could be possible to extend the logic used to map filesystem locations to wheel install paths to also understand {prefix}/lib/.

Subprojects using the Meson's external project module to build and autotools project do not have a list of the installed files, thus are supported by meson-python.

[dnicolodi]

I must take back some of the considerations above. I just put together a simple test package that installs an extension module that links to a shared library compiled as part of a cmake subproject and all works as expected. Apparently Meson is smart enough to correctly assign the shared library to the {libdir_shared} install location.

I don't know why in your case the install location for the shared library is {prefix}/lib/. We need much more information that what has been provided so far to understand where the issue is. I suspect it depends on how the cmake project is setup. However, I know little to nothing about cmake and I'm not sure I can debug this. We would at least need to know which library causes the issue.

[dnicolodi]

See #599. Interestingly, the test works on macOS but fails on Linux. There is a difference in how the Meson cmake modules maps installations paths on macOS and Linux. Some more investigation is required.

[tobiasdiez]

Oh wow @dnicolodi. Thanks a lot for diving so quickly into this issue. Do I understand it correctly, that with #599 you are able to reproduce the problem? Let me know if there is something I can do to assist in fixing or if you want me to try a new build etc.

To respond to your questions:

By the way, what's the purpose of building an editable wheel for running the tests in CI? The only advantage of editable build is to have the package automatic rebuilt on edit, but I think your CI jobs are not editing the source code.

No particular reason. I had various problems with the ci build and couldn't reproduce them locally within a devcontainer. So I tried to minimize the difference in the environments. Migrating sagemath to meson is a huge undertaking, still quite at the beginning and there are a lot of moving parts. Of course, normally you don't want/need to use editable builds for CI.

Subprojects using the Meson's external project module to build and autotools project do not have a list of the installed files, thus are supported by meson-python.

You probably meant not supported, right? This partly makes sense. On the other hand, an external project is also just a custom target, or not? And meson somehow understands where it should install the files afterwards.

I don't know why in your case the install location for the shared library is {prefix}/lib/. We need much more information that what has been provided so far to understand where the issue is. I suspect it depends on how the cmake project is setup. However, I know little to nothing about cmake and I'm not sure I can debug this. We would at least need to know which library causes the issue.

The subproject in question is flint as configured in https://github.com/sagemath/sage/blob/042c42ecfadce27d85be05ded583487dc570fca6/subprojects/flint/meson.build (whole PR sagemath/sage#37077). But it's a quite hacky setup and it turns out there are even issues with plain meson (meson claims to "Installing subprojects/flint-3.0.1/libflint.so to /usr/local/lib" but afterwards it cannot be found/its shadowed by a older system version). So I don't think it serves as a good isolated testing environment.

[dnicolodi]

Thanks a lot for diving so quickly into this issue.

No problem. Some of my early comments are based on wrong assumptions about how Meson's support for CMake subprojects works. Sorry for the confusion.

Do I understand it correctly, that with #599 you are able to reproduce the problem?

Yes, this is the case. I don't understand yet the root cause of the issue but I have what I need to investigate.

You probably meant not supported, right?

Yes, I meant not supported. But the statement is based on a wrong assumption on my part.

This partly makes sense. On the other hand, an external project is also just a custom target, or not? And meson somehow understands where it should install the files afterwards.

Not really. For CMake subprojects, Meson uses a reimplementation of the CMake interpreter that translates the CMake build definition into a Meson build definition and runs that with the regular Meson interpreter, with some limitations. Consequently, Meson knows which build artifacts are installed where, and meson-python can use this information to build the Python wheel. Unfortunately, it seems that somewhere Meson gets confused and does not assign the correct installation location to shared libraries build via the CMake "emulation" on Linux. This is the source of the issue you observe.

For autotools subprojects included with the external_project Meson module, Meson runs the usual configure, make, make install steps specifying as $DESTDIR a directory inside the Meson build directory. All files installed there are copied into their final location when the Meson project is installed. For subprojects using the external_projects module, Meson does not know (and cannot easily know) which files are installed where. The only information available to meson-python is that there is a directory tree in the buildir that needs to be copied into {prefix}. This is currently not supported, but could be supported with some heuristic (and some limitations, for example assuming that the subproject does not install Python modules). However, we did some work to get rid of such heuristic, and I would like to avoid to bring it back, unless there are very important reasons. The support for including shared libraries in Python wheels does anyhow not currently work on Windows, thus if you need cross-platform support, the story is complicated anyway.

But it's a quite hacky setup

Indeed. In my experience, it is much less time consuming to write a proper Meson build definition for the external projects that to try to work around all quirks. Especially for mature projects, it is mostly a one time cost.

[tobiasdiez]

@dnicolodi I just experienced the same problem with an autoconf subproject:

meson-python: error: Could not map installation path to an equivalent wheel directory: '{prefix}'

see sagemath/sage#40038 for the code (https://github.com/sagemath/sage/actions/runs/14821384349/job/41608803714?pr=40038#step:7:81813)

Is there a workaround to make such subprojects work with meson-python?

[rgommers]

The support for including shared libraries in Python wheels does anyhow not currently work on Windows,

This works now in the 0.18.0 release: https://mesonbuild.com/meson-python/how-to-guides/shared-libraries.html

see sagemath/sage#40038 for the code (https://github.com/sagemath/sage/actions/runs/14821384349/job/41608803714?pr=40038#step:7:81813)

That's a long build log:) The issue is that you're building https://gitlab.com/embeddable-common-lisp/ecl as a subproject and would like to avoid having this in your pyproject.toml, right?

]
[tool.meson-python.args]
# Prevent meson from trying to install the autoconf subprojects
# otherwise we hit https://github.com/mesonbuild/meson-python/issues/598
# The subprojects have to set the rpath to the build directory, which should work as long as
# we don't use build isolation.
install = ['--skip-subprojects', 'ecl']

Indeed. In my experience, it is much less time consuming to write a proper Meson build definition for the external projects that to try to work around all quirks. Especially for mature projects, it is mostly a one time cost.

This is still the most reliable solution I think - I wouldn't want to rely on fragile autotools support for something as widely used as Sage.

[tobiasdiez]

The support for including shared libraries in Python wheels does anyhow not currently work on Windows,

This works now in the 0.18.0 release: https://mesonbuild.com/meson-python/how-to-guides/shared-libraries.html

see sagemath/sage#40038 for the code (https://github.com/sagemath/sage/actions/runs/14821384349/job/41608803714?pr=40038#step:7:81813)

That's a long build log:)

Yes, sorry, sage is indeed not a good test project for a "minimal non-working example". I'd hoped that the relatively self-contained changes in this PR give a good enough impression of the problem.

The issue is that you're building https://gitlab.com/embeddable-common-lisp/ecl as a subproject and would like to avoid having this in your pyproject.toml, right?

]
[tool.meson-python.args]
Prevent meson from trying to install the autoconf subprojects
otherwise we hit #598
The subprojects have to set the rpath to the build directory, which should work as long as
we don't use build isolation.
install = ['--skip-subprojects', 'ecl']

Exactly! This is the workaround I came up with for now. It kinda works if you don't use build isolation since then the "temporary" (DESTDIR=dist) installation of the autoconf project stays present in the build directory and thus can be used.
But, of course, it would be nicer if these files would be properly installed in the wheel.

For me, it would be also fine to manually tag the to-be-installed files so that meson-python can put them in the correct place.

Indeed. In my experience, it is much less time consuming to write a proper Meson build definition for the external projects that to try to work around all quirks. Especially for mature projects, it is mostly a one time cost.

This is still the most reliable solution I think - I wouldn't want to rely on fragile autotools support for something as widely used as Sage.

I understand this and it agrees also with the general philosophy of meson concerning their autotools support. However, sage has 40+ external dependencies (https://github.com/sagemath/sage/blob/7888c42db953cc5852fbdf7e21dc524a219ae5d2/pyproject.toml#L108-L162), none of them have a meson build backend and some are quite complicated to build. So it's very unlikely that we can implement even a reasonable subset as meson subprojects.

(sage currently has even its own "distribution" mechanism to build the complete dependency tree (using autotools). That comes with it's own problems and more often than not runs into build issues. I don't need a 'perfect' solution - just something that is better than the current fragile, custom-build solution.)