Build / environment issue related to `ucxx`

[TomAugspurger]

The last couple runs have failed (https://github.com/rapidsai/dask-upstream-testing/actions/runs/15145027688/job/42578549944)

When we try to import ucxx and obtain the commit (roughly here) we hit an error:

/__w/dask-upstream-testing/dask-upstream-testing
Traceback (most recent call last):
  File "/__w/dask-upstream-testing/dask-upstream-testing/./scripts/check-version.py", line 37, in <module>
    main()
  File "/__w/dask-upstream-testing/dask-upstream-testing/./scripts/check-version.py", line 25, in main
    sha = importlib.resources.files(dist).joinpath("GIT_COMMIT").read_text().strip()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/home/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/importlib/resources/_common.py", line 46, in wrapper
    return func(anchor)
           ^^^^^^^^^^^^
  File "/github/home/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/importlib/resources/_common.py", line 56, in files
    return from_package(resolve(anchor))
                        ^^^^^^^^^^^^^^^
  File "/github/home/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/functools.py", line 912, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/home/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/importlib/resources/_common.py", line 82, in _
    return importlib.import_module(cand)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/home/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/ucxx/__init__.py", line 30, in <module>
    from . import exceptions, types, testing  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/ucxx/exceptions.py", line 4, in <module>
    from ._lib.libucxx import (  # noqa
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/ucxx/_lib/__init__.py", line 4, in <module>
    from .libucxx import _create_exceptions
ImportError: libpython3.12.so.1.0: cannot open shared object file: No such file or directory

I'll take a look.

[pentschev]

Looks like Python 3.13 picking a Python 3.12 build, either the package Python pin is incorrect or it was built with the wrong Python version.

[TomAugspurger]

Thanks, that's probably it? We should be using (uv's) cpython 3.12: https://github.com/rapidsai/dask-upstream-testing/actions/runs/15145027688/job/42578549944#step:10:33. But this is running inside the 3.13 container:https://github.com/rapidsai/dask-upstream-testing/actions/runs/15145027688/job/42578549944#step:3:18 and so something must depend on that.

I'll try updating to 3.13, and if that doesn't work I'll update the matrix filter.

[TomAugspurger]

In https://github.com/rapidsai/dask-upstream-testing/actions/runs/15146772957/job/42584316087#step:3:18 (a manually triggered run after #57) we do get python 3.12. But at https://github.com/rapidsai/dask-upstream-testing/actions/runs/15146772957/job/42584316087#step:10:424 we still fail with the same error.

So this may be an issue with the built ucxx wheel. I'll reopen this till I can confirm that.

[pentschev]

Yes, it is definitely a problem with the UCXX wheel.

[TomAugspurger]

Here's a reproducer:

$ docker run -it --rm -v (pwd):/src --gpus all rapidsai/citestwheel:cuda12.9.0-ubuntu24.04-py3.13

In that container:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv --allow-existing --python 3.12 --managed-python tmpvenv
source ./tmpvenv/bin/activate

uv pip install --prerelease=allow --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple "ucxx-cu12>=0.44.0a0,<0.45.0a0" python -c 'import ucxx; print(ucxx.__version__)'

I see

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmpvenv/lib/python3.12/site-packages/ucxx/__init__.py", line 30, in <module>
    from . import exceptions, types, testing  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmpvenv/lib/python3.12/site-packages/ucxx/exceptions.py", line 4, in <module>
    from ._lib.libucxx import (  # noqa
  File "/tmpvenv/lib/python3.12/site-packages/ucxx/_lib/__init__.py", line 4, in <module>
    from .libucxx import _create_exceptions
ImportError: libpython3.12.so.1.0: cannot open shared object file: No such file or directory

Running apt-get update && apt-get install strace and

strace -e open,openat python -c "import ucxx" 2>&1 | tee out.log

I do see us looking for stuff in /usr/local/cuda/lib64. I'm not sure if that's intentional (it's in LD_LIBRARY_PATH for the container env).

...
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/libucxx.cpython-312-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/usr/local/cuda/lib64/libucxx_python.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/glibc-hwcaps/x86-64-v3/libucxx_python.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/glibc-hwcaps/x86-64-v2/libucxx_python.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/libucxx_python.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/../lib64/glibc-hwcaps/x86-64-v3/libucxx_python.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/../lib64/glibc-hwcaps/x86-64-v2/libucxx_python.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/../lib64/libucxx_python.so", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/usr/local/cuda/lib64/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/../lib64/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v3/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v2/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v3/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v2/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/libpython3.12.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.venv/lib/python3.12/site-packages/ucxx/__init__.py", line 30, in <module>
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/__init__.py", O_RDONLY|O_CLOEXEC) = 4
    from . import exceptions, types, testing  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/ucxx/exceptions.py", line 4, in <module>
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/exceptions.py", O_RDONLY|O_CLOEXEC) = 4
    from ._lib.libucxx import (  # noqa
  File "/.venv/lib/python3.12/site-packages/ucxx/_lib/__init__.py", line 4, in <module>
openat(AT_FDCWD, "/.venv/lib/python3.12/site-packages/ucxx/_lib/__init__.py", O_RDONLY|O_CLOEXEC) = 4
    from .libucxx import _create_exceptions
ImportError: libpython3.12.so.1.0: cannot open shared object file: No such file or directory
+++ exited with 1 +++

[vyasr]

Is this a new problem? Has anything changed recently in this repo? If not, I would guess that the change is in the images themselves. We probably need to remove the explicit linkage to libpython altogether and rely on the symbols already being loaded. Direct linkage is intentionally prohibited in wheels for this reason IIRC. My guess is that the libucxx_python is being built with that anyway, but we've been lucky up until now to only be installing in environments where libpython exists.

I am quite surprised that we have never seen this issue before, though...

[TomAugspurger]

Ah, astral-sh/python-build-standalone#619 might be relevant. Reading through that now.

(For posterity, I compared the logs of a successful and failed run. The both use the exact same container image: rapidsai/citestwheel:cuda12.9.0-ubuntu24.04-py3.13 with the same SH256 digest: Digest: sha256:53e1e7cffbc98baf6ad40e99af1523ef51cf58ab3a6c7ba3ddf540d8fe0227ed.

I diffed the two sets of packages installed and didn't find anything interesting. Neither ucxx nor libucxx changed.

[vyasr]

Yes, that is definitely the relevant change. Wheels are not supposed to link to libpython directly to support this kind of environment, but in practice we are hardly ever installed in such an environment so it hasn't been an issue. In theory we would have run into this if someone installed ucxx directly into the main sysroot on some Linux distros I believe, but in practice I'm guessing that nobody is doing that so far and so hasn't run into this. I'm not sure offhand what the cleanest fix for this is offhand; we probably need to update our linkage to avoid linking directly to libpython. It'll be a bit annoying since we'll have to hack the CMake a little to get includes without linkage, but it should be doable.

[geofft]

FYI (with my python-build-standalone hat on) I'm coming to the same conclusion about the wheel mentioned in that issue, there's some CMake change and if this is not the obvious thing to do with CMake, then we need to think a little harder.

If you find yourself doing a nontrivial technical change to address this, or if it seems like this is going to be disruptive in the wider ecosystem and we need a better rollout plan, please reach out (you can at-mention me in the Astral or PyPA Discords for something IM-like).

[geofft]

Ah, I raced with your comment in our issue tracker, I see that you are indeed raising this concern and will follow up there :)