[BUG][pathfinder]: nvJitLink loaded from system library instead of conda

[rwgk]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Silent Failure

Component

cuda.bindings

Describe the bug

@leofang wrote:

I noticed that on the system that I’m on, which has a system CTK 12.3 and I installed CTK 12.9 from conda, the pathfinder from either cuda.bindings 12.9.0 or cuda.pathfinder 1.0.0 would pick up nvJitLink 12.3 (the system one) instead of 12.9 (the conda one), which is not following the behavior that we documented.
I suspect that the logic in _load_nvidia_dynamic_library_no_cache might be wrong:

    # Find the library path
    found = _find_nvidia_dynamic_library(libname)
    if found.abs_path is None:
        loaded = load_with_system_search(libname, found.lib_searched_for)

because in _find_nvidia_dynamic_library we always do this on Linux:

    self.lib_searched_for = f"lib{libname}.so"

meaning we don’t search with the full soname (libnvJitLink.so.12), but the symlink name (libnvJitLink.so), which conda does not provide if we only install the libnvjitlink package and not the libnvjitlink-dev package.
Therefore, the load_with_system_search function behaves wrong because we fed it a wrong soname.

How to Reproduce

I think a simple reproducer would be:

• launch a vanilla Ubuntu container
• Install miniforge and then create a new conda env with only cuda-pathfinder (from pip) and libnvjitlink (from conda-forge) installed.
• Run the pathfinder

from cuda import pathfinder
pathfinder.load_nvidia_dynamic_lib("nvJitLink")

Expected behavior

The conda .so should be found.

Operating System

No response

nvidia-smi output

No response

[rwgk]

Here is a simple, complete reproducer (running under Windows11 WSL2 Ubuntu with CTK 12.9 in /usr/local/cuda):

#!/bin/bash
eval "$(conda shell.bash hook)"
set -euo pipefail
# conda env remove -y -n cuda-python_issues_763
conda create -y -n cuda-python_issues_763 python=3.12
conda activate cuda-python_issues_763
pip install cuda-pathfinder
set -x
cat run.py
python run.py
set +x
conda install -y -c conda-forge libnvjitlink
set -x
echo "$CONDA_PREFIX"
ls -l $CONDA_PREFIX/lib/libnvJitLink*
python run.py

Last part of output:

+ cat run.py
from cuda import pathfinder
print(pathfinder.load_nvidia_dynamic_lib("nvJitLink"))
+ python run.py
LoadedDL(abs_path='/usr/local/cuda-12.9/targets/x86_64-linux/lib/libnvJitLink.so', was_already_loaded_from_elsewhere=False, _handle_uint=102787379914384)
...
+ ls -l /home/rgrossekunst/miniforge3/lib/libnvJitLink.so.12 /home/rgrossekunst/miniforge3/lib/libnvJitLink.so.12.9.86
lrwxrwxrwx 1 rgrossekunst rgrossekunst 51 Jul 18 16:03 /home/rgrossekunst/miniforge3/lib/libnvJitLink.so.12 -> ../targets/x86_64-linux/lib/libnvJitLink.so.12.9.86
lrwxrwxrwx 1 rgrossekunst rgrossekunst 51 Jul 18 16:03 /home/rgrossekunst/miniforge3/lib/libnvJitLink.so.12.9.86 -> ../targets/x86_64-linux/lib/libnvJitLink.so.12.9.86
+ python run.py
LoadedDL(abs_path='/usr/local/cuda-12.9/targets/x86_64-linux/lib/libnvJitLink.so', was_already_loaded_from_elsewhere=False, _handle_uint=94767212164752)

This shows that $CONDA_PREFIX/lib/libnvJitLink.so.12 exists but is ignored by load_nvidia_dynamic_lib().

[rwgk]

The bug is here:

• cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py

  Line 40 in 248edca

  loaded = load_with_system_search(libname, found.lib_searched_for)

• cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_dl_linux.py

  Lines 85 to 105 in 248edca

  	def load_with_system_search(libname: str, soname: str) -> Optional[LoadedDL]:
  	"""Try to load a library using system search paths.
  	Args:
  	libname: The name of the library to load
  	soname: The soname to search for
  	Returns:
  	A LoadedDL object if successful, None if the library cannot be loaded
  	Raises:
  	RuntimeError: If the library is loaded but no expected symbol is found
  	"""
  	try:
  	handle = ctypes.CDLL(soname, CDLL_MODE)
  	abs_path = abs_path_for_dynamic_library(libname, handle)
  	if abs_path is None:
  	raise RuntimeError(f"No expected symbol for {libname=!r}")
  	return LoadedDL(abs_path, False, handle._handle)
  	except OSError:
  	return None

After adding a print statment in load_with_system_search():

    print(f"load_with_system_search({libname=!r}, {soname=!r})", flush=True)

Then running:

$ python ~/cuda-python_issues_763/run.py
load_with_system_search(libname='nvJitLink', soname='libnvJitLink.so')
LoadedDL(abs_path='/usr/local/cuda/lib64/libnvJitLink.so', was_already_loaded_from_elsewhere=False, _handle_uint=107606264761008)

This happens because libnvJitLink.so only exist here:

$ ls -l /usr/local/cuda/lib64/libnvJitLink*so* | sed 's/  */ /g' | cut -d' ' -f9-
/usr/local/cuda/lib64/libnvJitLink.so -> libnvJitLink.so.12
/usr/local/cuda/lib64/libnvJitLink.so.12 -> libnvJitLink.so.12.9.86
/usr/local/cuda/lib64/libnvJitLink.so.12.9.86

But not here:

$ ls -l $CONDA_PREFIX/lib/libnvJitLink*so* | sed 's/  */ /g' | cut -d' ' -f9-
/home/rgrossekunst/miniforge3/envs/fix763/lib/libnvJitLink.so.12 -> ../targets/x86_64-linux/lib/libnvJitLink.so.12.9.86
/home/rgrossekunst/miniforge3/envs/fix763/lib/libnvJitLink.so.12.9.86 -> ../targets/x86_64-linux/lib/libnvJitLink.so.12.9.86

We need to work around this somehow, by making the system search more specific.

But a question first, just in case that informs what we need to do in general:

@leofang @kkraus14 Do you know if conda (or us?) deliberately did not create the .so softlink? Why?

(What I fear is: Could libnvJitLink.so.12 and, e.g., a hypothetical future libnvJitLink.so.13 both be present in the same directory?)

[leofang]

Do you know if conda (or us?) deliberately did not create the .so softlink? Why?

When packaging the CUDA packages on conda-forge, we follow the Debian packaging convention and split out 3 packages per library: library, library-dev, and libraray-static. The *-dev packages contain the headers and (unversioned) symlinks. This is documented in https://github.com/conda-forge/cuda-feedstock/blob/main/recipe/doc/end_user_compile_guide.md#package-naming-conventions. In short, we should not assume that the (unversioned) symlinks exist, which is an assumption that's also applicable to wheels.

[leofang]

(I edited my reply above to emphasize "unversioned"; the versioned symlinks (ex: $CONDA_PREFIX/lib/libnvJitLink.so.12 -> $CONDA_PREFIX/targets/x86_64-linux/lib/libnvJitLink.so.12.9.86) do exist and we should use them like when dealing with wheels.)

[rwgk]

When packaging the CUDA packages on conda-forge, we follow the Debian packaging convention and split out 3 packages per library: library, library-dev, and libraray-static. The *-dev packages contain the headers and (unversioned) symlinks. This is documented in https://github.com/conda-forge/cuda-feedstock/blob/main/recipe/doc/end_user_compile_guide.md#package-naming-conventions. In short, we should not assume that the (unversioned) symlinks exist, which is an assumption that's also applicable to wheels.

Thanks @leofang!

For the minute I think it's best to focus completely on the system-load path. Once we're sure what we want for that, we can revisit loading from wheels.

The minimal fix that's currently under PR #767 passes all existing testing.

It simply tries to load libnvJitLink.so.12 first, and if that doesn't succeed, tries loading libnvJitLink.so.

Questions that pop up:

• Should we actually try to load libnvJitLink.so?

More generally, let's look at this:

    "cublas": (
        "libcublas.so.11",
        "libcublas.so.12",
    ),

With the current PR #767, this will try to load in this order: libcublas.so.11, libcublas.so.12, libcublas.so

What do we want ideally?

• Try to determine the driver version, and use that if available?

• If we cannot determine the driver version, do we want to keep the order above, or change it to libcublas.so.12, libcublas.so.11, libcublas.so?

(Either way seems rather arbitrary. But I think it'll be best to settle on deterministic behavior, and fully document it.)

[leofang]

• Try to determine the driver version, and use that if available?

Yes, I think we should address this together with #654. We should detect the driver version and then make a decision:

• if it's CUDA 12 driver: try .so.12 -> .so.11 -> .so
• if it's CUDA 11 driver: try .so.11 -> .so

xref: https://github.com/NVIDIA/nvmath-python/blob/e082fedb46a57072e63d3b8f0aae3b451eeddcbb/nvmath/bindings/_internal/utils.pyx#L126-L176

In general, the soname versions may not be aligned with the driver version (example), but we already have them tabulated in the supported sonames.

In case of the driver being not installed (ex: on a CPU-only build machine), we can offer an env var to inform the path finder which driver version we'd like to pretend with. Conda's CONDA_OVERRIDE_CUDA is one such example.

I can be convinced that we should drop the unversioned .so from the search order. I would agree that using it could be error-prone.