CUDA 13.2.1 libcudart causes SIGSEGV in cuInit on WSL2 (Driver 596.49, Blackwell)

[gymaira1990-jpg]

CUDA 13.2.1 libcudart.so.13.2.75 causes SIGSEGV in cuInit on WSL2 with NVIDIA Driver 596.49 (R595, Blackwell RTX 5070)

Environment

Component	Version
OS	Windows 11 24H2 + WSL2 (Ubuntu 24.04, kernel 6.6.87.2-microsoft-standard-WSL2)
GPU	NVIDIA GeForce RTX 5070 Laptop GPU (Blackwell, compute capability sm_120a)
NVIDIA Driver (Windows)	596.49 (R595 Production Branch)
CUDA Toolkit	13.2.1 (CUDA 13.2.78, installed via runfile cuda_13.2.1_595.58.03_linux.run --toolkit --silent --nox11)
WSL CUDA Driver lib	/usr/lib/wsl/lib/libcuda.so.1 (shipped with driver 596.49, reports CUDA 13.2 via nvidia-smi)
Application	llama.cpp b9305 (compiled from source with -DGGML_CUDA=ON -DGGML_CUDA_FA=OFF)

Bug Description

cuInit() (called from ggml_cuda_init() in llama.cpp) triggers a SIGSEGV (signal 11) inside libc.so.6 when linked against CUDA 13.2.1's libcudart.so.13.2.75.

The crash occurs regardless of:

• --n-gpu-layers setting (even with 0 layers, CUDA init still runs)
• CUDA_VISIBLE_DEVICES mode (happens with device 0, bypassed only with CUDA_VISIBLE_DEVICES="")
• --main-gpu flags

Crash Detail (from dmesg)

llama-cli[25370]: segfault at 1 ip 0000706d0299b95c sp 00007fff50a98898 error 4 in libc.so.6

• signal: 11 (SIGSEGV)
• fault address: 0x1 (null pointer + 1 offset)
• error code: 4 (user-mode read fault)
• location: libc.so.6 — indicating a NULL pointer dereference in a libc function called during CUDA runtime initialization

The crash happens during the ggml_cuda_init() path, before any model loading or GPU memory allocation occurs.

Workaround

Replace the CUDA runtime library with a prior version that is compatible with the WSL driver.

Using the CUDA 13.0 runtime library (libcudart.so.13.0.96) that ships with Ollama's bundled CUDA v13 runtime:

# Runtime environment:
LD_LIBRARY_PATH=/path/to/cuda_v13:/usr/lib/wsl/lib ./llama-server ...

This is binary-compatible because:

• CUDA minor version compatibility allows libcudart.so.13.0 to work with a CUDA 13.x driver
• libcudart is a pure user-space library wrapping the kernel driver interface (libcuda.so.1)
• The compiled SM_120a GPU code executes without any performance penalty (verified: 95.5 tokens/sec generation, identical to expected Blackwell throughput)

Root Cause Hypothesis

The bug appears to be in the CUDA Runtime library's initialization path (libcudart.so.13.2.75) when interacting with the WSL2 bridge driver (/usr/lib/wsl/lib/libcuda.so.1). Likely:

1. A new code path or internal API call in libcudart.so.13.2.75 that is incompatible with the WSL libcuda.so.1 interface (even though nvidia-smi correctly reports CUDA 13.2 compatibility)
2. The WSL DXG-mapped CUDA driver exposes a different internal interface than native Linux CUDA drivers, which the updated runtime may not handle correctly
3. A NULL pointer dereference during device property enumeration or context creation

Steps to Reproduce

1. Set up WSL2 on Windows 11 with NVIDIA Driver 596.49
2. Install CUDA Toolkit 13.2.1:

   wget https://developer.download.nvidia.com/compute/cuda/13.2.1/local_installers/cuda_13.2.1_595.58.03_linux.run
   sudo sh cuda_13.2.1_595.58.03_linux.run --toolkit --silent --nox11

3. Build any CUDA application that calls cuInit():

   export PATH=/usr/local/cuda/bin:$PATH
   # Build llama.cpp or any CUDA sample

4. Run the application:

   export LD_LIBRARY_PATH=/usr/local/cuda/targets/x86_64-linux/lib
   ./cuda_app

Expected: CUDA initializes successfully, GPU is detected.
Actual: SIGSEGV during cuInit().

Additional Context

• NVIDIA Driver 596.49 is from the R595 Production Branch, which NVIDIA officially lists as supporting CUDA 13.x
• The driver's libcuda.so.1 at /usr/lib/wsl/lib/ is 183 KB and is a WSL-specific virtualized CUDA driver
• nvidia-smi correctly reports CUDA Version: 13.2
• This is not a Blackwell architecture issue — it's a runtime library compatibility issue
• The workaround with libcudart.so.13.0.96 produces identical performance to expected CUDA 13.2 generation

Suggested Fix

Review the cuInit() and device discovery code path in libcudart.so.13.2.75 for WSL-specific handling. The WSL libcuda.so.1 may expose a different set of driver entry points or handle certain CUDA API calls differently than the native Linux CUDA driver, and the updated runtime library may be making assumptions that only hold for native Linux.