CUDA illegal memory access when running Segger on multi-GPU machine

[enric-bazz]

When running Segger CLI to segment Xenium version 4.0.0. data on a machine with 2 separate GPUs, I encounter a cascade of CUDA: illegal memory access errors when both GPUs are visible. This appears to be caused by automatic distributed process spawning. Limiting execution to a single GPU avoids the issue.

---

Steps to reproduce

1. Run Segger CLI on a 2× NVIDIA GeForce RTX 4090 machine with the default environment.
2. Observe output:

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
...
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------

3. Segger crashes with multiple errors such as:

RuntimeError: parallel_for: failed to synchronize: cudaErrorIllegalAddress
During handling of the above exception, another exception occurred:
RuntimeError: CUDA error: an illegal memory access was encountered
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS

4. Limiting the session to a single GPU resolves the issue:

export CUDA_VISIBLE_DEVICES=0

Output:

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
...
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

Execution completes without errors.

---

Environment

• Python: 3.11.14
• Segger: 0.2.0
• PyTorch: 2.5.0+cu121
• Lightning: 2.6.0
• CUDA: 12.2.0
• NVIDIA Drivers: 535.247.01
• GPU: 2 × NVIDIA GeForce RTX 4090

Relevant packages and versions:

Package	Version
torch_scatter	2.1.2+pt25cu121
cuml-cu12	25.4.0
cugraph-cu12	25.4.1
cuspatial-cu12	25.4.0
cudf-cu12	25.4.0
cupy-cuda12x	13.6.0

[Tobiaspk]

Hey, were you able to fix this?

On another system I got it to run using 24.10 (instead of 25.4). looks like 24.12 and later switched to a different memory allocator that just didn't work on our system. May be worth a shot.

[jpark27]

Hi, @Tobiaspk!
With 'release/v2-stable', most of dataset run fine with 1 GPU (NVIDIA L40S).

But one dataset needed 2 GPUs then I got same error as @enric-bazz
Unfortunately, my packages were already 24.10

Name: cudf-cu12
Version: 24.10.1
Name: cuml-cu12
Version: 24.10.0
Name: cugraph-cu12
Version: 24.10.0
Name: cuspatial-cu12
Version: 24.10.0

Do we have other possible solutions to fix memory allocating issue on multiple-GPU?

[enric-bazz]

Hi @Tobiaspk, downgrading RAPIDS to 24.10 unfortunately does not solve the issue, as @jpark27 also reported. Setting the argument devices=1 in all Lightning trainers forces single-GPU usage and prevents it from spawning distributed processes, achieving the same effect as setting CUDA_VISIBLE_DEVICES=0.

It seems that the current implementation of Segger with RAPIDS/CuPy may not support multi-GPU training.

[Tobiaspk]

Thanks @enric-bazz . Which RAPIDS version is working for you?

[enric-bazz]

@Tobiaspk I did not install the full RAPIDS stack, only the required packages. The version I’m using is 25.4, the full list is included in the issue description.