--enable_cameras casues unknown failure on Headless Server

[omgunjal2727]

IsaacSim

[ ] 6.0.0
[x] 5.1.0
[ ] 5.0.0
[ ] 4.5.0
[ ] 4.2.0
[ ] 4.1.0
[ ] 4.0.0
[ ] 4.5.0
[ ] 2023.1.1
[ ] 2023.1.0-hotfix.1
[ ] Other (please specify):

Operating System

[ ] Ubuntu 24.04
[x] Ubuntu 22.04
[ ] Ubuntu 20.04
[ ] Windows 11
[ ] Windows 10
[ ] Other (please specify):

GPU Information

• Model: NVIDIA L40S (46GB)
• Driver Version: 595.58.03 (CUDA 13.2)

Segmentation Fault when cameras are enabled

Detailed Description

I have installed IsaacSim 5.1.0 and IsaacLab 2.3.1 on a headless HPC node with L40s. I am able to run non-camera based scripts, however any scripts with cameras fail with the error messages given in the "Error Messages" section. I have seen a simillar issue at #5197 Howevver that seems to get stuck and ours seems to crash.

Steps to Reproduce

1. Run any camera enabled script on a headless server with L40S wiith IsaacSim 5.1.0 and IsaacLab 2.3.1
   Example:
   ./isaaclab.sh -p scripts/tutorials/04_sensors/add_sensors_on_robot.py --num_envs 2 --enable_cameras --headless

Error Messages

Segmentation fault logs when enabling cameras

(isaaclab) soham.patil@node14:~/IsaacLab$ ./isaaclab.sh -p scripts/tutorials/04_sensors/add_sensors_on_robot.py --num_envs 2 --enable_cameras --headless
[INFO] Using python from: /home/soham.patil/miniconda3/envs/isaaclab/bin/python                                                                                                                                   
[INFO][AppLauncher]: Using device: cuda:0
[INFO][AppLauncher]: Loading experience file: /home/soham.patil/IsaacLab/apps/isaaclab.python.headless.rendering.kit
Loading user config located at: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/user.config.json'
[Info] [carb] Logging to file: /home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/logs/Kit/Isaac-Sim/5.1/kit_20260417_124436.log
2026-04-17T12:44:36Z [3ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/9af75908-58d5-498d-6656ada4-d2823261.dmp'
2026-04-17T12:44:36Z [10ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/5d40c2a0-05a1-4ef3-5849ddb1-ff532f07.dmp'
2026-04-17T12:44:36Z [10ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/b1f6020d-6f62-4d5d-940badb4-f87c6766.dmp'
2026-04-17T12:44:36Z [10ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/7558295e-d673-4b60-8fc79f8d-577e5a55.dmp'
2026-04-17T12:44:37Z [1,011ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:37Z [1,011ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:37Z [1,053ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.
2026-04-17T12:44:37Z [1,057ms] [Warning] [carb.cudainterop.plugin] CUDA_VISIBLE_DEVICES environment variable is set.
2026-04-17T12:44:37Z [1,057ms] [Warning] [carb.cudainterop.plugin] Note CUDA device enumeration and Omniverse device enumeration are different.
2026-04-17T12:44:37Z [1,057ms] [Warning] [carb.cudainterop.plugin] Setting CUDA_VISIBLE_DEVICES can lead to undesired behavior or crashes.

|---------------------------------------------------------------------------------------------|
| Driver Version: 595.58.03     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA L40S                      | Yes: 0 |     | 46068   MB | 10de      | 0          |
|     |                                  |        |     |            | 26b9      | bfbe89df.. |
|     |                                  |        |     |            | 31        |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 5.15.0-173-generic
| Processor: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
| Cores: 32 | Logical Cores: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 515576 | Free Memory: 510131
| Total Page/Swap (MB): 16383 | Free Page/Swap: 16383
|---------------------------------------------------------------------------------------------|
2026-04-17T12:44:38Z [1,729ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 0
2026-04-17T12:44:39Z [2,709ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,709ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,730ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,730ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,740ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,740ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,767ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,767ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,769ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,769ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,772ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,772ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [3,222ms] [Warning] [omni.log] Source: omni.hydra was already registered.
Failed to create secure directory (/run/user/2024701005/pulse): No such file or directory
2026-04-17T12:44:40Z [3,709ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [omni::physx::IPhysxBenchmarks v1.0] (plugin: <default plugin>), by client: omni.physics.physx.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2026-04-17T12:44:40Z [3,730ms] [Warning] [omni.isaac.dynamic_control] omni.isaac.dynamic_control is deprecated as of Isaac Sim 4.5. No action is needed from end-users.
/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-04-17T12:44:43Z [6,408ms] [Warning] [pxr.Semantics] pxr.Semantics is deprecated - please use Semantics instead
2026-04-17T12:44:43Z [6,558ms] [Warning] [omni.graph.core.plugin] Found duplicate of category 'Replicator' - was 'Annotators', adding 'Fabric Reader'
2026-04-17T12:44:43Z [6,558ms] [Warning] [omni.graph.core.plugin] Category 'Replicator' not accepted on node type 'omni.replicator.core.FabricReader' in extension 'omni.replicator.core'
2026-04-17T12:44:43Z [6,559ms] [Warning] [omni.replicator.core.scripts.extension] No material configuration file, adding configuration to material settings directly.
2026-04-17T12:44:49Z [0ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred.  If a debugger should be attached, please set the '/crashreporter/debuggerAttachTimeoutMs' setting to a timeout in milliseconds.  This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
2026-04-17T12:44:50Z [61ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] Wrote dump file '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/badadab5-b991-457f-fb1a10b0-e9dee10d.dmp'
2026-04-17T12:44:50Z [62ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/badadab5-b991-457f-fb1a10b0-e9dee10d.dmp'
2026-04-17T12:44:50Z [63ms] [Error] [carb.crashreporter-breakpad.plugin] [crash]     dump file size is 5263312 bytes, file is readable.
2026-04-17T12:44:50Z [64ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Crash detected in pid 246307 thread 246307
2026-04-17T12:44:50Z [64ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] Crash metadata:
2026-04-17T12:44:50Z [64ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  BuildGitlabJobID = '209507876'
2026-04-17T12:44:50Z [64ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  CarbSdkVersion = '206.6+release.9587.07f17b1b.gl'
2026-04-17T12:44:50Z [65ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  CrashTime = 'Fri Apr 17 12:44:50 2026 GMT'
2026-04-17T12:44:50Z [65ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  DriverShaderCacheWrapper = 'disabled'
2026-04-17T12:44:50Z [65ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  DumpId = 'badadab5-b991-457f-fb1a10b0-e9dee10d'
2026-04-17T12:44:50Z [66ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  LastUploadStatus = '0'
2026-04-17T12:44:50Z [66ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  ProductName = 'OmniverseKit'
2026-04-17T12:44:50Z [66ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  PythonTracebackStatus = ''py-spy' successfully wrote info to '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/badadab5-b991-457f-fb1a10b0-e9dee10d.py.txt''
2026-04-17T12:44:50Z [67ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  RetryCount = '0'
2026-04-17T12:44:50Z [67ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  StartupTime = '1776429876'
2026-04-17T12:44:50Z [67ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  UploadSuccessful = '0'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  UptimeSeconds = '14'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  UserStory = ''
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  UserStoryStatus = 'Running in headless mode; not gathering user story'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  Version = '107.3.3+production.229672.69cbf6ad.gl'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_enabled = '1'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_flags = '3'
2026-04-17T12:44:50Z [69ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_status = 'auto-enabled'
2026-04-17T12:44:50Z [69ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_version = '2024.1'
2026-04-17T12:44:50Z [69ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  appName = 'Isaac-Sim'
2026-04-17T12:44:50Z [70ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  appState = 'started'
2026-04-17T12:44:50Z [70ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  appVersion = '5.1.0'
2026-04-17T12:44:50Z [70ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  autoloadExts = ''
2026-04-17T12:44:50Z [71ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildBranch = 'production'
2026-04-17T12:44:50Z [71ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildCi = 'gl'
2026-04-17T12:44:50Z [71ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildConfig = 'release'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildGitlabJobName = 'kit-build-release-linux-x86_64'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildGitlabJobStage = 'kit-build'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildGitlabPipelineID = '34875094'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildHash = '69cbf6ad'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildMajor = '107'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildMinor = '3'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildNumber = '229672'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildPatch = '3'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildVersion = '107.3.3'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  carboniteFrameworkVersion = '206.6+release.9587.07f17b1b.gl'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  carboniteSdkVersion = '206.6+release.9587.07f17b1b.gl'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  commandLine = '/home/<username>/miniconda3/envs/isaaclab/bin/python scripts/tutorials/04_sensors/add_sensors_on_robot.py --num_envs 2 --enable_cameras --headless'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuCoreLimited = '64'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuId = 'Intel64 Family 6 Model 106 Stepping 6'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuLogicalCoresBareMetal = '64'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuName = 'Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuPhysicalCoresBareMetal = '32'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuUsageQuota = '-1.000000'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuVendor = 'GenuineIntel'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  environmentName = 'Individual'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  experience = 'Isaac Sim'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  externalBuild = '1'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  gpuDriver_0 = '595.58'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  gpuVRAM_0 = '48305799168'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  gpu_0 = 'NVIDIA L40S'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  kitRendererDriverVersion = '595.58'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildBranch = 'release'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildDate = 'Fri Oct 17 03:55:10 2025 +0000'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildHash = '9c81211'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildRepo = 'https://gitlab-master.nvidia.com/omniverse/isaac/omni_isaac_sim.git'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildVersion = '5.1.0-rc.19'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildBranch = 'HEAD'
2026-04-17T12:44:50Z [78ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildDate = 'Oct-03-2025'
2026-04-17T12:44:50Z [78ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildHash = '3a61992'
2026-04-17T12:44:50Z [78ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildRepo = 'gitlab-master.nvidia.com/omniverse/physics'
2026-04-17T12:44:50Z [78ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildVersion = '107.3.26'
2026-04-17T12:44:50Z [79ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  memoryStats = '(avail/total) RAM: 496.566/503.493GB, Swap: 16/16GB, VM: 1.71799e+10/1.71799e+10GB'
2026-04-17T12:44:50Z [79ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  nvidia-smi = '
==============NVSMI LOG==============

Timestamp                                              : Fri Apr 17 12:44:37 2026
Driver Version                                         : 595.58.03
CUDA Version                                           : 13.2

Attached GPUs                                          : 1
GPU 00000000:31:00.0
    Product Name                                       : NVIDIA L40S
    Product Brand                                      : NVIDIA
    Product Architecture                               : Ada Lovelace
    Display Mode                                       : Requested functionality has been deprecated
    Display Attached                                   : Yes
    Display Active                                     : Disabled
    Persistence Mode                                   : Enabled
    Addressing Mode                                    : None
    MIG Mode
        Current                                        : N/A
        Pending                                        : N/A
    Accounting Mode                                    : Disabled
    Accounting Mode Buffer Size                        : 4000
    Driver Model
        Current                                        : N/A
        Pending                                        : N/A
    Serial Number                                      : 1792325013960
    GPU UUID                                           : GPU-bfbe89df-81e3-6e55-f639-ebc1983b3cad
    GPU PDI                                            : 0xf6c17e3388f6ea06
    Minor Number                                       : 0
    VBIOS Version                                      : 95.02.66.00.15
    MultiGPU Board                                     : No
    Board ID                                           : 0x3100
    Board Part Number                                  : 900-2G133-0080-000
    GPU Part Number                                    : 26B9-896-A1
    FRU Part Number                                    : N/A
    Platform Info
        Chassis Serial Number                          : N/A
        Slot Number                                    : N/A
        Tray Index                                     : N/A
        Host ID                                        : N/A
        Peer Type                                      : N/A
        Module Id                                      : 1
        GPU Fabric GUID                                : N/A
    Inforom Version
        Image Version                                  : G133.0242.00.03
        OEM Object                                     : 2.1
        ECC Object                                     : 6.16
        Power Management Object                        : N/A
    Inforom BBX Object Flush
        Latest Timestamp                               : N/A
        Latest Duration                                : N/A
    GPU Operation Mode
        Current                                        : N/A
        Pending                                        : N/A
    GPU C2C Mode                                       : N/A
    GPU Virtualization Mode
        Virtualization Mode                            : None
        Host VGPU Mode                                 : N/A
        vGPU Heterogeneous Mode                        : N/A
    GPU Recovery Action                                : None
    GSP Firmware Version                               : 595.58.03
    IBMNPU
        Relaxed Ordering Mode                          : N/A
    PCI
        Bus                                            : 0x31
        Device                                         : 0x00
        Domain                                         : 0x0000
        Base Classcode                                 : 0x3
        Sub Classcode                                  : 0x2
        Device Id                                      : 0x26B910DE
        Bus Id                                         : 00000000:31:00.0
        Sub System Id                                  : 0x185110DE
        GPU Link Info
            PCIe Generation
                Max                                    : 4
                Current                                : 1
                Device Current                         : 1
                Device Max                             : 4
                Host Max                               : 4
            Link Width
                Max                                    : 16x
                Current                                : 16x
        Bridge Chip
            Type                                       : N/A
            Firmware                                   : N/A
        Replays Since Reset                            : 0
        Replay Number Rollovers                        : 0
        Tx Throughput                                  : 634 KB/s
        Rx Throughput                                  : 341 KB/s
        Atomic Caps Outbound                           : N/A
        Atomic Caps Inbound                            : N/A
    Fan Speed                                          : N/A
    Performance State                                  : P8
    Clocks Event Reasons
        Idle                                           : Active
        Applications Clocks Setting                    : Not Active
        SW Power Cap                                   : Not Active
        HW Slowdown                                    : Not Active
            HW Thermal Slowdown                        : Not Active
            HW Power Brake Slowdown                    : Not Active
        Sync Boost                                     : Not Active
        SW Thermal Slowdown                            : Not Active
        Display Clock Setting                          : Not Active
    Clocks Event Reasons Counters
        SW Power Capping                               : 270280264 us
        Sync Boost                                     : 0 us
        SW Thermal Slowdown                            : 0 us
        HW Thermal Slowdown                            : 0 us
        HW Power Braking                               : 0 us
    Sparse Operation Mode                              : N/A
    FB Memory Usage
        Total                                          : 46068 MiB
        Reserved                                       : 609 MiB
        Used                                           : 1 MiB
        Free                                           : 45460 MiB
    BAR1 Memory Usage
        Total                                          : 65536 MiB
        Used                                           : 0 MiB
        Free                                           : 65536 MiB
    Conf Compute Protected Memory Usage
        Total                                          : 0 MiB
        Used                                           : 0 MiB
        Free                                           : 0 MiB
    Compute Mode                                       : Default
    Utilization
        GPU                                            : 0 %
        Memory                                         : 0 %
        Encoder                                        : 0 %
        Decoder                                        : 0 %
        JPEG                                           : 0 %
        OFA                                            : 0 %
    Encoder Stats
        Active Sessions                                : 0
        Average FPS                                    : 0
        Average Latency                                : 0
    FBC Stats
        Active Sessions                                : 0
        Average FPS                                    : 0
        Average Latency                                : 0
    DRAM Encryption Mode
        Current                                        : N/A
        Pending                                        : N/A
    ECC Mode
        Current                                        : Enabled
        Pending                                        : Enabled
    ECC Errors
        Volatile
            SRAM Correctable                           : 0
            SRAM Uncorrectable Parity                  : 0
            SRAM Uncorrectable SEC-DED                 : 0
            DRAM Correctable                           : 0
            DRAM Uncorrectable                         : 0
        Aggregate
            SRAM Correctable                           : 0
            SRAM Uncorrectable Parity                  : 0
            SRAM Uncorrectable SEC-DED                 : 0
            DRAM Correctable                           : 0
            DRAM Uncorrectable                         : 0
            SRAM Threshold Exceeded                    : No
        Aggregate Uncorrectable SRAM Sources
            SRAM L2                                    : 0
            SRAM SM                                    : 0
            SRAM Microcontroller                       : 0
            SRAM PCIE                                  : 0
            SRAM Other                                 : 0
        Channel Repair Pending                         : No
        TPC Repair Pending                             : No
        Unrepairable Memory                            : N/A
    Retired Pages
        Single Bit ECC                                 : N/A
        Double Bit ECC                                 : N/A
        Pending Page Blacklist                         : N/A
    Remapped Rows
        Correctable Error                              : 0
        Uncorrectable Error                            : 0
        Pending                                        : No
        Remapping Failure Occurred                     : No
        Bank Remap Availability Histogram
            Max                                        : 192 bank(s)
            High                                       : 0 bank(s)
            Partial                                    : 0 bank(s)
            Low                                        : 0 bank(s)
            None                                       : 0 bank(s)
    Temperature
        GPU Current Temp                               : 40 C
        GPU T.Limit Temp                               : 48 C
        GPU Shutdown T.Limit Temp                      : -5 C
        GPU Slowdown T.Limit Temp                      : -2 C
        GPU Max Operating T.Limit Temp                 : 0 C
        GPU Target Temperature                         : N/A
        Memory Current Temp                            : N/A
        Memory Max Operating T.Limit Temp              : N/A
    GPU Power Readings
        Average Power Draw                             : 36.67 W
        Instantaneous Power Draw                       : 36.67 W
        Current Power Limit                            : 350.00 W
        Requested Power Limit                          : 350.00 W
        Default Power Limit                            : 350.00 W
        Min Power Limit                                : 100.00 W
        Max Power Limit                                : 350.00 W
    GPU Memory Power Readings 
        Average Power Draw                             : N/A
        Instantaneous Power Draw                       : N/A
    Module Power Readings
        Average Power Draw                             : N/A
        Instantaneous Power Draw                       : N/A
        Current Power Limit                            : N/A
        Requested Power Limit                          : N/A
        Default Power Limit                            : N/A
        Min Power Limit                                : N/A
        Max Power Limit                                : N/A
    Power Smoothing                                    : N/A
    Workload Power Profiles
        Requested Profiles                             : N/A
        Enforced Profiles                              : N/A
    EDPp Multiplier                                    : N/A
    Clocks
        Graphics                                       : 210 MHz
        SM                                             : 210 MHz
        Memory                                         : 405 MHz
        Video                                          : 1185 MHz
    Applications Clocks
        Graphics                                       : Requested functionality has been deprecated
        Memory                                         : Requested functionality has been deprecated
    Default Applications Clocks
        Graphics                                       : Requested functionality has been deprecated
        Memory                                         : Requested functionality has been deprecated
    Deferred Clocks
        Memory                                         : N/A
    Max Clocks
        Graphics                                       : 2520 MHz
        SM                                             : 2520 MHz
        Memory                                         : 9001 MHz
        Video                                          : 1965 MHz
    Max Customer Boost Clocks
        Graphics                                       : 2520 MHz
    Clock Policy
        Auto Boost                                     : N/A
        Auto Boost Default                             : N/A
    Fabric
        State                                          : N/A
        Status                                         : N/A
        CliqueId                                       : N/A
        ClusterUUID                                    : N/A
        Health
            Summary                                    : N/A
            Bandwidth                                  : N/A
            Route Recovery in progress                 : N/A
            Route Unhealthy                            : N/A
            Access Timeout Recovery                    : N/A
            Incorrect Configuration                    : N/A
            Partition Assigned                         : N/A
    Processes
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 246307
            Type                          : G
            Name                          : ............................................../python
            Used GPU Memory               : 0 MiB
    Capabilities
        EGM                                            : disabled

'
2026-04-17T12:44:50Z [79ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osDistro = 'ubuntu'
2026-04-17T12:44:50Z [80ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osName = '22.04.5 LTS (Jammy Jellyfish)'
2026-04-17T12:44:50Z [80ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osVersion = '22.04.5'
2026-04-17T12:44:50Z [80ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  portableMode = '1'
2026-04-17T12:44:50Z [80ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  runEnvironment = 'Individual'
2026-04-17T12:44:50Z [81ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  runningInContainer = '0'
2026-04-17T12:44:50Z [81ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_debugSymbols = '0'
2026-04-17T12:44:50Z [81ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_dumpIncludeOverrides = '0'
2026-04-17T12:44:50Z [81ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_dumpIntermediates = '0'
2026-04-17T12:44:50Z [82ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_obfuscateCode = '1'
2026-04-17T12:44:50Z [82ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_optimizationLevel = '1'
2026-04-17T12:44:50Z [82ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  systemInfo = '
|---------------------------------------------------------------------------------------------|
| Driver Version: 595.58.03     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA L40S                      | Yes: 0 |     | 46068   MB | 10de      | 0          |
|     |                                  |        |     |            | 26b9      | bfbe89df.. |
|     |                                  |        |     |            | 31        |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 5.15.0-173-generic
| Processor: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
| Cores: 32 | Logical Cores: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 515576 | Free Memory: 510131
| Total Page/Swap (MB): 16383 | Free Page/Swap: 16383
|---------------------------------------------------------------------------------------------|
'
2026-04-17T12:44:50Z [82ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  telemetrySessionId = '1467242400602034763'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  terminatedByAbort = '0'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  totalRamBareMetalMB = '515576'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  totalRamLimitedMB = '515576'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  totalSwapBareMetalMB = '16383'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  totalSwapLimitedMB = '16383'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  userId = 'default'
2026-04-17T12:44:50Z [84ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  workingDirectory = '/home/<username>/IsaacLab'
2026-04-17T12:44:50Z [84ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Thread 246307 backtrace follows:
2026-04-17T12:44:50Z [91ms] [Fatal] [carb.crashreporter-breakpad.plugin] 000: libc.so.6!__sigaction+0x50 (??:?)
2026-04-17T12:44:50Z [95ms] [Fatal] [carb.crashreporter-breakpad.plugin] 001: librtx.scenedb.plugin.so!void std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>*, std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > > >, std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>&&)+0x123ef (??:?)
2026-04-17T12:44:50Z [99ms] [Fatal] [carb.crashreporter-breakpad.plugin] 002: librtx.scenedb.plugin.so!void std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>*, std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > > >, std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>&&)+0x12766 (??:?)
2026-04-17T12:44:50Z [102ms] [Fatal] [carb.crashreporter-breakpad.plugin] 003: librtx.scenedb.plugin.so!void std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>*, std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > > >, std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>&&)+0x14a14 (??:?)
2026-04-17T12:44:50Z [107ms] [Fatal] [carb.crashreporter-breakpad.plugin] 004: librtx.scenedb.plugin.so!carbOnPluginStartup+0x3b4de (??:?)
2026-04-17T12:44:50Z [111ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: librtx.scenedb.plugin.so!void std::vector<int, std::allocator<int> >::_M_realloc_insert<int>(__gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >, int&&)+0x7ce1 (??:?)
2026-04-17T12:44:50Z [115ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x4655f (??:?)
2026-04-17T12:44:50Z [119ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x64486 (??:?)
2026-04-17T12:44:50Z [124ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: libcarb.scenerenderer-rtx.plugin.so!carbOnPluginShutdown+0xe4b (??:?)
2026-04-17T12:44:50Z [127ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: libomni.hydra.rtx.plugin.so!+0x4459
2026-04-17T12:44:50Z [131ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: libomni.usd.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)+0x10da (??:?)
2026-04-17T12:44:50Z [135ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: libomni.usd.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)+0x58ce (??:?)
2026-04-17T12:44:50Z [139ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: libomni.usd.so!omni::usd::UsdManager::destroyHydraEngine(unsigned int)+0x3b7 (??:?)
2026-04-17T12:44:50Z [143ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: libomni.usd.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)+0x8807 (??:?)
2026-04-17T12:44:50Z [147ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: libomni.usd.so!omni::usd::UsdManager::createHydraEngine(char const*, omni::usd::UsdContext*)+0x85 (??:?)
2026-04-17T12:44:50Z [151ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: _usd.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)+0x15ff4 (??:?)
2026-04-17T12:44:50Z [155ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: _usd.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)+0x9cf2 (??:?)

Additional Information

What I’ve Tried

1. The hpc node doesn't seem to have libvulkan.so.1 so tried to set LD_LIBRARY_PATH to use the bundled Vulkan loader
2. Set VK_ICD_FILENAMES and __EGL_VENDOR_LIBRARY_FILENAMES to NVIDIA paths
3. Ran in headless mode with display checks disabled
4. Cleared shader/cache directories
5. We followed the same setup procedure on a local 5090 Workstation with GUI and that works

Additional Context

1. Running this on a shared HPC cluster so don't have sudo access. The node has 4xL40s, but I am only requesting 1.
2. As mentioned, thought it was a missing libvulkan.so.1 issues, I attempted to work around this by using bundled libraries and setting environment variables, but the issue persists when enabling cameras, maybe I am doing this wrong?

[SouvickC]

Same problem. Posted an issue as well #5171 for quite some time.

[omgunjal2727]

Got it solved by - Forum . We tried it on a 570 driver version and it worked for me.

[omgunjal2727]

Got it solved by - Forum . I tried it on a 570 driver version and it worked for me.