[Isaac Sim 6.0.1] Recurring segfaults with ROS 2 sensor publishing: `textureCopyToRawBufferKernel<float>` (depth image writer) and `rtx.rtxsensor.plugin` (RTX lidar)

[llagoeiro-botbot]

Description

Environment

Isaac Sim	6.0.1 (lib_isaacSim_buildVersion 6.0.1-rc.7, Kit 110.1.2+production.326809)
OS	Ubuntu 22.04.5 LTS, kernel 6.8.0-124-generic, X11
GPU	NVIDIA RTX A5500 (Ampere, 24 GB), driver 580.159.03, CUDA 13.0
CPU / RAM	Intel i7-14700K, 64 GB
ROS 2	Humble via isaacsim.ros2.bridge 5.1.2 with internal ROS libs (isaacsim.ros2.core 1.9.4), rmw_cyclonedds_cpp
Run mode	Standalone SimulationApp Python script ({"headless": False}), physics 200 Hz (CPU device), rendering 25 fps

Scene

A saved USD world containing the NVIDIA Spot asset
(.../Isaac/6.0/Isaac/Robots/BostonDynamics/spot/spot.usd) driven by
SpotFlatTerrainPolicy, with a custom sensor payload mounted on the robot:

• 2 cameras (640×480), each publishing rgb + depth (32FC1) + camera_info
  through ROS2CameraHelper / ROS2CameraInfoHelper OmniGraph nodes;
• 1 RTX lidar based on the Ouster OS0 asset
  (.../Isaac/6.0/Isaac/Sensors/Ouster/OS0/OS0.usd) with a customized
  OmniLidar profile (40 channels authored as attribute overrides),
  publishing point cloud via ROS2PublishPointCloud.

The setup is Software-in-the-Loop (SIL): the ROS 2 topics published by
the simulation are consumed by a second machine connected over Ethernet,
with rmw_cyclonedds_cpp on both ends (CycloneDDS configured via XML with
explicit network interfaces and enlarged receive/defrag buffers for the large
image and point-cloud samples).

The crashes occur when the world is run through a standalone
SimulationApp Python script (launched with ~/isaacsim-6.0.1/python.sh,
which loads the stage, attaches the policy and steps the app in a
simulation_app.update() loop). So far we have not observed them when the
same world is opened interactively in the GUI editor.

We get two distinct, recurring segfaults (exit code 139). In all of them,
py-spy shows the Python main thread idle inside SimulationApp.update() —
the crash is always in a native worker thread.

Crash A — depth image writer (most frequent, reliably triggered)

002: libcuda.so!cuEGLApiInit+0x57f1b
003: libcuda.so!cuMemRetainAllocationHandle+0xaab4
004: libcuda.so!cuMipmappedArrayGetLevel+0x20
005: libisaacsim.ros2.nodes.plugin.so!void textureCopyToRawBufferKernel<float>(unsigned long long, float*, unsigned int, unsigned int)+0x76c8
006: libisaacsim.ros2.nodes.plugin.so!void textureCopyToRawBufferKernel<float>(...)+0x4bc17

Trigger: the crash only happens when the ROS 2 publishers are consumed
from the second device over Ethernet (the SIL consumer described above) —
subscribing on the sim host alone does not reproduce it. At one point it
crashed every time the remote machine subscribed to a depth image topic
(i.e., the moment the lazy publisher performs its first GPU texture copy).
It also crashed seconds after starting to drive the robot while the remote
consumers were attached. Uptime at crash ranged from ~136 s to ~20 min. The
<float> instantiation and the SDG pipeline commands in the crash metadata
point at the 32FC1 depth writers
(*_Pseudo_Depth_render_product_NodeWriterWriter, ROS2PublishImage).

What we tried:

• /rtx/ecoMode/enabled = false — crashes still occurred;
• resolution is standard 640×480; GPU healthy (24 GB free VRAM, 55–63 °C,
  no ECC errors, no OOM);
• workaround that avoids the crash entirely: disabling the depth
  ROS2CameraHelper nodes and reading depth via the Replicator
  distance_to_image_plane annotator attached to the same render products,
  publishing sensor_msgs/Image from Python. rgb + camera_info writers keep
  running fine. This isolates the fault to the bridge's GPU texture-copy path
  for float images.

Frequency is intermittent across sessions (some days it triggers on every
remote subscribe, later the same setup ran clean), which suggests a race on
the render-product texture handle rather than a deterministic config error.

Crash B — RTX lidar plugin (long-running sessions)

002: librtx.rtxsensor.plugin.so!std::_Rb_tree<unsigned int, ...>::_M_get_insert_hint_unique_pos(...)+0x13d2d   [low-confidence symbols]
003: librtx.rtxsensor.plugin.so!std::_Rb_tree<unsigned int, ...>::_M_get_insert_hint_unique_pos(...)+0x141d8
004-006: libgpu.foundation.plugin.so!...
007-010: libcarb.tasking.plugin.so!... (fiber, make_fcontext)

Happened twice: once early in a session and once after 3499 s (~58 min) of
continuous simulation with the robot being teleoperated. Possibly relevant:
our OS0 lidar profile is customized to 40 channels via attribute overrides on
top of the stock OS0_REV6_128ch10hz1024res profile — if any per-channel
array length is read from the base profile while numberOfChannels is
overridden, an out-of-bounds access inside the plugin would match this
signature.

Questions

1. Is the textureCopyToRawBufferKernel crash on first depth-image
   subscription a known issue in 6.0.x, and is there a fix planned?
2. Is publishing depth through the Replicator annotator (our workaround) the
   recommended path until then?
3. For the RTX lidar: are partial attribute overrides of a stock sensor
   profile (changing numberOfChannels/emitter arrays) supported, or must a
   custom profile be complete and self-consistent?

Isaac Sim version

6.0.1

Operating System (OS)

Ubuntu 22.04

GPU Name

RTX A5500

GPU Driver and CUDA versions

driver 580.159.03, CUDA 13.0

Logs

No response

Additional information

No response

[zwdoescode]

Hi @llagoeiro-botbot, for the crash A, could you please try disabling async publication and retest with the the following flags:

- `/exts/isaacsim.ros2.bridge/publish_multithreading_disabled`
- `/exts/isaacsim.ros2.bridge/publish_with_queue_thread`

You can refer to this page.

I will also reach out to the internal team to see if they are aware of this issue.

For crash B, could you please share your OS0 lidar profile? For structural lidar-profile fields, a custom profile needs to be complete and self-consistent. Partial surgery on a stock profile like changing numberOfChannels and only some emitter arrays is not a supported workflow I would rely on.

[llagoeiro-botbot]

Hey @zwdoescode. I will tryout with disabling async publication.

About my OS0 LiDAR profile is >

                        over "sensor"
                        {
                            custom string[] _replicator:rendervar:GenericModelOutput:channels = ["NONE"]
                            float omni:sensor:Core:avgPowerW = 0.002
                            float omni:sensor:Core:azimuthErrorMean = 0.1
                            float omni:sensor:Core:azimuthErrorStd = 0.5
                            float omni:sensor:Core:elevationErrorMean = 0.1
                            float omni:sensor:Core:elevationErrorStd = 0.5
                            int[] omni:sensor:Core:emitterState:s001:azimuthDeg = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                            uint[] omni:sensor:Core:emitterState:s001:channelId = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40]
                            float[] omni:sensor:Core:emitterState:s001:distanceCorrectionM = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                            float[] omni:sensor:Core:emitterState:s001:elevationDeg = [-7, -5.525, -4.05, -2.575, -1.1004, 0.374, 1.849, 3.324, 4.799, 6.274, 7.7494, 9.2249, 10.699, 12.174, 13.645, 15.1243, 16.5999, 18.074, 19.5499, 21.024, 22.493, 23.9749, 25.44, 26.924, 28.39, 29.8743, 31.3499, 32.824, 34.29, 35.774, 37.2486, 38.724, 40.19, 41.674, 43.14, 44.624, 46.09, 47.574, 49.048, 50.524]
                            uint[] omni:sensor:Core:emitterState:s001:fireTimeNs = [0, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, 16000, 17000, 18000, 19000, 20000, 21000, 22000, 23000, 24000, 25000, 26000, 27000, 28000, 29000, 30000, 31000, 32000, 33000, 34000, 35000, 36000, 37000, 38000, 39000]
                            float[] omni:sensor:Core:emitterState:s001:focalDistM = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                            float[] omni:sensor:Core:emitterState:s001:focalSlope = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                            float[] omni:sensor:Core:emitterState:s001:horOffsetM = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                            float[] omni:sensor:Core:emitterState:s001:reportRateDiv = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                            float[] omni:sensor:Core:emitterState:s001:vertOffsetM = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                            float omni:sensor:Core:farRangeM = 40
                            uniform token omni:sensor:Core:intensityMappingType = "LINEAR"
                            uniform token omni:sensor:Core:intensityProcessing = "NORMALIZATION"
                            uint omni:sensor:Core:maxReturns = 1
                            float omni:sensor:Core:minReflectance = 0.1
                            float omni:sensor:Core:minReflectanceRange = 70
                            float omni:sensor:Core:nearRangeM = 0.1
                            uint omni:sensor:Core:numberOfChannels = 40
                            uint omni:sensor:Core:numberOfEmitters = 40
                            uint omni:sensor:Core:pulseTimeNs = 6
                            float omni:sensor:Core:rangeAccuracyM = 0.025
                            float omni:sensor:Core:rangeOffset = 0.03
                            float omni:sensor:Core:rangeResolutionM = 0.004
                            uniform token omni:sensor:Core:rayType = "IDEALIZED"
                            uint omni:sensor:Core:reportRateBaseHz = 7761
                            uniform token omni:sensor:Core:rotationDirection = "CW"
                            int omni:sensor:Core:scanRateBaseHz = 10
                            uniform token omni:sensor:Core:scanType = "ROTARY"
                            float omni:sensor:Core:wavelengthNm = 905
                            quatd xformOp:orient = (1, 5.727600616012296e-10, -1.3025435841029775e-8, 1.6555940618445675e-8)
                            double3 xformOp:translate = (3.725290298461914e-9, 7.450580596923828e-9, 0.036220014095306396)
                            uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:orient", "xformOp:scale"]
                        }

Thanks.

[llagoeiro-botbot]

Hey!

Still getting a native crash (SIGSEGV) when publishing RTX sensors over the ROS 2 bridge even with the threading mitigations enabled. I suspect it's my OS0 lidar config and would like to confirm.

Thanks.