Isaac Sim 6.0.1: two RTX lidars with different scanRateBaseHz kill the CUDA context seconds after play (CudaHelperMem.h:324 invalid-argument memcpy -> CUDA 700 / SIGSEGV in librtx.sensors.lidar.core) — uniform rates are stable

[firetix]

Environment

• Isaac Sim 6.0.1 (build 6.0.1-rc.7, hash 32955d8, release branch), headless, containerized (base image nvcr.io/nvidia/isaac-sim:6.0.1)
• Driver 580.126.20, CUDA 13.0; reproduced on BOTH NVIDIA L4 (non-MIG) and RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb (gfx) instances, multiple physical hosts; PhysX 110.1.13; Ubuntu 24.04.3
• ROS 2 Jazzy bridge (isaacsim.ros2.bridge), stock ROS2RtxLidarHelper graphs

Summary

Wiring two or more RTX lidar publishers whose sensor profiles author different omni:sensor:Core:scanRateBaseHz values into one scene reliably destroys the run within ~5–10 s of timeline play (first rendered frames). The same number of lidars at a uniform scan rate is stable in every combination we tested — same robot or different robots, 2D laser-scan helper route or 3D point-cloud writer route, profile composed at runtime via reference or shipped inside a robot USD.

Concretely: a lidar composed from the stock Example_Rotary_2D.usda profile (scanRateBaseHz = 30, reportRateBaseHz = 32000) plus one wired Slamtec RPLidar S2E (scanRateBaseHz = 10, reportRateBaseHz = 32000, e.g. the prims shipped in nova_carter.usd) crashes every time; S2E + S2E, S2E + Hesai XT-32 (XT32_SD10, also 10 Hz), and Rotary + Rotary are all stable. Note the per-scan return-buffer size (reportRate/scanRate) differs ~3× between the 30 Hz and 10 Hz sensors.

Crash signature (one fault, three surfaces observed)

1. Breakpad SIGSEGV, crashing thread inside librtx.sensors.lidar.core.plugin.so → librtx.rtxsensor.plugin.so → librtx.memutil.plugin.so / libgpu.foundation.plugin.so; process terminates with exit code 139.
2. Leading edge in the sensor CUDA path:

   CUDA error 1 in .../source/include/omni/sensors/cuda/CudaHelperMem.h:324:
   invalid argument (cudaMemcpyAsync((void*)dstPtr, (void*)srcPtr, sizeInBytes, kind, cudaStream))
   

   repeated 4×, then within milliseconds a storm of [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress + Failed to signal external semaphore on Render graph command list … frame submission index 0/1/2, then breakpad or:
3. Surviving-process wedge: the Kit main loop permanently stalls at simulation step ~2, later logging [omni.rtx] Failed to start frame in pooled data manager. Semaphore wait timed out after 60 seconds. every 60 s while the HTTP status endpoint keeps serving (heartbeat stale, zero captured frames).
   Variant marker also observed: getResourceData(): device index 0 out of range (size = 0) + RTXMemUtil … Assuming device was lost.

Minimal repro

• Warehouse-class scene (static props). Replicator continuous capture was running in most arms but is NOT required — the crash reproduces with zero capture render products (arm 13).
• Lidar A: a plain Xform mount on any articulated robot; compose an OmniLidar child by referencing the stock Example_Rotary_2D.usda profile (30 Hz); wire the stock IsaacCreateRenderProduct + ROS2RtxLidarHelper + ROS2PublishLaserScan graph.
• Lidar B: one of nova_carter.usd's RPLidar S2E prims (10 Hz), wired through the same stock helper graph. (Mounting BOTH lidars on the same robot — or even composing both from profiles onto the same mount prim — crashes identically; two robots are NOT required.)
• Play the timeline → crash ~5–10 s later, at the first rendered frames. 6/6 reproduction across our matrix arms with a 30 Hz + 10 Hz pair; 0/8 with uniform-rate configurations of equal or larger sensor counts.

Discriminator matrix (each arm changes ONE dimension; all on the same scene/content, same container, same node class)

Arm	Wired lidars (scanRateBaseHz)	Placement	Result
1	Example_Rotary composed (30)	1 robot	clean 30 s capture
2	S2E ×2 shipped (10+10)	1 robot	clean
3	S2E ×2 shipped (10+10)	1 robot + second lidar-less robot present	clean
4	Rotary composed ×2 (30+30)	2 robots (1+1)	clean
5	S2E ×2 shipped + XT-32 3D writer route (10+10+10)	1 robot	clean (writer attached, 2D+3D mixed routes)
6	S2E ×2 shipped + S2E composed-from-profile (10+10+10)	1 robot	clean (provenance mix, uniform rate)
7	S2E ×2 shipped (10+10) + two skinned characters animating (GPU animation workload)	1 robot	clean — character/animation GPU work is NOT required for the crash and does not trigger it at uniform rates
8	Rotary composed + S2E shipped (30+10)	2 robots (1+1)	CRASH (SIGSEGV surface 1)
9	Rotary composed + S2E ×2 shipped (30+10+10)	2 robots	CRASH
10	S2E ×2 shipped + Rotary composed (10+10+30)	1 robot	CRASH (surface 1)
11	Rotary composed + S2E composed (30+10), both composed from profile USDs onto the same mount	1 robot	CRASH (surfaces 2+3: memcpy invalid-argument → CUDA-700 storm → pooled-data-manager wedge)
12	Rotary composed ×2 (30+30) on the same mount as arm 11	1 robot	clean (mount-sharing control for arm 11)
13	Rotary composed + S2E ×2 shipped (30+10+10), ZERO Replicator capture render products	2 robots	CRASH (surface 1) — continuous capture is not required

Eliminated by the matrix: robot count, wired-lidar count (≤3 tested), cross-robot vs same-robot distribution, 2D-only vs 2D+3D, helper vs writer route, runtime-composed vs USD-shipped prims, animated characters, Replicator continuous capture, and GPU memory pressure — in-container nvidia-smi telemetry at 5 s cadence shows crash arms dying at ~3.1–3.4 GiB of 24 GiB while clean arms run steady at 4.3–5.6 GiB (the highest-memory arm passed). Earlier rounds on the same content had already eliminated hardware/host (3+ GPUs across L4 and RTX PRO 6000 MIG, fresh hosts, single-tenant MIG), raytracing motion BVH on/off, and per-prim attribute state (accumulateOutputs=true, skipDroppingInvalidPoints=true, patternFiringRateHz authored on every OmniLidar prim, verified composed at play → identical crash).

Two more facts that localize the fault:

• Unwired OmniLidar prims are inert: the same mixed-rate prims composed on stage WITHOUT helper graphs/render products complete cleanly. The crash requires wired publishers.
• Removing the publisher graph prims from the stage after rigging but BEFORE play still crashes: sensor-core state created at graph-build time survives graph removal. Only never building the publisher graphs avoids the crash.
• One covariance we could not separate: our rig sets omni:sensor:tickRate := scanRateBaseHz on the runtime-composed prims (per the multi-tick guidance), so composed sensors also tick at their scan rate; shipped prims keep their vendor tick config. Mismatched scanRateBaseHz therefore implies mismatched tick rates in every crash arm.

Suspected mechanism

Per-sensor CUDA bookkeeping in the RTX lidar sensor core appears to share sizing/indexing state across sensors with different scan periods: with a 30 Hz and a 10 Hz sensor wired concurrently, per-scan return-buffer sizes differ ~3×, and the first frames after play produce a cudaMemcpyAsync with an invalid argument in CudaHelperMem.h:324 (or an out-of-range device index in getResourceData()), poisoning the process CUDA context.

Asks

1. Root-cause the multi-rate sensor path on 6.0.1 — a bad memcpy argument should fail loudly per-sensor rather than poison the process CUDA context / SIGSEGV the process.
2. Confirm whether heterogeneous scanRateBaseHz (and/or tickRate) across concurrently wired RTX lidars is intended to be supported; if it is not, a startup validation error naming the offending sensors would save integrators days of triage (mixed-vendor robots and multi-robot scenes hit this naturally).
3. Guidance on whether any 6.0.x point release or 6.1 changes this path.

[zwdoescode]

Thanks for bringing this issue up @firetix. I am able to reproduce this issue on my end.
I will create an internal ticket for the internal team to investigate.
As a temporary workaround, you can try the following

• keep all concurrently wired RTX lidars on the same scanRateBaseHz
• keep tickRate == scanRateBaseHz
• if mixed rates are unavoidable, isolate them into separate Kit processes

Internal ticket: 6500099

[zwdoescode]

I confirmed with the internal team that this issue is fixed. The fix will be released in Isaac Sim 6.1. Please stay tuned!