[Isaac Sim 6.0.1] Hesai XT-32 (OmniLidar) RTX lidar: GenericModelOutput numElements==0 on every frame — PointCloud2 topic advertised but never publishes, while single-emitter OmniLidar LaserScan is healthy in the same session

[firetix]

Description

On stock Isaac Sim 6.0.1 (headless), an RTX lidar based on the stock Hesai XT-32 asset (/Isaac/Sensors/HESAI/XT32_SD10/HESAI_XT32_SD10.usd, OmniLidar prim with OmniSensorGenericLidarCoreAPI) publishing point_cloud through ROS2RtxLidarHelper advertises its PointCloud2 topic but never publishes a single message for the entire session (30–60 s runs, repeated across many sessions). Instrumenting the SDG pipeline shows the cause locus precisely:

• GenericModelOutput buffers do reach OgnIsaacExtractRTXSensorPointCloud every dispatched frame, with a valid GMO header but numElements == 0, on every frame, for the whole session.
• The node then zero-skips by design (resetOutputs(db); return false; — source/extensions/isaacsim.sensors.rtx.nodes/nodes/OgnIsaacExtractRTXSensorPointCloud.cpp:137-142), so ROS2PublishPointCloud never fires. This path is completely silent — there is no log at any level for a permanently-empty GMO stream, which makes the failure look like a ROS/DDS problem when it is a sensor-output problem.

Meanwhile, in the same session, same stage, same graph topology, 2D rotary OmniLidar prims (stock Slamtec RPLidar S2E, e.g. the two shipped inside the stock Nova Carter robot asset alongside the XT-32) publish laser_scan at their native 10 Hz without issues. So rendering, the ROS 2 bridge, timeline, and the writer plumbing are all healthy — only the XT-32's GMO production is empty.

The full 6.0.1 chain we verified in this repo's source while diagnosing:

• Writer RtxLidarROS2PublishPointCloud = annotator IsaacExtractRTXSensorPointCloud + PostProcessDispatchIsaacSimulationGate (source/extensions/isaacsim.ros2.nodes/python/impl/extension.py:245-266).
• The annotator consumes render var GenericModelOutputPtr (source/extensions/isaacsim.sensors.rtx.nodes/python/impl/extension.py:74-81).
• With omni:sensor:Core:accumulateOutputs = true and omni:sensor:tickRate == omni:sensor:Core:scanRateBaseHz, zero-element GMO frames between scan completions are expected; a full-scan GMO should arrive at 10 Hz (matching test_rtx_sensor.py::_get_expected_message_count: duration * 10 messages for full scans). We never see a single non-empty GMO.

Reproduction

Minimal form of the failing pipeline (the observed failures come from this exact graph topology around the stock XT-32 asset, both standalone and embedded in the stock Nova Carter robot asset; environment is a stock warehouse scene so the sensor has abundant geometry in range):

from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": True})  # fails identically with enable_motion_bvh=True

import isaacsim.core.experimental.utils.app as app_utils
import isaacsim.core.experimental.utils.stage as stage_utils
import omni.graph.core as og
import omni.timeline
from isaacsim.core.simulation_manager import SimulationManager
from isaacsim.storage.native import get_assets_root_path

app_utils.enable_extension("isaacsim.ros2.bridge")
simulation_app.update()

assets_root = get_assets_root_path()
stage_utils.add_reference_to_stage(
    assets_root + "/Isaac/Environments/Simple_Warehouse/full_warehouse.usd", "/background"
)
stage_utils.add_reference_to_stage(
    assets_root + "/Isaac/Sensors/HESAI/XT32_SD10/HESAI_XT32_SD10.usd", "/XT32"
)
# OmniLidar prim composes at /XT32/PandarXT_32_10hz with the asset-authored
# tickRate=10 == scanRateBaseHz=10, accumulateOutputs=1

og.Controller.edit(
    {"graph_path": "/lidar_graph", "evaluator_name": "execution"},
    {
        og.Controller.Keys.CREATE_NODES: [
            ("Tick", "omni.graph.action.OnPlaybackTick"),
            ("RunOnce", "isaacsim.core.nodes.OgnIsaacRunOneSimulationFrame"),
            ("RenderProduct", "isaacsim.core.nodes.IsaacCreateRenderProduct"),
            ("Helper", "isaacsim.ros2.bridge.ROS2RtxLidarHelper"),
        ],
        og.Controller.Keys.CONNECT: [
            ("Tick.outputs:tick", "RunOnce.inputs:execIn"),
            ("RunOnce.outputs:step", "RenderProduct.inputs:execIn"),
            ("RenderProduct.outputs:execOut", "Helper.inputs:execIn"),
            ("RenderProduct.outputs:renderProductPath", "Helper.inputs:renderProductPath"),
        ],
        og.Controller.Keys.SET_VALUES: [
            ("RenderProduct.inputs:cameraPrim", "/XT32/PandarXT_32_10hz"),
            ("Helper.inputs:type", "point_cloud"),
            ("Helper.inputs:topicName", "/points"),
            ("Helper.inputs:frameId", "lidar"),
        ],
    },
)

SimulationManager.setup_simulation(dt=1.0 / 60.0, device="cpu")
omni.timeline.get_timeline_interface().play()
for _ in range(1800):  # 30 s
    simulation_app.update()
simulation_app.close()

Observed: ros2 topic list shows /points advertised; ros2 topic hz /points receives nothing for the entire run. Attaching the GenericModelOutput annotator to the same render product shows a valid GMO header (correct magic number) with numElements == 0 on every dispatched frame.

What we ruled out (each verified on hardware, one variable at a time)

Hypothesis	Result
ROS2RtxLidarHelper.inputs:fullScan	No effect — deprecated and ignored in 6.0.1 (OgnROS2RtxLidarHelper.py:300-304: "RTX Lidar now always produces full scans via accumulateOutputs")
omni:sensor:Core:skipDroppingInvalidPoints = True (the #110 workaround)	No effect — topic advertised, still 0 messages
Motion BVH off (default) vs on	No effect. Default boot logs [rtx.rtxsensor.plugin] Multi-tick is enabled but motion BVH is not active. This is not supported.; enabling --/renderer/raytracingMotion/enabled=true (+ enableHydraEngineMasking, enabledForHydraEngines) removes the warning — verified present in boot args — but GMO frames still carry 0 elements
Interference from other render products / Replicator capture	No effect — a control run with no cameras and no Replicator capture at all (lidar render products only) reproduces the identical advertise-then-silence failure
Sensor pose/occlusion (rays starting inside geometry)	Ruled out — the same prim family and mount produce healthy 2D scans, and the XT-32 sits in open space with a warehouse in range
accumulateOutputs / tickRate mismatch (documented multi-tick known issues)	Not applicable — the stock asset authors accumulateOutputs=1 and tickRate=10 == scanRateBaseHz=10, and we verified those values on the composed prim

Differential between the failing and working stock profiles

Both are ROTARY, accumulateOutputs=1, tickRate=10, scanRateBaseHz=10, skipDroppingInvalidPoints=0, rayType=IDEALIZED, CW:

Attribute	Hesai XT-32 (fails)	RPLidar S2E (works)
numberOfEmitters / numberOfChannels	32 / 32	1 / 1
maxReturns	2	1
reportRateBaseHz	20000	32000
emitter-state arrays	32-entry elevationDeg +15..-16, fireTimeNs 368..47240	single-entry

Adjacent reports and docs we checked (not duplicates, but likely related)

• Cannot retrieve RTX lidar data using headless Python script #110 — empty annotator data headless; resolved via skipDroppingInvalidPoints in 5.x. That workaround does not help here (and the failing stage is upstream of dropping: the GMO header itself reports 0 elements).
• [BUG]: Lidar Points not Published Properly using ROS2 Omnigraph #46 — all-zero published points in 5.0; maintainers described it as a hardware/timing-dependent race in the sensor pipeline, later fixed. Symptom differs (we get no messages at all, not zeroed points), but the "some hardware, deterministic per machine" character matches.
• RTX LiDAR: timestamp length is ~half of point count (XT32 / dual-return), cannot map 1:1 to XYZ #525 — XT-32 dual-return aux/timestamp bookkeeping wrong in 5.1 (numEchos reported 0), fixed in 6.0 GA. Our failing profile is exactly the dual-return XT-32 class.
• [Isaac Sim 6.0.1] Recurring segfaults with ROS 2 sensor publishing: textureCopyToRawBufferKernel<float> (depth image writer) and rtx.rtxsensor.plugin (RTX lidar) #708 — 6.0.1 rtx.rtxsensor.plugin segfaults with a customized OmniLidar profile under ROS 2 publishing.
• Multi-Tick Rendering docs, known issues: the tickRate != scanRateBaseHz partial-scan fallback does not apply (values match, and it would produce non-empty partial scans, not empty ones). The radar+lidar frames-in-flight (FIF) race ("the Lidar's per-frame trace begins before its sensor profile has been initialized") is the closest documented relative — and test_multi_sensor_warmup.py in this repo documents a non-crash manifestation where the racing sensor yields zero-element GMO frames for the entire collection window unless its render product/annotator attachment is deferred a few frames (WARMUP_FRAMES "is load-bearing on affected hardware"). Our topology creates render products for all three lidars in the same first played frame via IsaacCreateRenderProduct, which is exactly the concurrent-attachment pattern that test works around — but the docs scope that issue to Radar+Lidar+Motion BVH, whereas we see the permanent-empty behavior with lidars only, and with Motion BVH both off and on.

Questions for the team

1. Can the FIF/attachment race starve an OmniLidar's GMO output (permanent numElements==0, no crash) in a lidar-only scene, without radar and regardless of Motion BVH? If so, the multi-tick known-issue docs understate its scope, and a supported "defer attach"/warmup API would be very welcome.
2. If not, is there a known 6.0.1 defect in omni.sensors.nv.lidar full-scan emission for multi-emitter, dual-return rotary profiles (XT-32 class) under multi-tick + per-sensor-tick TLAS? The single-emitter, single-return profile is fine in the same session; the failure tracks the profile class, not the graph.
3. Diagnosability: would you accept a PR adding a throttled warn-once to OgnIsaacExtractRTXSensorPointCloud after N consecutive valid-header/zero-element GMO frames? Today this failure mode is invisible: the topic advertises and everything downstream silently no-ops. (Happy to open that PR if the approach is acceptable — it does not fix the producer, which is in the closed-source sensor plugins, so I have not opened one speculatively.)

We can run instrumented builds/probes on the affected hardware and provide full logs on request.

Isaac Sim version

6.0.1 (official nvcr.io/nvidia/isaac-sim:6.0.1 container, Kit 110)

Operating System (OS)

Linux (official Isaac Sim 6.0.1 container, headless, on a GKE node)

GPU Name

NVIDIA L4

GPU Driver and CUDA versions

Driver 580.126.20, CUDA 13.0 (per nvidia-smi)

Logs

Default boot (no Motion BVH args) logs, once per session:

[rtx.rtxsensor.plugin] Multi-tick is enabled but motion BVH is not active. This is not supported.

With --/renderer/raytracingMotion/enabled=true --/renderer/raytracingMotion/enableHydraEngineMasking=true --/renderer/raytracingMotion/enabledForHydraEngines=0,1,2,3 the warning is gone (args verified in the booted process) and the symptom is unchanged. No other sensor-related warnings or errors are emitted; the point-cloud path is silent by construction when GMO frames are empty.

Additional information

Headless container runs only (no GUI available on this fleet). The same container and scene publish healthy 10 Hz laser_scan from single-emitter OmniLidar prims in the same session, so the report is specifically about GMO production for the XT-32-class profile, not the ROS 2 bridge.

[firetix]

Additional evidence narrowing this down (Isaac Sim 6.0.1, headless, Hesai XT-32 OmniLidar-schema prim, RTX lidar):

We instrumented the GenericModelOutput directly via an isaacsim.sensors.rtx annotator on the sensor's render product, sampling every 300 frames for a 60 s (sim time) run:

report=0..5  magic_ok=True  numElements=0  coordsType=SPHERICAL  nonzeroRanges=0

• GMO magic/header is valid the whole run, but numElements stays 0 permanently — so OgnIsaacExtractRTXSensorPointCloud correctly early-returns and downstream ROS 2 publishers emit nothing.
• Motion BVH explicitly enabled for hydra engines 0,1,2,3 (matching your test setup) — no change.
• fullScan/skipDroppingInvalidPoints config variations — no change (as expected given the deprecated no-op in OgnROS2RtxLidarHelper).
• A 2D RTX lidar (Slamtec RPLidar S2E, LaserScan route) on the SAME stage/build produces healthy scans all run — this is specific to the 32-emitter/maxReturns=2 XT-32 profile (possibly related to RTX LiDAR: timestamp length is ~half of point count (XT32 / dual-return), cannot map 1:1 to XYZ #525).
• Consistent with the FIF warmup behavior documented in test_multi_sensor_warmup.py: we attach the render product before first play; we are now testing a deferred attach (~5 rendered frames after play) and will report back.

Happy to provide the full probe harness if useful.

[firetix]

Follow-up: we tested the deferred-attach hypothesis. Render product + RtxLidarROS2PublishPointCloud writer attach held until 5 consecutive playing frames after the timeline started (matching the warmup ordering in test_multi_sensor_warmup.py), on an otherwise identical run:

deferred attach EXECUTED at playing-frame 5
GMO probe report=0..5: magic_ok=True numElements=0 coordsType=SPHERICAL nonzeroRanges=0  (60 s sim time)

Identical result to the pre-play attach — so attach timing / FIF warmup is eliminated as the cause on our side. Summary of what's now refuted on GPU: fullScan, skipDroppingInvalidPoints, Motion-BVH engine masking (0,1,2,3 enabled), capture mode, ROS2RtxLidarHelper vs direct writer/annotator construction, and pre-play vs post-play attach. The GMO header is always valid; numElements is always 0; timestamps advance with sim time. A Slamtec RPLidar S2E (2D profile) on the same stage and build produces healthy returns all run — the failure is specific to the Hesai XT-32 profile (32 emitters, maxReturns=2). This smells adjacent to #525.

For reproduction, the probe is a small standalone sampler on the sensor's render product — roughly:

import omni.replicator.core as rep
from isaacsim.sensors.rtx import get_gmo_data  # GenericModelOutput helper

rp = rep.create.render_product("<XT-32 OmniLidar prim path>", (128, 128))
annotator = rep.AnnotatorRegistry.get_annotator("GenericModelOutput")
annotator.attach([rp.path])
# every N rendered frames while playing:
gmo = get_gmo_data(annotator.get_data())
print(gmo.magicNumber, gmo.numElements, gmo.coordsType)  # magic valid, numElements stays 0

Happy to share the full harness or run further instrumented repros on our side if that helps.

[zwdoescode]

Hi @firetix! I tried with both locally built Isaac Sim 6.0.1 and binary package but I cannot replicate your issue.
Could you please try:

1. Check your log and see if there is any notable errors
2. Check if you set up parameters ROS_DISTRO, RMW_IMPLEMENTATION and LD_LIBRARY_PATH correctly
3. Clear your local asset content cache and download asset again
4. Locally build Isaac Sim 6.0.1 from source and see if the issue still persists
5. Try the script I attached here and see if the issue still persists (make sure you update the path correctly)

repro_issue_712_user_graph.py

[firetix]

Hi @zwdoescode — thank you for the script and the checklist. We ran everything you asked, and the result is conclusive: you were right that stock 6.0.1 is healthy — the root cause was on our side, and we have isolated it to a single USD attribute. Full evidence below so the thread has a complete record. From our perspective this issue can be closed (one small diagnosability suggestion at the end).

1. Your script in our environment: it works

We ran repro_issue_712_user_graph.py byte-identical (md5 9ee83c354415484f8914bac46b107f46; only --output passed on the CLI — the ml_archive prebundle was discovered via ISAAC_PATH so no path edits were needed) inside our container (built FROM nvcr.io/nvidia/isaac-sim:6.0.1, Kit 6.0.1-rc.7+release.42383.32955d8d.gl, Ubuntu 24.04.3, headless on GKE), on both GPU types we operate:

Run	GPU	Cache state	PointCloud2 msgs (30 s)	GMO frames valid/zero	numElements (first valid)
1	RTX PRO 6000 Blackwell SE, MIG 1g.24gb, driver 580.126.20	cold pod, assets freshly downloaded	299	300 / 0	22914
2	same	warm re-run	300	300 / 0	22919
3	NVIDIA L4, driver 580.126.20	cold pod, fresh download	299	300 / 0	22905

/points at a clean 10 Hz, full scans from frame 0. We then extended your graph (only additively) and it kept working with: the XT-32 embedded in the stock Nova Carter asset (41336 pts — our production topology), all three lidars (XT-32 point_cloud + 2× RPLidar S2E laser_scan) with concurrent render products in the same first frame, and a timeline stop→play cycle before the measurement window. So multi-lidar attach, robot embedding, and lifecycle were all fine too.

2. Your checklist

1. Log errors: zero [Error] lines in every run — including the failing configuration below. This failure mode is completely silent (more on that at the end).
2. Env: ROS_DISTRO=jazzy, LD_LIBRARY_PATH includes /opt/ros/jazzy/lib + the Isaac ROS 2 bridge libs. One deliberate difference from stock: we run RMW_IMPLEMENTATION=rmw_zenoh_cpp (with an rmw_zenohd router alongside) — that is our supported configuration on 6.0.1. Results are RMW-independent here: the GMO instrumentation sits upstream of ROS, and with the healthy asset your script publishes fine over rmw_zenoh as well.
3. Asset cache: runs 1 and 3 were fresh pods with empty caches; get_assets_root_path() resolved to the production content root and all assets were freshly downloaded. Same result warm.
4. Source build: not needed — see below; the binary container is exonerated.

3. The actual root cause: a 5.1-generation XT-32 asset under 6.0.1

Our asset pipeline serves robots from a mirror that was populated from the Isaac 5.1 content tree. Our Nova Carter's sensor variant payload-references the 5.1-era standalone sensor asset (Isaac/Sensors/HESAI/XT-32.usd), not the 6.0 Isaac/Sensors/HESAI/XT32_SD10/HESAI_XT32_SD10.usd your script uses. Diffing the composed OmniLidar prims of the two assets (all other omni:sensor:* attributes are identical — ROTARY, 32 emitters/channels, maxReturns 2, tickRate 10 == scanRateBaseHz 10):

Attribute	5.1 XT-32.usd	6.0 HESAI_XT32_SD10.usd
omni:sensor:Core:accumulateOutputs	unauthored → false	authored true
omni:sensor:Core:patternFiringRateHz	unauthored → 36000	authored 20000
omni:sensor:Core:skipDroppingInvalidPoints	authored true	authored false

A/B with your unmodified graph in the same warm session/pod (RTX PRO 6000 MIG, 1800 frames):

XT-32 asset composed	accumulateOutputs on composed prim	PointCloud2 msgs	GMO
6.0 XT32_SD10	true	300	300 full scans (22.9k pts)
5.1 XT-32.usd	false	0	valid header, numElements == 0, on all 1800 frames
5.1 XT-32.usd + we author accumulateOutputs=true and patternFiringRateHz=20000 on the prim	true	300	300 full scans (128000 pts — invalid points retained per skipDroppingInvalidPoints=true)

That middle row is exactly the signature we reported: topic advertised, never publishes, GMO header valid with permanent numElements == 0, no log output at any level. The single-attribute repair in the last row fully restores 10 Hz full scans.

This also explains the confusing 2D/3D asymmetry in our report: the 5.1 RPLidar S2E asset also leaves accumulateOutputs unauthored, but the single-emitter/single-return LaserScan profile still produces scans in that mode — only the multi-emitter, dual-return XT-32-class profile goes permanently silent without full-scan accumulation.

Correction to our original report: we stated we had "verified accumulateOutputs=1 on the composed prim". That was wrong — the value we checked came from the stock 6.0 asset, not from the asset our pipeline actually composed. Apologies for the misdirection; the FIF/warmup and multi-emitter-defect hypotheses in our report are both withdrawn.

4. One suggestion (question 3 from our report, now sharpened)

Everything downstream behaves "correctly" here — OgnIsaacExtractRTXSensorPointCloud zero-skips by design — so a profile that can never complete a scan is indistinguishable from a healthy sensor with no returns, for days. Either of these would have cut this hunt from weeks to minutes:

• a throttled warn-once after N consecutive valid-header/zero-element GMO frames on an attached annotator, and/or
• a validation warning at sensor-creation time when an OmniLidar prim in multi-tick mode composes with accumulateOutputs unauthored/false on a multi-emitter profile (a plausibly-common state for anyone carrying pre-6.0 assets forward).

Happy to open a PR for the warn-once if the approach is acceptable to you. Otherwise, thanks again for the script — it was the discriminator that cracked this — and feel free to close.

[zwdoescode]

Thanks for confirming. Please feel free to open an PR with your suggestion. I will notify the internal team to review / cherry pick. I will close this issue.