OnPlaybackTick-driven ROS2 camera OmniGraph added as `over` on payload-referenced robot stops publishing after 1-2 frames (render product/data valid, ROS2CameraHelper output stalls)

[nimra-pixel]

Description

Summary

Following up on a previously closed forum thread: Isaac Sim 5.1.0 - ROS2CameraHelper reports "Render product not valid" despite valid render product and image data

We've since narrowed the issue significantly with additional diagnostics and believe we've isolated the actual distinguishing factor, which was an open question in that thread.

Environment

• Isaac Sim 5.1.0
• Ubuntu on AWS EC2, NVIDIA L40S GPU
• ROS2 Humble (in Docker, --network host)
• Custom robot (Nova Carter) referenced via payload in a City Massing demo environment

Pipeline

OnPlaybackTick → ROS2Context → IsaacCreateRenderProduct → ROS2CameraHelper (×2: depth, semantic_segmentation)

authored as a USD over override on Nova_Carter_ROS, a prim composed from an external payload (World_CityMassingDemopack_custom1.usd).

The key distinguishing factor we've isolated

Every pre-existing camera graph on this same robot (front_hawk, back_hawk, left_hawk, right_hawk stereo pairs) publishes continuously and correctly. These graphs are authored directly in the payload's own defining USD layer.

Our new graph, added identically in structure and node types/versions, but authored as an over override on top of the already-composed payload (not in the defining layer), publishes only 1-2 frames after Play, then goes silent — no further errors, no further data, indefinitely, while the native graphs on the same robot keep running at full rate.

Diagnostics performed

1. DDS/transport layer — ruled out. Initially all topics (including pre-existing, previously-working ones) went completely silent after an EC2 instance stop/start. Root cause: AWS VPC blocks UDP multicast, which both rmw_fastrtps_cpp and default rmw_cyclonedds_cpp discovery rely on. Fixed with:

   <CycloneDDS><Domain><General><AllowMulticast>false</AllowMulticast></General><Discovery><Peers><Peer address="localhost"/></Peers></Discovery></Domain></CycloneDDS>

forcing unicast discovery to localhost (both processes on the same host, --network host). This fully restored the pre-existing camera/lidar/imu topics to sustained, correct rates. Our new graph's topics were unaffected by this fix — still only 1-2 frames.

2. Node attribute values — verified via og.Controller.attribute(...).get() (not raw prim.GetAttributes(), which incorrectly shows None for connection-driven inputs):

   • gpr_render_product.inputs:cameraPrim → correctly resolves to the target camera
   • gpr_render_product.outputs:renderProductPath → valid, distinct path (e.g. /Render/OmniverseKit/HydraTextures/Replicator_02), confirmed via UsdRender.Product inspection: resolution=(640,480), camera relationship target correct
   • ROS2CameraHelper.inputs:context/renderProductPath/nodeNamespace/topicName/type → all correctly resolved on both depth and segmentation helpers

3. QoS profile — found empty ("") on our nodes vs. a full profile JSON on the working reference nodes; authored a matching QoS profile. No change in behavior.

4. Execution wiring — hypothesized ROS2CameraHelper.inputs:execIn chained through IsaacCreateRenderProduct.outputs:execOut (rather than directly from OnPlaybackTick) might only pulse once. Rewired both helpers' execIn directly to OnPlaybackTick.outputs:tick in parallel via og.Controller.connect(). No change in behavior.

5. Render product resource limits — this robot has ~10 concurrent render products (4 stereo pairs + LiDAR + viewport). Disabled an existing camera (back_hawk) to free a slot; no change in our graph's behavior. GPU memory was not under pressure (5GB/40GB used).

6. Type mismatches — found and fixed inputs:height/inputs:width on IsaacCreateRenderProduct incorrectly typed uint64 instead of schema-required uint. Fixed; eliminated load-time errors, no effect on the runtime stalling behavior.

Question

Is there a known difference in how OnPlaybackTick-driven OmniGraphs are scheduled/re-evaluated when:

• authored in a payload's defining layer, vs.
• added afterward as an over override on the already-composed prim

We can provide the full .usda scene, diagnostic scripts, and Console logs on request.

Isaac Sim version

5.1.0

Operating System (OS)

Ubuntu 22.04

GPU Name

NVIDIA L40S

GPU Driver and CUDA versions

nvidia-smi

Logs

No response

Additional information

Full .usda scene, gpr_camera_node.py, and step-by-step reproduction session available on request.

[zwdoescode]

@nimra-pixel could you please try our latest stable version Isaac Sim 6.0.1 and see if this issue reproducible? Thanks!

[nimra-chorerobots]

hi sure but is there any way we can still proceed with this version and complete our work ?

[zwdoescode]

I tried to reproduce the suspected payload-override scheduling issue in Isaac Sim 5.1 using two local repros:

1. A minimal payloaded /World/Nova_Carter_ROS stage with one graph authored in the payload layer and an equivalent graph authored as a stronger over layer.
2. The packaged 5.1 Nova_Carter_ROS.usd asset, payloaded into a root stage, with a new over-authored graph targeting /World/Nova_Carter_ROS/chassis_link/sensors/front_hawk/left/camera_left.

In both cases the OnPlaybackTick -> ROS2Context -> IsaacCreateRenderProduct -> ROS2CameraHelper pipeline continued publishing beyond the first frames. The Nova Carter override repro published both depth and semantic topics continuously through 360 app updates, with 120 messages received for each topic. The render product was valid after play at /Render/OmniverseKit/HydraTextures/Replicator_03.

Based on that, I cannot reproduce a generic issue where an OmniGraph authored as an over under a payloaded prim stops being scheduled. Once ROS2CameraHelper successfully receives a valid render product, it attaches the SDG/Replicator ROS writer once; ongoing publishing is then driven by the generated writer graph rather than by repeated helper reattachment. So if your scene publishes 1-2 frames and then goes silent while the render product remains valid, the next thing to inspect is likely scene-specific SDG/render-product state, a generated gate/writer node being disabled or overwritten, or a ROS environment/library mismatch.

Could you share a minimized package that includes:

• World_CityMassingDemopack_custom1.usd or a reduced scene preserving the same payload composition.
• The exact .usda override layer that adds the GPR camera graph.
• gpr_camera_node.py.
• Full Kit logs from launch through at least 10 seconds after Play.
• Your container ROS environment values for ROS_DISTRO, RMW_IMPLEMENTATION, LD_LIBRARY_PATH, AMENT_PREFIX_PATH, and any CycloneDDS/FastDDS discovery config.

[zwdoescode]

I will close this issue due to inactivity. Please feel free to reopen it if you have more updates. Thanks!