Add Debug Visualization Markers to Newton Based Visualizers (#5473)

# Description

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html

💡 Please try to keep PRs small and focused. Large PRs are harder to
review and merge.
-->

Extend debug Visualization Markers, which are supported in the Kit
Visualizer, to the Newton Visualizers.

These Visualization Markers are various shapes and models which can be
added to envs for debugging / showing extra information.

Also added filtering for partial visualization (when we filtered which
envs are shown the in the visualizer, we also filter the markers)

For general USD mesh marker support in Newton, a followup PR will be
required, once a Newton API for general USD -> Newton Mesh conversion is
added (see newton-physics/newton#2667)

Checked velocity arrows, dexcubes, raycasts, frames, goal markers


<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- New feature (non-breaking change which adds functionality)
- Documentation update

## Screenshots

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

## Checklist

- [ ] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------

Signed-off-by: matthewtrepte <mtrepte@nvidia.com>

Author: matthewtrepte

docs/source/features/visualization.rst

@@ -22,16 +22,16 @@ Isaac Lab supports four visualizer backends, each optimized for different use ca
      - Key Features
    * - **Omniverse**
      - High-fidelity, Isaac Sim integration
-     - USD, visual markers, live plots
+     - USD, visualization markers, live plots
    * - **Newton**
      - Fast iteration
-     - Low overhead, visual markers
+     - Low overhead, visualization markers
    * - **Rerun**
      - Remote viewing, replay
-     - Webviewer, time scrubbing, recording export
+     - Webviewer, time scrubbing, recording export, visualization markers
    * - **Viser**
      - Web-based remote visualization, sharing, recording
-     - Warp-based rendering, browser-based, share URL
+     - Warp-based rendering, browser-based, share URL, visualization markers
 
 
 *The following visualizers are shown training the Isaac-Velocity-Flat-Anymal-D-v0 environment.*
@@ -284,9 +284,9 @@ Omniverse Visualizer
 **Main Features:**
 
 - Native USD stage integration
-- Visualization markers for debugging (arrows, frames, points, etc.)
 - Live plots for monitoring training metrics
 - Full Isaac Sim rendering capabilities and tooling
+- Visualization markers for debugging (arrows, frames, object targets, etc.)
 
 **Core Configuration:**
 
@@ -316,10 +316,10 @@ Newton Visualizer
 **Main Features:**
 
 - Lightweight OpenGL rendering with low overhead
-- Visualization markers (joints, contacts, springs, COM)
 - Simulation and rendering pause controls
 - Adjustable update frequency for performance tuning
 - Some customizable rendering options (shadows, sky, wireframe)
+- Visualization markers (joints, contacts, springs, COM, debug markers)
 
 
 **Interactive Controls:**
@@ -388,6 +388,7 @@ Rerun Visualizer
 - Metadata logging and filtering
 - Recording to .rrd files for offline replay (.rrd files can be opened with ctrl+O from the web viewer)
 - Timeline scrubbing and playback controls of recordings
+- Visualization debug markers
 
 **Core Configuration:**
 
@@ -432,6 +433,7 @@ server, allowing you to view and interact with the scene from any browser.
 - Optional public share URL for remote viewing
 - Recording to ``.viser`` format for replay
 - Environment filtering to control which environments are rendered
+- Visualization debug markers
 
 **Launch with Viser:**
 
@@ -464,7 +466,7 @@ server, allowing you to view and interact with the scene from any browser.
 
 .. note::
 
-   The Viser visualizer does not currently support markers or live plots.
+   The Viser visualizer does not currently support live plots.
 
 
 Performance Note
@@ -497,15 +499,9 @@ the num of environments can be overwritten and decreased using ``--num_envs``:
 The FPS control in the Rerun visualizer UI may not affect the visualization frame rate in all configurations.
 
 
-**Newton Visualizer Contact and Center of Mass Markers**
+**Live Plots**
 
-Contact and center of mass markers are not yet supported in the Newton visualizer. This will be addressed in a future release.
-
-
-**Viser Visualizer Markers and Live Plots**
-
-The Viser visualizer does not currently support visualization markers or live plots. For these features, use the
-Omniverse or Newton visualizers.
+Currently, live plots are only available in the Kit Visualizer.
 
 
 **Viser Visualizer Renderer Requirement**

source/isaaclab/changelog.d/mtrepte-expand_viz_markers-2.skip

@@ -205,16 +205,13 @@ def set_debug_vis(self, debug_vis: bool) -> bool:
         if debug_vis:
             if self._debug_vis_handle is None:
                 sim_ctx = SimulationContext.instance()
-                if "physx" in sim_ctx.physics_manager.__name__.lower():
-                    import omni.kit.app
-
-                    app_interface = omni.kit.app.get_app_interface()
-                    self._debug_vis_handle = app_interface.get_post_update_event_stream().create_subscription_to_pop(
-                        lambda event, obj=weakref.proxy(self): obj._debug_vis_callback(event)
-                    )
+                if sim_ctx is not None:
+                    self._debug_vis_handle = sim_ctx.vis_marker_registry.add_debug_vis_callback(self)
         else:
-            if self._debug_vis_handle is not None:
-                self._debug_vis_handle.unsubscribe()
+            sim_ctx = SimulationContext.instance()
+            if sim_ctx is not None:
+                sim_ctx.vis_marker_registry.clear_debug_vis_callback(self)
+            else:
                 self._debug_vis_handle = None
         # return success
         return True
@@ -404,8 +401,10 @@ def _initialize_callback(self, event):
     def _invalidate_initialize_callback(self, event):
         """Invalidates the scene elements."""
         self._is_initialized = False
-        if self._debug_vis_handle is not None:
-            self._debug_vis_handle.unsubscribe()
+        sim_ctx = SimulationContext.instance()
+        if sim_ctx is not None:
+            sim_ctx.vis_marker_registry.clear_debug_vis_callback(self)
+        else:
             self._debug_vis_handle = None
 
     def _on_prim_deletion(self, event) -> None:
@@ -435,6 +434,8 @@ def _clear_callbacks(self) -> None:
         if self._prim_deletion_handle is not None:
             self._prim_deletion_handle.deregister()
             self._prim_deletion_handle = None
-        if self._debug_vis_handle is not None:
-            self._debug_vis_handle.unsubscribe()
+        sim_ctx = SimulationContext.instance()
+        if sim_ctx is not None:
+            sim_ctx.vis_marker_registry.clear_debug_vis_callback(self)
+        else:
             self._debug_vis_handle = None

source/isaaclab/isaaclab/assets/asset_base.py

@@ -9,19 +9,13 @@
 
 import inspect
 import re
-import weakref
 from abc import abstractmethod
 from collections.abc import Sequence
 from typing import TYPE_CHECKING, Any
 
 import torch
 from prettytable import PrettyTable
 
-from isaaclab.utils.version import has_kit
-
-if has_kit():
-    import omni.kit.app
-
 from isaaclab.envs.utils.io_descriptors import GenericActionIODescriptor
 
 from .manager_base import ManagerBase, ManagerTermBase
@@ -66,9 +60,11 @@ def __init__(self, cfg: ActionTermCfg, env: ManagerBasedEnv):
 
     def __del__(self):
         """Unsubscribe from the callbacks."""
-        if self._debug_vis_handle:
-            self._debug_vis_handle.unsubscribe()
-            self._debug_vis_handle = None
+        env = getattr(self, "_env", None)
+        sim = getattr(env, "sim", None)
+        registry = getattr(sim, "vis_marker_registry", None)
+        if registry is not None:
+            registry.clear_debug_vis_callback(self)
 
     """
     Properties.
@@ -135,15 +131,10 @@ def set_debug_vis(self, debug_vis: bool) -> bool:
         if debug_vis:
             # create a subscriber for the post update event if it doesn't exist
             if self._debug_vis_handle is None:
-                app_interface = omni.kit.app.get_app_interface()
-                self._debug_vis_handle = app_interface.get_post_update_event_stream().create_subscription_to_pop(
-                    lambda event, obj=weakref.proxy(self): obj._debug_vis_callback(event)
-                )
+                self._debug_vis_handle = self._env.sim.vis_marker_registry.add_debug_vis_callback(self)
         else:
             # remove the subscriber if it exists
-            if self._debug_vis_handle is not None:
-                self._debug_vis_handle.unsubscribe()
-                self._debug_vis_handle = None
+            self._env.sim.vis_marker_registry.clear_debug_vis_callback(self)
         # return success
         return True
 

source/isaaclab/isaaclab/managers/action_manager.py

@@ -8,25 +8,19 @@
 from __future__ import annotations
 
 import inspect
-import weakref
 from abc import abstractmethod
 from collections.abc import Sequence
 from typing import TYPE_CHECKING
 
 import torch
 from prettytable import PrettyTable
 
-from isaaclab.utils.version import has_kit
-
 from .manager_base import ManagerBase, ManagerTermBase
 from .manager_term_cfg import CommandTermCfg
 
 if TYPE_CHECKING:
     from isaaclab.envs import ManagerBasedRLEnv
 
-if has_kit():
-    import omni.kit.app
-
 
 class CommandTerm(ManagerTermBase):
     """The base class for implementing a command term.
@@ -65,9 +59,11 @@ def __init__(self, cfg: CommandTermCfg, env: ManagerBasedRLEnv):
 
     def __del__(self):
         """Unsubscribe from the callbacks."""
-        if self._debug_vis_handle:
-            self._debug_vis_handle.unsubscribe()
-            self._debug_vis_handle = None
+        env = getattr(self, "_env", None)
+        sim = getattr(env, "sim", None)
+        registry = getattr(sim, "vis_marker_registry", None)
+        if registry is not None:
+            registry.clear_debug_vis_callback(self)
 
     """
     Properties
@@ -106,18 +102,12 @@ def set_debug_vis(self, debug_vis: bool) -> bool:
         # toggle debug visualization objects
         self._set_debug_vis_impl(debug_vis)
         # toggle debug visualization handles
-        if debug_vis and has_kit():
-            # create a subscriber for the post update event if it doesn't exist
+        if debug_vis:
             if self._debug_vis_handle is None:
-                app_interface = omni.kit.app.get_app_interface()
-                self._debug_vis_handle = app_interface.get_post_update_event_stream().create_subscription_to_pop(
-                    lambda event, obj=weakref.proxy(self): obj._debug_vis_callback(event)
-                )
+                self._debug_vis_handle = self._env.sim.vis_marker_registry.add_debug_vis_callback(self)
         else:
             # remove the subscriber if it exists
-            if self._debug_vis_handle is not None:
-                self._debug_vis_handle.unsubscribe()
-                self._debug_vis_handle = None
+            self._env.sim.vis_marker_registry.clear_debug_vis_callback(self)
         # return success
         return True
 

source/isaaclab/isaaclab/managers/command_manager.py

@@ -0,0 +1,66 @@
+# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""Registry for visualization marker state."""
+
+from __future__ import annotations
+
+import weakref
+from collections.abc import Callable
+from typing import Any
+
+
+class VisMarkerRegistry:
+    """Tracks visualization marker callbacks and active marker groups."""
+
+    def __init__(self):
+        self._callbacks: dict[str, Callable[[Any], None]] = {}
+        self._groups: dict[str, Any] = {}
+
+    def add_callback(self, name: str, callback: Callable[[Any], None]) -> str:
+        """Register a callback invoked before marker-capable visualizers step each render tick."""
+        self._callbacks[name] = callback
+        return name
+
+    def add_debug_vis_callback(self, owner: Any) -> str:
+        """Register an owner's debug visualization callback.
+
+        Args:
+            owner: Object implementing ``_debug_vis_callback(event)``.
+
+        Returns:
+            Callback identifier that can be passed to :meth:`remove_callback`.
+        """
+        callback_id = f"visualization_marker:{type(owner).__name__}:{id(owner)}"
+        owner_ref = weakref.proxy(owner)
+        return self.add_callback(callback_id, lambda event: owner_ref._debug_vis_callback(event))
+
+    def clear_debug_vis_callback(self, owner: Any) -> None:
+        """Clear an owner's registered debug visualization callback, if any."""
+        callback_id = getattr(owner, "_debug_vis_handle", None)
+        if callback_id is not None:
+            self.remove_callback(callback_id)
+            owner._debug_vis_handle = None
+
+    def remove_callback(self, callback_id: str) -> None:
+        """Remove a visualization marker callback if it exists."""
+        self._callbacks.pop(callback_id, None)
+
+    def dispatch_callbacks(self, event: Any = None) -> None:
+        """Invoke all registered visualization marker callbacks."""
+        for callback in list(self._callbacks.values()):
+            callback(event)
+
+    def set_group(self, group_id: str, state: Any) -> None:
+        """Set or replace one visualization marker group state."""
+        self._groups[group_id] = state
+
+    def remove_group(self, group_id: str) -> None:
+        """Remove one visualization marker group state if present."""
+        self._groups.pop(group_id, None)
+
+    def get_groups(self) -> dict[str, Any]:
+        """Return all active visualization marker groups."""
+        return self._groups