Sync video recorder's (--video) renderer backend with the active visualizer (--visualizer) (#5474)

# Description

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html

💡 Please try to keep PRs small and focused. Large PRs are harder to
review and merge.
-->

Extending Brian's original PR:
Sync video recorder's (--video) renderer backend with the active
visualizer (--visualizer)

Change enable cross recording support for Renderers and Visualizers

Also fixes
https://nvbugs/6121118


<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)

## Screenshots

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

## Checklist

- [ ] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------

Signed-off-by: bdilinila <148156773+bdilinila@users.noreply.github.com>
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Signed-off-by: matthewtrepte <mtrepte@nvidia.com>
Co-authored-by: Brian Dilinila <bdilinila@nvidia.com>
Co-authored-by: HuiDong Chen <huidongc@nvidia.com>
Co-authored-by: bdilinila <148156773+bdilinila@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Piotr Barejko <pbarejko@nvidia.com>
Co-authored-by: myurasov-nv <168484206+myurasov-nv@users.noreply.github.com>
Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com>
Co-authored-by: Antoine Richard <antoiner@nvidia.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: ClemensSchwarke <clemens.schwarke@gmail.com>
Co-authored-by: rwiltz <165190220+rwiltz@users.noreply.github.com>
Co-authored-by: Kelly Guo <kellyg@nvidia.com>
Co-authored-by: Kelly Guo <kelly@nvidia.com>
Co-authored-by: ooctipus <zhengyuz@nvidia.com>
Co-authored-by: hougantc-nvda <127865892+hougantc-nvda@users.noreply.github.com>
Co-authored-by: Piotr Barejko <piotrbarejko@protonmail.com>

Author: 16 people

apps/isaaclab.python.headless.rendering.kit

@@ -17,6 +17,7 @@ keywords = ["experience", "app", "isaaclab", "python", "camera", "minimal"]
 [dependencies]
 # Isaac Lab minimal app
 "isaaclab.python.headless" = {}
+"isaacsim.core.rendering_manager" = {}
 "omni.replicator.core" = {}
 
 # Rendering

docs/source/features/visualization.rst

@@ -185,6 +185,96 @@ Also, there is a CLI arg ``--max_visible_envs`` that overrides ``VisualizerCfg.m
      - any
      - Run headless; ``--headless`` takes precedence.
 
+Video Recording
+---------------
+
+Video recording is enabled with the ``--video`` flag. When combined with ``--visualizer``,
+the visualizer selection also determines which backend captures the video frames:
+
+- ``--visualizer kit`` enables ``--video`` capture through the Isaac RTX renderer (Omniverse Replicator).
+- ``--visualizer newton`` enables ``--video`` capture through the Newton OpenGL renderer.
+- ``--visualizer rerun`` does not produce ``--video`` clips; it records Rerun ``.rrd`` data for replay
+  through the Rerun visualizer.
+- ``--visualizer viser`` does not currently provide a ``--video`` recording backend.
+
+When both Kit and Newton visualizers are active, Isaac Lab records a single ``--video`` stream and
+Kit takes precedence. To record from the renderer/physics stack instead of the active visualizer,
+set ``VideoRecorderCfg.backend_source = "renderer"`` in the task configuration.
+
+.. list-table:: ``--video`` compatibility: visualizer × renderer preset
+   :header-rows: 1
+   :widths: 28 36 36
+
+   * - Renderer preset
+     - ``--visualizer kit --video``
+     - ``--visualizer newton --video``
+   * - ``isaacsim_rtx_renderer``
+     - ✅ Kit RTX captures video *(default, no change)*
+     - ✅ Newton GL captures video *(overrides RTX backend)*
+   * - ``newton_renderer``
+     - ✅ Kit RTX captures video *(overrides Newton backend)*
+     - ✅ Newton GL captures video *(default, no change)*
+   * - ``ovrtx_renderer``
+     - ❌ **Raises an error** — see note below
+     - ✅ Newton GL captures video; ovrtx provides camera sensor data
+
+.. note::
+
+   ``--visualizer kit`` combined with ``ovrtx_renderer`` raises a ``ValueError`` at startup.
+   Both Kit (Isaac Sim) and ovrtx ship conflicting RTX hydra libraries compiled against
+   different USD namespaces (``pxrInternal_v0_25_11`` vs ``ovInternal_v0_25_11``), which
+   causes a dynamic-linker crash when loaded into the same process.
+   Use ``--visualizer newton`` instead — it is compatible with all renderer presets.
+
+**Record video with the ovrtx renderer preset**
+
+.. code-block:: bash
+
+   ./isaaclab.sh -p scripts/benchmarks/benchmark_rsl_rl.py \
+     --task=Isaac-Repose-Cube-Shadow-Vision-Direct-v0 \
+     --enable_cameras \
+     --visualizer newton \
+     --video \
+     --video_length=300 \
+     --video_interval=2000 \
+     --max_iterations=5 \
+     --num_envs=1024 \
+     --benchmark_backend=summary \
+     "presets=newton,ovrtx_renderer,rgb"
+
+**Record video with the Isaac RTX renderer preset using the Newton video backend**
+
+.. code-block:: bash
+
+   ./isaaclab.sh -p scripts/benchmarks/benchmark_rsl_rl.py \
+     --task=Isaac-Repose-Cube-Shadow-Vision-Direct-v0 \
+     --enable_cameras \
+     --visualizer newton \
+     --video \
+     --video_length=300 \
+     --video_interval=2000 \
+     --max_iterations=5 \
+     --num_envs=1024 \
+     --benchmark_backend=summary \
+     "presets=physx,isaacsim_rtx_renderer,rgb"
+
+**Record video with the Isaac RTX renderer preset using the Kit video backend**
+
+.. code-block:: bash
+
+   ./isaaclab.sh -p scripts/benchmarks/benchmark_rsl_rl.py \
+     --task=Isaac-Repose-Cube-Shadow-Vision-Direct-v0 \
+     --enable_cameras \
+     --visualizer kit \
+     --video \
+     --video_length=300 \
+     --video_interval=2000 \
+     --max_iterations=5 \
+     --num_envs=1024 \
+     --benchmark_backend=summary \
+     "presets=physx,isaacsim_rtx_renderer,rgb"
+
+
 Visualizer Backends
 -------------------
 

docs/source/how-to/record_video.rst

@@ -3,9 +3,10 @@ Recording video clips during training
 
 Isaac Lab supports recording video clips during training using the
 `gymnasium.wrappers.RecordVideo <https://gymnasium.farama.org/main/_modules/gymnasium/wrappers/record_video/>`_ class.
-When the ``--video`` flag is enabled, Isaac Lab captures a perspective view of the scene. The backend
-is chosen automatically from the active physics and renderer stack: an Isaac Sim Kit camera or a
-Newton GL headless viewer.
+When the ``--video`` flag is enabled, Isaac Lab captures a perspective view of the scene. If a Kit or
+Newton visualizer is active, that visualizer selects the video backend by default. Otherwise, the
+backend is chosen automatically from the active physics and renderer stack: an Isaac Sim Kit camera or
+a Newton GL headless viewer.
 
 This feature can be enabled by installing ``ffmpeg`` and using the following command line arguments with the training
 script:
@@ -32,8 +33,8 @@ Overview
 
 The video recording feature is implemented using the ``VideoRecorder`` class. This class is responsible for resolving the video backend from the scene, capturing the video frames, and saving them to a file.
 
-* ``VideoRecorderCfg`` (``isaaclab.envs.utils.video_recorder_cfg``) holds resolution and world-space
-  perspective parameters ``camera_position`` and ``camera_target`` (defaults to a diagonal view of the
+* ``VideoRecorderCfg`` (``isaaclab.envs.utils.video_recorder_cfg``) holds resolution, backend source,
+  and world-space perspective parameters ``eye`` and ``lookat`` (defaults to a diagonal view of the
   scene).
 * ``VideoRecorder`` (``isaaclab.envs.utils.video_recorder``) picks a video backend from the scene
   (Kit vs Newton GL), builds the matching low-level capture object, and returns RGB frames via
@@ -47,12 +48,12 @@ The video recording feature is implemented using the ``VideoRecorder`` class. Th
 Configuration: ``VideoRecorderCfg``
 ------------------------------------
 
-The dataclass lives in ``isaaclab.envs.utils.video_recorder_cfg``. Fields ``camera_position`` and
-``camera_target`` are the perspective ``eye`` and ``lookat`` points in meters.
+The dataclass lives in ``isaaclab.envs.utils.video_recorder_cfg``. Fields ``eye`` and ``lookat`` are
+the perspective camera position and target in meters.
 
 .. literalinclude:: ../../../source/isaaclab/isaaclab/envs/utils/video_recorder_cfg.py
    :language: python
-   :lines: 20-48
+   :lines: 20-58
 
 
 Task framing: ``ViewerCfg``
@@ -72,11 +73,20 @@ Backend selection: Kit vs Newton GL
 -------------------------------------
 
 ``VideoRecorder`` resolves the implementation from the live :class:`~isaaclab.scene.InteractiveScene`.
-If the user provides the PhysX physics (``presets=physx,...``) or Isaac RTX (``presets=isaac_rtx_renderer,...``) in the sensor stack, the Kit path is selected (``omni.replicator`` on
-``/OmniverseKit_Persp``). The Newton GL path is selected when Newton physics is active (``presets=newton,...``) or the Newton
-Warp renderer (``presets=newton_renderer,...``) appears in the sensor stack - and neither PhysX nor Isaac RTX is present to claim the
-Kit path. OVRTX (``presets=ovrtx_renderer,...`` from ``isaaclab_ov``) can pair with IsaacSim or Newton physics; in that case the video backend is
-selected via the physics preset. If both Kit and Newton GL signals are present (e.g., ``presets=physx,isaac_rtx_renderer,...`` or ``presets=newton,newton_renderer,...``), the Kit path is chosen.
+With the default ``VideoRecorderCfg.backend_source = "visualizer"``, an active ``--visualizer kit``
+selects the Kit path (``omni.replicator`` on ``/OmniverseKit_Persp``), and an active
+``--visualizer newton`` selects the Newton GL path. If both visualizers are active, Kit takes
+precedence and only one ``--video`` stream is recorded. Rerun records ``.rrd`` replay data through
+the Rerun visualizer rather than producing ``--video`` clips, and Viser does not currently provide a
+``--video`` recording backend.
+
+Set ``VideoRecorderCfg.backend_source = "renderer"`` to ignore active visualizers and choose from the
+physics/renderer stack instead. In that mode, PhysX physics (``presets=physx,...``) or Isaac RTX
+(``presets=isaac_rtx_renderer,...``) selects the Kit path. Newton physics (``presets=newton,...``) or
+the Newton Warp renderer (``presets=newton_renderer,...``) selects the Newton GL path when no Kit
+signal is present. OVRTX (``presets=ovrtx_renderer,...`` from ``isaaclab_ov``) can pair with IsaacSim
+or Newton physics; in that case the video backend is selected via the physics preset. If both Kit and
+Newton GL signals are present, the Kit path is chosen.
 
 .. literalinclude:: ../../../source/isaaclab/isaaclab/envs/utils/video_recorder.py
    :language: python
@@ -87,8 +97,8 @@ Construction and dispatch
 --------------------------
 
 When ``env_render_mode`` is ``"rgb_array"`` (as when wrappers or scripts request RGB frames for
-video), the recorder instantiates the backend-specific helper and passes through ``camera_position``,
-``camera_target``, and window size.
+video), the recorder instantiates the backend-specific helper and passes through ``eye``, ``lookat``,
+and window size.
 
 .. literalinclude:: ../../../source/isaaclab/isaaclab/envs/utils/video_recorder.py
    :language: python
@@ -98,18 +108,19 @@ video), the recorder instantiates the backend-specific helper and passes through
 Customising the camera view
 ----------------------------
 
-When ``--video`` is passed, the recording camera uses the same
-position and look-at target as the interactive viewer. The defaults come from
+When ``--video`` is passed, the recording camera uses the same configured
+position and look-at target as the active Kit or Newton visualizer when that visualizer drives backend
+selection. Otherwise, the defaults come from
 :class:`~isaaclab.envs.common.ViewerCfg`:
 
 * ``eye = (7.5, 7.5, 7.5)`` — camera position in world space (metres)
 * ``lookat = (0.0, 0.0, 0.0)`` — camera look-at target in world space (metres)
 * Resolution ``1280x720``
 
-To change the recording angle, override the ``viewer`` field in your task's environment config.
-The RL base classes automatically copy ``eye`` and ``lookat`` into ``VideoRecorderCfg`` before
-recording starts (when ``origin_type`` is ``"world"``), so the video clip uses the same viewpoint
-as the interactive viewport:
+To change the recording angle without a visualizer, override the ``viewer`` field in your task's
+environment config. The RL base classes automatically copy ``eye`` and ``lookat`` into
+``VideoRecorderCfg`` before recording starts (when ``origin_type`` is ``"world"``), so the video clip
+uses the same configured viewpoint as the interactive viewport:
 
 .. code-block:: python
 
@@ -144,6 +155,12 @@ Summary
    * - ``newton,...,ovrtx_renderer,...`` (OVRTX + Newton physics)
      - Newton GL (``"newton_gl"``)
      - ``newton.viewer.ViewerGL`` on the SDP Newton model
+   * - ``--visualizer kit`` with default ``backend_source``
+     - Kit (``"kit"``)
+     - Visualizer ``eye`` / ``lookat`` copied to ``/OmniverseKit_Persp`` + Replicator RGB
+   * - ``--visualizer newton`` with default ``backend_source``
+     - Newton GL (``"newton_gl"``)
+     - Visualizer ``eye`` / ``lookat`` initially, then live Newton viewer camera sync per frame
 
 
 See also

source/isaaclab/changelog.d/mtrepte-expand_viz_markers.minor.rst

@@ -0,0 +1,5 @@
+Added
+^^^^^
+
+* Added backend-agnostic :class:`~isaaclab.markers.VisualizationMarkers` support for
+  marker-capable Kit, Newton, Rerun, and Viser visualizers.

source/isaaclab/isaaclab/envs/direct_marl_env.py

@@ -176,8 +176,8 @@ def _init_sim(self, render_mode: str | None = None, **kwargs):
         if self.cfg.video_recorder is not None:
             self.cfg.video_recorder.env_render_mode = render_mode
             vr = self.cfg.video_recorder
-            vr.camera_position = tuple(float(x) for x in self.cfg.viewer.eye)
-            vr.camera_target = tuple(float(x) for x in self.cfg.viewer.lookat)
+            vr.eye = tuple(float(x) for x in self.cfg.viewer.eye)
+            vr.lookat = tuple(float(x) for x in self.cfg.viewer.lookat)
             self.video_recorder: VideoRecorder = self.cfg.video_recorder.class_type(self.cfg.video_recorder, self.scene)
         else:
             self.video_recorder = None

source/isaaclab/isaaclab/envs/direct_rl_env.py

@@ -182,8 +182,8 @@ def _init_sim(self, render_mode: str | None = None, **kwargs):
             self.cfg.video_recorder.env_render_mode = render_mode
             # Perspective --video uses same eye/lookat as task viewer (Kit persp + Newton GL).
             vr = self.cfg.video_recorder
-            vr.camera_position = tuple(float(x) for x in self.cfg.viewer.eye)
-            vr.camera_target = tuple(float(x) for x in self.cfg.viewer.lookat)
+            vr.eye = tuple(float(x) for x in self.cfg.viewer.eye)
+            vr.lookat = tuple(float(x) for x in self.cfg.viewer.lookat)
             self.video_recorder: VideoRecorder = self.cfg.video_recorder.class_type(self.cfg.video_recorder, self.scene)
         else:
             self.video_recorder = None

source/isaaclab/isaaclab/envs/manager_based_env.py

@@ -195,7 +195,7 @@ def _init_sim(self):
         # Instantiate the video recorder before sim.reset() so that any fallback Camera
         # (used for state-based envs without an observation camera) is spawned into the USD
         # stage and registered for the PHYSICS_READY callback before physics initialises.
-        # env_render_mode and camera_position/camera_target are forwarded by subclasses (e.g. ManagerBasedRLEnv)
+        # env_render_mode and eye/lookat are forwarded by subclasses (e.g. ManagerBasedRLEnv)
         # into cfg.video_recorder before calling super().__init__().
         if self.cfg.video_recorder is not None:
             self.video_recorder: VideoRecorder = self.cfg.video_recorder.class_type(self.cfg.video_recorder, self.scene)

source/isaaclab/isaaclab/envs/manager_based_rl_env.py

@@ -81,8 +81,8 @@ def __init__(self, cfg: ManagerBasedRLEnvCfg, render_mode: str | None = None, **
         # (env_render_mode="rgb_array") and the perspective view matches the task viewport.
         if cfg.video_recorder is not None:
             cfg.video_recorder.env_render_mode = render_mode
-            cfg.video_recorder.camera_position = tuple(float(x) for x in cfg.viewer.eye)
-            cfg.video_recorder.camera_target = tuple(float(x) for x in cfg.viewer.lookat)
+            cfg.video_recorder.eye = tuple(float(x) for x in cfg.viewer.eye)
+            cfg.video_recorder.lookat = tuple(float(x) for x in cfg.viewer.lookat)
 
         # initialize the base class to setup the scene.
         super().__init__(cfg=cfg)