Reuse Newton visualizer frames in videos

[ooctipus]

Summary

• reuse the active Newton visualizer framebuffer directly for --video
• preserve its live camera, selected environments, and viewer-side scene markers without a second render pass
• keep Kit and renderer-selected standalone Newton captures eager so pre-reset camera setup is unchanged
• remove duplicate-viewer camera reconstruction, world-selection synchronization, overlay callbacks, and the associated standalone capture APIs
• document that active-Newton capture uses the visualizer resolution and excludes Newton UI panels such as the tiled-camera panel

With VideoRecorderCfg.backend_source = "renderer", recording remains independent of active visualizers and uses the existing standalone capture path.

Testing

• 38 passed: test_video_recorder.py and test_newton_adapter.py
• fail-before-fix verification: the old recorder returned the standalone capture frame instead of binding the active Newton visualizer framebuffer
• ./isaaclab.sh -d passed; rendered HTML contains both stable :pyobject: excerpts
• scoped pre-commit passed, including the LFS hook
• repository-wide pre-commit passed with only the known unrelated golden-image LFS hook skipped
• changelog fragment check passed

[greptile-apps]

Greptile Summary

This PR extends the Newton GL video capture pipeline to mirror the active Newton visualizer's world-visibility selection, visual world spacing, and marker overlays into recorded frames. The synchronization runs per frame in _sync_newton_camera and propagates into a new set of public API methods on NewtonGlPerspectiveVideo.

• VideoRecorder._sync_newton_camera is extended to resolve get_visualized_env_ids() + max_visible_envs each frame and forward the result (plus world spacing and an overlay callback) to the capture object before rendering.
• NewtonGlPerspectiveVideo gains set_visible_worlds, set_world_offsets, and set_frame_overlay_callback, all of which support pre-initialization queuing and the first two include change-deduplication to avoid redundant Newton GL cache rebuilds.
• render_rgb_array wraps log_state + overlay callback in a try/finally so viewer.end_frame() is guaranteed even when a marker callback throws.

Confidence Score: 4/5

Safe to merge; all changes are additive and well-tested, with no modifications to existing public APIs.

The synchronization logic in _sync_newton_camera is correct for all common cases and is backed by thorough unit tests. The try/finally in render_rgb_array is a genuine correctness improvement. The three findings are cosmetic: the num_envs=0 edge case is documented behavior that only affects a degenerate scene configuration, the missing deduplication in set_frame_overlay_callback is a per-frame attribute store with no downstream side effects, and the BOM in the RST file did not break the documented build.

No files require special attention beyond the minor notes on video_recorder.py (num_envs=0 edge case), newton_gl_perspective_video.py (callback deduplication), and the RST BOM.

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/envs/utils/video_recorder.py	Extends _sync_newton_camera to resolve visible env IDs + cap, world spacing, and overlay callback each frame; logic is correct but the num_envs=0 edge case forwards None instead of [] when max_visible_envs is set.
source/isaaclab_newton/isaaclab_newton/video_recording/newton_gl_perspective_video.py	Adds set_visible_worlds, set_world_offsets, set_frame_overlay_callback with pre-init queuing; render_rgb_array wraps log_state+callback in try/finally; minor asymmetry: set_world_offsets is called directly in _ensure_viewer but guarded in the method itself.
source/isaaclab/test/envs/test_video_recorder.py	Adds 9 focused unit tests covering visible-env ID forwarding, capping, edge cases, world-spacing sync, marker enable/disable, and legacy capture compatibility; all well-structured.
source/isaaclab_newton/test/video_recording/test_newton_gl_perspective_video.py	New test file with 10 tests covering pre-init deferral, deduplication, mutation safety, ordering guarantees, and overlay callback frame ordering; comprehensively exercises the new public API.
docs/source/api/lab_newton/isaaclab_newton.video_recording.rst	New RST stub for the video_recording module; contains a UTF-8 BOM at the start of the file which is unusual and may cause docutils/Sphinx warnings in stricter environments.
docs/source/how-to/record_video.rst	Adds documentation for Newton visualizer world-spacing and marker-overlay recording; prose and example are clear and accurate.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant VR as VideoRecorder
    participant NV as NewtonVisualizer (live)
    participant CAP as NewtonGlPerspectiveVideo
    participant GL as ViewerGL (headless)

    VR->>VR: render_rgb_array()
    VR->>VR: _sync_newton_camera()
    VR->>NV: get_visualized_env_ids()
    NV-->>VR: [0,1,2,3] or None
    VR->>VR: apply max_visible_envs cap
    VR->>CAP: set_visible_worlds([0,1,2,3])
    CAP->>CAP: dedup check
    CAP->>GL: set_visible_worlds([0,1,2,3])
    VR->>CAP: set_world_offsets((2.0,2.0,0.0))
    CAP->>CAP: dedup check
    CAP->>GL: set_world_offsets((2.0,2.0,0.0))
    VR->>CAP: set_frame_overlay_callback(render_markers)
    CAP->>CAP: store callback
    VR->>CAP: update_camera(pos, target)
    CAP->>GL: set_camera(pos, pitch, yaw)

    VR->>CAP: render_rgb_array()
    CAP->>GL: begin_frame(dt)
    CAP->>GL: log_state(state)
    CAP->>GL: render_markers(viewer) [overlay callback]
    CAP->>GL: end_frame() [always, via finally]
    CAP->>GL: get_frame()
    GL-->>VR: np.ndarray (RGB frame)

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant VR as VideoRecorder
    participant NV as NewtonVisualizer (live)
    participant CAP as NewtonGlPerspectiveVideo
    participant GL as ViewerGL (headless)

    VR->>VR: render_rgb_array()
    VR->>VR: _sync_newton_camera()
    VR->>NV: get_visualized_env_ids()
    NV-->>VR: [0,1,2,3] or None
    VR->>VR: apply max_visible_envs cap
    VR->>CAP: set_visible_worlds([0,1,2,3])
    CAP->>CAP: dedup check
    CAP->>GL: set_visible_worlds([0,1,2,3])
    VR->>CAP: set_world_offsets((2.0,2.0,0.0))
    CAP->>CAP: dedup check
    CAP->>GL: set_world_offsets((2.0,2.0,0.0))
    VR->>CAP: set_frame_overlay_callback(render_markers)
    CAP->>CAP: store callback
    VR->>CAP: update_camera(pos, target)
    CAP->>GL: set_camera(pos, pitch, yaw)

    VR->>CAP: render_rgb_array()
    CAP->>GL: begin_frame(dt)
    CAP->>GL: log_state(state)
    CAP->>GL: render_markers(viewer) [overlay callback]
    CAP->>GL: end_frame() [always, via finally]
    CAP->>GL: get_frame()
    GL-->>VR: np.ndarray (RGB frame)

Loading

_{Reviews (1): Last reviewed commit: "Record Newton visualizer overlays in vid..." | Re-trigger Greptile}

[greptile-apps]

num_envs == 0 with a cap set forwards None (render all) instead of [] (render none)

When visible_env_ids is None (no explicit selection) and max_visible_envs is set but self._scene.num_envs == 0, neither branch is taken, so visible_env_ids remains None and set_visible_worlds(None) is called — meaning "render all worlds." A cap of, say, 4 with 0 known environments arguably should forward [] rather than None, since there is nothing to show. The parametrized test (0, 4, None) documents this as expected, but the semantics could surprise callers who interpret None as "unconstrained."

[greptile-apps]

set_frame_overlay_callback stores unconditionally on every call

set_visible_worlds and set_world_offsets both deduplicate before touching the underlying viewer, explicitly to avoid unnecessary Newton GL cache rebuilds. set_frame_overlay_callback skips this guard entirely — it always overwrites self._frame_overlay_callback regardless of whether the value changed. Because _sync_newton_camera calls this every frame, the attribute is re-assigned on each render even when enable_markers and the bound method are stable. A simple identity check (if callback is self._frame_overlay_callback: return) would make the three setters consistent and avoid the per-frame churn.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

UTF-8 BOM at start of file

The file begins with a UTF-8 BOM (U+FEFF). While Sphinx strips it before parsing in most configurations (which is why the documented build passed), docutils in stricter or older environments may count the BOM as a title character, making the title one character longer than the underline and triggering a "Title underline too short" warning or error. The BOM is not meaningful in UTF-8 files and is best removed to keep the RST consistent with the other files in the same directory.