Add GUI support for object-reference

[qianl-nv]

Summary

Improve GUI to work for the Iightwheel kitchen camera + object-ref snapshots

Detailed description

• Improve review GUI USD thumbnails for Lightwheel robocasa kitchen backgrounds and object references.
• Add get_viewer_cfg to the Lightwheel kitchen, and apply the view cfg in the SimApp background + object reference snapshot path when available. Default ViewCfg looks at the kitchen back wall. Set the custom get_viewer_cfg to look from the front opening.
• Capture object_reference nodes after the background: resolve prim paths to parent USD subtrees, frame the viewport on the highlighted prim, and show collision-mesh previews in asset cards.
• Add the background USD prim tree and the object reference AABB in visualization panel.

Searchable prim tree display

[qianl-nv]

self review 1

[arena-review-bot]

🤖 Isaac Lab-Arena Review Bot

Summary

This PR extends the agentic-environment review GUI to render object_reference snapshots (collision-mesh highlights framed on the picked subtree), adds a per-background viewer camera to frame kitchen thumbnails, and introduces a collapsible/searchable background prim-tree panel. It also refactors thumbnail capture to instantiate assets once from the spec rather than re-resolving specs through the registry. The change is well-factored and almost entirely contained in the review_gui example package; the core touch points are small and sensible.

Design, Boundaries & Scope

get_viewer_cfg is only defined on the Lightwheel kitchen subclass and consumed via getattr(background, "get_viewer_cfg", None). That is capability-by-accident rather than a declared interface — worth promoting to a base Background.get_viewer_cfg() with a None default so every background answers explicitly (inline). The deferred IsaacLabArenaEnvironment/Scene imports in arena_env_graph_conversion_utils.py read as intentional (letting the no-sim visualization service import _instantiate_assets_from_spec without dragging in sim), so no concern there.

Findings

🟡 Warning: thumbnail_capture.py:53 — get_viewer_cfg is detected by name via getattr(..., None); declare it on the base Background so it is a real interface.
🔵 Improvement: thumbnail_capture.py:19 — the example imports the private _instantiate_assets_from_spec from core; if it is now shared API, drop the underscore.

Test Coverage

Good: the renamed test_includes_object_references now asserts references appear as cards, and new TestPrimTreeView/TestBackgroundPrimTree cover the prim-tree nesting, HTML render, and the load path (with monkeypatched USD resolution). These are pure-Python and land in Phase 1 correctly. The Kit-viewport capture path (_capture_usd_snapshot_job, viewer-cfg framing, collider toggling) remains untested, which is understandable given it needs a live SimApp — no action required.

Verdict

Minor fixes needed

[greptile-apps]

Greptile Summary

This PR extends the review GUI to support Lightwheel kitchen camera framing and object-reference snapshots. It refactors the thumbnail pipeline from a spec-registry lookup model to a live-asset model, adds _UsdSnapshotJob batching so one USD stage open serves both asset and object-reference captures, and adds a new collapsible/searchable prim-tree widget.

• Thumbnail capture now calls _instantiate_assets_from_spec to get live assets, resolves AABB from them, and applies a per-background ViewerCfg camera position; object references get a dedicated pass with collision-mesh visualization and frame_viewport_prims framing.
• Prim tree view (prim_tree_view.py) is a new self-contained HTML widget rendered in a Streamlit iframe, built from sorted UsdPrimRecord paths with path-prefix nesting.
• Asset cards now include ObjectReferenceSpec entries alongside backgrounds and objects, and the build_asset_cards_with_thumbnails function is changed to return a (cards, prim_tree) tuple propagated through session state.

Confidence Score: 3/5

Safe to merge for the prim-tree widget and visualization improvements, but the refactored thumbnail path has two unguarded dict accesses that will raise unhandled exceptions whenever any asset fails to instantiate.

The two direct dict accesses — assets_by_node_id[spec.background.id] in thumbnail_capture.py and assets_by_node_id[node_id] in resolve_node_usd_paths — will produce KeyError if _instantiate_assets_from_spec omits any node. The previous code handled these paths gracefully per-asset with try/except; the new code moves all instantiation to a single call and then accesses the result without checking. Real environments regularly have missing or mis-registered assets, so this is a realistic failure path. The rest of the change — prim-tree widget, viewer-cfg framing, collision-mesh overlay, asset cards for object references — looks correct and well-tested.

simapp/asset_usd.py (unchecked node_id lookup in resolve_node_usd_paths) and simapp/thumbnail_capture.py (unchecked background asset access on line 53) need attention before merge.

Important Files Changed

Filename	Overview
isaaclab_arena_examples/agentic_environment_generation/review_gui/simapp/asset_usd.py	Refactored from spec-based to live-asset-based resolution; introduces a KeyError on missing node IDs in resolve_node_usd_paths and uses assert instead of a proper exception in absolute_prim_path.
isaaclab_arena_examples/agentic_environment_generation/review_gui/simapp/thumbnail_capture.py	Major rework adding _UsdSnapshotJob batching and object_reference snapshot support; direct dict access for background asset on line 53 raises KeyError if instantiation fails, and imports a private function across module boundaries.
isaaclab_arena_examples/agentic_environment_generation/review_gui/spec_visualization/prim_tree_view.py	New file; renders a collapsible, searchable HTML prim tree. Properly escapes node text with html.escape; recursive _render_nodes is straightforward and unlikely to hit depth limits in practice.
isaaclab_arena_examples/agentic_environment_generation/review_gui/visualization_service.py	Adds resolve_background_prim_tree with graceful try/except fallback; changes return type of build_asset_cards_with_thumbnails to a tuple and updates cache to include prim tree. Logic is clean and handles missing background correctly.
isaaclab_arena_examples/agentic_environment_generation/review_gui/visualization_panel.py	Correctly unpacks the new tuple return from build_asset_cards_with_thumbnails and stores prim_tree in session state. Session state reads fall back to [] when prim_tree is absent.
isaaclab_arena_examples/agentic_environment_generation/review_gui/spec_visualization/asset_cards.py	Renames asset field to spec and adds ObjectReferenceSpec support; now includes object references in cards; background null-guard added. Clean changes with matching test updates.
isaaclab_arena/assets/background_library.py	Adds get_viewer_cfg to LightwheelKitchenBackground returning a fixed camera position; straightforward, well-commented addition.
isaaclab_arena/environment_spec/arena_env_graph_conversion_utils.py	Moves two imports inside build_arena_env_from_graph_spec to break a circular import; no logic changes.
isaaclab_arena_examples/tests/test_review_gui.py	Tests updated to reflect renamed asset → spec field and inverted expectation for object references; adds new TestPrimTreeView and TestBackgroundPrimTree test classes with good coverage.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant UI as visualization_panel
    participant SVC as visualization_service
    participant CAP as thumbnail_capture (SimApp)
    participant KIT as Kit Viewport

    UI->>SVC: build_asset_cards_with_thumbnails(spec)
    SVC->>SVC: resolve_background_prim_tree(spec)
    SVC->>CAP: client.render_spec(spec)
    CAP->>CAP: _instantiate_assets_from_spec(spec, registry)
    CAP->>CAP: resolve_node_usd_paths(assets, node_ids)
    loop Per USD stage (_UsdSnapshotJob)
        CAP->>KIT: open_stage / wait_for_stage_load
        alt asset_captures present
            CAP->>KIT: _apply_viewer_cfg OR frame_viewport_prims
            KIT-->>CAP: capture_viewport_png → PNG bytes
        end
        alt ref_captures present
            CAP->>KIT: set_selected_prim_paths + frame_viewport_prims
            KIT-->>CAP: capture_viewport_png → PNG bytes
            CAP->>KIT: clear_selected_prim_paths + disable collision viz
        end
    end
    CAP-->>SVC: "{node_id: path} + {node_id: aabb}"
    SVC->>SVC: build_asset_cards(spec, thumbnails, aabb)
    SVC-->>UI: (asset_cards, prim_tree)
    UI->>UI: render_visualization_widgets(spec, cards, prim_tree)

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant UI as visualization_panel
    participant SVC as visualization_service
    participant CAP as thumbnail_capture (SimApp)
    participant KIT as Kit Viewport

    UI->>SVC: build_asset_cards_with_thumbnails(spec)
    SVC->>SVC: resolve_background_prim_tree(spec)
    SVC->>CAP: client.render_spec(spec)
    CAP->>CAP: _instantiate_assets_from_spec(spec, registry)
    CAP->>CAP: resolve_node_usd_paths(assets, node_ids)
    loop Per USD stage (_UsdSnapshotJob)
        CAP->>KIT: open_stage / wait_for_stage_load
        alt asset_captures present
            CAP->>KIT: _apply_viewer_cfg OR frame_viewport_prims
            KIT-->>CAP: capture_viewport_png → PNG bytes
        end
        alt ref_captures present
            CAP->>KIT: set_selected_prim_paths + frame_viewport_prims
            KIT-->>CAP: capture_viewport_png → PNG bytes
            CAP->>KIT: clear_selected_prim_paths + disable collision viz
        end
    end
    CAP-->>SVC: "{node_id: path} + {node_id: aabb}"
    SVC->>SVC: build_asset_cards(spec, thumbnails, aabb)
    SVC-->>UI: (asset_cards, prim_tree)
    UI->>UI: render_visualization_widgets(spec, cards, prim_tree)

Loading

_{Reviews (1): Last reviewed commit: "Trim kitchen viewer and gui_runner comme..." | Re-trigger Greptile}