Skip to content

feat(examples): migrate LIBERO Isaac examples + shared harness from robots-sim#1276

Open
yinsong1986 wants to merge 2 commits into
strands-labs:mainfrom
yinsong1986:feat/libero-isaac-examples
Open

feat(examples): migrate LIBERO Isaac examples + shared harness from robots-sim#1276
yinsong1986 wants to merge 2 commits into
strands-labs:mainfrom
yinsong1986:feat/libero-isaac-examples

Conversation

@yinsong1986

Copy link
Copy Markdown
Contributor

What

Migrates the Isaac-specific LIBERO examples and their shared harness from
strands-labs/robots-sim/examples/libero/ into this repo's
examples/libero/, now that the Isaac backend is vendored in-tree (epic #1144).

New files under examples/libero/:

File Role
run_isaac.py LIBERO eval on the Isaac backend via create_simulation("isaac"); two grep-stable output lines + synchronous rollout-MP4 recorder
run_isaac_agent.py Same eval driven by a Strands Agent (single @tool wrapper)
libero_backend_matrix.py Subprocess-and-parse side-by-side backend-matrix table
gr00t_server_deterministic_wrapper.py Container-side strict-determinism wrapper for run_gr00t_server (imports gr00t.*/torch/tyro; runs inside the GR00T container, not on the host)
README.md Deps (via extras), run commands, env vars

Why

Follow-up to backend-absorb epic #1144 / migration epic #1269. With the
Isaac backend in-tree, create_simulation("isaac", ...) resolves to
strands_robots.simulation.isaac.IsaacSimulation, so these examples no
longer need the external strands-robots-sim package.

Key changes vs the robots-sim originals

  • Rewrote imports: from strands_robots_sim.isaac import IsaacSimulation
    -> from strands_robots.simulation.isaac import IsaacSimulation in both
    run files. Updated stale strands_robots_sim/... module-path references
    in docstrings/comments and the strands-robots-sim[isaac] install hints
    to strands-robots[sim-isaac,benchmark-libero].
  • Repo conventions (epic [EPIC] Migrate robots-sim examples into this repo #1269): multi-file example in a named subdir
    with its own README.md; deps documented via extras (no per-dir
    requirements.txt); indexed the three host-side scripts in
    examples/README.md; documented the new STRANDS_ISAAC_RTX_PATHTRACING
    env var.
  • Lint: applied this repo's ruff rules (UP037 quoted-annotation removal
    • format) to the migrated files.

Test surface

  • create_simulation("isaac", headless=True, num_envs=1, render_mode="rtx_realtime")
    constructs the in-tree IsaacSimulation and destroys cleanly on this
    non-Isaac (no-GPU) host.
  • IsaacSimulation.is_available() short-circuits with the install-hint
    reason string (no omni.*/isaacsim.* import needed) — safe to
    import/lint on CPU-only CI.
  • run_isaac._suite_for_task doctests pass.
  • hatch run lint: clean (Success: no issues found in 755 source files).
  • hatch run test: 8077 passed, 45 skipped. The only failures are the
    pre-existing OpenGL/rendering environmental ones
    (test_sim_camera_mesh_publish, test_lerobot_e2e::test_record_train_load_loop,
    test_rendering.py/test_policy_runner.py) — confirmed identical on clean
    upstream/main (mjr_makeContext / "an OpenGL platform library has not
    been loaded"). This PR touches only examples/, so it cannot affect
    library tests.

Acceptance criteria (from #1273)

  • Migrate libero/run_isaac.py, run_isaac_agent.py as runnable (no skip marker — feat: dataset format bridge — LeRobot recording to GR00T training format #140 RTX fix is upstream)
  • Migrate shared deps libero/libero_backend_matrix.py, gr00t_server_deterministic_wrapper.py
  • Do NOT migrate run_mujoco.py/run_mujoco_agent.py
  • Update strands_robots_sim imports -> strands_robots
  • Confirm create_simulation("isaac") path works now that the backend is in-tree
  • Place scripts under examples/libero/, no NN_*.py renumbering
  • No per-dir requirements.txt; deps documented via pip install "strands-robots[sim-isaac,...]"
  • Add rows to examples/README.md

Out-of-scope follow-ups

Parent: #1269. Board: this issue should move to In review on
Strands Labs - Robots.

…obots-sim (closes strands-labs#1273)

Port the Isaac-specific LIBERO examples and their shared harness from
strands-labs/robots-sim into examples/libero/, now that the Isaac
backend is vendored in-tree (create_simulation("isaac") ->
strands_robots.simulation.isaac.IsaacSimulation):

- run_isaac.py            LIBERO eval on the Isaac backend (grep-stable
                          output lines, synchronous rollout-MP4 recorder)
- run_isaac_agent.py      same eval driven by a Strands Agent (single
                          @tool wrapper)
- libero_backend_matrix.py  subprocess-and-parse side-by-side backend
                          matrix table
- gr00t_server_deterministic_wrapper.py  container-side strict-determinism
                          wrapper for run_gr00t_server

Changes vs the robots-sim originals:
- Rewrote `from strands_robots_sim.isaac import IsaacSimulation` to
  `from strands_robots.simulation.isaac import IsaacSimulation` in both
  run files; updated stale strands_robots_sim/ module-path references and
  `strands-robots-sim[isaac]` install hints to
  `strands-robots[sim-isaac,benchmark-libero]`.
- Per epic strands-labs#1269 conventions: multi-file example lives in a named subdir
  with its own README.md (deps via extras, no per-dir requirements.txt);
  indexed the three host-side scripts in examples/README.md; documented
  the STRANDS_ISAAC_RTX_PATHTRACING env var.
- Applied this repo's ruff rules (UP037 quoted-annotation removal, format).

Per the issue, run_mujoco.py/run_mujoco_agent.py are NOT migrated (handled
under a separate epic-strands-labs#1269 child); the matrix script prints
`unavailable (no <driver>.py)` rows for the not-yet-migrated mujoco and
isaac-4096 drivers and evaluates the isaac-1 (single-env) row. strands-labs#140's RTX
warm-up fix is upstream, so these migrate as runnable with no skip marker.

Verified create_simulation("isaac") resolves the in-tree backend and
IsaacSimulation.is_available() short-circuits cleanly on this non-Isaac
(no-GPU) host. `hatch run lint` clean; `hatch run test` 8077 passed with
only the pre-existing OpenGL/rendering environmental failures.
@yinsong1986
yinsong1986 requested a review from cagataycali July 11, 2026 09:13
Comment thread examples/libero/libero_backend_matrix.py Fixed
main()
except SystemExit:
raise
except BaseException: # noqa: BLE001 - top-level: log + force non-zero exit
Comment thread examples/libero/run_isaac_agent.py Fixed
main()
except SystemExit:
raise
except BaseException: # noqa: BLE001 - top-level: log + force non-zero exit
@cagataycali

Copy link
Copy Markdown
Member

Review — LIBERO Isaac examples + shared harness

Reviewed against a local checkout. Independent of the other examples; lands last in the roll-up ordering.

Verified

  • No strands_robots_sim refs, no per-example dep files, all three scripts (run_isaac.py, run_isaac_agent.py, libero_backend_matrix.py) indexed in examples/README.md, no host paths, no emojis in runtime strings.

CodeQL triage

  • libero_backend_matrix.py:129Unused global _RE_BENCHMARK: real dead code. The comment says it guards against print-format drift, but _RE_RESULT is the one actually parsed. Either wire _RE_BENCHMARK into the parse (so a missing benchmark_name= line is detected) or drop it. One-liner.
  • run_isaac.py:888 and run_isaac_agent.py:871Except handles BaseException: defensible, keep as-is. Both are top-level if __name__ == "__main__" crash handlers that re-raise SystemExit first and os._exit(1) to bypass Isaac Sim's atexit hook (which otherwise swallows non-zero status into a misleading exit 0). They carry # noqa: BLE001 and an explanatory comment — this is the correct pattern for the SimulationApp fast-shutdown problem.
  • run_isaac_agent.py:849Empty except: this is except Exception: pass around _sim.destroy() in a finally teardown. Recommend narrowing to the concrete destroy error or adding a one-line logger.debug(...) + comment so a real teardown failure isn't silently swallowed (project convention: clean up on partial failure, but don't bury the reason).

No blockers. The shared-harness + backend-matrix approach (one LIBERO task across every installed backend with a side-by-side success_rate / wall_time table) is a genuinely useful addition. Thanks Yin.

cagataycali's CodeQL triage on the LIBERO Isaac examples flagged two
items. (1) libero_backend_matrix.py:129 - _RE_BENCHMARK was unused dead
code despite a comment claiming it guards print-format drift; wired it
into _parse_driver_output so a driver that exits 0 without a
benchmark_name= announce line is surfaced as 'ok (... output format
drift?)' in the row status instead of masquerading as a silent early
exit. (2) run_isaac_agent.py:849 - narrowed the empty
'except Exception: pass' around _sim.destroy() in finally teardown to
the concrete (RuntimeError, OSError, AttributeError) and added a
logger.debug + comment so a real teardown failure is logged rather than
silently swallowed.
@yinsong1986

Copy link
Copy Markdown
Contributor Author

Thanks for the CodeQL triage @cagataycali. Addressed the two actionable items in db4480f:

  • libero_backend_matrix.py:129 (unused _RE_BENCHMARK) — wired it into _parse_driver_output. The parser now checks the benchmark_name= announce line: if a driver exits 0 with neither the metrics line nor the announce line, that's genuine output-format drift and the row status reads ok (no benchmark_name= line (output format drift?)) instead of silently looking like an early exit. A driver that announced its benchmark but exited before the metrics line still gets the benign -- cells. Updated the comment block to match the new semantics.
  • run_isaac_agent.py:849 (empty except) — narrowed except Exception: pass around _sim.destroy() in the finally teardown to the concrete (RuntimeError, OSError, AttributeError) (matching what IsaacSimulation.destroy()'s own internal cleanup catches) and added a logger.debug(...) + comment so a real teardown failure is logged rather than buried, while keeping teardown best-effort so it never masks the eval result.

Left the two top-level BaseException crash handlers (run_isaac.py:888 / run_isaac_agent.py:871) as-is per your note.

hatch run lint is clean. hatch run test hits a pre-existing native-extension segfault on this box that reproduces on the unmodified branch too (the changes only touch examples/ scripts, which the test suite does not import).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-sim migration Isaac backend migration

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

3 participants