Skip to content
208 changes: 208 additions & 0 deletions docs/backends/isaac.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
# Isaac Sim Backend

GPU-native photorealistic simulation backend for `strands-robots-sim` using NVIDIA Isaac Sim.

## Overview

`IsaacSimulation` provides:

- **Photorealistic rendering** via Omniverse RTX (path-traced, ground-truth depth, semantic segmentation)
- **Asset pipeline**: USD-native scenes, NVIDIA Nucleus assets
- **Sensors**: RTX cameras, LiDAR, depth, contact, IMU (GPU-batched)
- **Synthetic data generation**: Replicator pipeline for domain randomization
- **Fleet replication**: parallel environments via `omni.isaac.cloner.Cloner`
- **Isaac Lab integration**: GPU-accelerated RL environments


> **Phase 1 status (skeleton).** This release ships the SimEngine-shaped surface and lazy-import scaffolding only. Several methods on `IsaacSimulation` (`add_robot` on the procedural branch, `_load_usd_robot`, `_load_urdf_robot`, `add_object`, `add_camera`, `replicate`) currently return `status: "success"` without instantiating the underlying USD prim or articulation handle. Following the documented Quick Start on a real Isaac Sim install will therefore observe `get_observation()` returning `{}` and `render()` returning blank frames -- no exception is raised. The data-plane wiring (USD stage management, articulation construction, sensor / replicator integration) lands in Phase 2 and later. Treat the Phase-1 surface as an integration contract, not as a working physics path.

## Installation

Isaac Sim is **not installable from PyPI**. It is an NVIDIA Omniverse Kit application that must be installed separately.

### Option 1: NVIDIA Omniverse Launcher (recommended)

1. Download [NVIDIA Omniverse Launcher](https://developer.nvidia.com/omniverse)
2. Install **Isaac Sim 2024.x** (or newer) from the Exchange tab
3. Install Python dependencies:

```bash
pip install 'strands-robots-sim[isaac]'
```

### Option 2: Isaac Lab

```bash
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab
./isaaclab.sh -i
pip install 'strands-robots-sim[isaac]'
```

### Option 3: Docker

```bash
docker pull nvcr.io/nvidia/isaac-sim:4.5.0
docker run --gpus all -it nvcr.io/nvidia/isaac-sim:4.5.0
# Inside container:
pip install 'strands-robots-sim[isaac]'
```

## Requirements

- NVIDIA GPU (RTX 2070+ or A100/H100 for fleet training)
- CUDA 12.0+
- Isaac Sim 2024.x or newer
- Linux (Ubuntu 22.04+ recommended)
- Python 3.10+

## Quick Start

```python
from strands_robots_sim.isaac import IsaacSimulation, IsaacConfig

# Check availability
available, reason = IsaacSimulation.is_available()
if not available:
print(f"Isaac Sim not available: {reason}")
exit(1)

# Create simulation
config = IsaacConfig(
num_envs=1,
headless=True,
render_mode="rtx_realtime",
)
sim = IsaacSimulation(config)

# Create world and add robot
sim.create_world()
sim.add_robot("so100")
sim.add_camera("front_cam", position=[1.0, 0.0, 0.5])

# Step and render
sim.step(100)
result = sim.render("front_cam")
rgb = result["rgb"] # (H, W, 3) uint8

# Clean up
sim.destroy()
```

## Fleet Training (Multi-Env)

```python
config = IsaacConfig(num_envs=1024, headless=True)
sim = IsaacSimulation(config)
sim.create_world()
sim.add_robot("so100")
sim.replicate(1024) # 1024 parallel environments

for step in range(10000):
sim.step(1)
obs = sim.get_observation("so100") # batched across all envs
# ... RL training loop ...

sim.destroy()
```

## Entry-Point Discovery

Isaac Sim registers as a `strands_robots.backends` entry point:

```python
from importlib.metadata import entry_points

for ep in entry_points(group='strands_robots.backends'):
print(ep.name, '->', ep.value)
# isaac -> strands_robots_sim.isaac.simulation:IsaacSimulation
# isaac_sim -> strands_robots_sim.isaac.simulation:IsaacSimulation
```

## Configuration

### `IsaacConfig` Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `num_envs` | 1 | Number of parallel environments |
| `device` | "cuda:0" | CUDA device |
| `headless` | True | Run without GUI |
| `physics_dt` | 1/120 s | Physics timestep |
| `rendering_dt` | 1/30 s | Rendering timestep |
| `render_mode` | "headless" | "headless", "rtx_realtime", or "rtx_pathtracing" |
| `gravity` | (0, 0, -9.81) | Gravity vector (Z-up) |
| `camera_width` | 640 | Default camera width |
| `camera_height` | 480 | Default camera height |
| `enable_rtx_sensors` | True | Enable RTX-accelerated sensors |

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `STRANDS_ISAAC_HEADLESS` | - | Override headless mode ("true"/"false") |
| `STRANDS_ISAAC_RTX_PATHTRACING` | - | Enable RTX pathtracing ("true"/"false") |
| `STRANDS_ISAAC_NUCLEUS_URL` | - | Override Omniverse Nucleus server URL |

## Procedural Robots

The following robots can be added without any asset files:

- `so100` (aliases: `so-100`, `so_100`, `so101`) -- 6-DOF tabletop arm
- `panda` (aliases: `franka`, `franka_panda`) -- 7-DOF manipulator
- `unitree_g1` (aliases: `g1`) -- 21-DOF humanoid (simplified)

```python
sim.add_robot("so100") # procedural, no asset files needed
sim.add_robot("panda")
sim.add_robot("g1", data_config="unitree_g1")
```

## Comparison with Newton Backend

| Feature | Newton (Warp) | Isaac Sim |
|---------|:---:|:---:|
| Physics parallelism | 4096+ envs | 1024 envs |
| Rendering | OpenGL/null | RTX photorealistic |
| USD native | Partial | Full |
| Sensors (camera, LiDAR) | Basic | RTX GPU-batched |
| Synthetic data gen | No | Replicator |
| Soft body/cloth | Yes (VBD) | Yes (PhysX) |
| Differentiable sim | Yes (Warp tape) | No |
| Install size | ~500MB | ~30GB |
| Use case | Fast RL training | Photorealistic sim2real |

## Architecture

```
strands_robots_sim/isaac/
__init__.py PEP 562 lazy exports (zero omni overhead on import)
config.py IsaacConfig dataclass
simulation.py IsaacSimulation(SimEngine) -- main backend class
procedural.py SO-100 / Panda / G1 USD prim builders
stages.py USD stage management (Phase 2)
sensors.py RTX camera, LiDAR wrappers (Phase 3)
replicator.py Domain randomization (Phase 3)
tests/
test_unit.py Mocked tests (no GPU)
test_entrypoint.py Entry-point verification
test_gpu_integ.py GPU tests (STRANDS_GPU_TEST=1)
```

## Thread Safety

- All mutable state protected by `threading.RLock`
- `step()` must not run concurrently with `add_robot()`
- `SimulationApp` is a process-wide singleton (never create more than one)
- `destroy()` clears the World but does NOT shut down `SimulationApp`

## Testing

```bash
# Unit tests (no GPU required)
pytest strands_robots_sim/isaac/tests/test_unit.py -v
pytest strands_robots_sim/isaac/tests/test_entrypoint.py -v

# GPU integration tests (requires Isaac Sim)
STRANDS_GPU_TEST=1 pytest strands_robots_sim/isaac/tests/test_gpu_integ.py -v
```
37 changes: 31 additions & 6 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,31 @@ dev = [
"black",
"isort",
"flake8",
"pytest>=7.0",
]
# Isaac Sim is NOT installable from PyPI -- it's an Omniverse Kit app.
# The user must install it separately:
# - via NVIDIA Omniverse Launcher (Isaac Sim 2024.x), OR
# - via Isaac Lab: git clone IsaacLab && ./isaaclab.sh -i, OR
# - via Docker: nvcr.io/nvidia/isaac-sim:4.5.0
# We declare lightweight deps that Isaac ships and that ARE pip-installable:
isaac = [
"usd-core>=24.5,<26.0",
"warp-lang>=1.13.0",
]
# NOTE: the lightweight `sim` extra (libero / robosuite / mujoco / gymnasium)
# was removed in 0.2.0 — that backend now lives in `strands-robots` and is
# installed via `pip install 'strands-robots[sim-mujoco]'`. Heavy GPU-only
# backends (Isaac Sim, Newton) will land here behind `[isaac]` / `[newton]`
# extras in upcoming releases. See examples/MIGRATION.md and
# https://github.com/strands-labs/robots-sim/issues/8.
all = [
"strands-robots-sim[isaac]",
]

[project.entry-points."strands_robots.backends"]
isaac = "strands_robots_sim.isaac.simulation:IsaacSimulation"
isaac_sim = "strands_robots_sim.isaac.simulation:IsaacSimulation"

[project.urls]
Homepage = "https://github.com/strands-labs/robots-sim"
Expand All @@ -65,18 +83,20 @@ packages = ["strands_robots_sim"]

[tool.hatch.envs.default]
dependencies = [
"pytest>=7.0",
"black",
"isort",
"flake8",
]

[tool.hatch.envs.default.scripts]
# No tests until backend code lands in R7 (Isaac) / R11 (Newton). For now
# `hatch run test` is an import smoke that exercises the no-op stub.
test = "python -c \"import strands_robots_sim; print('strands-robots-sim', strands_robots_sim.__version__, 'OK')\""
# Lint extends to `examples/` so the runnable example files (which copy-
# paste into PR docstrings + the README matrix) catch the same drift the
# package source does. Keeps each new R8/R12/R23 backend example honest.
# Lint/format scope extends to `examples/` so the runnable example files
# (which copy-paste into PR docstrings + the README matrix) catch the same
# drift the package source does. Keeps each new R8/R12/R23 backend example
# honest. The `test` script targets isaac/tests/ now that R7 lands the
# first backend with real pytest coverage; superseded the prior import-
# smoke placeholder.
test = "pytest strands_robots_sim/isaac/tests/ -v --ignore=strands_robots_sim/isaac/tests/test_gpu_integ.py"
lint = [
"black --check strands_robots_sim examples",
"isort --check-only strands_robots_sim examples",
Expand All @@ -96,3 +116,8 @@ include = '\.pyi?$'
profile = "black"
line_length = 120
multi_line_output = 3

[tool.pytest.ini_options]
markers = [
"gpu: tests requiring NVIDIA GPU + Isaac Sim (gated on STRANDS_GPU_TEST=1)",
]
43 changes: 43 additions & 0 deletions strands_robots_sim/isaac/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
"""strands_robots_sim.isaac -- GPU-native Isaac Sim simulation backend.

This subpackage provides :class:`IsaacSimulation`, a ``SimEngine`` backend
built on **NVIDIA Isaac Sim / Omniverse** for photorealistic rendering,
synthetic data generation, and GPU-batched sensor simulation.

Usage::

from strands_robots_sim.isaac import IsaacSimulation, IsaacConfig
config = IsaacConfig(num_envs=1, headless=True)
sim = IsaacSimulation(config)
ok, msg = IsaacSimulation.is_available()

Requires NVIDIA Isaac Sim 2024.x+ (not pip-installable).
Install via Omniverse Launcher or ``nvcr.io/nvidia/isaac-sim:4.5.0``.
"""

from __future__ import annotations

__all__ = ["IsaacSimulation", "IsaacConfig"]


def _lazy_isaac_simulation():
"""Lazy import to avoid pulling omni/Isaac at module-import time."""
from strands_robots_sim.isaac.simulation import IsaacSimulation

return IsaacSimulation


def _lazy_isaac_config():
"""Lazy import to avoid pulling dataclass internals at import time."""
from strands_robots_sim.isaac.config import IsaacConfig

return IsaacConfig


def __getattr__(name: str):
"""PEP 562 lazy attribute access."""
if name == "IsaacSimulation":
return _lazy_isaac_simulation()
if name == "IsaacConfig":
return _lazy_isaac_config()
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
Loading
Loading