Skip to content

Commit 8b75c1a

Browse files
committed
docs: add LeRobot and SO-101 getting-started guide
New multi-page section under docs/source/getting_started/lerobot/ for the end-to-end SO-101 pipeline on Isaac Teleop, LeRobot, and Isaac Lab: - index: overview, end-to-end workflow, and teleop/policy figures - devices: the XR controller and SO-101 Leader teleop devices - data_collection_real: record LeRobot datasets on hardware with either device, via the isaac_teleop_to_so101 example - data_collection_sim: record in Isaac Lab with the XR controller (the SO-101 Leader and LeRobot export are work-in-progress) - training_groot: GR00T N1.7 training (work-in-progress) Also wires the landing page into the Getting Started toctree and adds a so101-leader-plugin ref label in device/joint_space.rst. Builds clean under sphinx-build -W. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
1 parent 41ef6d6 commit 8b75c1a

11 files changed

Lines changed: 544 additions & 0 deletions
Lines changed: 3 additions & 0 deletions
Loading
Lines changed: 3 additions & 0 deletions
Loading
Lines changed: 3 additions & 0 deletions
Loading
Lines changed: 3 additions & 0 deletions
Loading

docs/source/device/joint_space.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,8 @@ The gripper is just another named DOF (conventionally ``"gripper"``). ``velocity
6969
and ``ee_pose`` are optional/reserved: the reference plugin and ``JointStateSource`` populate and
7070
surface joint **positions** only.
7171

72+
.. _so101-leader-plugin:
73+
7274
The SO-101 leader plugin
7375
------------------------
7476

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
.. SPDX-License-Identifier: Apache-2.0
3+
4+
Data Collection in Real
5+
=======================
6+
7+
Record demonstrations on a physical SO-101 into a `LeRobot dataset
8+
<https://huggingface.co/docs/lerobot/en/index>`_, driving the follower with either teleop device
9+
(see :doc:`devices`). The example scripts live in ``examples/isaac_teleop_to_so101/`` in the
10+
`LeRobot <https://github.com/huggingface/lerobot>`_ repository: ``teleoperate.py`` drives the arm
11+
live, and ``record.py`` does the same while saving a dataset. Both take the same
12+
``--robot.*`` / ``--teleop.*`` flags; ``--teleop.type`` selects the device
13+
(``xr_controller`` | ``so101_leader``).
14+
15+
Before you start
16+
----------------
17+
18+
#. A working **SO-101 follower** — assembled, motors set up, and calibrated. See
19+
`SO-101 support in LeRobot`_.
20+
21+
#. The **isaac-teleop extra** installed (``isaacteleop`` ships on public PyPI; its
22+
``[cloudxr,retargeters]`` extras pull the CloudXR runtime bindings and the retargeter library):
23+
24+
.. code-block:: bash
25+
26+
uv pip install -e '.[isaac-teleop]'
27+
28+
#. Run the scripts from the example directory, and log in to the Hugging Face Hub — recorded
29+
datasets are pushed to the Hub by default (pass ``--dataset.push_to_hub=false`` to keep them
30+
local):
31+
32+
.. code-block:: bash
33+
34+
cd examples/isaac_teleop_to_so101
35+
huggingface-cli login
36+
37+
Then follow the steps for your teleop device:
38+
39+
.. tab-set::
40+
41+
.. tab-item:: XR controller
42+
43+
The controller pose drives the follower's end-effector through the clutch + IK pipeline,
44+
streamed over CloudXR.
45+
46+
#. **Fetch the robot model.** The XR path solves inverse kinematics, so it needs the SO-101
47+
URDF and meshes (downloaded into ``./SO101/``):
48+
49+
.. code-block:: bash
50+
51+
python download_assets.py
52+
53+
#. **Connect a headset.** Bring up CloudXR and connect your XR headset — follow the
54+
:doc:`/getting_started/quick_start`.
55+
56+
#. **(Optional) Try teleoperation without recording.** A good way to check the setup before
57+
committing to a dataset:
58+
59+
.. code-block:: bash
60+
61+
python teleoperate.py \
62+
--robot.type=so101_follower \
63+
--robot.port=/dev/ttyACM0 \
64+
--robot.id=so101_follower_arm \
65+
--teleop.type=xr_controller
66+
67+
Squeeze and hold the grip to engage the clutch and move the arm; the trigger controls the
68+
gripper. Release the grip to pause.
69+
70+
#. **Record a dataset.** Add cameras and the dataset parameters:
71+
72+
.. code-block:: bash
73+
74+
python record.py \
75+
--robot.type=so101_follower \
76+
--robot.port=/dev/ttyACM0 \
77+
--robot.id=so101_follower_arm \
78+
--teleop.type=xr_controller \
79+
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
80+
--dataset.repo_id=<hf_user>/<dataset_name> \
81+
--dataset.single_task="Pick up vial from rack on the left side" \
82+
--dataset.num_episodes=3 \
83+
--dataset.episode_time_s=20 \
84+
--dataset.reset_time_s=5
85+
86+
.. note::
87+
88+
**Customizing the reset pose.** On startup the XR path slews the arm to a built-in default
89+
reset pose (a comfortable mid-range pose) before handing control to the clutch — you do
90+
**not** need to record anything. To tailor it to your setup, back-drive the arm to the
91+
pose you want and run ``python override_reset_pose.py``; it writes ``reset_pose.json``
92+
(git-ignored, user-local), which takes priority over the default on the next run. Pass
93+
``--reset_to_origin=false`` to skip the slew and keep the arm where it is.
94+
95+
.. tab-item:: SO-101 Leader
96+
97+
A back-drivable SO-101 leader arm mirrored 1:1 to the follower. Its joints are streamed by
98+
Isaac Teleop's ``so101_leader`` plugin, which the script launches for you.
99+
100+
#. **Build the so101_leader plugin.** It is part of Isaac Teleop's C++ source, not the
101+
``isaacteleop`` pip package, so build it from an Isaac Teleop checkout:
102+
103+
.. code-block:: bash
104+
105+
cmake -B build && cmake --build build --parallel && cmake --install build
106+
107+
The binary lands at ``install/plugins/so101_leader/so101_leader_plugin``. For details see
108+
:ref:`so101-leader-plugin` and :doc:`/getting_started/build_from_source/index`.
109+
110+
#. **Calibrate the leader** so the leader and follower agree on each joint's zero and range.
111+
This reuses the serial SO-101 leader's calibration (stored under ``so_leader/<id>.json``
112+
and reused on every run):
113+
114+
.. code-block:: bash
115+
116+
lerobot-calibrate \
117+
--teleop.type=so101_leader \
118+
--teleop.port=/dev/ttyACM1 \
119+
--teleop.id=so101_leader_arm
120+
121+
#. **(Optional) Try teleoperation without recording.** ``--launch_plugin`` spawns the plugin
122+
after CloudXR is up; ``--teleop.port`` is the leader's serial port:
123+
124+
.. code-block:: bash
125+
126+
python teleoperate.py \
127+
--robot.type=so101_follower \
128+
--robot.port=/dev/ttyACM0 \
129+
--robot.id=so101_follower_arm \
130+
--teleop.type=so101_leader \
131+
--teleop.port=/dev/ttyACM1 \
132+
--teleop.id=so101_leader_arm \
133+
--launch_plugin=/path/to/IsaacTeleop/install/plugins/so101_leader/so101_leader_plugin
134+
135+
Back-drive the leader arm by hand to move the follower.
136+
137+
#. **Record a dataset.** Same flags as teleoperation, plus the cameras and dataset parameters
138+
(keep ``--launch_plugin`` so the plugin is started):
139+
140+
.. code-block:: bash
141+
142+
python record.py \
143+
--robot.type=so101_follower \
144+
--robot.port=/dev/ttyACM0 \
145+
--robot.id=so101_follower_arm \
146+
--teleop.type=so101_leader \
147+
--teleop.port=/dev/ttyACM1 \
148+
--teleop.id=so101_leader_arm \
149+
--launch_plugin=/path/to/IsaacTeleop/install/plugins/so101_leader/so101_leader_plugin \
150+
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
151+
--dataset.repo_id=<hf_user>/<dataset_name> \
152+
--dataset.single_task="Pick up vial from rack on the left side" \
153+
--dataset.num_episodes=3 \
154+
--dataset.episode_time_s=20 \
155+
--dataset.reset_time_s=5
156+
157+
Recording controls
158+
------------------
159+
160+
``record.py`` records ``--dataset.num_episodes`` episodes of ``--dataset.episode_time_s`` seconds
161+
each, with a ``--dataset.reset_time_s`` window between episodes to reposition the scene. While
162+
it is running, **press** these keys in the terminal where ``record.py`` is running — the example
163+
reads them from that terminal, so they work over SSH and in a plain terminal (Linux/macOS), with
164+
no desktop session required:
165+
166+
.. list-table::
167+
:header-rows: 1
168+
:widths: 22 78
169+
170+
* - Key
171+
- Action
172+
* - Right arrow →
173+
- End the current episode early and save it.
174+
* - Left arrow ←
175+
- Discard the current take and re-record it.
176+
* - Escape
177+
- Stop after the current episode (already-saved episodes are kept).
178+
179+
Set ``LEROBOT_KEYBOARD_BACKEND`` to override how keys are read — ``auto`` (the default; uses the
180+
terminal when one is attached, otherwise a global listener), ``stdin``, ``pynput``, or ``none``.
181+
The dataset is written under ``$HF_LEROBOT_HOME/<repo_id>`` and pushed to the Hub when recording
182+
finishes (unless ``--dataset.push_to_hub=false``). Next, train a policy on it:
183+
:doc:`training_groot`.
184+
185+
.. _SO-101 support in LeRobot: https://huggingface.co/docs/lerobot/en/so101
Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
.. SPDX-License-Identifier: Apache-2.0
3+
4+
Data Collection in Sim
5+
======================
6+
7+
Collect SO-101 demonstrations in simulation with `NVIDIA Isaac Lab
8+
<https://isaac-sim.github.io/IsaacLab>`_, on the cube-stacking task. You drive the simulated
9+
follower through Isaac Teleop (see :doc:`devices`) and record episodes to an HDF5 dataset.
10+
11+
Two SO-101 stack tasks are registered in Isaac Lab:
12+
13+
.. list-table::
14+
:header-rows: 1
15+
:widths: 45 55
16+
17+
* - Task id
18+
- Use
19+
* - ``IsaacContrib-Stack-Cube-SO101-IK-Abs-v0``
20+
- Absolute-pose IK + Isaac Teleop teleoperation (use this for data collection).
21+
* - ``IsaacContrib-Stack-Cube-SO101-v0``
22+
- Joint-position control baseline (no teleop).
23+
24+
Before you start
25+
----------------
26+
27+
.. important::
28+
29+
Both steps below are **required** — complete them first. The teleoperation and recording
30+
commands later on will not work until you have.
31+
32+
**Step 1 — Install Isaac Lab.** Follow the `Isaac Lab installation guide`_ to set up the ``Lab``
33+
repository, then run every script through its launcher: ``./isaaclab.sh -p <script> ...`` (or
34+
plain ``python`` inside the activated Isaac Lab environment). The SO-101 USD assets stream from
35+
the NVIDIA Nucleus server, so there is no manual asset download.
36+
37+
**Step 2 — Set up CloudXR and connect a headset.** XR teleoperation needs CloudXR and a headset,
38+
the same as the real flow — follow the :doc:`/getting_started/quick_start` and the
39+
`CloudXR teleoperation in Isaac Lab`_ guide. CloudXR auto-launches by default; pick the profile
40+
with ``--cloudxr_env`` (``cloudxrjs`` for Quest/Pico, ``avp`` for Apple Vision Pro, ``none`` to
41+
disable). No physical headset? Open the CloudXR web client in a desktop browser, which emulates a
42+
headset.
43+
44+
Collect Teleop Data
45+
-------------------
46+
47+
.. tab-set::
48+
49+
.. tab-item:: XR controller
50+
51+
The controller pose drives the simulated follower's end-effector through the clutch + IK
52+
pipeline, streamed over CloudXR — the same controls as on real hardware.
53+
54+
#. **(Optional) Try teleoperation without recording.** A good way to check the setup first:
55+
56+
.. code-block:: bash
57+
58+
./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
59+
--task IsaacContrib-Stack-Cube-SO101-IK-Abs-v0 \
60+
--xr \
61+
--viz kit
62+
63+
``--xr`` enables the XR/CloudXR path and ``--viz kit`` opens the Omniverse Kit viewport.
64+
Squeeze and hold the grip to engage the clutch and move the arm; the trigger controls the
65+
gripper.
66+
67+
#. **Record a dataset.** ``record_demos.py`` runs the same teleoperation while saving
68+
episodes to HDF5. It records ``--num_demos`` demonstrations, marking one successful after
69+
``--num_success_steps`` consecutive success frames:
70+
71+
.. code-block:: bash
72+
73+
./isaaclab.sh -p scripts/tools/record_demos.py \
74+
--task IsaacContrib-Stack-Cube-SO101-IK-Abs-v0 \
75+
--dataset_file ./datasets/so101_stack_demos.hdf5 \
76+
--num_demos 10 \
77+
--step_hz 30 \
78+
--xr \
79+
--viz kit
80+
81+
The demos are written to the ``--dataset_file`` path in HDF5 format.
82+
83+
.. tab-item:: SO-101 Leader
84+
85+
.. admonition:: 🚧 Work in progress
86+
:class: caution
87+
88+
Driving the simulated follower from an **SO-101 Leader** arm is **not yet supported in
89+
Isaac Lab** — Isaac Lab has no joint-space leader device, and the SO-101 stack task is
90+
wired only for the XR controller. Sim leader support is planned; until then, use the
91+
leader arm on the real robot (:doc:`data_collection_real`).
92+
93+
Convert to LeRobot Dataset
94+
--------------------------
95+
96+
.. admonition:: 🚧 Work in progress
97+
:class: caution
98+
99+
**Export to a LeRobot dataset.** Converting these sim HDF5 demos to the
100+
:doc:`LeRobot dataset format <training_groot>` is **not yet provided** for the stack task. The
101+
closest reference is the locomanipulation converter `convert_dataset.py`_ from the ``develop``
102+
branch in Isaac Lab, which targets a different task and must be adapted.
103+
104+
..
105+
References
106+
.. _Isaac Lab installation guide: https://isaac-sim.github.io/IsaacLab/develop/source/setup/installation/index.html#isaaclab-installation-root
107+
.. _CloudXR teleoperation in Isaac Lab: https://isaac-sim.github.io/IsaacLab/develop/source/how-to/cloudxr_teleoperation.html
108+
.. _convert_dataset.py: https://github.com/isaac-sim/IsaacLab/blob/develop/scripts/imitation_learning/locomanipulation_sdg/gr00t/convert_dataset.py

0 commit comments

Comments
 (0)