|
| 1 | +.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. |
| 2 | +.. SPDX-License-Identifier: Apache-2.0 |
| 3 | +
|
| 4 | +Data Collection in Real |
| 5 | +======================= |
| 6 | + |
| 7 | +Record demonstrations on a physical SO-101 into a `LeRobot dataset |
| 8 | +<https://huggingface.co/docs/lerobot/en/index>`_, driving the follower with either teleop device |
| 9 | +(see :doc:`devices`). The example scripts live in ``examples/isaac_teleop_to_so101/`` in the |
| 10 | +`LeRobot <https://github.com/huggingface/lerobot>`_ repository: ``teleoperate.py`` drives the arm |
| 11 | +live, and ``record.py`` does the same while saving a dataset. Both take the same |
| 12 | +``--robot.*`` / ``--teleop.*`` flags; ``--teleop.type`` selects the device |
| 13 | +(``xr_controller`` | ``so101_leader``). |
| 14 | + |
| 15 | +Before you start |
| 16 | +---------------- |
| 17 | + |
| 18 | +#. A working **SO-101 follower** — assembled, motors set up, and calibrated. See |
| 19 | + `SO-101 support in LeRobot`_. |
| 20 | + |
| 21 | +#. The **isaac-teleop extra** installed (``isaacteleop`` ships on public PyPI; its |
| 22 | + ``[cloudxr,retargeters]`` extras pull the CloudXR runtime bindings and the retargeter library): |
| 23 | + |
| 24 | + .. code-block:: bash |
| 25 | +
|
| 26 | + uv pip install -e '.[isaac-teleop]' |
| 27 | +
|
| 28 | +#. Run the scripts from the example directory, and log in to the Hugging Face Hub — recorded |
| 29 | + datasets are pushed to the Hub by default (pass ``--dataset.push_to_hub=false`` to keep them |
| 30 | + local): |
| 31 | + |
| 32 | + .. code-block:: bash |
| 33 | +
|
| 34 | + cd examples/isaac_teleop_to_so101 |
| 35 | + huggingface-cli login |
| 36 | +
|
| 37 | +Then follow the steps for your teleop device: |
| 38 | + |
| 39 | +.. tab-set:: |
| 40 | + |
| 41 | + .. tab-item:: XR controller |
| 42 | + |
| 43 | + The controller pose drives the follower's end-effector through the clutch + IK pipeline, |
| 44 | + streamed over CloudXR. |
| 45 | + |
| 46 | + #. **Fetch the robot model.** The XR path solves inverse kinematics, so it needs the SO-101 |
| 47 | + URDF and meshes (downloaded into ``./SO101/``): |
| 48 | + |
| 49 | + .. code-block:: bash |
| 50 | +
|
| 51 | + python download_assets.py |
| 52 | +
|
| 53 | + #. **Connect a headset.** Bring up CloudXR and connect your XR headset — follow the |
| 54 | + :doc:`/getting_started/quick_start`. |
| 55 | + |
| 56 | + #. **(Optional) Try teleoperation without recording.** A good way to check the setup before |
| 57 | + committing to a dataset: |
| 58 | + |
| 59 | + .. code-block:: bash |
| 60 | +
|
| 61 | + python teleoperate.py \ |
| 62 | + --robot.type=so101_follower \ |
| 63 | + --robot.port=/dev/ttyACM0 \ |
| 64 | + --robot.id=so101_follower_arm \ |
| 65 | + --teleop.type=xr_controller |
| 66 | +
|
| 67 | + Squeeze and hold the grip to engage the clutch and move the arm; the trigger controls the |
| 68 | + gripper. Release the grip to pause. |
| 69 | + |
| 70 | + #. **Record a dataset.** Add cameras and the dataset parameters: |
| 71 | + |
| 72 | + .. code-block:: bash |
| 73 | +
|
| 74 | + python record.py \ |
| 75 | + --robot.type=so101_follower \ |
| 76 | + --robot.port=/dev/ttyACM0 \ |
| 77 | + --robot.id=so101_follower_arm \ |
| 78 | + --teleop.type=xr_controller \ |
| 79 | + --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \ |
| 80 | + --dataset.repo_id=<hf_user>/<dataset_name> \ |
| 81 | + --dataset.single_task="Pick up vial from rack on the left side" \ |
| 82 | + --dataset.num_episodes=3 \ |
| 83 | + --dataset.episode_time_s=20 \ |
| 84 | + --dataset.reset_time_s=5 |
| 85 | +
|
| 86 | + .. note:: |
| 87 | + |
| 88 | + **Customizing the reset pose.** On startup the XR path slews the arm to a built-in default |
| 89 | + reset pose (a comfortable mid-range pose) before handing control to the clutch — you do |
| 90 | + **not** need to record anything. To tailor it to your setup, back-drive the arm to the |
| 91 | + pose you want and run ``python override_reset_pose.py``; it writes ``reset_pose.json`` |
| 92 | + (git-ignored, user-local), which takes priority over the default on the next run. Pass |
| 93 | + ``--reset_to_origin=false`` to skip the slew and keep the arm where it is. |
| 94 | + |
| 95 | + .. tab-item:: SO-101 Leader |
| 96 | + |
| 97 | + A back-drivable SO-101 leader arm mirrored 1:1 to the follower. Its joints are streamed by |
| 98 | + Isaac Teleop's ``so101_leader`` plugin, which the script launches for you. |
| 99 | + |
| 100 | + #. **Build the so101_leader plugin.** It is part of Isaac Teleop's C++ source, not the |
| 101 | + ``isaacteleop`` pip package, so build it from an Isaac Teleop checkout: |
| 102 | + |
| 103 | + .. code-block:: bash |
| 104 | +
|
| 105 | + cmake -B build && cmake --build build --parallel && cmake --install build |
| 106 | +
|
| 107 | + The binary lands at ``install/plugins/so101_leader/so101_leader_plugin``. For details see |
| 108 | + :ref:`so101-leader-plugin` and :doc:`/getting_started/build_from_source/index`. |
| 109 | + |
| 110 | + #. **Calibrate the leader** so the leader and follower agree on each joint's zero and range. |
| 111 | + This reuses the serial SO-101 leader's calibration (stored under ``so_leader/<id>.json`` |
| 112 | + and reused on every run): |
| 113 | + |
| 114 | + .. code-block:: bash |
| 115 | +
|
| 116 | + lerobot-calibrate \ |
| 117 | + --teleop.type=so101_leader \ |
| 118 | + --teleop.port=/dev/ttyACM1 \ |
| 119 | + --teleop.id=so101_leader_arm |
| 120 | +
|
| 121 | + #. **(Optional) Try teleoperation without recording.** ``--launch_plugin`` spawns the plugin |
| 122 | + after CloudXR is up; ``--teleop.port`` is the leader's serial port: |
| 123 | + |
| 124 | + .. code-block:: bash |
| 125 | +
|
| 126 | + python teleoperate.py \ |
| 127 | + --robot.type=so101_follower \ |
| 128 | + --robot.port=/dev/ttyACM0 \ |
| 129 | + --robot.id=so101_follower_arm \ |
| 130 | + --teleop.type=so101_leader \ |
| 131 | + --teleop.port=/dev/ttyACM1 \ |
| 132 | + --teleop.id=so101_leader_arm \ |
| 133 | + --launch_plugin=/path/to/IsaacTeleop/install/plugins/so101_leader/so101_leader_plugin |
| 134 | +
|
| 135 | + Back-drive the leader arm by hand to move the follower. |
| 136 | + |
| 137 | + #. **Record a dataset.** Same flags as teleoperation, plus the cameras and dataset parameters |
| 138 | + (keep ``--launch_plugin`` so the plugin is started): |
| 139 | + |
| 140 | + .. code-block:: bash |
| 141 | +
|
| 142 | + python record.py \ |
| 143 | + --robot.type=so101_follower \ |
| 144 | + --robot.port=/dev/ttyACM0 \ |
| 145 | + --robot.id=so101_follower_arm \ |
| 146 | + --teleop.type=so101_leader \ |
| 147 | + --teleop.port=/dev/ttyACM1 \ |
| 148 | + --teleop.id=so101_leader_arm \ |
| 149 | + --launch_plugin=/path/to/IsaacTeleop/install/plugins/so101_leader/so101_leader_plugin \ |
| 150 | + --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \ |
| 151 | + --dataset.repo_id=<hf_user>/<dataset_name> \ |
| 152 | + --dataset.single_task="Pick up vial from rack on the left side" \ |
| 153 | + --dataset.num_episodes=3 \ |
| 154 | + --dataset.episode_time_s=20 \ |
| 155 | + --dataset.reset_time_s=5 |
| 156 | +
|
| 157 | +Recording controls |
| 158 | +------------------ |
| 159 | + |
| 160 | +``record.py`` records ``--dataset.num_episodes`` episodes of ``--dataset.episode_time_s`` seconds |
| 161 | +each, with a ``--dataset.reset_time_s`` window between episodes to reposition the scene. While |
| 162 | +it is running, **press** these keys in the terminal where ``record.py`` is running — the example |
| 163 | +reads them from that terminal, so they work over SSH and in a plain terminal (Linux/macOS), with |
| 164 | +no desktop session required: |
| 165 | + |
| 166 | +.. list-table:: |
| 167 | + :header-rows: 1 |
| 168 | + :widths: 22 78 |
| 169 | + |
| 170 | + * - Key |
| 171 | + - Action |
| 172 | + * - Right arrow → |
| 173 | + - End the current episode early and save it. |
| 174 | + * - Left arrow ← |
| 175 | + - Discard the current take and re-record it. |
| 176 | + * - Escape |
| 177 | + - Stop after the current episode (already-saved episodes are kept). |
| 178 | + |
| 179 | +Set ``LEROBOT_KEYBOARD_BACKEND`` to override how keys are read — ``auto`` (the default; uses the |
| 180 | +terminal when one is attached, otherwise a global listener), ``stdin``, ``pynput``, or ``none``. |
| 181 | +The dataset is written under ``$HF_LEROBOT_HOME/<repo_id>`` and pushed to the Hub when recording |
| 182 | +finishes (unless ``--dataset.push_to_hub=false``). Next, train a policy on it: |
| 183 | +:doc:`training_groot`. |
| 184 | + |
| 185 | +.. _SO-101 support in LeRobot: https://huggingface.co/docs/lerobot/en/so101 |
0 commit comments