Skip to content

Commit c0c6518

Browse files
DozgulbasDozgulbas
committed
Working on adding VR teleop on Isaac Sim
1 parent 4309d93 commit c0c6518

2 files changed

Lines changed: 252 additions & 0 deletions

File tree

‎scripts/isaacsim/README.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# VR Teleoperation Setup — OpenArm Isaac Sim
2+
3+
This document describes how to set up VR teleoperation for the OpenArm bimanual robot in Isaac Sim using a Meta Quest 2 on Ubuntu 22.04.
4+
5+
## System Requirements
6+
7+
- Ubuntu 22.04
8+
- NVIDIA GPU (RTX 3060 or better)
9+
- NVIDIA driver 535.129+ (tested with 580.x)
10+
- CUDA 12.x
11+
- Meta Quest 2 with developer mode enabled
12+
- Steam installed
13+
14+
## Overview
15+
16+
The VR pipeline is:
17+
18+
```
19+
Meta Quest 2 → ALVR (WiFi streaming) → SteamVR (OpenXR runtime) → Isaac Sim XR extensions → Robot visualization
20+
```
21+
22+
---
23+
24+
## 1. Enable Developer Mode on Quest 2
25+
26+
1. Install the **Meta app** on your phone
27+
2. Go to **Menu → Devices → [your Quest 2]**
28+
3. Enable **Developer Mode**
29+
4. On the Quest 2, accept the USB debugging prompt when connecting via USB-C
30+
31+
Verify ADB connection:
32+
```bash
33+
sudo apt install android-tools-adb
34+
adb devices
35+
# Should show your device serial number
36+
```
37+
38+
Add udev rules for Quest 2 USB:
39+
```bash
40+
cat << 'EOF' | sudo tee /etc/udev/rules.d/51-oculus.rules
41+
SUBSYSTEM=="usb", ATTR{idVendor}=="2833", ATTR{idProduct}=="5010", MODE="0666", GROUP="plugdev"
42+
SUBSYSTEM=="usb", ATTR{idVendor}=="2833", ATTR{idProduct}=="5011", MODE="0666", GROUP="plugdev"
43+
EOF
44+
sudo udevadm control --reload-rules
45+
sudo udevadm trigger
46+
sudo usermod -aG plugdev $USER
47+
```
48+
49+
---
50+
51+
## 2. Install Steam and SteamVR
52+
53+
```bash
54+
sudo apt install steam -y
55+
```
56+
57+
Launch Steam, log in, then install **SteamVR** from the Steam store (App ID: 250820).
58+
59+
Launch SteamVR once to initialize its config files, then close it:
60+
```bash
61+
steam steam://run/250820
62+
```
63+
64+
Add the following to SteamVR's launch options in Steam (right-click SteamVR → Properties → Launch Options):
65+
```
66+
/home/vision/.steam/debian-installation/steamapps/common/SteamVR/bin/vrmonitor.sh %command%
67+
```
68+
69+
---
70+
71+
## 3. Install ALVR
72+
73+
ALVR streams Quest 2 over WiFi and registers as a SteamVR driver.
74+
75+
### Download ALVR streamer (PC side)
76+
```bash
77+
wget --no-check-certificate \
78+
https://github.com/alvr-org/ALVR/releases/download/v20.14.1/alvr_streamer_linux.tar.gz \
79+
-O ~/Downloads/alvr_streamer.tar.gz
80+
mkdir -p ~/alvr
81+
tar -xzf ~/Downloads/alvr_streamer.tar.gz -C ~/alvr
82+
```
83+
84+
### Install ALVR client on Quest 2 (via ADB)
85+
```bash
86+
wget --no-check-certificate \
87+
https://github.com/alvr-org/ALVR/releases/download/v20.14.1/alvr_client_android.apk \
88+
-O ~/Downloads/alvr_client.apk
89+
adb install ~/Downloads/alvr_client.apk
90+
```
91+
92+
### Configure ALVR
93+
94+
Open the ALVR dashboard:
95+
```bash
96+
chmod +x ~/alvr/alvr_streamer_linux/bin/alvr_dashboard
97+
~/alvr/alvr_streamer_linux/bin/alvr_dashboard
98+
```
99+
100+
In the dashboard:
101+
1. Go to **Installation** tab
102+
2. Click **Register ALVR driver**
103+
3. Click **Set firewall rules**
104+
105+
Also open firewall ports manually:
106+
```bash
107+
sudo ufw allow 9943/udp
108+
sudo ufw allow 9944/udp
109+
sudo ufw allow 9943/tcp
110+
sudo ufw allow 9944/tcp
111+
```
112+
113+
---
114+
115+
## 4. Configure OpenXR Runtime
116+
117+
Set SteamVR as the system OpenXR runtime:
118+
119+
```bash
120+
mkdir -p ~/.config/openxr/1
121+
cat > ~/.config/openxr/1/active_runtime.json << 'EOF'
122+
{
123+
"file_format_version": "1.0.0",
124+
"runtime": {
125+
"library_path": "/home/vision/.steam/debian-installation/steamapps/common/SteamVR/bin/linux64/libopenxr_loader.so",
126+
"name": "SteamVR"
127+
}
128+
}
129+
EOF
130+
```
131+
132+
---
133+
134+
## 5. Launch VR Session
135+
136+
Every time you want to use VR:
137+
138+
### Step 1 — Start ALVR dashboard
139+
```bash
140+
~/alvr/alvr_streamer_linux/bin/alvr_dashboard &
141+
```
142+
143+
### Step 2 — Start SteamVR
144+
Launch SteamVR from Steam GUI or:
145+
```bash
146+
steam steam://run/250820
147+
```
148+
149+
### Step 3 — Connect Quest 2
150+
- On the Quest 2, open **App Library → Unknown Sources → ALVR**
151+
- Launch ALVR on the headset
152+
- In the ALVR dashboard on theia, click **Trust** next to the device entry
153+
154+
### Step 4 — Launch Isaac Sim in VR mode
155+
```bash
156+
source ~/workspace/simlab/activate-isaacsim.sh
157+
isaacsim --experience ~/workspace/simlab/.venv-isaacsim/lib/python3.11/site-packages/isaacsim/apps/isaacsim.exp.base.xr.vr.kit
158+
```
159+
160+
### Step 5 — Enable XR extensions in Isaac Sim
161+
In Isaac Sim:
162+
1. Go to **Window → Extensions**
163+
2. Search for `xr`
164+
3. Enable:
165+
- `omni.kit.xr.profile.vr`
166+
- `omni.kit.xr.core`
167+
- `omni.kit.xr.system.openxr`
168+
169+
### Step 6 — Load OpenArm robot
170+
Open the bimanual USD:
171+
```
172+
~/humanoids/openarm_isaac_lab/source/openarm/openarm/tasks/manager_based/openarm_manipulation/usds/openarm_bimanual/openarm_bimanual.usd
173+
```
174+
175+
Or run the sim script:
176+
```bash
177+
source ~/workspace/simlab/activate-isaacsim.sh
178+
python ~/humanoids/openarm_module/scripts/isaacsim/openarm_sim.py
179+
```
180+
181+
---
182+
183+
## File Locations
184+
185+
| Item | Path |
186+
|------|------|
187+
| ALVR streamer | `~/alvr/alvr_streamer_linux/` |
188+
| ALVR client APK | `~/Downloads/alvr_client.apk` |
189+
| OpenXR runtime config | `~/.config/openxr/1/active_runtime.json` |
190+
| Isaac Sim VR kit | `~/.venv-isaacsim/lib/python3.11/site-packages/isaacsim/apps/isaacsim.exp.base.xr.vr.kit` |
191+
| OpenArm bimanual USD | `~/humanoids/openarm_isaac_lab/source/openarm/.../openarm_bimanual.usd` |
192+
| Sim scripts | `~/humanoids/openarm_module/scripts/isaacsim/` |
193+
194+
---
195+
196+
## Troubleshooting
197+
198+
**Quest 2 not detected by ADB**
199+
- Unplug and replug USB cable
200+
- Put on headset and accept the "Allow USB debugging" prompt
201+
- Run `adb kill-server && adb devices`
202+
203+
**ALVR connection timeout**
204+
- Ensure Quest 2 and theia are on the same WiFi network
205+
- Re-run firewall rules in ALVR dashboard
206+
- Check `sudo ufw status`
207+
208+
**SteamVR not starting properly**
209+
- Make sure launch option is set: `vrmonitor.sh %command%`
210+
- Launch SteamVR from Steam GUI first to initialize config
211+
212+
**OpenXR not found by Isaac Sim**
213+
- Verify `~/.config/openxr/1/active_runtime.json` exists
214+
- Make sure SteamVR is running before launching Isaac Sim
215+
216+
**rclpy not loading in Isaac Sim**
217+
- Source the activate script before launching: `source ~/workspace/simlab/activate-isaacsim.sh`
218+
- The activate script sets the required `LD_LIBRARY_PATH` for the ROS2 bridge
219+
220+
---
221+
222+
## Next Steps
223+
224+
- Write VR teleoperation script using Isaac Sim XR input APIs to read controller poses
225+
- Implement IK (using cuRobo) to convert end-effector poses to joint angles
226+
- Bridge joint commands to real OpenArm hardware via ROS2 → CAN bus

‎scripts/isaacsim/openarm_sim.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
from isaaclab.app import AppLauncher
2+
3+
app_launcher = AppLauncher(headless=False)
4+
simulation_app = app_launcher.app
5+
6+
import isaaclab.sim as sim_utils
7+
8+
BIMANUAL_USD = "/home/vision/humanoids/openarm_isaac_lab/source/openarm/openarm/tasks/manager_based/openarm_manipulation/usds/openarm_bimanual/openarm_bimanual.usd"
9+
10+
def main():
11+
sim_cfg = sim_utils.SimulationCfg(dt=0.01)
12+
sim = sim_utils.SimulationContext(sim_cfg)
13+
sim.set_camera_view([2.0, 2.0, 2.0], [0.0, 0.0, 0.5])
14+
15+
cfg = sim_utils.UsdFileCfg(usd_path=BIMANUAL_USD)
16+
cfg.func("/World/OpenArm", cfg)
17+
18+
sim.reset()
19+
print("Bimanual OpenArm loaded.")
20+
21+
while simulation_app.is_running():
22+
sim.step()
23+
24+
if __name__ == "__main__":
25+
main()
26+
simulation_app.close()

0 commit comments

Comments
 (0)