Skip to content

Commit bf89eb5

Browse files
committed
update readme
1 parent 1101561 commit bf89eb5

5 files changed

Lines changed: 40 additions & 16 deletions

File tree

README.md

Lines changed: 38 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ VisionProTeleop
22
===========
33

44

5-
> **🎉 UPDATE: Now supporting Low-Latency Video Streaming!** You can now stream back your robot's camera feed back to Vision Pro via webRTC protocol, alongside the original hand tracking data stream. No complicated network setting required. Download the app, `pip install avp_stream`, and you're done!
5+
> **🎉 UPDATE: Now supporting Low-Latency Video Streaming!** You can now stream back your robot's camera feed back to Vision Pro via webRTC protocol, alongside the original hand tracking data stream. No complicated network setting required. Update the app, `pip install --upgrade avp_stream`, and you're done!
66
77

88
<div align="center">
@@ -40,26 +40,30 @@ If you use this repository in your work, consider citing:
4040

4141
### Step 1. Install the app on Vision Pro
4242

43-
![](assets/visionpro_main.png)
4443

45-
This app is now officially on VisionOS App Store! You can search for **[Tracking Streamer](https://apps.apple.com/us/app/tracking-streamer/id6478969032)** from the App Store and install the app.
44+
This app is officially on VisionOS App Store! You can search for **[Tracking Streamer](https://apps.apple.com/us/app/tracking-streamer/id6478969032)** from the App Store and install the app.
4645

47-
If you want to play around with the app, you can build/install the app yourself too. To learn how to do that, take a look at this [documentation](/how_to_install.md). This requires (a) Apple Developer Account, (b) Vision Pro Developer Strap, and (c) a Mac with Xcode installed.
46+
47+
> If you want to play around with the app, you can build/install the app yourself too. To learn how to do that, take a look at this [documentation](/how_to_install.md). This requires (a) Apple Developer Account, (b) Vision Pro Developer Strap, and (c) a Mac with Xcode installed.
4848
4949

5050
### Step 2. Run the app on Vision Pro
5151

52-
After installation, click on the app on Vision Pro and click `Start`. That's it! Vision Pro is now streaming the tracking data over your wifi network.
52+
After installation, click and open the app on Vision Pro. It should show something like this. Click **START with Video Streaming** if you want to stream back videos from your robot over webRTC. Otherwise, click the left button. That's it!
53+
54+
Vision Pro is now streaming the tracking data over your wifi network via gRPC, and ready to receive video stream via webRTC.
55+
56+
57+
![](assets/app-screenshot.png)
5358

54-
**Tip** Remember the IP address before you click start; you need to specify this IP address to subscribe to the data. Once you click start, the app will immediately enter into pass-through mode. Click on the digital crown to stop streaming.
5559

5660

5761
### Step 3. Receive the hand tracking data from anywhere
5862

5963
The following python package allows you to receive the data stream from any device that's connected to the same WiFi network. First, install the package:
6064

6165
```
62-
pip install avp_stream
66+
pip install --upgrade avp_stream
6367
```
6468

6569
Then, add this code snippet to any of your projects you were developing:
@@ -74,9 +78,9 @@ while True:
7478
print(r['head'], r['right_wrist'], r['right_fingers'])
7579
```
7680

77-
### Step 4. [🎉NEW FEATURE] Stream video feeds back to Vision Pro!
81+
### Step 4. [🎉V2 Update🎉] Stream video feeds back to Vision Pro!
7882

79-
Streaming your robot's video feed back to Vision Pro requires one additional line: `start_video_streaming`.
83+
Streaming your robot's video feed back to Vision Pro requires one additional line: `start_video_streaming`. This feature is only supported on the latest version of the VisionOS app, and python package. Make sure you upgrade both python library / visionOS app.
8084

8185
```python
8286
from avp_stream import VisionProStreamer
@@ -86,7 +90,7 @@ s = VisionProStreamer(ip = avp_ip)
8690
# you can simply start a video stream
8791
# by defining which video device you want to use
8892
s.start_video_streaming(device="/dev/video0", format="v4l2", \
89-
size="640x480", fps=30)
93+
size="640x480", fps=30, stereo=False)
9094

9195
while True:
9296
r = s.latest
@@ -103,7 +107,16 @@ You can also:
103107
s.start_video_streaming(device="/dev/video0", format="v4l2", \
104108
size="640x480", fps=30)
105109
```
106-
- work without a physical camera and send over synthetically generated frames (i.e., rendered from simulation)
110+
111+
- send over as a stereo camera feed (assumes side-by-side concatenated image)
112+
113+
```python
114+
s = VisionProStreamer(ip = avp_ip)
115+
s.start_video_streaming(device="/dev/video0", format="v4l2", \
116+
size="640x480", fps=30, stereo=True)
117+
```
118+
119+
- work without a physical camera and send over synthetically generated frames (i.e., simulation renderings, or purely synthetic images)
107120
```python
108121
s = VisionProStreamer(ip = avp_ip)
109122
# define your own image generating function, and register
@@ -114,7 +127,14 @@ You can also:
114127

115128
which is explained in detail in [examples](examples) folder.
116129

117-
> **NOTE**: Finding the right combination of `device`, `format`, `size`, and `fps` can sometime be tricky, since camera models only support certain combination. Also, depending on your operation system and how you plugged the camera, the `device` might have been mounted on a different directory.
130+
**Note:** Finding the right combination of `device`, `format`, `size`, and `fps` can sometime be tricky, since camera models only support certain combination. Also, depending on your operation system and how you plugged the camera, the camera might have been mounted on a different directory. To find the right combination of these parameters, you can use the following script:
131+
132+
```bash
133+
python test_video_devices.py --live
134+
```
135+
When it's the right combination, it'll show up a live feed of the selected camera.
136+
137+
118138

119139

120140

@@ -168,15 +188,19 @@ Refer to the image above to see what order the joints are represented in each ha
168188

169189
## App Features
170190

171-
### Status Window
191+
### Status / Control Window
172192

193+
Status window is a good way to monitor the networking status, and control the life cycle of the app. It is shown in full view when you first start the app, but automatically gets minimized when video frame comes in. You can also minimize / maximize as you need.
194+
195+
![](assets/status-window.png)
173196

174197

175198
### Modifying the Video Viewport
176199

200+
You can also modify the video viewport -- how far they are from your eyes.
201+
177202

178203

179-
### Temporarily Hiding Video Stream
180204

181205

182206
## Acknowledgements

assets/app-screenshot.png

5.23 MB
Loading

assets/status-window.png

526 KB
Loading

examples/01_stream_from_camera.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
import time
1414

1515
def main():
16-
VISION_PRO_IP = "10.29.249.251"
16+
VISION_PRO_IP = "192.168.86.21"
1717

1818
streamer = VisionProStreamer(ip=VISION_PRO_IP, record=True)
1919

examples/03_synthetic_frames.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
import time
1313

1414
# Replace with your Vision Pro's IP address
15-
VISION_PRO_IP = '10.29.249.251'
15+
VISION_PRO_IP = '192.168.86.21'
1616

1717

1818
# Example 1: Animated gradient

0 commit comments

Comments
 (0)