Skip to content

Commit d11e627

Browse files
younghyoparkyounghyopark
committed
add docs
1 parent 0ab9143 commit d11e627

18 files changed

Lines changed: 2342 additions & 1 deletion

‎.github/workflows/deploy-docs.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: Deploy Documentation to GitHub Pages
2+
3+
on:
4+
push:
5+
branches:
6+
- visual # Change to your default branch if different
7+
- main
8+
paths:
9+
- 'docs/build/html/**'
10+
workflow_dispatch: # Allow manual triggering
11+
12+
permissions:
13+
contents: write
14+
15+
jobs:
16+
deploy:
17+
runs-on: ubuntu-latest
18+
19+
steps:
20+
- name: Checkout repository
21+
uses: actions/checkout@v4
22+
with:
23+
fetch-depth: 0 # Fetch all history for all branches
24+
25+
- name: Check if docs/build/html exists
26+
run: |
27+
if [ ! -d "docs/build/html" ]; then
28+
echo "Error: docs/build/html directory not found"
29+
echo "Please build the documentation locally first using ./build_docs.sh"
30+
exit 1
31+
fi
32+
echo "Documentation found in docs/build/html"
33+
ls -la docs/build/html
34+
35+
- name: Deploy to GitHub Pages
36+
uses: peaceiris/actions-gh-pages@v3
37+
with:
38+
github_token: ${{ secrets.GITHUB_TOKEN }}
39+
publish_dir: ./docs/build/html
40+
publish_branch: gh-pages
41+
force_orphan: true # Keep gh-pages clean
42+
user_name: 'github-actions[bot]'
43+
user_email: 'github-actions[bot]@users.noreply.github.com'
44+
commit_message: 'Deploy documentation'
45+
46+
- name: Summary
47+
run: |
48+
echo "✓ Documentation deployed to GitHub Pages"
49+
echo "Visit: https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/"

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
11
/*.egg-info
22
/dist
3-
*.pyc
3+
*.pyc
4+
5+
# Sphinx documentation build files
6+
docs/build/

‎build_docs.sh‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
#!/bin/bash
2+
3+
# Script to build Sphinx documentation for VisionProTeleop
4+
# This script builds the HTML documentation from RST source files
5+
6+
set -e # Exit on error
7+
8+
# Colors for output
9+
GREEN='\033[0;32m'
10+
YELLOW='\033[1;33m'
11+
RED='\033[0;31m'
12+
NC='\033[0m' # No Color
13+
14+
echo -e "${GREEN}Building VisionProTeleop Documentation${NC}"
15+
echo "========================================"
16+
17+
# Check if sphinx-build is installed
18+
if ! command -v sphinx-build &> /dev/null; then
19+
echo -e "${RED}Error: sphinx-build not found${NC}"
20+
echo "Please install Sphinx and the RTD theme:"
21+
echo " pip install sphinx sphinx_rtd_theme"
22+
exit 1
23+
fi
24+
25+
# Get the script directory
26+
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
27+
DOCS_DIR="$SCRIPT_DIR/docs"
28+
SOURCE_DIR="$DOCS_DIR/source"
29+
BUILD_DIR="$DOCS_DIR/build"
30+
31+
# Check if source directory exists
32+
if [ ! -d "$SOURCE_DIR" ]; then
33+
echo -e "${RED}Error: Source directory not found: $SOURCE_DIR${NC}"
34+
exit 1
35+
fi
36+
37+
# Clean previous build
38+
if [ -d "$BUILD_DIR" ]; then
39+
echo -e "${YELLOW}Cleaning previous build...${NC}"
40+
rm -rf "$BUILD_DIR"
41+
fi
42+
43+
# Create build directory
44+
mkdir -p "$BUILD_DIR"
45+
46+
# Build the documentation
47+
echo -e "${GREEN}Building HTML documentation...${NC}"
48+
sphinx-build -b html "$SOURCE_DIR" "$BUILD_DIR/html"
49+
50+
# Check if build was successful
51+
if [ $? -eq 0 ]; then
52+
echo ""
53+
echo -e "${GREEN}✓ Documentation built successfully!${NC}"
54+
echo ""
55+
echo "Documentation location:"
56+
echo " $BUILD_DIR/html/index.html"
57+
echo ""
58+
echo "To view the documentation, open in your browser:"
59+
echo -e " ${YELLOW}open $BUILD_DIR/html/index.html${NC}"
60+
echo ""
61+
else
62+
echo -e "${RED}✗ Documentation build failed${NC}"
63+
exit 1
64+
fi

‎docs/source/acknowledgements.rst‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
Acknowledgements
2+
================
3+
4+
We acknowledge support from:
5+
6+
- **Hyundai Motor Company**
7+
- **ARO MURI grant number W911NF-23-1-0277**
8+
9+
Research Team
10+
-------------
11+
12+
**Authors:**
13+
14+
- Younghyo Park (MIT)
15+
- Pulkit Agrawal (MIT)
16+
17+
**Affiliation:**
18+
19+
Improbable AI Lab, Massachusetts Institute of Technology
20+
21+
Contributing
22+
------------
23+
24+
Contributions to VisionProTeleop are welcome! Please visit the `GitHub repository <https://github.com/Improbable-AI/VisionProTeleop>`_ to:
25+
26+
- Report issues
27+
- Submit pull requests
28+
- Suggest features
29+
- Share your use cases
30+
31+
Community
32+
---------
33+
34+
Join the community:
35+
36+
- GitHub Discussions: Share your projects and ask questions
37+
- Issues: Report bugs or request features
38+
- Pull Requests: Contribute improvements
39+
40+
Related Projects
41+
----------------
42+
43+
This project builds upon:
44+
45+
- **gRPC**: High-performance RPC framework
46+
- **WebRTC**: Real-time communication protocol
47+
- **VisionOS**: Apple's spatial computing platform
48+
- **Isaac Gym**: NVIDIA's physics simulation platform
49+
50+
Publications
51+
------------
52+
53+
For a detailed explanation of the system, check out the `short paper <https://github.com/Improbable-AI/VisionProTeleop/blob/main/assets/short_paper_new.pdf>`_.
54+
55+
If you use this work in your research, please cite:
56+
57+
.. code-block:: bibtex
58+
59+
@software{park2024avp,
60+
title={Using Apple Vision Pro to Train and Control Robots},
61+
author={Park, Younghyo and Agrawal, Pulkit},
62+
year={2024},
63+
url = {https://github.com/Improbable-AI/VisionProTeleop},
64+
}
65+
66+
Contact
67+
-------
68+
69+
For questions or collaboration inquiries:
70+
71+
- **Email**: younghyo@mit.edu
72+
- **Lab Website**: `Improbable AI Lab <https://people.csail.mit.edu/pulkitag/>`_
73+
- **GitHub**: `Improbable-AI <https://github.com/Improbable-AI>`_

‎docs/source/api/isaac_env.rst‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
Isaac Environment
2+
=================
3+
4+
Integration with NVIDIA Isaac Gym for simulation.
5+
6+
.. automodule:: avp_stream.isaac_env
7+
:members:
8+
:undoc-members:
9+
:show-inheritance:
10+
11+
Classes and Functions
12+
---------------------
13+
14+
.. autoclass:: avp_stream.isaac_env.IsaacEnv
15+
:members:
16+
:undoc-members:
17+
:show-inheritance:
18+
:special-members: __init__
19+
20+
Overview
21+
--------
22+
23+
The Isaac environment module provides integration between VisionProTeleop and NVIDIA Isaac Gym simulation environments. This allows you to control simulated robots using Vision Pro hand tracking data.
24+
25+
Example Usage
26+
-------------
27+
28+
Basic Isaac Gym integration::
29+
30+
from avp_stream import VisionProStreamer
31+
from avp_stream.isaac_env import IsaacEnv
32+
33+
# Initialize Vision Pro streamer
34+
avp_ip = "10.31.181.201"
35+
streamer = VisionProStreamer(ip=avp_ip)
36+
37+
# Create Isaac environment
38+
env = IsaacEnv(streamer)
39+
40+
# Training/control loop
41+
while True:
42+
# Get tracking data
43+
data = streamer.latest
44+
45+
# Update simulation with tracking data
46+
env.update(data)
47+
48+
# Step simulation
49+
env.step()
50+
51+
See Also
52+
--------
53+
54+
- :doc:`streamer` - Main streamer interface
55+
- :doc:`utils` - Utility functions for Isaac integration

‎docs/source/api/modules.rst‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
API Reference
2+
=============
3+
4+
This section provides detailed API documentation for all modules.
5+
6+
.. toctree::
7+
:maxdepth: 2
8+
9+
streamer
10+
isaac_env
11+
utils
12+
visual

‎docs/source/api/streamer.rst‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
VisionProStreamer
2+
=================
3+
4+
The main interface for receiving tracking data and streaming video.
5+
6+
.. automodule:: avp_stream.streamer
7+
:members:
8+
:undoc-members:
9+
:show-inheritance:
10+
:special-members: __init__
11+
12+
Classes
13+
-------
14+
15+
.. autoclass:: avp_stream.streamer.VisionProStreamer
16+
:members:
17+
:undoc-members:
18+
:show-inheritance:
19+
:special-members: __init__
20+
21+
.. automethod:: __init__
22+
.. automethod:: start_video_streaming
23+
.. automethod:: register_frame_callback
24+
.. automethod:: stop
25+
.. autoproperty:: latest
26+
27+
Main Methods
28+
^^^^^^^^^^^^
29+
30+
.. py:method:: __init__(ip: str, port: int = 50051)
31+
32+
Initialize the Vision Pro streamer.
33+
34+
:param ip: IP address of the Vision Pro device
35+
:type ip: str
36+
:param port: gRPC port number (default: 50051)
37+
:type port: int
38+
39+
.. py:method:: start_video_streaming(device, format, size, fps, stereo=False)
40+
41+
Start streaming video back to Vision Pro.
42+
43+
:param device: Path to video device (e.g., "/dev/video0") or None for synthetic
44+
:type device: str or None
45+
:param format: Video format (e.g., "v4l2", "dshow", "avfoundation") or None
46+
:type format: str or None
47+
:param size: Frame size as string (e.g., "640x480")
48+
:type size: str
49+
:param fps: Frames per second
50+
:type fps: int
51+
:param stereo: Whether to stream stereo video (side-by-side format)
52+
:type stereo: bool
53+
54+
.. py:method:: register_frame_callback(callback)
55+
56+
Register a function to process frames before streaming.
57+
58+
:param callback: Function that takes a frame (numpy array) and returns processed frame
59+
:type callback: callable
60+
61+
.. py:method:: stop()
62+
63+
Stop the streamer and release resources.
64+
65+
.. py:property:: latest
66+
67+
Get the latest tracking data.
68+
69+
:return: Dictionary containing tracking data
70+
:rtype: dict
71+
72+
Dictionary keys:
73+
74+
- ``head``: (1, 4, 4) numpy array - head pose in ground frame
75+
- ``right_wrist``: (1, 4, 4) numpy array - right wrist pose in ground frame
76+
- ``left_wrist``: (1, 4, 4) numpy array - left wrist pose in ground frame
77+
- ``right_fingers``: (25, 4, 4) numpy array - right finger joints in wrist frame
78+
- ``left_fingers``: (25, 4, 4) numpy array - left finger joints in wrist frame
79+
- ``right_pinch_distance``: float - distance between right thumb and index finger
80+
- ``left_pinch_distance``: float - distance between left thumb and index finger
81+
- ``right_wrist_roll``: float - rotation angle of right wrist around arm axis
82+
- ``left_wrist_roll``: float - rotation angle of left wrist around arm axis
83+
84+
Example Usage
85+
^^^^^^^^^^^^^
86+
87+
Basic usage::
88+
89+
from avp_stream import VisionProStreamer
90+
91+
avp_ip = "10.31.181.201"
92+
streamer = VisionProStreamer(ip=avp_ip)
93+
94+
while True:
95+
data = streamer.latest
96+
print(data['head'], data['right_wrist'])
97+
98+
With video streaming::
99+
100+
streamer = VisionProStreamer(ip=avp_ip)
101+
streamer.start_video_streaming(
102+
device="/dev/video0",
103+
format="v4l2",
104+
size="640x480",
105+
fps=30
106+
)
107+
108+
while True:
109+
data = streamer.latest
110+
# Process tracking data
111+
112+
With frame processing::
113+
114+
def process_frame(frame):
115+
# Add overlay or processing
116+
return frame
117+
118+
streamer = VisionProStreamer(ip=avp_ip)
119+
streamer.register_frame_callback(process_frame)
120+
streamer.start_video_streaming(
121+
device="/dev/video0",
122+
format="v4l2",
123+
size="640x480",
124+
fps=30
125+
)

0 commit comments

Comments
 (0)