Skip to content

README.md

Latest commit

 

History

History
154 lines (119 loc) · 9.85 KB

README.md

File metadata and controls

154 lines (119 loc) · 9.85 KB

QBotix Rover - Arm Feed Subsystem

This document provides a detailed technical overview of the Arm Feed Subsystem for the rover - Orbitron built by Team Qbotix Rover, developed for the International Rover Challenge 2026.

1. Overview

The Arm Feed subsystem is a real-time video streaming solution designed to transmit live video from two cameras mounted on the rover's robotic arm to the base station. This provides operators with crucial visual feedback for manipulation tasks, sample collection, and navigating complex terrain.

The system is architected for low-latency, high-framerate streaming over a standard IP network, using WebRTC as its core communication protocol. It consists of two main components: a Python-based server running on the rover's onboard computer (Raspberry Pi) and a web-based client viewer on the base station.

2. System Architecture

The architecture is a simple client-server model:

  • Edge Device (Server): A Raspberry Pi connected to two USB cameras. It runs a Python script that captures the video feeds and hosts a WebRTC signaling server.
  • Base Station (Client): Any computer with a modern web browser. It hosts a simple HTML page that acts as the video receiver and player.
+---------------------------------+      +---------------------------------+
|      Rover (Raspberry Pi)       |      |      Base Station (Computer)    |
|                                 |      |                                 |
|  +-----------+   +-----------+  |      |                                 |
|  | USB Cam 1 |   | USB Cam 2 |  |      |                                 |
|  +-----+-----+   +-----+-----+  |      |                                 |
|        |               |        |      |                                 |
|        v               v        |      |                                 |
|  +-------------------------+    |      |      +----------------------+   |
|  |    sender_armfeed.py    |    |      |      |     armfeed.html     |   |
|  |(OpenCV, aiohttp, aiortc)| ----TCP/IP---->  |(Browser, WebRTC JS)  |   |
|  +-------------------------+    |      |      +----------------------+   |
|                                 |      |                                 |
+---------------------------------+      +---------------------------------+

Data Flow

  1. Camera frames captured via OpenCV.
  2. Frames converted to a WebRTC-compatible format using PyAV.
  3. SDP offer/answer metadata is exchanged between client and server via HTTP.
  4. A direct peer-to-peer connection is established.
  5. Video tracks are streamed directly to the browser and rendered.

3. How It Works

  1. Initialization: The sender_armfeed.py script starts on the Raspberry Pi. It initializes two separate camera capture loops using OpenCV and starts an aiohttp web server.
  2. Signaling: The operator on the base station opens armfeed.html. Clicking "Stream" triggers a JavaScript function that creates a WebRTC "offer". This offer, containing session description protocol (SDP) information, is sent via an HTTP POST request to the /offer endpoint on the Python server.
  3. Negotiation: The Python server receives the offer, generates a corresponding WebRTC "answer", and sends it back to the client. This handshake establishes a direct peer-to-peer connection between the browser and the Python script.
  4. Streaming: Once the connection is established, the Python script continuously grabs frames from the cameras, converts them, and streams them directly to the browser via the established WebRTC PeerConnection.
  5. Display: The browser receives the two separate video tracks and renders them in their respective <video> elements on the HTML page.

4. Key Features

  • Dual Camera Support: Streams two camera feeds simultaneously.
  • Low Latency: Utilizes WebRTC for near real-time video transmission.
  • Robust Reconnection: If a camera feed is lost, the script initiates a retry loop, attempting to reconnect every 30 seconds without halting the other active stream.
  • Bandwidth Management: Includes server-side logic to limit the stream bitrate, preventing network congestion.
  • Web-Based Client: Requires no special software on the base station besides a standard web browser.
  • High Compatibility: Based on standard web technologies (Python, asyncio, JS).

5. Technology Stack

  • Server-Side (Edge Device):

    • Language: Python 3
    • Libraries:
      • opencv-python: For capturing video from USB cameras.
      • aiohttp: Asynchronous web server for signaling.
      • aiortc: Python implementation of WebRTC.
      • numpy: For numerical operations on video frames.
      • av: For video frame format conversion (PyAV).

  • Client-Side (Base Station):

    • HTML5
    • CSS3
    • JavaScript (with the native RTCPeerConnection API)

6. Design Decisions

  • WebRTC over RTSP/HTTP Streaming: Chosen for its low-latency, peer-to-peer communication and native browser support, which eliminates the need for special client software.
  • aiortc (Python) over native C++: Enables rapid development and seamless integration with Python-based computer vision pipelines.
  • aiohttp for Signaling: A lightweight asynchronous server is sufficient for the SDP exchange and avoids introducing heavier frameworks.
  • Dual Independent Camera Loops: This design prevents the entire capture process from blocking and ensures that the failure of one camera does not halt the stream from the other.

7. Core Components

  • sender_armfeed.py: Handles camera capture, WebRTC negotiation, and the streaming pipeline.
  • Camera Capture Loops: Independent asyncio tasks for main and side cameras using OpenCV.
  • WebRTC Signaling Endpoint (/offer): An aiohttp route that handles the SDP exchange between client and server.
  • armfeed.html: The browser-based viewer that handles the client-side WebRTC connection and renders the incoming video streams.

8. Internal Execution Model

The server operates on a non-blocking, asynchronous model powered by asyncio.

  • Concurrent Capture: Each camera runs in an independent async capture loop (arm_cam_grab_loop). This prevents a slow or disconnected camera from blocking the main thread or the other camera's feed.
  • Frame Buffering & Handling: The most recent frame from each camera is stored in a global variable, acting as a simple in-memory buffer. When the WebRTC track handler (.recv()) is ready for a new frame, it pulls the latest one from this buffer.
  • Frame Conversion: The aiortc library does not natively consume OpenCV's NumPy frame format. Therefore, each frame is first converted to a PyAV VideoFrame object (av.VideoFrame.from_ndarray) before being passed to the WebRTC stack.
  • Integrated Event Loop: The aiohttp web server and aiortc peer connections all run on the same asyncio event loop. This allows the application to efficiently manage both HTTP signaling and WebRTC streaming concurrently within a single thread.

9. Setup and Usage

On the Edge Device (Raspberry Pi)

  1. Prerequisites: Ensure Python 3 and pip are installed.
  2. Install Dependencies: 
    pip install opencv-python aiohttp aiortc numpy av
  3. Configuration:
    • Open sender_armfeed.py in a text editor.
    • Adjust ARM_MAIN_CAM_INDEX and ARM_SIDE_CAM_INDEX to match your system's camera setup (e.g., found via ls /dev/video* on Linux).
    • (Optional) Modify the HOST and PORT. The default 0.0.0.0 allows connections from any IP on the network.
  4. Run the Server: 
    python sender_armfeed.py

On the Base Station

  1. Prerequisites: A modern web browser (Chrome, Firefox, Edge).
  2. Configuration:
    • Open armfeed.html in a text editor.
    • Modify the ARM_SERVER_URL constant to match the IP address and port of your Raspberry Pi.
    const ARM_SERVER_URL = "http://<YOUR_RASPBERRY_PI_IP>:8080";
  3. Launch the Viewer:
    • Open the modified armfeed.html file in your browser.
    • Click "Stream" to start the video feeds.

10. Performance Considerations

  • Latency: Typically sub-second under stable local network conditions.
  • Bandwidth: ~2–4 Mbps for dual 480p streams, depending on bitrate settings.
  • Scalability: Designed for a single client. Multiple simultaneous clients will increase CPU and bandwidth usage significantly.

11. Limitations

  • Single-Client Focus: The system is not optimized for multiple concurrent viewers.
  • Network Dependent: Performance is highly sensitive to network stability and bandwidth.
  • Basic Security: The implementation does not include authentication or authorization layers beyond the default security provided by WebRTC (DTLS-SRTP).
  • Hardware Constraints: Stream resolution and framerate are limited by the Raspberry Pi's processing power.

12. Troubleshooting

  • No video appears:
    • Verify the ARM_SERVER_URL in armfeed.html is correct and that the Raspberry Pi is reachable (ping <YOUR_RASPBERRY_PI_IP>).
    • Check the browser's developer console (F12) for JavaScript or WebRTC connection errors.
    • Check the console output of sender_armfeed.py for camera initialization errors.
  • "Failed to open camera" error:
    • Ensure the camera indices in the Python script are correct.
    • Verify the cameras are properly connected and not in use by another application.
  • Video is choppy or lagging:
    • Ensure a strong, stable network connection (Ethernet is preferred over Wi-Fi).
    • Consider lowering the ARM_CAM_TARGET_FPS or resolution constants in the Python script to reduce bandwidth requirements.