Merge pull request #3318 from matt-cossins/reachy

Build an Edge AI Reachy Mini App with Raspberry Pi, MediaPipe, and MuJoCo

Author: pareenaverma

assets/contributors.csv

@@ -122,5 +122,6 @@ Johnny Nunez,NVIDIA,johnnynunez,johnnycano,,
 Raymond Lo,NVIDIA,raymondlo84,raymondlo84,,
 Kavya Sri Chennoju,Arm,kavya-chennoju,kavya-sri-chennoju,,
 Akash Malik,Arm,akashmalik19973,akash-malik-a65bab219,,
+Matt Cossins,Arm,matt-cossins,,,
 Sue Wu,Arm,,,,
 Sabika Tasneem,Memgraph,,,,

content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_index.md

@@ -0,0 +1,79 @@
+---
+title: Build an Edge AI Reachy Mini App with Raspberry Pi, MediaPipe, and MuJoCo
+
+draft: true
+cascade:
+    draft: true
+    
+description: Run MediaPipe gesture inference on a Raspberry Pi 5, connect to a Reachy Mini MuJoCo simulation on a development machine, and use a browser dashboard to decide Reachy's fate with a thumbs-up or thumbs-down.
+
+minutes_to_complete: 60
+
+who_is_this_for: This Learning Path is for developers interested in edge AI, robotics simulation, and physical AI applications. You can complete the main path without owning a physical Reachy Mini.
+
+learning_objectives:
+    - Understand why simulation environments can aid Edge AI and robotics development.
+    - Run a simulated Reachy Mini robot on a laptop or desktop.
+    - Use MediaPipe and TensorFlow Lite gesture recognition on Raspberry Pi 5.
+    - Connect an edge inference node to a robot daemon over the network.
+    - Display results over a web dashboard.
+    - (Optional) Extend the project toward a physical Reachy Mini, audio or multimodal interaction, or your own app.
+
+prerequisites:
+    - A Raspberry Pi 5, ideally with 16 GB RAM.
+    - A USB webcam connected to the Raspberry Pi.
+    - A macOS or Linux machine, or a Windows machine with WSL2, capable of running the Reachy Mini MuJoCo simulation.
+    - Basic Python and Bash terminal experience.
+    - (Optional) [Reachy Mini](https://huggingface.co/reachy-mini)
+
+author: Matt Cossins
+
+### Tags
+skilllevels: Introductory
+subjects: ML
+armips:
+    - Cortex-A
+tools_software_languages:
+    - Raspberry Pi
+    - Reachy Mini
+    - Python
+    - MediaPipe
+    - FastAPI
+    - MuJoCo
+operatingsystems:
+    - Linux
+    - macOS
+    - Windows
+
+### Cross-platform metadata only
+shared_path: true
+shared_between:
+    - embedded-and-microcontrollers
+    - laptops-and-desktops
+
+further_reading:
+    - resource:
+        title: Make and publish your Reachy Mini app
+        link: https://huggingface.co/blog/pollen-robotics/make-and-publish-your-reachy-mini-apps
+        type: blog
+    - resource:
+        title: Reachy Mini Python SDK documentation
+        link: https://pollen-robotics.github.io/reachy-mini-sdk/
+        type: documentation
+    - resource:
+        title: Reachy Mini project examples
+        link: https://github.com/pollen-robotics/reachy-mini-sdk/tree/main/examples
+        type: website
+    - resource:
+        title: MediaPipe Gesture Recognizer guide
+        link: https://ai.google.dev/edge/mediapipe/solutions/vision/gesture_recognizer
+        type: documentation
+
+
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1                       # _index.md always has weight of 1 to order correctly
+layout: "learningpathall"       # All files under learning paths have this same wrapper
+learning_path_main_page: "yes"  # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
+---

content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_next-steps.md

@@ -0,0 +1,8 @@
+---
+# ================================================================================
+#       FIXED, DO NOT MODIFY THIS FILE
+# ================================================================================
+weight: 21                  # The weight controls the order of the pages. _index.md always has weight 1.
+title: "Next Steps"         # Always the same, html page title.
+layout: "learningpathall"   # All files under learning paths have this same wrapper for Hugo processing.
+---

content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/battle-cry.gif

@@ -0,0 +1,132 @@
+---
+title: (Optional) Extend the project
+weight: 7
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Try focused changes
+
+You now have a working physical AI pattern: sensor input at the edge, local
+inference, robot commands, and a dashboard for visibility. A good next step is
+to make small changes where the effect is easy to observe:
+
+- Change `VERDICT_MIN_CONFIDENCE` in `main.py` to make thumbs detection
+  stricter or more forgiving.
+- Change `MOVE_REPETITIONS` in `main.py` so each move plays more or fewer
+  times.
+- Add a fifth move in `moves.py` and register it in `MOVE_CATALOGUE`.
+- Try a different webcam with `REACHY_GLADIATOR_CAMERA_INDEX=1`.
+
+These changes are deliberately small. They help you learn which part of the
+system owns each behavior before you make larger changes to perception, robot
+motion, or packaging.
+
+## Change the classifier
+
+Trying adding new gesture controls. Start in `gesture.py`, where the app maps MediaPipe labels such as `Thumb_Up` and `Thumb_Down` to app labels. Then add matching robot behavior in `moves.py` and branch on the new label in `main.py`.
+
+Some game-themed ideas:
+
+- Map `Closed_Fist` to `challenge`, and make Reachy repeat the current move
+- Map `Pointing_Up` to `reroll`, and make Reachy reject the current move and
+  choose another one.
+- Map `Number of fingers` to choosing a specific move.
+
+## Add audio output
+
+The gladiator theme is a good fit for sound. Try adding audio cues such as:
+
+- a crowd cheer during victory,
+- a dramatic sound during defeat,
+- a short drum hit before each move,
+- a spoken move name before Reachy performs it.
+
+Keep audio output separate from `moves.py` at first. For example, create an
+`audio.py` helper and call it from `main.py` when the state changes. This keeps
+robot motion and sound effects easy to change independently.
+
+## Replace thumbs with audio input
+
+The vision-based verdict is one edge AI input modality. You can replace
+or complement it with audio - many webcams include microphones or you can use a USB microphone.
+
+- say "yes" for victory and "no" for defeat,
+- clap once for victory and twice for defeat,
+
+A lightweight keyword-spotting model can map spoken commands to the same game states currently triggered by MediaPipe gestures.
+
+## Try the packaged app on a physical Reachy
+
+If you have a physical Reachy Mini, the quickest way to try the finished
+experience is to install the packaged [Reachy Gladiator app](https://huggingface.co/spaces/cossinsmatthew/reachy_gladiator) through the [Reachy Mini Control app](https://github.com/pollen-robotics/reachy-mini-desktop-app).
+
+Simply install Reachy Mini Control on a supported machine, connected to your Reachy, and search for the reachy gladiator app.
+
+{{% notice Warning %}}
+If using a physical Reachy Mini, use caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health & safety risk. You are responsible for your safety and the safety of others around you when using physical robotic devices
+{{% /notice %}}
+
+## Adapt this source project for physical Reachy
+
+The main learning path uses the Raspberry Pi USB webcam for perception and a
+remote MuJoCo daemon for robot motion. A physical Reachy route changes two
+things:
+
+- camera frames come from the Reachy daemon instead of the Pi USB webcam,
+- the Pi app connects to the physical Reachy daemon instead of the simulation
+  daemon.
+
+The source project exposes these switches as environment variables, so you do
+not need to edit the Python source:
+
+```bash
+REACHY_GLADIATOR_MEDIA_BACKEND=reachy \
+REACHY_GLADIATOR_CAMERA=reachy \
+REACHY_GLADIATOR_DAEMON_PORT=8000 \
+./scripts/run_pi_app.sh localhost
+```
+
+Use `localhost` only when the physical daemon runs on the same Pi as the app.
+If the daemon runs on another machine, replace `localhost` with that machine's
+IP address and set `REACHY_GLADIATOR_DAEMON_PORT` to the daemon port.
+
+These variables map to the code in two places:
+
+- `REACHY_GLADIATOR_MEDIA_BACKEND=reachy` lets `ReachyMiniApp` request daemon
+  camera media.
+- `REACHY_GLADIATOR_CAMERA=reachy` tells `camera.py` to use
+  `ReachyMediaFrameSource` instead of `OpenCVCameraFrameSource`.
+
+## Build your own Reachy Mini app
+
+A Reachy Mini app is a Python package with a class that inherits from `ReachyMiniApp`, implements `run()`, and exposes an entry point in `pyproject.toml`.
+
+To build a fresh app:
+
+1. Start with one `ReachyMiniApp` class.
+2. Add one safe motion such as `neutral()`.
+3. Add one input source such as a camera gesture, audio command, button, or web
+   endpoint.
+4. Add a dashboard only after the core loop works.
+5. Test in simulation before physical hardware.
+6. Package the app when it is stable enough for repeated use.
+
+The Reachy Mini tooling can scaffold and validate a shareable app:
+
+```bash
+reachy-mini-app-assistant create my_app ~/reachy_projects
+reachy-mini-app-assistant check ~/reachy_projects/my_app
+```
+
+The [Reachy Mini app publishing guide](https://huggingface.co/blog/pollen-robotics/make-and-publish-your-reachy-mini-apps)
+explains the packaging and publishing workflow in more detail.
+
+Use the Reachy Mini SDK documentation and examples to understand available
+motion, media, and daemon APIs. If you use an AI coding agent, give it the
+Pollen Robotics `AGENTS.md` instructions, provided by the [Reachy Mini project](https://github.com/pollen-robotics/reachy_mini) so it follows the expected app structure.
+
+## What you learned
+
+You explored options for extending from simulation to a physical Reachy, as well ideas for changing the project to include audio, new vision gestures, or different behaviours.

content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/extend.md

@@ -0,0 +1,123 @@
+---
+title: Learn about Reachy and understand the application
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Reachy Gladiator - Decide Reachy's Fate
+
+In this Learning Path, Reachy is stepping out onto the sands of the Arena. Reachy has practiced their gladiator moves, but are they good enough?
+
+You will decide Reachy's fate in the classic Roman way - a 👍 for **Victory**, or a 👎 for **Defeat**.
+
+![Illustration of Reachy Mini in a gladiator style pose. The app uses this arena theme for its moves, verdicts, and dashboard.#center](reachy_gladiator.png "Reachy Mini Gladiator App Concept")
+
+## What is Reachy Mini?
+
+Reachy Mini is a small open robotics platform from Pollen Robotics. It is designed for expressive head, antenna, and body motion, and it can be controlled from Python with the Reachy Mini SDK. The Reachy Mini Wireless version includes an onboard Arm-powered Raspberry Pi 4 Compute Module, and the Lite version is operated with external compute (e.g., Raspberry Pi, DGX Spark, Mac/PC).
+
+Reachy can also be simulated using MuJoCo software. Most developers do not have a physical Reachy Mini robot on their desk, and it is often useful to develop software before hardware is available. Extrapolating from Reachy to more industrial robotics, it is also important to test applications in simulation in advance for safety.
+
+{{% notice Warning %}}
+If using a physical Reachy Mini, use caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health & safety risk. You are responsible for your safety and the safety of others around you when using physical robotic devices
+{{% /notice %}}
+
+## What will you build?
+
+The workflow of this learning path is split across two machines:
+
+**Laptop/Desktop: macOS, Linux, or Windows with WSL2**
+  - Runs the Reachy Mini daemon
+  - Runs the MuJoCo simulation
+  - Displays simulated Reachy movement
+  - Displays the Pi-hosted dashboard at `http://<pi-ip-address>:8042`
+
+**Raspberry Pi 5: Raspberry Pi OS**
+  - Captures frames from a USB webcam
+  - Runs the Edge AI application (local MediaPipe gesture recognition)
+  - Serves a dashboard on port 8042
+  - Sends robot movement commands to the simulation host daemon
+
+This split is a common edge/physical AI pattern:
+
+- A small edge device handles sensors and inference close to the user.
+- A robot API or daemon receives movement commands.
+- A dashboard gives visibility into the live state of the system.
+- Digital twin simulation reduces hardware access as a blocker and allows for safer development.
+
+This is similar to how larger industrial robotics systems are often built.
+Keeping perception, robot control, and observability as separate pieces makes it easier to test, replace, and deploy parts of the system independently.
+
+## What does the app do?
+
+The app is called Reachy Gladiator. Reachy (in simulation or otherwise) performs a randomly-chosen scripted gladiator move. You provide a 👍 for **Victory**, or a 👎 for **Defeat**. **Victory** makes Reachy celebrate, **Defeat** makes Reachy react sadly.
+
+This learning path starts from the complete `reachy_gladiator_lp` project instead of asking you to create every file from scratch. The simulation host only needs a launcher script, but the Raspberry Pi will clone and run the full project. You will inspect the different parts of the system so you can recreate your own apps running on Reachy or in simulation.
+
+The Reachy Gladiator app runs a repeated loop:
+
+1. Start with a 10-second preparing countdown
+2. Randomly pick one gladiator move
+3. Repeat the selected move three times
+4. Return Reachy to a neutral pose
+5. Watch for a 👍 or a 👎
+6. Run a victory or defeat reaction
+7. Repeat with another move
+
+There are four preset moves:
+
+- `Salute`
+![Salute Move#center](salute.gif "Salute Move")
+- `Sword Swing`
+![Sword Swing Move#center](sword.gif "Sword Swing Move")
+- `Shield Up`
+![Shield Up Move#center](shield.gif "Shield Up Move")
+- `Battle Cry`
+![Battle Cry Move#center](battle-cry.gif "Battle Cry Move")
+
+The app shuffles all four moves and performs each once before any move repeats. When the bag is empty, it reshuffles and avoids repeating the same move at the shuffle boundary.
+
+## Use a compatible terminal
+
+The commands in this learning path use a Bash-style shell. They work directly on macOS and Linux.
+
+On Windows, use WSL2 with an Ubuntu distribution for the simulation host commands. This keeps the commands almost identical to the macOS and Linux flow:
+
+- Use the WSL terminal for `python3`, `source .venv/bin/activate`, and `./scripts/start_sim.sh`.
+- Keep the browser on Windows if you prefer; open the Pi dashboard from any browser that can reach the Raspberry Pi.
+- If the Pi cannot reach a daemon running inside WSL2, check Windows firewall and WSL networking. WSL2 uses virtualized networking, so inbound access from another device on your LAN may require Windows port forwarding.
+
+The rest of this learning path shows the common Bash commands and calls out the one IP-address command that differs by host operating system.
+
+## Project structure
+
+The simulation host does not need the full project checkout. In the next
+section, you will download only the simulation launcher script on that machine.
+
+You will clone the full `reachy_gladiator_lp` project on the Raspberry Pi later on.
+
+The key files are:
+
+```text
+reachy_gladiator_lp/
+├── pyproject.toml
+├── scripts/
+│   ├── start_sim.sh
+│   ├── setup_pi.sh
+│   ├── run_pi_app.sh
+│   └── check_pi_camera.sh
+└── reachy_gladiator_lp/
+    ├── main.py
+    ├── camera.py
+    ├── gesture.py
+    ├── moves.py
+    ├── assets/
+    │   └── gesture_recognizer.task
+    └── static/           # dashboard HTML, CSS, JavaScript, and media
+```
+
+## What you learned and what is next
+
+You learned what Reachy Mini is, why simulation is useful for edge/physical AI development, and how the app splits work between a simulation host and a Raspberry Pi. You are now ready to start the simulation host with a lightweight launcher script.