You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_index.md
+2-6Lines changed: 2 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,9 +1,5 @@
1
1
---
2
-
title: Build an Edge AI Reachy Mini App with Raspberry Pi, MediaPipe, and MuJoCo
3
-
4
-
draft: true
5
-
cascade:
6
-
draft: true
2
+
title: Build an edge AI Reachy Mini app with Raspberry Pi, MediaPipe, and MuJoCo
7
3
8
4
description: Run MediaPipe gesture inference on a Raspberry Pi 5, connect to a Reachy Mini MuJoCo simulation on a development machine, and use a browser dashboard to decide Reachy's fate with a thumbs-up or thumbs-down.
9
5
@@ -17,7 +13,7 @@ learning_objectives:
17
13
- Use MediaPipe and TensorFlow Lite gesture recognition on Raspberry Pi 5.
18
14
- Connect an edge inference node to a robot daemon over the network.
19
15
- Display results over a web dashboard.
20
-
- (Optional) Extend the project toward a physical Reachy Mini, audio or multimodal interaction, or your own app.
16
+
- Optionally extend the project toward a physical Reachy Mini, audio or multimodal interaction, or your own app.
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/extend.md
+22-21Lines changed: 22 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,6 @@
1
1
---
2
2
title: (Optional) Extend the project
3
+
description: Explore approaches you can take to extend Reachy Gladiator with new gestures, audio output, physical Reachy Mini support, and custom Reachy Mini app packaging.
3
4
weight: 7
4
5
5
6
### FIXED, DO NOT MODIFY
@@ -38,10 +39,10 @@ Some game-themed ideas:
38
39
39
40
The gladiator theme is a good fit for sound. Try adding audio cues such as:
40
41
41
-
- a crowd cheer during victory,
42
-
- a dramatic sound during defeat,
43
-
- a short drum hit before each move,
44
-
- a spoken move name before Reachy performs it.
42
+
- a crowd cheer during victory
43
+
- a dramatic sound during defeat
44
+
- a short drum hit before each move
45
+
- a spoken move name before Reachy performs it
45
46
46
47
Keep audio output separate from `moves.py` at first. For example, create an
47
48
`audio.py` helper and call it from `main.py` when the state changes. This keeps
@@ -52,34 +53,35 @@ robot motion and sound effects easy to change independently.
52
53
The vision-based verdict is one edge AI input modality. You can replace
53
54
or complement it with audio. Many webcams include microphones, or you can use a USB microphone.
54
55
55
-
- say "yes" for victory and "no" for defeat,
56
-
- clap once for victory and twice for defeat,
56
+
Try adding audio inputs such as:
57
+
58
+
- say "yes" for victory and "no" for defeat
59
+
- clap once for victory and twice for defeat
57
60
58
61
A lightweight keyword-spotting model can map spoken commands to the same game states currently triggered by MediaPipe gestures.
59
62
60
63
## Try the packaged app on a physical Reachy
61
64
62
-
If you have a physical Reachy Mini, the quickest way to try the finished
63
-
experience is to install the packaged [Reachy Gladiator app](https://huggingface.co/spaces/cossinsmatthew/reachy_gladiator) through the [Reachy Mini Control app](https://github.com/pollen-robotics/reachy-mini-desktop-app).
65
+
If you have a physical Reachy Mini, you can try the finished
66
+
experience by installing the packaged [Reachy Gladiator app](https://huggingface.co/spaces/cossinsmatthew/reachy_gladiator) through the [Reachy Mini Control app](https://github.com/pollen-robotics/reachy-mini-desktop-app).
64
67
65
68
Install Reachy Mini Control on a supported machine, connect it to your Reachy, and search for the Reachy Gladiator app.
66
69
67
70
{{% notice Warning %}}
68
-
If using a physical Reachy Mini, use caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health & safety risk. You are responsible for your safety and the safety of others around you when using physical robotic devices.
71
+
If you use a physical Reachy Mini, exercise caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health and safety risk. You're responsible for your safety and the safety of others around you when using physical robotic devices.
69
72
{{% /notice %}}
70
73
71
74
## Adapt this source project for physical Reachy
72
75
73
-
The main learning path uses the Raspberry Pi USB webcam for perception and a
76
+
The main Learning Path uses the Raspberry Pi USB webcam for perception and a
74
77
remote MuJoCo daemon for robot motion. A physical Reachy route changes two
75
78
things:
76
79
77
-
- camera frames come from the Reachy daemon instead of the Pi USB webcam,
80
+
- camera frames come from the Reachy daemon instead of the Pi USB webcam
78
81
- the Pi app connects to the physical Reachy daemon instead of the simulation
79
-
daemon.
82
+
daemon
80
83
81
-
The source project exposes these switches as environment variables, so you do
82
-
not need to edit the Python source:
84
+
The source project exposes these switches as environment variables, so you don't need to edit the Python source:
83
85
84
86
```bash
85
87
REACHY_GLADIATOR_MEDIA_BACKEND=reachy \
@@ -95,9 +97,9 @@ IP address and set `REACHY_GLADIATOR_DAEMON_PORT` to the daemon port.
The [Reachy Mini app publishing guide](https://huggingface.co/blog/pollen-robotics/make-and-publish-your-reachy-mini-apps)
124
-
explains the packaging and publishing workflow in more detail.
125
+
For more information about the packaging and publishing workflow, see the [Reachy Mini app publishing guide](https://huggingface.co/blog/pollen-robotics/make-and-publish-your-reachy-mini-apps).
125
126
126
127
Use the Reachy Mini SDK documentation and examples to understand available
127
128
motion, media, and daemon APIs. If you use an AI coding agent, give it the
128
129
Pollen Robotics `AGENTS.md` instructions, provided by the [Reachy Mini project](https://github.com/pollen-robotics/reachy_mini) so it follows the expected app structure.
129
130
130
-
## What you learned
131
+
## What you've learned
131
132
132
-
You explored options for extending from simulation to a physical Reachy, as well as ideas for changing the project to include audio, new vision gestures, or different behaviors.
133
+
You've now explored options for extending from simulation to a physical Reachy, as well as ideas for changing the project to include audio, new vision gestures, or different behaviors.
title: Learn about Reachy and understand the application
2
+
title: Learn about Reachy and the Reachy Gladiator application
3
+
description: Learn how the Reachy Gladiator app splits gesture inference, robot control, simulation, and dashboard display between a simulation host and a Raspberry Pi.
3
4
weight: 2
4
5
5
6
### FIXED, DO NOT MODIFY
@@ -8,37 +9,37 @@ layout: learningpathall
8
9
9
10
## Reachy Gladiator - Decide Reachy's fate
10
11
11
-
In this Learning Path, you build a distributed edge AI application where a Raspberry Pi runs local gesture inference and sends robot motion commands to a simulated Reachy Mini. The app follows a gladiator arena theme: Reachy performs a move, and you give a thumbs-up 👍 for **Victory** or thumbs-down 👎 for **Defeat** to trigger a victory or defeat reaction.
12
+
In this Learning Path, you'll build a distributed edge AI application where a Raspberry Pi runs local gesture inference and sends robot motion commands to a simulated Reachy Mini. The app follows a gladiator arena theme: Reachy performs a move, and you give a thumbs-up for victory or thumbs-down for defeat to trigger a victory or defeat reaction.
12
13
13
-
![Illustration of Reachy Mini in a gladiator style pose. The app uses this arena theme for its moves, verdicts, and dashboard.#center](reachy_gladiator.png"Reachy Mini Gladiator App Concept")
14
+
![Illustration of Reachy Mini in a gladiator style pose. The app uses this arena theme for its moves, verdicts, and dashboard.#center](reachy_gladiator.png"Reachy Mini Gladiator app concept")
14
15
15
16
## What is Reachy Mini?
16
17
17
-
Reachy Mini is a small open robotics platform from Pollen Robotics. It is designed for expressive head, antenna, and body motion, and it can be controlled from Python with the Reachy Mini SDK. The Reachy Mini Wireless version includes an onboard Arm-powered Raspberry Pi 4 Compute Module, and the Lite version is operated with external compute such as a Raspberry Pi, DGX Spark, or Mac/PC.
18
+
Reachy Mini is a small open robotics platform from Pollen Robotics. It's designed for expressive head, antenna, and body motion, and it can be controlled with Python using the Reachy Mini SDK. The Reachy Mini Wireless version includes an onboard Arm-powered Raspberry Pi 4 Compute Module, and the Lite version is operated with external compute such as a Raspberry Pi, DGX Spark, or a Mac or PC.
18
19
19
-
Reachy can also be simulated using MuJoCo software. Most developers do not have a physical Reachy Mini robot on their desk, and it is often useful to develop software before hardware is available. Extrapolating from Reachy to more industrial robotics, it is also important to test applications in simulation in advance for safety.
20
+
Reachy can also be simulated using MuJoCo software. Most developers don't have a physical Reachy Mini robot on their desk, and it's often useful to develop software before hardware is available. Extrapolating from Reachy to more industrial robotics, it's also important to test applications in simulation in advance for safety.
20
21
21
22
{{% notice Warning %}}
22
-
If using a physical Reachy Mini, use caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health & safety risk. You are responsible for your safety and the safety of others around you when using physical robotic devices.
23
+
If using a physical Reachy Mini, exercise caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health and safety risk. You're responsible for your safety and the safety of others around you when using physical robotic devices.
23
24
{{% /notice %}}
24
25
25
-
## What will you build?
26
+
## What you'll build
26
27
27
-
The workflow of this learning path is split across two machines:
28
+
The workflow of this Learning Path is split across two machines:
28
29
29
-
**Laptop/Desktop: macOS, Linux, or Windows with WSL2**
30
+
Laptop or Desktop: macOS, Linux, or Windows with WSL2
30
31
- Runs the Reachy Mini daemon
31
32
- Runs the MuJoCo simulation
32
33
- Displays simulated Reachy movement
33
34
- Displays the Pi-hosted dashboard at `http://<pi-ip-address>:8042`
34
35
35
-
**Raspberry Pi 5: Raspberry Pi OS**
36
+
Raspberry Pi 5: Raspberry Pi OS
36
37
- Captures frames from a USB webcam
37
38
- Runs the Edge AI application (local MediaPipe gesture recognition)
38
39
- Serves a dashboard on port 8042
39
40
- Sends robot movement commands to the simulation host daemon
40
41
41
-
This split is a common edge/physical AI pattern:
42
+
This split is a common edge or physical AI pattern:
42
43
43
44
- A small edge device handles sensors and inference close to the user.
44
45
- A robot API or daemon receives movement commands.
@@ -48,53 +49,53 @@ This split is a common edge/physical AI pattern:
48
49
This is similar to how larger industrial robotics systems are often built.
49
50
Keeping perception, robot control, and observability as separate pieces makes it easier to test, replace, and deploy parts of the system independently.
50
51
51
-
## What does the app do?
52
+
## What the app does
52
53
53
-
The app is called Reachy Gladiator. Reachy (in simulation or otherwise) performs a randomly-chosen scripted gladiator move. You provide a 👍 for **Victory**, or a 👎 for **Defeat**. **Victory** makes Reachy celebrate, **Defeat** makes Reachy react sadly.
54
+
The app is called Reachy Gladiator. Reachy (in simulation or otherwise) performs a randomly-chosen scripted gladiator move. You provide a thumbs up for victory, or a thumbs down for defeat. Victory makes Reachy celebrate, and defeat makes Reachy react sadly.
54
55
55
-
This Learning Path starts from the complete [`reachy_gladiator_lp`](https://github.com/matt-cossins/reachy_gladiator_lp) project instead of asking you to create every file from scratch. The simulation host only needs a launcher script, but the Raspberry Pi will clone and run the full project. You will inspect the different parts of the system so you can recreate your own apps running on Reachy or in simulation.
56
+
This Learning Path starts from the complete [`reachy_gladiator_lp`](https://github.com/matt-cossins/reachy_gladiator_lp) project, so that you don't have to create every file from scratch. The simulation host needs only a launcher script, but the Raspberry Pi will clone and run the full project. You'll inspect the different parts of the system so you can recreate your own apps running on Reachy or in simulation.
56
57
57
58
The Reachy Gladiator app runs a repeated loop:
58
59
59
60
1. Start with a 10-second preparing countdown
60
61
2. Randomly pick one gladiator move
61
62
3. Repeat the selected move three times
62
63
4. Return Reachy to a neutral pose
63
-
5. Watch for a 👍 or a 👎
64
+
5. Watch for a thumbs up or a thumbs down
64
65
6. Run a victory or defeat reaction
65
66
7. Repeat with another move
66
67
67
68
There are four preset moves:
68
69
69
70
-`Salute`
70
-
![Salute Move#center](salute.gif"Salute Move")
71
+
![Animated MuJoCo simulation showing Reachy Mini performing the Salute preset move. This shows one of the scripted motions that the app can select during a round.#center](salute.gif"Salute move")
![Animated MuJoCo simulation showing Reachy Mini performing the Sword Swing preset move. This demonstrates another scripted gladiator motion sent through the Reachy Mini SDK.#center](sword.gif"Sword swing move")
73
74
-`Shield Up`
74
-
![Shield Up Move#center](shield.gif"Shield Up Move")
75
+
![Animated MuJoCo simulation showing Reachy Mini performing the Shield Up preset move. This confirms that the app can play defensive-themed robot motions in simulation.#center](shield.gif"Shield up move")
![Animated MuJoCo simulation showing Reachy Mini performing the Battle Cry preset move. This shows the expressive motion style used before the thumbs-up or thumbs-down verdict.#center](battle-cry.gif"Battle cry move")
77
78
78
79
The app shuffles all four moves and performs each once before any move repeats. When the bag is empty, it reshuffles and avoids repeating the same move at the shuffle boundary.
79
80
80
-
## Use a compatible terminal
81
+
## Compatible terminals for this project
81
82
82
-
The commands in this learning path use a Bash-style shell. They work directly on macOS and Linux.
83
+
The commands in this learning path use a Bash-style shell and work directly on macOS and Linux.
83
84
84
85
On Windows, use WSL2 with an Ubuntu distribution for the simulation host commands. This keeps the commands almost identical to the macOS and Linux flow:
85
86
86
87
- Use the WSL terminal for `python3`, `source .venv/bin/activate`, and `./scripts/start_sim.sh`.
87
88
- Keep the browser on Windows if you prefer; open the Pi dashboard from any browser that can reach the Raspberry Pi.
88
-
- If the Pi cannot reach a daemon running inside WSL2, check Windows firewall and WSL networking. WSL2 uses virtualized networking, so inbound access from another device on your LAN may require Windows port forwarding.
89
+
- If the Pi can't reach a daemon running inside WSL2, check Windows firewall and WSL networking. WSL2 uses virtualized networking, so inbound access from another device on your LAN might require Windows port forwarding.
89
90
90
-
The rest of this learning path shows the common Bash commands and calls out the one IP-address command that differs by host operating system.
91
+
In the rest of this learning path, you'll see common Bash commands and note the one IP-address command that differs by host operating system.
91
92
92
93
## Project structure
93
94
94
-
The simulation host does not need the full project checkout. In the next
95
-
section, you will download only the simulation launcher script on that machine.
95
+
The simulation host doesn't need the full project checkout. In the next
96
+
section, you'll download only the simulation launcher script on that machine.
96
97
97
-
You will clone the full `reachy_gladiator_lp` project on the Raspberry Pi later.
98
+
You'll clone the full `reachy_gladiator_lp` project on the Raspberry Pi later.
98
99
99
100
The key files are:
100
101
@@ -116,6 +117,8 @@ reachy_gladiator_lp/
116
117
└── static/ # dashboard HTML, CSS, JavaScript, and media
117
118
```
118
119
119
-
## What you learned and what is next
120
+
## What you've learned and what's next
120
121
121
-
You learned what Reachy Mini is, why simulation is useful for edge/physical AI development, and how the app splits work between a simulation host and a Raspberry Pi. You are now ready to start the simulation host with a lightweight launcher script.
122
+
You learned what Reachy Mini is, why simulation is useful for edge or physical AI development, and how the app splits work between a simulation host and a Raspberry Pi.
123
+
124
+
Next, you'll start the simulation host with a lightweight launcher script.
0 commit comments