You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_index.md
+2-6Lines changed: 2 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,9 +1,5 @@
1
1
---
2
-
title: Build an Edge AI Reachy Mini App with Raspberry Pi, MediaPipe, and MuJoCo
3
-
4
-
draft: true
5
-
cascade:
6
-
draft: true
2
+
title: Build an edge AI Reachy Mini app with Raspberry Pi, MediaPipe, and MuJoCo
7
3
8
4
description: Run MediaPipe gesture inference on a Raspberry Pi 5, connect to a Reachy Mini MuJoCo simulation on a development machine, and use a browser dashboard to decide Reachy's fate with a thumbs-up or thumbs-down.
9
5
@@ -17,7 +13,7 @@ learning_objectives:
17
13
- Use MediaPipe and TensorFlow Lite gesture recognition on Raspberry Pi 5.
18
14
- Connect an edge inference node to a robot daemon over the network.
19
15
- Display results over a web dashboard.
20
-
- (Optional) Extend the project toward a physical Reachy Mini, audio or multimodal interaction, or your own app.
16
+
- Optionally extend the project toward a physical Reachy Mini, audio or multimodal interaction, or your own app.
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/extend.md
+26-25Lines changed: 26 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,6 @@
1
1
---
2
2
title: (Optional) Extend the project
3
+
description: Explore approaches you can take to extend Reachy Gladiator with new gestures, audio output, physical Reachy Mini support, and custom Reachy Mini app packaging.
3
4
weight: 7
4
5
5
6
### FIXED, DO NOT MODIFY
@@ -25,23 +26,23 @@ motion, or packaging.
25
26
26
27
## Change the classifier
27
28
28
-
Trying adding new gesture controls. Start in `gesture.py`, where the app maps MediaPipe labels such as `Thumb_Up` and `Thumb_Down` to app labels. Then add matching robot behavior in `moves.py` and branch on the new label in `main.py`.
29
+
Try adding new gesture controls. Start in `gesture.py`, where the app maps MediaPipe labels such as `Thumb_Up` and `Thumb_Down` to app labels. Then add matching robot behavior in `moves.py` and branch on the new label in `main.py`.
29
30
30
31
Some game-themed ideas:
31
32
32
33
- Map `Closed_Fist` to `challenge`, and make Reachy repeat the current move
33
34
- Map `Pointing_Up` to `reroll`, and make Reachy reject the current move and
34
35
choose another one.
35
-
- Map `Number of fingers`to choosing a specific move.
36
+
- Map the number of fingers shown to choose a specific move.
36
37
37
38
## Add audio output
38
39
39
40
The gladiator theme is a good fit for sound. Try adding audio cues such as:
40
41
41
-
- a crowd cheer during victory,
42
-
- a dramatic sound during defeat,
43
-
- a short drum hit before each move,
44
-
- a spoken move name before Reachy performs it.
42
+
- a crowd cheer during victory
43
+
- a dramatic sound during defeat
44
+
- a short drum hit before each move
45
+
- a spoken move name before Reachy performs it
45
46
46
47
Keep audio output separate from `moves.py` at first. For example, create an
47
48
`audio.py` helper and call it from `main.py` when the state changes. This keeps
@@ -50,36 +51,37 @@ robot motion and sound effects easy to change independently.
50
51
## Replace thumbs with audio input
51
52
52
53
The vision-based verdict is one edge AI input modality. You can replace
53
-
or complement it with audio - many webcams include microphones or you can use a USB microphone.
54
+
or complement it with audio. Many webcams include microphones, or you can use a USB microphone.
54
55
55
-
- say "yes" for victory and "no" for defeat,
56
-
- clap once for victory and twice for defeat,
56
+
Try adding audio inputs such as:
57
+
58
+
- say "yes" for victory and "no" for defeat
59
+
- clap once for victory and twice for defeat
57
60
58
61
A lightweight keyword-spotting model can map spoken commands to the same game states currently triggered by MediaPipe gestures.
59
62
60
63
## Try the packaged app on a physical Reachy
61
64
62
-
If you have a physical Reachy Mini, the quickest way to try the finished
63
-
experience is to install the packaged [Reachy Gladiator app](https://huggingface.co/spaces/cossinsmatthew/reachy_gladiator) through the [Reachy Mini Control app](https://github.com/pollen-robotics/reachy-mini-desktop-app).
65
+
If you have a physical Reachy Mini, you can try the finished
66
+
experience by installing the packaged [Reachy Gladiator app](https://huggingface.co/spaces/cossinsmatthew/reachy_gladiator) through the [Reachy Mini Control app](https://github.com/pollen-robotics/reachy-mini-desktop-app).
64
67
65
-
Simply install Reachy Mini Control on a supported machine, connected to your Reachy, and search for the reachy gladiator app.
68
+
Install Reachy Mini Control on a supported machine, connect it to your Reachy, and search for the Reachy Gladiator app.
66
69
67
70
{{% notice Warning %}}
68
-
If using a physical Reachy Mini, use caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health & safety risk. You are responsible for your safety and the safety of others around you when using physical robotic devices
71
+
If you use a physical Reachy Mini, exercise caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health and safety risk. You're responsible for your safety and the safety of others around you when using physical robotic devices.
69
72
{{% /notice %}}
70
73
71
74
## Adapt this source project for physical Reachy
72
75
73
-
The main learning path uses the Raspberry Pi USB webcam for perception and a
76
+
The main Learning Path uses the Raspberry Pi USB webcam for perception and a
74
77
remote MuJoCo daemon for robot motion. A physical Reachy route changes two
75
78
things:
76
79
77
-
- camera frames come from the Reachy daemon instead of the Pi USB webcam,
80
+
- camera frames come from the Reachy daemon instead of the Pi USB webcam
78
81
- the Pi app connects to the physical Reachy daemon instead of the simulation
79
-
daemon.
82
+
daemon
80
83
81
-
The source project exposes these switches as environment variables, so you do
82
-
not need to edit the Python source:
84
+
The source project exposes these switches as environment variables, so you don't need to edit the Python source:
83
85
84
86
```bash
85
87
REACHY_GLADIATOR_MEDIA_BACKEND=reachy \
@@ -95,9 +97,9 @@ IP address and set `REACHY_GLADIATOR_DAEMON_PORT` to the daemon port.
The [Reachy Mini app publishing guide](https://huggingface.co/blog/pollen-robotics/make-and-publish-your-reachy-mini-apps)
124
-
explains the packaging and publishing workflow in more detail.
125
+
For more information about the packaging and publishing workflow, see the [Reachy Mini app publishing guide](https://huggingface.co/blog/pollen-robotics/make-and-publish-your-reachy-mini-apps).
125
126
126
127
Use the Reachy Mini SDK documentation and examples to understand available
127
128
motion, media, and daemon APIs. If you use an AI coding agent, give it the
128
129
Pollen Robotics `AGENTS.md` instructions, provided by the [Reachy Mini project](https://github.com/pollen-robotics/reachy_mini) so it follows the expected app structure.
129
130
130
-
## What you learned
131
+
## What you've learned
131
132
132
-
Youexplored options for extending from simulation to a physical Reachy, as well ideas for changing the project to include audio, new vision gestures, or different behaviours.
133
+
You've now explored options for extending from simulation to a physical Reachy, as well as ideas for changing the project to include audio, new vision gestures, or different behaviors.
0 commit comments