Skip to content

Commit 2e28239

Browse files
lxowallelxowalle
committed
* add picoclaw doc
1 parent 179a28c commit 2e28239

4 files changed

Lines changed: 251 additions & 1 deletion

File tree

‎docs/doc/en/basic/picoclaw.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
---
2+
title: Using PicoClaw with MaixCAM / MaixCAM2
3+
---
4+
5+
## What Is PicoClaw
6+
7+
[PicoClaw](https://github.com/sipeed/picoclaw) is a lightweight open-source AI Agent project initiated by Sipeed. It is written in Go and can run on PCs, MaixCAM / MaixCAM2, Raspberry Pi, LicheeRV-Nano, and other Linux devices.
8+
9+
After PicoClaw is running on MaixCAM / MaixCAM2, you can ask it from the PicoClaw conversation UI to operate the device itself, such as creating and running MaixPy scripts, opening the camera, running a YOLO model, reading the detection result, and replying to you.
10+
11+
## Preparation
12+
13+
1. Connect MaixCAM / MaixCAM2 to the network.
14+
2. Install the `picoclaw` executable for the current device architecture according to the official PicoClaw documentation. See [PicoClaw documentation](https://docs.picoclaw.io/).
15+
3. Configure the model provider, API key, and other PicoClaw settings according to the PicoClaw documentation.
16+
17+
You can check the device architecture in the MaixCAM / MaixCAM2 terminal:
18+
19+
```bash
20+
uname -m
21+
```
22+
23+
After configuration, run a simple test on the device:
24+
25+
```bash
26+
picoclaw agent -m "Hello, reply with one sentence to confirm you are running on this device."
27+
```
28+
29+
You can also use the web UI or terminal UI provided by PicoClaw. Follow the PicoClaw documentation for the exact usage.
30+
31+
## Run One Person Detection Through PicoClaw
32+
33+
The workflow of this example is:
34+
35+
1. You send a task in the PicoClaw conversation UI.
36+
2. PicoClaw creates a MaixPy script on the MaixCAM / MaixCAM2 device.
37+
3. PicoClaw runs the script.
38+
4. The script opens the camera and uses YOLO to detect `person`.
39+
5. If a person is detected, the script prints the detection result and saves an annotated image. PicoClaw then replies to you according to the output.
40+
41+
You can send this prompt to PicoClaw:
42+
43+
```txt
44+
Refer to https://wiki.sipeed.com/maixpy/doc/en/index.html, create /root/picoclaw_person_detect.py on the current device, and do the following:
45+
1. Write a MaixPy program: open the camera, use a YOLO model to detect person, and run for at most 15 seconds.
46+
2. If person is detected, save the annotated image to /root/person_detected.jpg, and output PICOCLAW_PERSON_DETECTED, confidence, and coordinates; if no person is detected, output PICOCLAW_NO_PERSON.
47+
3. Then run python /root/picoclaw_person_detect.py and notify me according to the output.
48+
```
49+
50+
After thinking, PicoClaw generates a script like the following. It is recommended to choose a capable model; otherwise, it may generate incorrect code:
51+
52+
```python
53+
from maix import camera, display, image, nn, app, time
54+
55+
MODEL_PATH = "/root/models/yolov5s.mud"
56+
TIMEOUT_MS = 15000
57+
OUT_IMAGE = "/root/person_detected.jpg"
58+
59+
detector = nn.YOLOv5(model=MODEL_PATH, dual_buff=True)
60+
# To use a YOLO11 model, change it to:
61+
# detector = nn.YOLO11(model="/root/models/yolo11n.mud", dual_buff=True)
62+
63+
cam = camera.Camera(detector.input_width(), detector.input_height(), detector.input_format())
64+
disp = display.Display()
65+
66+
start = time.ticks_ms()
67+
detected = False
68+
69+
while not app.need_exit():
70+
img = cam.read()
71+
objs = detector.detect(img, conf_th=0.5, iou_th=0.45)
72+
73+
for obj in objs:
74+
class_name = detector.labels[obj.class_id]
75+
img.draw_rect(obj.x, obj.y, obj.w, obj.h, color=image.COLOR_RED)
76+
img.draw_string(obj.x, obj.y, f"{class_name}: {obj.score:.2f}", color=image.COLOR_RED)
77+
78+
if class_name == "person":
79+
img.save(OUT_IMAGE)
80+
print(
81+
"PICOCLAW_PERSON_DETECTED "
82+
f"score={obj.score:.2f} "
83+
f"x={obj.x} y={obj.y} w={obj.w} h={obj.h} "
84+
f"image={OUT_IMAGE}"
85+
)
86+
detected = True
87+
break
88+
89+
disp.show(img)
90+
91+
if detected:
92+
break
93+
if time.ticks_ms() - start > TIMEOUT_MS:
94+
break
95+
96+
if not detected:
97+
print(f"PICOCLAW_NO_PERSON timeout_ms={TIMEOUT_MS}")
98+
```
99+
100+
If PicoClaw sees output similar to:
101+
102+
```txt
103+
PICOCLAW_PERSON_DETECTED score=0.86 x=120 y=60 w=80 h=180 image=/root/person_detected.jpg
104+
```
105+
106+
it means a person was detected, and the annotated image is saved to `/root/person_detected.jpg`.
107+
108+
## Model Path
109+
110+
The example uses the common MaixCAM model path by default:
111+
112+
```python
113+
MODEL_PATH = "/root/models/yolov5s.mud"
114+
```
115+
116+
If you are using MaixCAM2, or if a YOLO11 model is already available on your system, change it to:
117+
118+
```python
119+
MODEL_PATH = "/root/models/yolo11n.mud"
120+
detector = nn.YOLO11(model=MODEL_PATH, dual_buff=True)
121+
```
122+
123+
The actual model path depends on the files in `/root/models/` on your device. For more YOLO usage, see [YOLO object detection](../vision/yolov5.md).

‎docs/doc/en/sidebar.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ items:
2323
label: Auto start
2424
- file: basic/view_src_code.md
2525
label: View API source code
26+
- file: basic/picoclaw.md
27+
label: PicoClaw usage
2628

2729
- label: Basic images and algorithms
2830
items:
@@ -262,4 +264,3 @@ items:
262264
- file: pro/compile_os.md
263265
label: Build OS
264266

265-

‎docs/doc/zh/basic/picoclaw.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
title: MaixCAM / MaixCAM2 使用 PicoClaw
3+
---
4+
5+
## PicoClaw 是什么
6+
7+
[PicoClaw](https://github.com/sipeed/picoclaw) 是 Sipeed 发起的轻量级开源 AI Agent 项目，使用 Go 编写，可以运行在 PC、MaixCAM / MaixCAM2、树莓派、LicheeRV-Nano 等 Linux 设备上。
8+
9+
在 MaixCAM / MaixCAM2 中运行 PicoClaw 后，你可以在 PicoClaw 的对话界面里让它直接操作设备本机，例如创建并运行 MaixPy 脚本、调用摄像头、运行 YOLO 模型、读取检测结果，然后把结果回复给你。
10+
11+
## 使用前准备
12+
13+
1. 让 MaixCAM / MaixCAM2 连接网络。
14+
2. 根据 PicoClaw 官方文档安装适合当前设备架构的 `picoclaw` 可执行文件，参考 [PicoClaw 文档](https://docs.picoclaw.io/)。
15+
3. 按 PicoClaw 文档完成模型供应商、API Key 等配置。
16+
17+
设备架构可以在 MaixCAM / MaixCAM2 终端中查看：
18+
19+
```bash
20+
uname -m
21+
```
22+
23+
配置完成后，可以先在设备上运行一次简单对话确认 PicoClaw 可用：
24+
25+
```bash
26+
picoclaw agent -m "你好，请回复一句话说明你已经在这台设备上运行"
27+
```
28+
29+
也可以使用 PicoClaw 提供的网页或终端界面进行交互，具体以 PicoClaw 官方文档为准。
30+
31+
## 通过 PicoClaw 运行一次人体检测
32+
33+
下面示例的思路是：
34+
35+
1. 你在 PicoClaw 对话界面中提出任务。
36+
2. PicoClaw 在 MaixCAM / MaixCAM2 本机创建一个 MaixPy 脚本。
37+
3. PicoClaw 执行这个脚本。
38+
4. 脚本打开摄像头，使用 YOLO 检测 `person`。
39+
5. 如果检测到人体，脚本打印检测结果并保存一张标注图，PicoClaw 根据输出提醒你。
40+
41+
你可以把下面这段话发给 PicoClaw：
42+
43+
44+
```txt
45+
请参考https://wiki.sipeed.com/maixpy/doc/en/index.html, 在当前创建 /root/picoclaw_person_detect.py, 并执行以下操作:
46+
1. 写入一个 MaixPy 程序：打开摄像头，使用 YOLO 模型检测 person，最多运行 15 秒。
47+
2. 如果检测到 person，请保存标注图到 /root/person_detected.jpg，并输出 PICOCLAW_PERSON_DETECTED、置信度和坐标；如果没有检测到，请输出 PICOCLAW_NO_PERSON。
48+
3. 然后执行 python /root/picoclaw_person_detect.py，并根据输出结果通知我。
49+
```
50+
51+
PicoClaw经过思考后, 生成的脚本如下(注意建议选一些能力强的模型, 否则有可能生成一些错误代码):
52+
53+
```python
54+
from maix import camera, display, image, nn, app, time
55+
56+
MODEL_PATH = "/root/models/yolov5s.mud"
57+
TIMEOUT_MS = 15000
58+
OUT_IMAGE = "/root/person_detected.jpg"
59+
60+
detector = nn.YOLOv5(model=MODEL_PATH, dual_buff=True)
61+
# 如果使用 YOLO11 模型，可以改成：
62+
# detector = nn.YOLO11(model="/root/models/yolo11n.mud", dual_buff=True)
63+
64+
cam = camera.Camera(detector.input_width(), detector.input_height(), detector.input_format())
65+
disp = display.Display()
66+
67+
start = time.ticks_ms()
68+
detected = False
69+
70+
while not app.need_exit():
71+
img = cam.read()
72+
objs = detector.detect(img, conf_th=0.5, iou_th=0.45)
73+
74+
for obj in objs:
75+
class_name = detector.labels[obj.class_id]
76+
img.draw_rect(obj.x, obj.y, obj.w, obj.h, color=image.COLOR_RED)
77+
img.draw_string(obj.x, obj.y, f"{class_name}: {obj.score:.2f}", color=image.COLOR_RED)
78+
79+
if class_name == "person":
80+
img.save(OUT_IMAGE)
81+
print(
82+
"PICOCLAW_PERSON_DETECTED "
83+
f"score={obj.score:.2f} "
84+
f"x={obj.x} y={obj.y} w={obj.w} h={obj.h} "
85+
f"image={OUT_IMAGE}"
86+
)
87+
detected = True
88+
break
89+
90+
disp.show(img)
91+
92+
if detected:
93+
break
94+
if time.ticks_ms() - start > TIMEOUT_MS:
95+
break
96+
97+
if not detected:
98+
print(f"PICOCLAW_NO_PERSON timeout_ms={TIMEOUT_MS}")
99+
```
100+
101+
如果 PicoClaw 执行后看到类似输出：
102+
103+
```txt
104+
PICOCLAW_PERSON_DETECTED score=0.86 x=120 y=60 w=80 h=180 image=/root/person_detected.jpg
105+
```
106+
107+
就表示已经检测到人体，并且标注图片保存在 `/root/person_detected.jpg`。
108+
109+
## 模型路径说明
110+
111+
示例默认使用 MaixCAM 常见的模型路径：
112+
113+
```python
114+
MODEL_PATH = "/root/models/yolov5s.mud"
115+
```
116+
117+
如果你的设备是 MaixCAM2，或者系统中已经准备了 YOLO11 模型，可以改成：
118+
119+
```python
120+
MODEL_PATH = "/root/models/yolo11n.mud"
121+
detector = nn.YOLO11(model=MODEL_PATH, dual_buff=True)
122+
```
123+
124+
实际可用模型以设备 `/root/models/` 目录为准。更多 YOLO 用法请看 [YOLO 物体检测](../vision/yolov5.md)。

‎docs/doc/zh/sidebar.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ items:
2323
label: 应用开机自启
2424
- file: basic/view_src_code.md
2525
label: 查看 API 对应源码
26+
- file: basic/picoclaw.md
27+
label: PicoClaw 使用
2628

2729
- label: 基础图像和算法
2830
items:

0 commit comments

Comments
 (0)