Skip to content

Commit e476cdf

Browse files
Corning-AIclaude
andcommitted
Add device tree explainer, camera+NPU plan, and multimedia build script
- docs/08: Plain-language walkthrough of imx8mp-evk.dts structure (I2C nodes, pinctrl, peripherals) with i2cdetect correlation - docs/07: Camera + NPU real-time object detection demo plan - scripts/build-multimedia.sh: Yocto build helper for imx-image-multimedia - procurement/bom.md: Fix formatting and correct camera connector to J12 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent bd96e16 commit e476cdf

4 files changed

Lines changed: 682 additions & 21 deletions

File tree

docs/07-camera-npu.md

Lines changed: 275 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,275 @@
1+
# 07 — Camera + NPU: Real-time Edge AI Object Detection
2+
3+
## Goal
4+
5+
Build a real-time object detection demo using the OV5640 MIPI CSI camera, the onboard Verisilicon VIP8000 NPU (2.3 TOPS INT8), and an HDMI display. This showcases the full heterogeneous processing capability of the i.MX 8M Plus: Cortex-A53 Linux for application logic, NPU for inference acceleration, and Cortex-M7 FreeRTOS for real-time coordination.
6+
7+
<!-- 目标:用 OV5640 摄像头 + NPU 做实时物体检测,HDMI 显示结果,展示异构 SoC 全栈能力 -->
8+
9+
## Demo Architecture
10+
11+
```text
12+
OV5640 Camera (MIPI CSI, J12)
13+
14+
15+
ISI / V4L2 capture
16+
17+
18+
GStreamer pipeline (frame acquisition)
19+
20+
21+
TFLite + VX Delegate ──→ NPU (MobileNet SSD, INT8)
22+
23+
24+
Overlay: bounding boxes + labels + FPS + NPU latency
25+
26+
27+
Wayland/Weston ──→ HDMI Display (J7)
28+
29+
┌──────────────────────────────────────┐
30+
│ Cortex-M7 (FreeRTOS) │
31+
│ └── heartbeat via RPMsg │
32+
│ → shown on display overlay │
33+
└──────────────────────────────────────┘
34+
```
35+
36+
## HDMI Output Layout
37+
38+
```text
39+
┌─────────────────────────────────────────────────────────┐
40+
│ ┌───────────────────────────────────┐ ┌────────────┐ │
41+
│ │ │ │ NPU: 8ms │ │
42+
│ │ Live Camera Feed │ │ CPU: 180ms │ │
43+
│ │ + Bounding Boxes │ │ FPS: 33 │ │
44+
│ │ + Labels + Confidence │ │ │ │
45+
│ │ │ │ M7: alive │ │
46+
│ │ [phone 92%] [cup 87%] │ │ uptime: 42s │ │
47+
│ │ │ └────────────┘ │
48+
│ └───────────────────────────────────┘ │
49+
└─────────────────────────────────────────────────────────┘
50+
```
51+
52+
## Skills Demonstrated
53+
54+
| Skill | How It's Shown |
55+
| ----- | -------------- |
56+
| Embedded Linux BSP | Custom Yocto image, kernel config, device tree |
57+
| Camera / video pipeline | MIPI CSI → ISI → V4L2 → GStreamer |
58+
| Hardware acceleration | TFLite + VX Delegate → NPU (2.3 TOPS) |
59+
| Real-time systems | FreeRTOS on M7, hard real-time heartbeat |
60+
| Inter-core communication | RPMsg between A53 (Linux) and M7 (FreeRTOS) |
61+
| Performance engineering | CPU vs NPU benchmark, FPS optimization |
62+
| System integration | End-to-end: camera → compute → display |
63+
64+
## Hardware
65+
66+
| Item | Connection | Notes |
67+
| ---- | ---------- | ----- |
68+
| OV5640 + MINISASTOCSI adapter | J12 (CSI1 MIPI) | Official NXP camera module; mini-SAS cable connection |
69+
| HDMI display + cable | J7 | Any HDMI monitor |
70+
| Ethernet cable | J10 or J11A | For file transfer (scp, wget) |
71+
72+
## Step-by-step
73+
74+
### Step 1 — Connect camera and display
75+
76+
1. Power off EVK (SW3 → OFF)
77+
2. Connect OV5640 camera to **J12** (CSI1 MIPI) via MINISASTOCSI adapter
78+
- Use the mini-SAS cable (included with the MINISASTOCSI kit)
79+
- Plug into J12 until the latch clicks
80+
3. Connect HDMI cable from **J7** to display
81+
4. Connect Ethernet cable to **J10**
82+
5. Power on, login as `root`
83+
84+
### Step 2 — Verify camera detection
85+
86+
```bash
87+
dmesg | grep -i ov5640
88+
ls /dev/video*
89+
v4l2-ctl --list-devices
90+
v4l2-ctl -d /dev/video0 --list-formats-ext
91+
```
92+
93+
Expected output: OV5640 detected on I2C1 (address 0x3c), `/dev/video0` created, supports YUYV/NV12 up to 1920x1080.
94+
95+
If `ov5640_check_chip_id: failed` appears, the camera isn't physically connected or the mini-SAS cable is loose.
96+
97+
### Step 3 — Test live preview on HDMI
98+
99+
```bash
100+
# Camera → HDMI display
101+
gst-launch-1.0 v4l2src device=/dev/video0 ! \
102+
video/x-raw,width=640,height=480,framerate=30/1 ! \
103+
waylandsink
104+
105+
# Alternative: NXP's optimized source element
106+
gst-launch-1.0 imxv4l2src device=/dev/video0 ! \
107+
video/x-raw,width=640,height=480 ! \
108+
waylandsink
109+
```
110+
111+
If Weston is not running: `systemctl start weston@root`
112+
113+
### Step 4 — Verify NPU stack
114+
115+
```bash
116+
# NPU kernel driver
117+
lsmod | grep galcore
118+
119+
# VX Delegate library
120+
ls /usr/lib/libvx_delegate.so
121+
122+
# TFLite runtime
123+
python3 -c "import tflite_runtime; print('TFLite OK')"
124+
125+
# Benchmark tool
126+
ls /usr/bin/tensorflow-lite-*/tools/benchmark_model
127+
```
128+
129+
### Step 5 — Get network access and download model
130+
131+
```bash
132+
# DHCP on eth0
133+
dhclient eth0 && ip addr show eth0
134+
135+
# Create directories
136+
mkdir -p /opt/models /opt/camera-detect
137+
138+
# Download MobileNet SSD v1 (COCO dataset, INT8 quantized)
139+
cd /opt/models
140+
wget https://storage.googleapis.com/download.tensorflow.org/models/tflite/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip
141+
unzip coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip
142+
```
143+
144+
If no internet on the board, download on PC and transfer:
145+
146+
```bash
147+
scp detect.tflite root@<EVK_IP>:/opt/models/
148+
```
149+
150+
### Step 6 — Benchmark NPU vs CPU
151+
152+
```bash
153+
BENCH=/usr/bin/tensorflow-lite-*/tools/benchmark_model
154+
MODEL=/opt/models/detect.tflite
155+
156+
# CPU-only inference
157+
$BENCH --graph=$MODEL --num_runs=50 --warmup_runs=5
158+
159+
# NPU-accelerated inference
160+
$BENCH --graph=$MODEL \
161+
--external_delegate_path=/usr/lib/libvx_delegate.so \
162+
--num_runs=50 --warmup_runs=5
163+
```
164+
165+
Expected results:
166+
167+
| Backend | Inference Latency | Notes |
168+
| ------- | ----------------- | ----- |
169+
| CPU (A53) | ~180–200 ms | 4 threads |
170+
| NPU (VX Delegate) | ~8–12 ms | INT8 quantized |
171+
| Speedup | ~20x | |
172+
173+
### Step 7 — Check for NXP pre-built demos
174+
175+
```bash
176+
find / -name "*eiq*" -o -name "*detection*" -o -name "*classify*" 2>/dev/null | head -20
177+
gst-inspect-1.0 | grep -i tensor
178+
ls /opt/gopoint* /usr/share/nxp* 2>/dev/null
179+
```
180+
181+
NXP's Yocto image may ship demo applications we can adapt instead of writing from scratch.
182+
183+
### Step 8 — Build the detection application
184+
185+
Python script: `/opt/camera-detect/detect_camera.py`
186+
187+
Functionality:
188+
189+
1. Capture frames from OV5640 via V4L2 / GStreamer
190+
2. Run MobileNet SSD on NPU via TFLite + VX Delegate
191+
3. Draw bounding boxes, class labels, confidence scores
192+
4. Overlay performance metrics: FPS, NPU vs CPU latency
193+
5. Show M7 heartbeat status (from RPMsg)
194+
6. Display on HDMI via Wayland
195+
196+
Key dependencies:
197+
198+
- `tflite_runtime` — inference with VX Delegate
199+
- GStreamer Python (`gi.repository.Gst`) — camera capture + display
200+
- `PIL` or `cv2` — drawing overlays
201+
202+
### Step 9 — Add M7 FreeRTOS heartbeat via RPMsg
203+
204+
On the Ubuntu build host, build the RPMsg echo example:
205+
206+
```bash
207+
cd mcux-sdk-examples/boards/evkmimx8mp/multicore_examples/rpmsg_lite_str_echo_rtos/armgcc
208+
export ARMGCC_DIR=/opt/arm-gnu-toolchain-13.3.rel1-x86_64-arm-none-eabi
209+
./build_release.sh
210+
```
211+
212+
Transfer and load on the EVK:
213+
214+
```bash
215+
cp rpmsg_lite_str_echo_rtos.elf /lib/firmware/
216+
echo rpmsg_lite_str_echo_rtos.elf > /sys/class/remoteproc/remoteproc0/firmware
217+
echo start > /sys/class/remoteproc/remoteproc0/state
218+
219+
# Verify
220+
cat /sys/class/remoteproc/remoteproc0/state # should say "running"
221+
ls /dev/ttyRPMSG* # should see ttyRPMSG0
222+
echo "ping" > /dev/ttyRPMSG0
223+
cat /dev/ttyRPMSG0 # should echo back
224+
```
225+
226+
The detection application reads M7 status from `/dev/ttyRPMSG0` and displays it on the HDMI overlay.
227+
228+
### Step 10 — Record demo video
229+
230+
1. Run the detection app
231+
2. Point camera at desk objects (phone, cup, keyboard, book)
232+
3. Move objects to show real-time tracking
233+
4. Record the HDMI display with a phone or capture card
234+
5. Upload as GIF or video to the GitHub repo
235+
236+
### Step 11 — Document results
237+
238+
Update this file with:
239+
240+
- Actual benchmark numbers
241+
- Screenshots / GIF of the running demo
242+
- Any issues encountered and workarounds
243+
244+
## File locations
245+
246+
**On the EVK:**
247+
248+
| What | Path |
249+
| ---- | ---- |
250+
| Detection model | `/opt/models/detect.tflite` |
251+
| COCO labels | `/opt/models/coco_labels.txt` |
252+
| Detection app | `/opt/camera-detect/detect_camera.py` |
253+
| VX Delegate | `/usr/lib/libvx_delegate.so` |
254+
| M7 firmware | `/lib/firmware/rpmsg_lite_str_echo_rtos.elf` |
255+
256+
**In the repo:**
257+
258+
| What | Path |
259+
| ---- | ---- |
260+
| This document | `docs/07-camera-npu.md` |
261+
| Detection app source | `app/camera-detect/detect_camera.py` |
262+
| Model download script | `scripts/download_model.sh` |
263+
| Demo video / GIF | `docs/assets/demo-detection.gif` |
264+
265+
## Troubleshooting
266+
267+
| Problem | Solution |
268+
| ------- | -------- |
269+
| OV5640 not detected | Re-seat mini-SAS cable on J12; check `dmesg` for I2C errors on bus 1 |
270+
| No video on HDMI | `systemctl status weston`; re-seat HDMI adapter board |
271+
| `waylandsink` fails | Try `fbdevsink` or run as root |
272+
| OpenCV not available | Use GStreamer Python bindings; or rebuild Yocto with `opencv` |
273+
| No internet for model download | Download on PC, `scp` to board |
274+
| VX Delegate fails | Check `lsmod \| grep galcore`; try `modprobe galcore` |
275+
| `/dev/ttyRPMSG*` missing | Check device tree rpmsg node and reserved memory regions |

0 commit comments

Comments
 (0)