|
| 1 | +# 07 — Camera + NPU: Real-time Edge AI Object Detection |
| 2 | + |
| 3 | +## Goal |
| 4 | + |
| 5 | +Build a real-time object detection demo using the OV5640 MIPI CSI camera, the onboard Verisilicon VIP8000 NPU (2.3 TOPS INT8), and an HDMI display. This showcases the full heterogeneous processing capability of the i.MX 8M Plus: Cortex-A53 Linux for application logic, NPU for inference acceleration, and Cortex-M7 FreeRTOS for real-time coordination. |
| 6 | + |
| 7 | +<!-- 目标:用 OV5640 摄像头 + NPU 做实时物体检测,HDMI 显示结果,展示异构 SoC 全栈能力 --> |
| 8 | + |
| 9 | +## Demo Architecture |
| 10 | + |
| 11 | +```text |
| 12 | +OV5640 Camera (MIPI CSI, J12) |
| 13 | + │ |
| 14 | + ▼ |
| 15 | +ISI / V4L2 capture |
| 16 | + │ |
| 17 | + ▼ |
| 18 | +GStreamer pipeline (frame acquisition) |
| 19 | + │ |
| 20 | + ▼ |
| 21 | +TFLite + VX Delegate ──→ NPU (MobileNet SSD, INT8) |
| 22 | + │ |
| 23 | + ▼ |
| 24 | +Overlay: bounding boxes + labels + FPS + NPU latency |
| 25 | + │ |
| 26 | + ▼ |
| 27 | +Wayland/Weston ──→ HDMI Display (J7) |
| 28 | +
|
| 29 | + ┌──────────────────────────────────────┐ |
| 30 | + │ Cortex-M7 (FreeRTOS) │ |
| 31 | + │ └── heartbeat via RPMsg │ |
| 32 | + │ → shown on display overlay │ |
| 33 | + └──────────────────────────────────────┘ |
| 34 | +``` |
| 35 | + |
| 36 | +## HDMI Output Layout |
| 37 | + |
| 38 | +```text |
| 39 | +┌─────────────────────────────────────────────────────────┐ |
| 40 | +│ ┌───────────────────────────────────┐ ┌────────────┐ │ |
| 41 | +│ │ │ │ NPU: 8ms │ │ |
| 42 | +│ │ Live Camera Feed │ │ CPU: 180ms │ │ |
| 43 | +│ │ + Bounding Boxes │ │ FPS: 33 │ │ |
| 44 | +│ │ + Labels + Confidence │ │ │ │ |
| 45 | +│ │ │ │ M7: alive │ │ |
| 46 | +│ │ [phone 92%] [cup 87%] │ │ uptime: 42s │ │ |
| 47 | +│ │ │ └────────────┘ │ |
| 48 | +│ └───────────────────────────────────┘ │ |
| 49 | +└─────────────────────────────────────────────────────────┘ |
| 50 | +``` |
| 51 | + |
| 52 | +## Skills Demonstrated |
| 53 | + |
| 54 | +| Skill | How It's Shown | |
| 55 | +| ----- | -------------- | |
| 56 | +| Embedded Linux BSP | Custom Yocto image, kernel config, device tree | |
| 57 | +| Camera / video pipeline | MIPI CSI → ISI → V4L2 → GStreamer | |
| 58 | +| Hardware acceleration | TFLite + VX Delegate → NPU (2.3 TOPS) | |
| 59 | +| Real-time systems | FreeRTOS on M7, hard real-time heartbeat | |
| 60 | +| Inter-core communication | RPMsg between A53 (Linux) and M7 (FreeRTOS) | |
| 61 | +| Performance engineering | CPU vs NPU benchmark, FPS optimization | |
| 62 | +| System integration | End-to-end: camera → compute → display | |
| 63 | + |
| 64 | +## Hardware |
| 65 | + |
| 66 | +| Item | Connection | Notes | |
| 67 | +| ---- | ---------- | ----- | |
| 68 | +| OV5640 + MINISASTOCSI adapter | J12 (CSI1 MIPI) | Official NXP camera module; mini-SAS cable connection | |
| 69 | +| HDMI display + cable | J7 | Any HDMI monitor | |
| 70 | +| Ethernet cable | J10 or J11A | For file transfer (scp, wget) | |
| 71 | + |
| 72 | +## Step-by-step |
| 73 | + |
| 74 | +### Step 1 — Connect camera and display |
| 75 | + |
| 76 | +1. Power off EVK (SW3 → OFF) |
| 77 | +2. Connect OV5640 camera to **J12** (CSI1 MIPI) via MINISASTOCSI adapter |
| 78 | + - Use the mini-SAS cable (included with the MINISASTOCSI kit) |
| 79 | + - Plug into J12 until the latch clicks |
| 80 | +3. Connect HDMI cable from **J7** to display |
| 81 | +4. Connect Ethernet cable to **J10** |
| 82 | +5. Power on, login as `root` |
| 83 | + |
| 84 | +### Step 2 — Verify camera detection |
| 85 | + |
| 86 | +```bash |
| 87 | +dmesg | grep -i ov5640 |
| 88 | +ls /dev/video* |
| 89 | +v4l2-ctl --list-devices |
| 90 | +v4l2-ctl -d /dev/video0 --list-formats-ext |
| 91 | +``` |
| 92 | + |
| 93 | +Expected output: OV5640 detected on I2C1 (address 0x3c), `/dev/video0` created, supports YUYV/NV12 up to 1920x1080. |
| 94 | + |
| 95 | +If `ov5640_check_chip_id: failed` appears, the camera isn't physically connected or the mini-SAS cable is loose. |
| 96 | + |
| 97 | +### Step 3 — Test live preview on HDMI |
| 98 | + |
| 99 | +```bash |
| 100 | +# Camera → HDMI display |
| 101 | +gst-launch-1.0 v4l2src device=/dev/video0 ! \ |
| 102 | + video/x-raw,width=640,height=480,framerate=30/1 ! \ |
| 103 | + waylandsink |
| 104 | + |
| 105 | +# Alternative: NXP's optimized source element |
| 106 | +gst-launch-1.0 imxv4l2src device=/dev/video0 ! \ |
| 107 | + video/x-raw,width=640,height=480 ! \ |
| 108 | + waylandsink |
| 109 | +``` |
| 110 | + |
| 111 | +If Weston is not running: `systemctl start weston@root` |
| 112 | + |
| 113 | +### Step 4 — Verify NPU stack |
| 114 | + |
| 115 | +```bash |
| 116 | +# NPU kernel driver |
| 117 | +lsmod | grep galcore |
| 118 | + |
| 119 | +# VX Delegate library |
| 120 | +ls /usr/lib/libvx_delegate.so |
| 121 | + |
| 122 | +# TFLite runtime |
| 123 | +python3 -c "import tflite_runtime; print('TFLite OK')" |
| 124 | + |
| 125 | +# Benchmark tool |
| 126 | +ls /usr/bin/tensorflow-lite-*/tools/benchmark_model |
| 127 | +``` |
| 128 | + |
| 129 | +### Step 5 — Get network access and download model |
| 130 | + |
| 131 | +```bash |
| 132 | +# DHCP on eth0 |
| 133 | +dhclient eth0 && ip addr show eth0 |
| 134 | + |
| 135 | +# Create directories |
| 136 | +mkdir -p /opt/models /opt/camera-detect |
| 137 | + |
| 138 | +# Download MobileNet SSD v1 (COCO dataset, INT8 quantized) |
| 139 | +cd /opt/models |
| 140 | +wget https://storage.googleapis.com/download.tensorflow.org/models/tflite/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip |
| 141 | +unzip coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip |
| 142 | +``` |
| 143 | + |
| 144 | +If no internet on the board, download on PC and transfer: |
| 145 | + |
| 146 | +```bash |
| 147 | +scp detect.tflite root@<EVK_IP>:/opt/models/ |
| 148 | +``` |
| 149 | + |
| 150 | +### Step 6 — Benchmark NPU vs CPU |
| 151 | + |
| 152 | +```bash |
| 153 | +BENCH=/usr/bin/tensorflow-lite-*/tools/benchmark_model |
| 154 | +MODEL=/opt/models/detect.tflite |
| 155 | + |
| 156 | +# CPU-only inference |
| 157 | +$BENCH --graph=$MODEL --num_runs=50 --warmup_runs=5 |
| 158 | + |
| 159 | +# NPU-accelerated inference |
| 160 | +$BENCH --graph=$MODEL \ |
| 161 | + --external_delegate_path=/usr/lib/libvx_delegate.so \ |
| 162 | + --num_runs=50 --warmup_runs=5 |
| 163 | +``` |
| 164 | + |
| 165 | +Expected results: |
| 166 | + |
| 167 | +| Backend | Inference Latency | Notes | |
| 168 | +| ------- | ----------------- | ----- | |
| 169 | +| CPU (A53) | ~180–200 ms | 4 threads | |
| 170 | +| NPU (VX Delegate) | ~8–12 ms | INT8 quantized | |
| 171 | +| Speedup | ~20x | | |
| 172 | + |
| 173 | +### Step 7 — Check for NXP pre-built demos |
| 174 | + |
| 175 | +```bash |
| 176 | +find / -name "*eiq*" -o -name "*detection*" -o -name "*classify*" 2>/dev/null | head -20 |
| 177 | +gst-inspect-1.0 | grep -i tensor |
| 178 | +ls /opt/gopoint* /usr/share/nxp* 2>/dev/null |
| 179 | +``` |
| 180 | + |
| 181 | +NXP's Yocto image may ship demo applications we can adapt instead of writing from scratch. |
| 182 | + |
| 183 | +### Step 8 — Build the detection application |
| 184 | + |
| 185 | +Python script: `/opt/camera-detect/detect_camera.py` |
| 186 | + |
| 187 | +Functionality: |
| 188 | + |
| 189 | +1. Capture frames from OV5640 via V4L2 / GStreamer |
| 190 | +2. Run MobileNet SSD on NPU via TFLite + VX Delegate |
| 191 | +3. Draw bounding boxes, class labels, confidence scores |
| 192 | +4. Overlay performance metrics: FPS, NPU vs CPU latency |
| 193 | +5. Show M7 heartbeat status (from RPMsg) |
| 194 | +6. Display on HDMI via Wayland |
| 195 | + |
| 196 | +Key dependencies: |
| 197 | + |
| 198 | +- `tflite_runtime` — inference with VX Delegate |
| 199 | +- GStreamer Python (`gi.repository.Gst`) — camera capture + display |
| 200 | +- `PIL` or `cv2` — drawing overlays |
| 201 | + |
| 202 | +### Step 9 — Add M7 FreeRTOS heartbeat via RPMsg |
| 203 | + |
| 204 | +On the Ubuntu build host, build the RPMsg echo example: |
| 205 | + |
| 206 | +```bash |
| 207 | +cd mcux-sdk-examples/boards/evkmimx8mp/multicore_examples/rpmsg_lite_str_echo_rtos/armgcc |
| 208 | +export ARMGCC_DIR=/opt/arm-gnu-toolchain-13.3.rel1-x86_64-arm-none-eabi |
| 209 | +./build_release.sh |
| 210 | +``` |
| 211 | + |
| 212 | +Transfer and load on the EVK: |
| 213 | + |
| 214 | +```bash |
| 215 | +cp rpmsg_lite_str_echo_rtos.elf /lib/firmware/ |
| 216 | +echo rpmsg_lite_str_echo_rtos.elf > /sys/class/remoteproc/remoteproc0/firmware |
| 217 | +echo start > /sys/class/remoteproc/remoteproc0/state |
| 218 | + |
| 219 | +# Verify |
| 220 | +cat /sys/class/remoteproc/remoteproc0/state # should say "running" |
| 221 | +ls /dev/ttyRPMSG* # should see ttyRPMSG0 |
| 222 | +echo "ping" > /dev/ttyRPMSG0 |
| 223 | +cat /dev/ttyRPMSG0 # should echo back |
| 224 | +``` |
| 225 | + |
| 226 | +The detection application reads M7 status from `/dev/ttyRPMSG0` and displays it on the HDMI overlay. |
| 227 | + |
| 228 | +### Step 10 — Record demo video |
| 229 | + |
| 230 | +1. Run the detection app |
| 231 | +2. Point camera at desk objects (phone, cup, keyboard, book) |
| 232 | +3. Move objects to show real-time tracking |
| 233 | +4. Record the HDMI display with a phone or capture card |
| 234 | +5. Upload as GIF or video to the GitHub repo |
| 235 | + |
| 236 | +### Step 11 — Document results |
| 237 | + |
| 238 | +Update this file with: |
| 239 | + |
| 240 | +- Actual benchmark numbers |
| 241 | +- Screenshots / GIF of the running demo |
| 242 | +- Any issues encountered and workarounds |
| 243 | + |
| 244 | +## File locations |
| 245 | + |
| 246 | +**On the EVK:** |
| 247 | + |
| 248 | +| What | Path | |
| 249 | +| ---- | ---- | |
| 250 | +| Detection model | `/opt/models/detect.tflite` | |
| 251 | +| COCO labels | `/opt/models/coco_labels.txt` | |
| 252 | +| Detection app | `/opt/camera-detect/detect_camera.py` | |
| 253 | +| VX Delegate | `/usr/lib/libvx_delegate.so` | |
| 254 | +| M7 firmware | `/lib/firmware/rpmsg_lite_str_echo_rtos.elf` | |
| 255 | + |
| 256 | +**In the repo:** |
| 257 | + |
| 258 | +| What | Path | |
| 259 | +| ---- | ---- | |
| 260 | +| This document | `docs/07-camera-npu.md` | |
| 261 | +| Detection app source | `app/camera-detect/detect_camera.py` | |
| 262 | +| Model download script | `scripts/download_model.sh` | |
| 263 | +| Demo video / GIF | `docs/assets/demo-detection.gif` | |
| 264 | + |
| 265 | +## Troubleshooting |
| 266 | + |
| 267 | +| Problem | Solution | |
| 268 | +| ------- | -------- | |
| 269 | +| OV5640 not detected | Re-seat mini-SAS cable on J12; check `dmesg` for I2C errors on bus 1 | |
| 270 | +| No video on HDMI | `systemctl status weston`; re-seat HDMI adapter board | |
| 271 | +| `waylandsink` fails | Try `fbdevsink` or run as root | |
| 272 | +| OpenCV not available | Use GStreamer Python bindings; or rebuild Yocto with `opencv` | |
| 273 | +| No internet for model download | Download on PC, `scp` to board | |
| 274 | +| VX Delegate fails | Check `lsmod \| grep galcore`; try `modprobe galcore` | |
| 275 | +| `/dev/ttyRPMSG*` missing | Check device tree rpmsg node and reserved memory regions | |
0 commit comments