Skip to content

Commit d93385c

Browse files
Corning-AIclaude
andcommitted
Update camera+NPU docs with verified results, add serial transfer scripts
- Fix HDMI port (J7→J17), video device (/dev/video0→/dev/video3) - Add verified NPU benchmark: CPU 45.3ms vs NPU 9.1ms (5x speedup) - Document correct Weston service name and video device mapping - Add two serial file transfer utilities (base64 heredoc + Python receiver) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent e476cdf commit d93385c

3 files changed

Lines changed: 314 additions & 31 deletions

File tree

docs/07-camera-npu.md

Lines changed: 61 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ TFLite + VX Delegate ──→ NPU (MobileNet SSD, INT8)
2424
Overlay: bounding boxes + labels + FPS + NPU latency
2525
2626
27-
Wayland/Weston ──→ HDMI Display (J7)
27+
Wayland/Weston ──→ HDMI Display (J17)
2828
2929
┌──────────────────────────────────────┐
3030
│ Cortex-M7 (FreeRTOS) │
@@ -66,7 +66,7 @@ Wayland/Weston ──→ HDMI Display (J7)
6666
| Item | Connection | Notes |
6767
| ---- | ---------- | ----- |
6868
| OV5640 + MINISASTOCSI adapter | J12 (CSI1 MIPI) | Official NXP camera module; mini-SAS cable connection |
69-
| HDMI display + cable | J7 | Any HDMI monitor |
69+
| HDMI display + cable | J17 | Any HDMI monitor |
7070
| Ethernet cable | J10 or J11A | For file transfer (scp, wget) |
7171

7272
## Step-by-step
@@ -77,38 +77,55 @@ Wayland/Weston ──→ HDMI Display (J7)
7777
2. Connect OV5640 camera to **J12** (CSI1 MIPI) via MINISASTOCSI adapter
7878
- Use the mini-SAS cable (included with the MINISASTOCSI kit)
7979
- Plug into J12 until the latch clicks
80-
3. Connect HDMI cable from **J7** to display
80+
3. Connect HDMI cable from **J17** to display
8181
4. Connect Ethernet cable to **J10**
8282
5. Power on, login as `root`
8383

8484
### Step 2 — Verify camera detection
8585

8686
```bash
8787
dmesg | grep -i ov5640
88-
ls /dev/video*
8988
v4l2-ctl --list-devices
90-
v4l2-ctl -d /dev/video0 --list-formats-ext
89+
v4l2-ctl -d /dev/video3 --list-formats-ext
9190
```
9291

93-
Expected output: OV5640 detected on I2C1 (address 0x3c), `/dev/video0` created, supports YUYV/NV12 up to 1920x1080.
92+
**IMPORTANT:** On i.MX8MP, the video device numbering is:
93+
94+
| Device | Function | Notes |
95+
| ------ | -------- | ----- |
96+
| `/dev/video0` | VPU encoder (vsi_v4l2enc) | H.264/HEVC encoding, NOT camera! |
97+
| `/dev/video1` | VPU decoder (vsi_v4l2dec) | H.264/HEVC decoding |
98+
| `/dev/video2` | ISI M2M (mxc-isi-m2m_v1) | Color space conversion |
99+
| `/dev/video3` | **ISI Capture (mxc-isi-cap_v1)** | **This is the camera!** |
100+
101+
Verified output (2026-03-10):
102+
103+
```
104+
[ 0.080573] platform 32e40000.csi: Fixed dependency cycle(s) with /soc@0/bus@30800000/i2c@30a30000/ov5640_mipi@3c
105+
[ 2.633919] ov5640 1-003c: supply DOVDD not found, using dummy regulator
106+
[ 8.525147] mx8-img-md: Registered sensor subdevice: ov5640 1-003c (1)
107+
[ 8.548408] mx8-img-md: created link [ov5640 1-003c] => [mxc-mipi-csi2.0]
108+
```
109+
110+
OV5640 detected at I2C address 0x3c, linked to MIPI CSI-2 receiver. The "supply not found, using dummy regulator" warnings are normal — the EVK uses fixed regulators not described in the device tree.
94111

95112
If `ov5640_check_chip_id: failed` appears, the camera isn't physically connected or the mini-SAS cable is loose.
96113

97114
### Step 3 — Test live preview on HDMI
98115

99116
```bash
100-
# Camera → HDMI display
101-
gst-launch-1.0 v4l2src device=/dev/video0 ! \
117+
# Camera → HDMI display (use /dev/video3, NOT video0!)
118+
gst-launch-1.0 v4l2src device=/dev/video3 ! \
102119
video/x-raw,width=640,height=480,framerate=30/1 ! \
103120
waylandsink
104121

105122
# Alternative: NXP's optimized source element
106-
gst-launch-1.0 imxv4l2src device=/dev/video0 ! \
123+
gst-launch-1.0 imxv4l2src device=/dev/video3 ! \
107124
video/x-raw,width=640,height=480 ! \
108125
waylandsink
109126
```
110127

111-
If Weston is not running: `systemctl start weston@root`
128+
If Weston is not running: `systemctl start weston.service`
112129

113130
### Step 4 — Verify NPU stack
114131

@@ -128,11 +145,9 @@ ls /usr/bin/tensorflow-lite-*/tools/benchmark_model
128145

129146
### Step 5 — Get network access and download model
130147

131-
```bash
132-
# DHCP on eth0
133-
dhclient eth0 && ip addr show eth0
148+
WiFi is available via `mlan0` (see [06-wifi-bluetooth.md](06-wifi-bluetooth.md)), or use Ethernet.
134149

135-
# Create directories
150+
```bash
136151
mkdir -p /opt/models /opt/camera-detect
137152

138153
# Download MobileNet SSD v1 (COCO dataset, INT8 quantized)
@@ -141,34 +156,49 @@ wget https://storage.googleapis.com/download.tensorflow.org/models/tflite/coco_s
141156
unzip coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip
142157
```
143158

144-
If no internet on the board, download on PC and transfer:
159+
If long URLs get corrupted via serial, download on PC and transfer via WiFi:
145160

146161
```bash
147-
scp detect.tflite root@<EVK_IP>:/opt/models/
162+
# On Windows host:
163+
scp ssd_model.zip root@192.168.1.98:/opt/models/
164+
# On EVK:
165+
cd /opt/models && unzip ssd_model.zip
148166
```
149167

168+
Model files: `detect.tflite` (4.2 MB) + `labelmap.txt` (COCO 90 classes).
169+
150170
### Step 6 — Benchmark NPU vs CPU
151171

152-
```bash
153-
BENCH=/usr/bin/tensorflow-lite-*/tools/benchmark_model
154-
MODEL=/opt/models/detect.tflite
172+
> **Note:** `benchmark_model` is not included in `imx-image-multimedia`. Use Python instead:
155173
156-
# CPU-only inference
157-
$BENCH --graph=$MODEL --num_runs=50 --warmup_runs=5
174+
```python
175+
import time, numpy as np
176+
import tflite_runtime.interpreter as tflite
158177

159-
# NPU-accelerated inference
160-
$BENCH --graph=$MODEL \
161-
--external_delegate_path=/usr/lib/libvx_delegate.so \
162-
--num_runs=50 --warmup_runs=5
178+
MODEL = "/opt/models/detect.tflite"
179+
180+
# CPU benchmark
181+
interp_cpu = tflite.Interpreter(model_path=MODEL, num_threads=4)
182+
interp_cpu.allocate_tensors()
183+
inp = interp_cpu.get_input_details()
184+
dummy = np.random.randint(0, 255, size=inp[0]["shape"], dtype=np.uint8)
185+
# ... warmup + timing loop ...
186+
187+
# NPU benchmark
188+
delegate = tflite.load_delegate("/usr/lib/libvx_delegate.so")
189+
interp_npu = tflite.Interpreter(model_path=MODEL, experimental_delegates=[delegate])
190+
# ... same warmup + timing loop ...
163191
```
164192

165-
Expected results:
193+
Verified results (2026-03-10, MobileNet SSD v1 INT8, 300x300 input):
194+
195+
| Backend | Avg Latency | Min | Max | Notes |
196+
| ------- | ----------- | --- | --- | ----- |
197+
| CPU (A53 x4, XNNPACK) | **45.3 ms** | 45.2 | 45.5 | NEON SIMD auto-enabled |
198+
| NPU (VX Delegate) | **9.1 ms** | 8.5 | 9.7 | 1 op fallback to CPU (PostProcess) |
199+
| Speedup | **5.0x** | | | CPU faster than expected due to XNNPACK |
166200

167-
| Backend | Inference Latency | Notes |
168-
| ------- | ----------------- | ----- |
169-
| CPU (A53) | ~180–200 ms | 4 threads |
170-
| NPU (VX Delegate) | ~8–12 ms | INT8 quantized |
171-
| Speedup | ~20x | |
201+
> **Note:** CPU is faster than initial estimates (~45ms vs ~180ms) because TFLite 2.16's XNNPACK delegate uses ARM NEON SIMD on the A53. The NPU's real advantage is **freeing CPU for other tasks** (camera capture, display, post-processing) in a concurrent pipeline.
172202
173203
### Step 7 — Check for NXP pre-built demos
174204

scripts/serial_transfer.py

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
"""Transfer files to i.MX8MP EVK via serial console (COM5).
2+
3+
Uses base64 + heredoc to send binary files over the debug UART.
4+
115200 baud ≈ 11.5 KB/s → ~4MB file takes ~8 minutes.
5+
"""
6+
7+
import serial
8+
import base64
9+
import hashlib
10+
import time
11+
import sys
12+
import tempfile
13+
14+
15+
def transfer_file(ser, local_path, remote_path):
16+
"""Transfer a single file via base64 heredoc."""
17+
with open(local_path, "rb") as f:
18+
data = f.read()
19+
20+
b64 = base64.b64encode(data).decode("ascii")
21+
md5_expected = hashlib.md5(data).hexdigest()
22+
total = len(b64)
23+
24+
print(f"Transferring {local_path}{remote_path}")
25+
print(f" Original: {len(data)} bytes, Base64: {total} bytes, MD5: {md5_expected}")
26+
27+
# Start heredoc: base64 -d writes decoded binary to remote_path
28+
cmd = f"base64 -d > {remote_path} << 'ENDOFFILE'\n"
29+
ser.write(cmd.encode())
30+
time.sleep(0.5)
31+
ser.read(ser.in_waiting)
32+
33+
# Send base64 data in 76-char lines (standard base64 line length)
34+
start = time.time()
35+
sent = 0
36+
line_count = 0
37+
38+
for i in range(0, total, 76):
39+
line = b64[i : i + 76] + "\n"
40+
ser.write(line.encode())
41+
sent += len(line)
42+
line_count += 1
43+
44+
# Every 200 lines (~15KB), pause briefly to avoid UART overflow
45+
if line_count % 200 == 0:
46+
time.sleep(0.1)
47+
ser.read(ser.in_waiting) # drain echo buffer
48+
elapsed = time.time() - start
49+
pct = sent / total * 100
50+
speed = sent / elapsed / 1024 if elapsed > 0 else 0
51+
print(f"\r {pct:5.1f}% {sent//1024:>5}KB / {total//1024}KB "
52+
f"{speed:.1f} KB/s {elapsed:.0f}s", end="", flush=True)
53+
54+
# End heredoc
55+
ser.write(b"ENDOFFILE\n")
56+
elapsed = time.time() - start
57+
print(f"\r 100.0% {total//1024:>5}KB / {total//1024}KB "
58+
f"Done in {elapsed:.0f}s ")
59+
60+
# Wait for base64 decode to finish
61+
time.sleep(3)
62+
ser.read(ser.in_waiting)
63+
64+
# Verify
65+
ser.write(f"ls -la {remote_path} && md5sum {remote_path}\n".encode())
66+
time.sleep(3)
67+
output = ser.read(ser.in_waiting).decode(errors="replace")
68+
print(f" Board output: {output.strip()}")
69+
print(f" Expected MD5: {md5_expected}")
70+
71+
if md5_expected in output:
72+
print(" ✓ Transfer verified OK!")
73+
return True
74+
else:
75+
print(" ✗ MD5 mismatch or verification failed")
76+
return False
77+
78+
79+
def main():
80+
port = sys.argv[1] if len(sys.argv) > 1 else "COM5"
81+
print(f"Opening {port} at 115200...")
82+
83+
ser = serial.Serial(port, 115200, timeout=5)
84+
time.sleep(0.2)
85+
ser.read(ser.in_waiting)
86+
87+
# Create target directory
88+
ser.write(b"mkdir -p /opt/models\n")
89+
time.sleep(1)
90+
ser.read(ser.in_waiting)
91+
92+
# Disable echo to double throughput
93+
ser.write(b"stty -echo\n")
94+
time.sleep(0.5)
95+
ser.read(ser.in_waiting)
96+
97+
try:
98+
# Transfer model
99+
# Windows paths (downloaded via curl in Git Bash /tmp → AppData\Local\Temp)
100+
temp = tempfile.gettempdir()
101+
transfer_file(ser, f"{temp}\\detect.tflite", "/opt/models/detect.tflite")
102+
103+
# Transfer labels (tiny file)
104+
transfer_file(ser, f"{temp}\\labelmap.txt", "/opt/models/labelmap.txt")
105+
106+
finally:
107+
# Always re-enable echo
108+
ser.write(b"stty echo\n")
109+
time.sleep(0.5)
110+
ser.read(ser.in_waiting)
111+
ser.close()
112+
print("\nDone. Serial closed.")
113+
114+
115+
if __name__ == "__main__":
116+
main()

0 commit comments

Comments
 (0)