Skip to content

Commit b22affb

Browse files
Corning-AIclaude
andcommitted
Add kernel drivers, V4L2 capture, device tree overlay, and rewrite README
Kernel modules: - hello: minimal loadable module (init/exit, printk) - chardev: character device with file_operations, ioctl, mutex - bme280: I2C client driver with device tree matching and sysfs Userspace: - V4L2 multi-planar capture program for i.MX8MP ISI - Camera detection app (TFLite + VX Delegate + GStreamer) - M7 RPMsg heartbeat firmware skeleton Device tree: - Annotated OV5640 camera overlay (MIPI CSI-2, OF graph, clocks) Docs: - Rewrite README with actual repo structure and verified results - Update camera+NPU guide with correct device paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent d93385c commit b22affb

17 files changed

Lines changed: 2090 additions & 135 deletions

File tree

README.md

Lines changed: 100 additions & 113 deletions
Original file line numberDiff line numberDiff line change
@@ -1,143 +1,130 @@
1-
# embedded-linux
1+
# Embedded Linux on i.MX 8M Plus
22

3-
An i.MX 8M Plus heterogeneous platform project: Yocto Linux on Cortex-A53 + FreeRTOS on Cortex-M7 + NPU edge inference.
3+
Heterogeneous SoC platform project built on the NXP i.MX 8M Plus EVK. Covers BSP bring-up, kernel driver development, camera/NPU pipelines, and real-time co-processing with FreeRTOS on the Cortex-M7.
44

5-
<!-- 异构多核嵌入式 Linux 项目:A53 跑 Linux,M7 跑 FreeRTOS,NPU 做边缘推理 -->
6-
7-
I'm building this on the NXP i.MX 8M Plus EVK to explore the full stack of embedded Linux development — from writing kernel drivers and custom Yocto images, to real-time firmware on a dedicated MCU core, to deploying quantized neural networks on the onboard NPU. The goal is a working sensor platform where the A53 and M7 cores each do what they're good at.
8-
9-
## Current Status
10-
11-
**Phase 1 complete** — the EVK boots a custom Yocto scarthgap image from SD card. All core peripherals verified (CPU, RAM, storage, network interfaces, I2C buses). See the [first boot verification log](docs/02-hardware-guide.md#first-boot-verification-2026-03-09) for details.
12-
13-
<!-- 当前进度:Phase 1 已完成,EVK 已成功从 SD 卡启动自定义 Yocto 镜像。所有基本外设验证通过。 -->
5+
## Hardware
146

15-
| Milestone | Status |
16-
| --------- | ------ |
17-
| Phase 1 — Yocto first boot & peripheral verification | **Done** |
18-
| Phase 2 — BME280 I2C kernel driver (A53 Linux) | Next up |
19-
| Phase 3 — FreeRTOS on M7 + RPMsg inter-core comms | Planned |
20-
| Phase 4 — NPU edge inference with eIQ / TFLite | Planned |
7+
- **SoC:** NXP i.MX 8M Plus — quad Cortex-A53 (1.8 GHz) + Cortex-M7 + 2.3 TOPS NPU
8+
- **Board:** 8MPLUSLPD4-EVK (6 GB LPDDR4, 32 GB eMMC)
9+
- **Camera:** OV5640 MIPI CSI-2 via MINISASTOCSI adapter (J12)
10+
- **Display:** HDMI 2.0a (J17), Weston/Wayland compositor
11+
- **Kernel:** 6.6.52-lts (Yocto Scarthgap)
2112

22-
## Architecture
13+
## Repository Structure
2314

2415
```text
25-
┌──────────────────────────────────────────────────────────────┐
26-
│ │
27-
│ Cortex-A53 (Linux / Yocto) │
28-
│ ├── Custom Yocto image (Scarthgap) │
29-
│ ├── BME280 I2C kernel driver module │
30-
│ ├── Userspace data collector daemon (C) │
31-
│ ├── NPU inference — anomaly detection (eIQ / TFLite) │
32-
│ └── Lightweight web UI for live sensor data │
33-
│ │
34-
│ ┌──────────────┐ │
35-
│ │ RPMsg / MU │ │
36-
│ │ (inter-core) │ │
37-
│ └──────────────┘ │
38-
│ │
39-
│ Cortex-M7 (FreeRTOS) │
40-
│ ├── MPU6050 6-axis real-time sampling (1 kHz) │
41-
│ ├── Hard real-time control loop │
42-
│ ├── SSD1306 OLED local display │
43-
│ └── Data packing + forwarding to A53 via RPMsg │
44-
│ │
45-
└──────────────────────────────────────────────────────────────┘
46-
47-
I2C buses:
48-
I2C3 (A53 Linux) ──── BME280 (temp / humidity / pressure)
49-
I2C2 (M7 FreeRTOS) ─┬─ MPU6050 (accel / gyro)
50-
└─ SSD1306 (OLED display)
16+
kernel-modules/
17+
├── hello/ Minimal loadable kernel module
18+
├── chardev/ Character device driver (/dev node, ioctl, mutex)
19+
└── bme280/ I2C client driver with sysfs interface
20+
21+
drivers/
22+
└── v4l2-capture/ V4L2 mmap frame capture (C, multi-planar API)
23+
24+
dts/
25+
└── imx8mp-evk-ov5640.dts Annotated device tree overlay (camera pipeline)
26+
27+
scripts/
28+
├── build-multimedia.sh Yocto build helper for imx-image-multimedia
29+
└── serial_transfer.py File transfer over debug UART (base64)
30+
31+
docs/
32+
├── 01-dev-environment.md Build host setup
33+
├── 02-hardware-guide.md EVK wiring, boot config, peripheral map
34+
├── 03-yocto-bsp.md Yocto BSP build (repo init → bitbake)
35+
├── 07-camera-npu.md Camera + NPU object detection pipeline
36+
└── 08-device-tree-explained.md Device tree walkthrough
5137
```
5238

53-
<!-- 架构说明:环境传感(BME280)走 Linux 内核驱动,运动传感(MPU6050)走 M7 实时采样,两核通过 RPMsg 通信 -->
39+
## What's Working
5440

55-
The split is deliberate: environmental sensing (BME280) goes through a proper Linux kernel driver so I can expose it via sysfs and feed it into the NPU pipeline. Motion sensing (MPU6050) needs deterministic sub-millisecond timing, so it runs bare-metal on the M7 under FreeRTOS. The two cores talk over RPMsg through the Messaging Unit (MU) hardware.
41+
| Subsystem | Status | Details |
42+
| --------- | ------ | ------- |
43+
| Yocto BSP | Boots from SD | `imx-image-multimedia`, Scarthgap branch |
44+
| OV5640 camera | Live HDMI preview | MIPI CSI-2 → ISI → GStreamer → Weston |
45+
| NPU (VIP8000) | Driver loaded | galcore 6.4.11, VX Delegate + TFLite 2.16.2 |
46+
| GPU (GC7000UL) | Weston @ 60 FPS | `weston-simple-egl` verified |
47+
| Debug UART | Working | J23, 3rd COM port = A53 console (115200 8N1) |
5648

57-
## Hardware
49+
## Quick Start
5850

59-
| Item | Part | Notes |
60-
| ---- | ---- | ----- |
61-
| Dev board | 8MPLUSLPD4-EVK | i.MX 8M Plus EVK, 6 GB LPDDR4, 32 GB eMMC |
62-
| Camera adapter + OV5640 | MINISASTOCSI | MIPI CSI camera module |
63-
| Industrial SD card | SanDisk SDXC 64 GB | SD boot for development |
64-
| Temp/humidity/pressure sensor | BME280 (I2C, 3.3 V) | Connected to I2C3 (A53 Linux) |
65-
| 6-axis IMU | MPU6050 GY-521 | Connected to I2C2 (M7 FreeRTOS) |
66-
| OLED display | SSD1306 0.96" I2C | Shares I2C2 bus with MPU6050 |
67-
68-
### Verified Specs (from first boot)
69-
70-
<!-- 首次启动验证数据 -->
71-
72-
| Parameter | Value |
73-
| --------- | ----- |
74-
| CPU | 4x Cortex-A53 @ 1.8 GHz (ARMv8-A) |
75-
| Usable RAM | 5.5 GiB (of 6 GB LPDDR4) |
76-
| eMMC | 29.1 GB on-board |
77-
| SPI NOR | 32 MB |
78-
| Ethernet | 2x GbE (FEC + DWMAC/TSN) |
79-
| I2C | 3 buses available (I2C1–I2C3) |
80-
| Kernel | 6.6.52-lts (Yocto scarthgap) |
51+
```bash
52+
# 1. Build the Yocto image (Ubuntu 22.04 host, ~2 hours first build)
53+
source scripts/build-multimedia.sh
54+
55+
# 2. Flash to SD card (use Rufus DD image mode on Windows)
56+
# Set SW4: OFF OFF ON ON (SD card boot)
57+
58+
# 3. Connect: USB-C to J5 (power), micro-USB to J23 (debug UART)
59+
# Serial: 3rd COM port, 115200 8N1
60+
61+
# 4. Camera preview (on the EVK, over serial console)
62+
export XDG_RUNTIME_DIR=/run/user/0
63+
gst-launch-1.0 v4l2src device=/dev/video3 ! \
64+
video/x-raw,width=640,height=480,framerate=30/1 ! \
65+
videoconvert ! autovideosink
66+
67+
# 5. Build and load the hello module (cross-compile or on-device)
68+
cd kernel-modules/hello
69+
make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- \
70+
KERNELDIR=/path/to/yocto/build/tmp/work/.../linux-imx/build
71+
scp hello.ko root@<EVK_IP>:/tmp/
72+
# On EVK:
73+
insmod /tmp/hello.ko && dmesg | tail -3
74+
```
8175

82-
## Roadmap
76+
## Kernel Modules
8377

84-
### Phase 1 — Environment + first Yocto boot ✓
78+
Three out-of-tree modules demonstrating progressive driver complexity:
8579

86-
- [x] Set up Ubuntu 22.04 build host with all Yocto dependencies
87-
- [x] `repo init` the NXP BSP manifest, sync sources
88-
- [x] Build a minimal custom image with `bitbake`
89-
- [x] Flash to SD card, boot the EVK, get a serial console
90-
- [x] Verify basic peripherals (CPU, RAM, storage, network, I2C)
80+
**hello** — Module lifecycle (`init`/`exit`), `printk`, section markers (`__init`/`__exit`).
9181

92-
### Phase 2 — Linux kernel driver development (BME280)
82+
**chardev** — Full character device with dynamic major allocation, `file_operations` (read/write/ioctl), `copy_to_user`/`copy_from_user`, mutex synchronization, and automatic `/dev` node creation via `class_create`/`device_create`.
9383

94-
- [ ] Write an out-of-tree I2C kernel driver module for BME280
95-
- [ ] Modify the device tree to describe the BME280 on I2C3
96-
- [ ] Expose temperature, humidity, and pressure readings via sysfs
97-
- [ ] Build a userspace data collector daemon in C
98-
- [ ] Integrate the driver into the Yocto build as a recipe
84+
**bme280** — I2C client driver using the `probe`/`remove` lifecycle, device tree `compatible` matching, `i2c_smbus_*` register access, sysfs attributes, and `devm_` managed allocation.
9985

100-
### Phase 3 — FreeRTOS dual-core development (M7 + RPMsg)
86+
## V4L2 Capture
10187

102-
<!-- M7 实时核开发:MCUXpresso SDK + FreeRTOS + RPMsg 核间通信 -->
88+
Userspace C program demonstrating the V4L2 multi-planar API as used by i.MX8MP's ISI:
10389

104-
- [ ] Set up the MCUXpresso SDK build environment for Cortex-M7
105-
- [ ] Port or write a FreeRTOS application for MPU6050 sampling at 1 kHz
106-
- [ ] Bring up RPMsg communication between A53 (Linux) and M7 (FreeRTOS)
107-
- [ ] Implement data packing on M7 and unpacking on the Linux side
108-
- [ ] Add SSD1306 OLED output for local IMU status display
90+
- `VIDIOC_QUERYCAP` → capability query
91+
- `VIDIOC_S_FMT` → format negotiation (multi-planar)
92+
- `VIDIOC_REQBUFS` + `mmap()` → zero-copy DMA buffer setup
93+
- `VIDIOC_QBUF`/`VIDIOC_DQBUF` → streaming capture loop
10994

110-
### Phase 4 — NPU edge AI integration
95+
## Device Tree
11196

112-
<!-- NPU 边缘推理:eIQ + TFLite + INT8 量化模型 -->
97+
The [`dts/imx8mp-evk-ov5640.dts`](dts/imx8mp-evk-ov5640.dts) overlay is annotated to explain how the camera pipeline is described in hardware:
11398

114-
- [ ] Explore the eIQ ML framework and its TFLite delegate for the NPU
115-
- [ ] Train and quantize a small anomaly detection model (INT8)
116-
- [ ] Deploy on the NPU, feed it sensor data from both cores
117-
- [ ] Build a lightweight web UI to display real-time data and inference results
118-
- [ ] Profile NPU inference latency and power consumption
99+
```text
100+
OV5640 (I2C, 0x3c) → MIPI CSI-2 RX → ISI ch0 → /dev/video3
101+
```
119102

120-
## Documentation
103+
Covers clock providers, regulator bindings, OF graph endpoint linking, and MIPI lane configuration.
121104

122-
| Doc | Content |
123-
| --- | ------- |
124-
| [01-dev-environment.md](docs/01-dev-environment.md) | Host PC setup: Ubuntu, Yocto deps, toolchains |
125-
| [02-hardware-guide.md](docs/02-hardware-guide.md) | EVK wiring, serial console, DIP switches, boot verification |
126-
| [03-yocto-bsp.md](docs/03-yocto-bsp.md) | Full Yocto build walkthrough (repo init to bitbake) |
127-
| [04-freertos-m7.md](docs/04-freertos-m7.md) | Cortex-M7 FreeRTOS development (MCUXpresso SDK) |
128-
| [05-npu-eiq.md](docs/05-npu-eiq.md) | NPU edge AI (eIQ, TFLite, quantized models) |
129-
| [06-references.md](docs/06-references.md) | Official docs, community resources, useful links |
105+
## i.MX8MP Video Device Map
130106

131-
## Quick Start
107+
On this SoC, `/dev/video0` is **not** the camera:
108+
109+
| Device | Function |
110+
| ------ | -------- |
111+
| /dev/video0 | VPU H.264/HEVC encoder |
112+
| /dev/video1 | VPU decoder |
113+
| /dev/video2 | ISI memory-to-memory (CSC) |
114+
| /dev/video3 | **ISI capture (camera)** |
132115

133-
1. **Set up the build host** — Install Ubuntu 22.04 and the required packages. See [01-dev-environment.md](docs/01-dev-environment.md).
134-
2. **Build the Yocto image** — Follow [03-yocto-bsp.md](docs/03-yocto-bsp.md) to `repo init`, sync, and `bitbake` a bootable image.
135-
3. **Flash and boot** — Write the `.wic` image to SD card with Rufus (use DD image mode), set SW4 to `OFF OFF ON ON`, power on via J5. Login as `root` (no password) over the serial console at J23 (3rd COM port, 115200 8N1).
136-
4. **Wire up sensors** — Connect BME280 to I2C3 and MPU6050 + SSD1306 to I2C2. Pinout details in [02-hardware-guide.md](docs/02-hardware-guide.md).
137-
5. **Build M7 firmware** — Compile the FreeRTOS application and load it onto the Cortex-M7. See [04-freertos-m7.md](docs/04-freertos-m7.md).
116+
## Roadmap
138117

139-
<!-- 快速开始提示:烧录用 Rufus DD 模式,SW4 拨到 OFF OFF ON ON,串口选第三个 COM 口 -->
118+
- [x] Yocto BSP bring-up and first boot verification
119+
- [x] Camera pipeline: OV5640 → ISI → GStreamer → HDMI
120+
- [x] NPU stack verification (galcore, VX Delegate, TFLite)
121+
- [x] Kernel module examples (hello, chardev, I2C driver)
122+
- [x] V4L2 capture program and device tree overlay
123+
- [ ] NPU benchmark: CPU vs NPU inference latency
124+
- [ ] Real-time object detection (MobileNet SSD + camera + NPU)
125+
- [ ] FreeRTOS on Cortex-M7 with RPMsg inter-core communication
126+
- [ ] End-to-end demo: camera → NPU → overlay → HDMI
140127

141128
## License
142129

143-
MIT — see [LICENSE](LICENSE).
130+
MIT

app/.gitkeep

Whitespace-only changes.

0 commit comments

Comments
 (0)