|
1 | | -# embedded-linux |
| 1 | +# Embedded Linux on i.MX 8M Plus |
2 | 2 |
|
3 | | -An i.MX 8M Plus heterogeneous platform project: Yocto Linux on Cortex-A53 + FreeRTOS on Cortex-M7 + NPU edge inference. |
| 3 | +Heterogeneous SoC platform project built on the NXP i.MX 8M Plus EVK. Covers BSP bring-up, kernel driver development, camera/NPU pipelines, and real-time co-processing with FreeRTOS on the Cortex-M7. |
4 | 4 |
|
5 | | -<!-- 异构多核嵌入式 Linux 项目:A53 跑 Linux,M7 跑 FreeRTOS,NPU 做边缘推理 --> |
6 | | - |
7 | | -I'm building this on the NXP i.MX 8M Plus EVK to explore the full stack of embedded Linux development — from writing kernel drivers and custom Yocto images, to real-time firmware on a dedicated MCU core, to deploying quantized neural networks on the onboard NPU. The goal is a working sensor platform where the A53 and M7 cores each do what they're good at. |
8 | | - |
9 | | -## Current Status |
10 | | - |
11 | | -**Phase 1 complete** — the EVK boots a custom Yocto scarthgap image from SD card. All core peripherals verified (CPU, RAM, storage, network interfaces, I2C buses). See the [first boot verification log](docs/02-hardware-guide.md#first-boot-verification-2026-03-09) for details. |
12 | | - |
13 | | -<!-- 当前进度:Phase 1 已完成,EVK 已成功从 SD 卡启动自定义 Yocto 镜像。所有基本外设验证通过。 --> |
| 5 | +## Hardware |
14 | 6 |
|
15 | | -| Milestone | Status | |
16 | | -| --------- | ------ | |
17 | | -| Phase 1 — Yocto first boot & peripheral verification | **Done** | |
18 | | -| Phase 2 — BME280 I2C kernel driver (A53 Linux) | Next up | |
19 | | -| Phase 3 — FreeRTOS on M7 + RPMsg inter-core comms | Planned | |
20 | | -| Phase 4 — NPU edge inference with eIQ / TFLite | Planned | |
| 7 | +- **SoC:** NXP i.MX 8M Plus — quad Cortex-A53 (1.8 GHz) + Cortex-M7 + 2.3 TOPS NPU |
| 8 | +- **Board:** 8MPLUSLPD4-EVK (6 GB LPDDR4, 32 GB eMMC) |
| 9 | +- **Camera:** OV5640 MIPI CSI-2 via MINISASTOCSI adapter (J12) |
| 10 | +- **Display:** HDMI 2.0a (J17), Weston/Wayland compositor |
| 11 | +- **Kernel:** 6.6.52-lts (Yocto Scarthgap) |
21 | 12 |
|
22 | | -## Architecture |
| 13 | +## Repository Structure |
23 | 14 |
|
24 | 15 | ```text |
25 | | -┌──────────────────────────────────────────────────────────────┐ |
26 | | -│ │ |
27 | | -│ Cortex-A53 (Linux / Yocto) │ |
28 | | -│ ├── Custom Yocto image (Scarthgap) │ |
29 | | -│ ├── BME280 I2C kernel driver module │ |
30 | | -│ ├── Userspace data collector daemon (C) │ |
31 | | -│ ├── NPU inference — anomaly detection (eIQ / TFLite) │ |
32 | | -│ └── Lightweight web UI for live sensor data │ |
33 | | -│ │ |
34 | | -│ ┌──────────────┐ │ |
35 | | -│ │ RPMsg / MU │ │ |
36 | | -│ │ (inter-core) │ │ |
37 | | -│ └──────────────┘ │ |
38 | | -│ │ |
39 | | -│ Cortex-M7 (FreeRTOS) │ |
40 | | -│ ├── MPU6050 6-axis real-time sampling (1 kHz) │ |
41 | | -│ ├── Hard real-time control loop │ |
42 | | -│ ├── SSD1306 OLED local display │ |
43 | | -│ └── Data packing + forwarding to A53 via RPMsg │ |
44 | | -│ │ |
45 | | -└──────────────────────────────────────────────────────────────┘ |
46 | | -
|
47 | | -I2C buses: |
48 | | - I2C3 (A53 Linux) ──── BME280 (temp / humidity / pressure) |
49 | | - I2C2 (M7 FreeRTOS) ─┬─ MPU6050 (accel / gyro) |
50 | | - └─ SSD1306 (OLED display) |
| 16 | +kernel-modules/ |
| 17 | +├── hello/ Minimal loadable kernel module |
| 18 | +├── chardev/ Character device driver (/dev node, ioctl, mutex) |
| 19 | +└── bme280/ I2C client driver with sysfs interface |
| 20 | +
|
| 21 | +drivers/ |
| 22 | +└── v4l2-capture/ V4L2 mmap frame capture (C, multi-planar API) |
| 23 | +
|
| 24 | +dts/ |
| 25 | +└── imx8mp-evk-ov5640.dts Annotated device tree overlay (camera pipeline) |
| 26 | +
|
| 27 | +scripts/ |
| 28 | +├── build-multimedia.sh Yocto build helper for imx-image-multimedia |
| 29 | +└── serial_transfer.py File transfer over debug UART (base64) |
| 30 | +
|
| 31 | +docs/ |
| 32 | +├── 01-dev-environment.md Build host setup |
| 33 | +├── 02-hardware-guide.md EVK wiring, boot config, peripheral map |
| 34 | +├── 03-yocto-bsp.md Yocto BSP build (repo init → bitbake) |
| 35 | +├── 07-camera-npu.md Camera + NPU object detection pipeline |
| 36 | +└── 08-device-tree-explained.md Device tree walkthrough |
51 | 37 | ``` |
52 | 38 |
|
53 | | -<!-- 架构说明:环境传感(BME280)走 Linux 内核驱动,运动传感(MPU6050)走 M7 实时采样,两核通过 RPMsg 通信 --> |
| 39 | +## What's Working |
54 | 40 |
|
55 | | -The split is deliberate: environmental sensing (BME280) goes through a proper Linux kernel driver so I can expose it via sysfs and feed it into the NPU pipeline. Motion sensing (MPU6050) needs deterministic sub-millisecond timing, so it runs bare-metal on the M7 under FreeRTOS. The two cores talk over RPMsg through the Messaging Unit (MU) hardware. |
| 41 | +| Subsystem | Status | Details | |
| 42 | +| --------- | ------ | ------- | |
| 43 | +| Yocto BSP | Boots from SD | `imx-image-multimedia`, Scarthgap branch | |
| 44 | +| OV5640 camera | Live HDMI preview | MIPI CSI-2 → ISI → GStreamer → Weston | |
| 45 | +| NPU (VIP8000) | Driver loaded | galcore 6.4.11, VX Delegate + TFLite 2.16.2 | |
| 46 | +| GPU (GC7000UL) | Weston @ 60 FPS | `weston-simple-egl` verified | |
| 47 | +| Debug UART | Working | J23, 3rd COM port = A53 console (115200 8N1) | |
56 | 48 |
|
57 | | -## Hardware |
| 49 | +## Quick Start |
58 | 50 |
|
59 | | -| Item | Part | Notes | |
60 | | -| ---- | ---- | ----- | |
61 | | -| Dev board | 8MPLUSLPD4-EVK | i.MX 8M Plus EVK, 6 GB LPDDR4, 32 GB eMMC | |
62 | | -| Camera adapter + OV5640 | MINISASTOCSI | MIPI CSI camera module | |
63 | | -| Industrial SD card | SanDisk SDXC 64 GB | SD boot for development | |
64 | | -| Temp/humidity/pressure sensor | BME280 (I2C, 3.3 V) | Connected to I2C3 (A53 Linux) | |
65 | | -| 6-axis IMU | MPU6050 GY-521 | Connected to I2C2 (M7 FreeRTOS) | |
66 | | -| OLED display | SSD1306 0.96" I2C | Shares I2C2 bus with MPU6050 | |
67 | | - |
68 | | -### Verified Specs (from first boot) |
69 | | - |
70 | | -<!-- 首次启动验证数据 --> |
71 | | - |
72 | | -| Parameter | Value | |
73 | | -| --------- | ----- | |
74 | | -| CPU | 4x Cortex-A53 @ 1.8 GHz (ARMv8-A) | |
75 | | -| Usable RAM | 5.5 GiB (of 6 GB LPDDR4) | |
76 | | -| eMMC | 29.1 GB on-board | |
77 | | -| SPI NOR | 32 MB | |
78 | | -| Ethernet | 2x GbE (FEC + DWMAC/TSN) | |
79 | | -| I2C | 3 buses available (I2C1–I2C3) | |
80 | | -| Kernel | 6.6.52-lts (Yocto scarthgap) | |
| 51 | +```bash |
| 52 | +# 1. Build the Yocto image (Ubuntu 22.04 host, ~2 hours first build) |
| 53 | +source scripts/build-multimedia.sh |
| 54 | + |
| 55 | +# 2. Flash to SD card (use Rufus DD image mode on Windows) |
| 56 | +# Set SW4: OFF OFF ON ON (SD card boot) |
| 57 | + |
| 58 | +# 3. Connect: USB-C to J5 (power), micro-USB to J23 (debug UART) |
| 59 | +# Serial: 3rd COM port, 115200 8N1 |
| 60 | + |
| 61 | +# 4. Camera preview (on the EVK, over serial console) |
| 62 | +export XDG_RUNTIME_DIR=/run/user/0 |
| 63 | +gst-launch-1.0 v4l2src device=/dev/video3 ! \ |
| 64 | + video/x-raw,width=640,height=480,framerate=30/1 ! \ |
| 65 | + videoconvert ! autovideosink |
| 66 | + |
| 67 | +# 5. Build and load the hello module (cross-compile or on-device) |
| 68 | +cd kernel-modules/hello |
| 69 | +make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- \ |
| 70 | + KERNELDIR=/path/to/yocto/build/tmp/work/.../linux-imx/build |
| 71 | +scp hello.ko root@<EVK_IP>:/tmp/ |
| 72 | +# On EVK: |
| 73 | +insmod /tmp/hello.ko && dmesg | tail -3 |
| 74 | +``` |
81 | 75 |
|
82 | | -## Roadmap |
| 76 | +## Kernel Modules |
83 | 77 |
|
84 | | -### Phase 1 — Environment + first Yocto boot ✓ |
| 78 | +Three out-of-tree modules demonstrating progressive driver complexity: |
85 | 79 |
|
86 | | -- [x] Set up Ubuntu 22.04 build host with all Yocto dependencies |
87 | | -- [x] `repo init` the NXP BSP manifest, sync sources |
88 | | -- [x] Build a minimal custom image with `bitbake` |
89 | | -- [x] Flash to SD card, boot the EVK, get a serial console |
90 | | -- [x] Verify basic peripherals (CPU, RAM, storage, network, I2C) |
| 80 | +**hello** — Module lifecycle (`init`/`exit`), `printk`, section markers (`__init`/`__exit`). |
91 | 81 |
|
92 | | -### Phase 2 — Linux kernel driver development (BME280) |
| 82 | +**chardev** — Full character device with dynamic major allocation, `file_operations` (read/write/ioctl), `copy_to_user`/`copy_from_user`, mutex synchronization, and automatic `/dev` node creation via `class_create`/`device_create`. |
93 | 83 |
|
94 | | -- [ ] Write an out-of-tree I2C kernel driver module for BME280 |
95 | | -- [ ] Modify the device tree to describe the BME280 on I2C3 |
96 | | -- [ ] Expose temperature, humidity, and pressure readings via sysfs |
97 | | -- [ ] Build a userspace data collector daemon in C |
98 | | -- [ ] Integrate the driver into the Yocto build as a recipe |
| 84 | +**bme280** — I2C client driver using the `probe`/`remove` lifecycle, device tree `compatible` matching, `i2c_smbus_*` register access, sysfs attributes, and `devm_` managed allocation. |
99 | 85 |
|
100 | | -### Phase 3 — FreeRTOS dual-core development (M7 + RPMsg) |
| 86 | +## V4L2 Capture |
101 | 87 |
|
102 | | -<!-- M7 实时核开发:MCUXpresso SDK + FreeRTOS + RPMsg 核间通信 --> |
| 88 | +Userspace C program demonstrating the V4L2 multi-planar API as used by i.MX8MP's ISI: |
103 | 89 |
|
104 | | -- [ ] Set up the MCUXpresso SDK build environment for Cortex-M7 |
105 | | -- [ ] Port or write a FreeRTOS application for MPU6050 sampling at 1 kHz |
106 | | -- [ ] Bring up RPMsg communication between A53 (Linux) and M7 (FreeRTOS) |
107 | | -- [ ] Implement data packing on M7 and unpacking on the Linux side |
108 | | -- [ ] Add SSD1306 OLED output for local IMU status display |
| 90 | +- `VIDIOC_QUERYCAP` → capability query |
| 91 | +- `VIDIOC_S_FMT` → format negotiation (multi-planar) |
| 92 | +- `VIDIOC_REQBUFS` + `mmap()` → zero-copy DMA buffer setup |
| 93 | +- `VIDIOC_QBUF`/`VIDIOC_DQBUF` → streaming capture loop |
109 | 94 |
|
110 | | -### Phase 4 — NPU edge AI integration |
| 95 | +## Device Tree |
111 | 96 |
|
112 | | -<!-- NPU 边缘推理:eIQ + TFLite + INT8 量化模型 --> |
| 97 | +The [`dts/imx8mp-evk-ov5640.dts`](dts/imx8mp-evk-ov5640.dts) overlay is annotated to explain how the camera pipeline is described in hardware: |
113 | 98 |
|
114 | | -- [ ] Explore the eIQ ML framework and its TFLite delegate for the NPU |
115 | | -- [ ] Train and quantize a small anomaly detection model (INT8) |
116 | | -- [ ] Deploy on the NPU, feed it sensor data from both cores |
117 | | -- [ ] Build a lightweight web UI to display real-time data and inference results |
118 | | -- [ ] Profile NPU inference latency and power consumption |
| 99 | +```text |
| 100 | +OV5640 (I2C, 0x3c) → MIPI CSI-2 RX → ISI ch0 → /dev/video3 |
| 101 | +``` |
119 | 102 |
|
120 | | -## Documentation |
| 103 | +Covers clock providers, regulator bindings, OF graph endpoint linking, and MIPI lane configuration. |
121 | 104 |
|
122 | | -| Doc | Content | |
123 | | -| --- | ------- | |
124 | | -| [01-dev-environment.md](docs/01-dev-environment.md) | Host PC setup: Ubuntu, Yocto deps, toolchains | |
125 | | -| [02-hardware-guide.md](docs/02-hardware-guide.md) | EVK wiring, serial console, DIP switches, boot verification | |
126 | | -| [03-yocto-bsp.md](docs/03-yocto-bsp.md) | Full Yocto build walkthrough (repo init to bitbake) | |
127 | | -| [04-freertos-m7.md](docs/04-freertos-m7.md) | Cortex-M7 FreeRTOS development (MCUXpresso SDK) | |
128 | | -| [05-npu-eiq.md](docs/05-npu-eiq.md) | NPU edge AI (eIQ, TFLite, quantized models) | |
129 | | -| [06-references.md](docs/06-references.md) | Official docs, community resources, useful links | |
| 105 | +## i.MX8MP Video Device Map |
130 | 106 |
|
131 | | -## Quick Start |
| 107 | +On this SoC, `/dev/video0` is **not** the camera: |
| 108 | + |
| 109 | +| Device | Function | |
| 110 | +| ------ | -------- | |
| 111 | +| /dev/video0 | VPU H.264/HEVC encoder | |
| 112 | +| /dev/video1 | VPU decoder | |
| 113 | +| /dev/video2 | ISI memory-to-memory (CSC) | |
| 114 | +| /dev/video3 | **ISI capture (camera)** | |
132 | 115 |
|
133 | | -1. **Set up the build host** — Install Ubuntu 22.04 and the required packages. See [01-dev-environment.md](docs/01-dev-environment.md). |
134 | | -2. **Build the Yocto image** — Follow [03-yocto-bsp.md](docs/03-yocto-bsp.md) to `repo init`, sync, and `bitbake` a bootable image. |
135 | | -3. **Flash and boot** — Write the `.wic` image to SD card with Rufus (use DD image mode), set SW4 to `OFF OFF ON ON`, power on via J5. Login as `root` (no password) over the serial console at J23 (3rd COM port, 115200 8N1). |
136 | | -4. **Wire up sensors** — Connect BME280 to I2C3 and MPU6050 + SSD1306 to I2C2. Pinout details in [02-hardware-guide.md](docs/02-hardware-guide.md). |
137 | | -5. **Build M7 firmware** — Compile the FreeRTOS application and load it onto the Cortex-M7. See [04-freertos-m7.md](docs/04-freertos-m7.md). |
| 116 | +## Roadmap |
138 | 117 |
|
139 | | -<!-- 快速开始提示:烧录用 Rufus DD 模式,SW4 拨到 OFF OFF ON ON,串口选第三个 COM 口 --> |
| 118 | +- [x] Yocto BSP bring-up and first boot verification |
| 119 | +- [x] Camera pipeline: OV5640 → ISI → GStreamer → HDMI |
| 120 | +- [x] NPU stack verification (galcore, VX Delegate, TFLite) |
| 121 | +- [x] Kernel module examples (hello, chardev, I2C driver) |
| 122 | +- [x] V4L2 capture program and device tree overlay |
| 123 | +- [ ] NPU benchmark: CPU vs NPU inference latency |
| 124 | +- [ ] Real-time object detection (MobileNet SSD + camera + NPU) |
| 125 | +- [ ] FreeRTOS on Cortex-M7 with RPMsg inter-core communication |
| 126 | +- [ ] End-to-end demo: camera → NPU → overlay → HDMI |
140 | 127 |
|
141 | 128 | ## License |
142 | 129 |
|
143 | | -MIT — see [LICENSE](LICENSE). |
| 130 | +MIT |
0 commit comments