LP: Multimodal On-Device Inference on Armv9 with MNN for Audio and Vision

Author: odinshen-git

content/learning-paths/cross-platform/multimodel_mnn_v9/1_mnn_v9.md

@@ -0,0 +1,34 @@
+---
+title: Understand MNN and Omni for Armv9 on-device inference
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Why MNN for on-device deployment
+
+- **Portable inference runtime** with strong support for mobile and edge platforms
+- **CPU-first workflow** that maps naturally to Armv9 servers and development boards
+- Native builds allow you to benefit from **Arm-specific optimizations** (such as KleidiAI)
+- Single codebase can target x86, Arm Linux, Android, and iOS
+
+## Why Omni multimodal
+
+- **Single model** that can handle **image, audio, and text** in one pipeline
+- Enables **cross-modal tasks** such as “look at this shelf + listen to this note → generate an operational ticket”
+- **Prompt-based control** with tags like `<img>…</img>` and `<audio>…</audio>`
+- No need to maintain separate models for vision and speech
+
+## Scope boundaries for reproducibility
+
+To keep this Learning Path simple and repeatable, we deliberately limit the scope:
+
+- **CPU-only**  
+  No GPU or NPU acceleration is required.
+
+- **No quantization**  
+  You use a **prebuilt MNN Omni model package** as-is. No additional export or quantization steps are needed.
+
+- **No CPU–NPU heterogeneous pipeline**  
+  All compute runs on the Armv9 CPU. Focus is on **correctness and reproducibility**, not maximum throughput.

content/learning-paths/cross-platform/multimodel_mnn_v9/2_mnn_build.md

@@ -0,0 +1,105 @@
+---
+title: Build MNN and prepare an Omni model on Armv9
+weight: 3
+layout: "learningpathall"
+---
+
+## Get source and native build
+
+Create a working directory under your home folder:
+
+```bash
+cd ~
+mkdir -p ~/mnn_lp
+cd ~/mnn_lp
+```
+
+Clone MNN and build:
+
+```bash
+git clone https://github.com/alibaba/MNN.git
+cd MNN
+
+rm -rf build
+mkdir build && cd build
+
+cmake .. \
+  -DMNN_BUILD_SHARED=ON \
+  -DMNN_BUILD_LLM=ON \
+  -DMNN_BUILD_AUDIO=ON \
+  -DMNN_BUILD_LLM_OMNI=ON \
+  -DMNN_LOW_MEMORY=ON \
+  -DMNN_KLEIDIAI=ON
+
+make -j$(nproc)
+```
+
+**Key CMake options:**
+
+- `MNN_BUILD_LLM=ON` – enables LLM support
+- `MNN_BUILD_AUDIO=ON` – enables audio features needed for Omni
+- `MNN_BUILD_LLM_OMNI=ON` – enables multimodal Omni support
+- `MNN_LOW_MEMORY=ON` – uses lower memory configurations where possible
+- `MNN_KLEIDIAI=ON` – enables KleidiAI-related optimizations on supported Arm platforms
+
+Verify that `llm_demo` was built:
+
+```bash
+ls -l ~/mnn_lp/MNN/build/llm_demo
+```
+
+---
+
+## Avoid library mismatch
+
+On systems with multiple MNN builds, a common pitfall is a **binary from one build tree** loading a **shared library from another**.
+
+Check which libraries `llm_demo` will use:
+
+```bash
+cd ~/mnn_lp/MNN/build
+ldd ./llm_demo
+```
+
+Ensure that `libMNN.so` and related libraries point to your **current build directory**.
+
+If you see a mismatch, set `LD_LIBRARY_PATH` explicitly:
+
+```bash
+export LD_LIBRARY_PATH=~/mnn_lp/MNN/build:$LD_LIBRARY_PATH
+```
+
+Re-run `ldd` or `./llm_demo` to confirm the correct shared libraries are used.
+
+---
+
+## Download a prebuilt MNN Omni model
+
+This Learning Path assumes you have a prebuilt Omni model package, for example:
+
+```bash
+~/mnn_lp/Qwen2.5-Omni-7B-MNN
+```
+
+The directory should contain at least a `config.json` and the associated model files.
+
+Verify the config path:
+
+```bash
+ls -l ~/mnn_lp/Qwen2.5-Omni-7B-MNN/config.json
+```
+
+**Important:** This is an **MNN-ready Omni package**. You **do not** need to export from PyTorch or perform additional quantization.
+
+---
+
+## Checkpoint
+
+You should be able to start `llm_demo` with your config:
+
+```bash
+cd ~/mnn_lp/MNN/build
+./llm_demo ~/mnn_lp/Qwen2.5-Omni-7B-MNN/config.json
+```
+
+If the binary starts without `undefined symbol` or library errors, you are ready for the next module.

content/learning-paths/cross-platform/multimodel_mnn_v9/3_mnn_text_baseline.md

@@ -0,0 +1,32 @@
+---
+title: Run a text baseline with an Omni model on MNN
+weight: 4
+layout: "learningpathall"
+---
+
+## Steps
+
+Create a simple text prompt file:
+
+```bash
+cat > ~/mnn_lp/text_baseline_prompt.txt <<'EOF'
+You are an on-device assistant. In one short sentence, describe the goal of this tutorial.
+EOF
+```
+
+Run `llm_demo`:
+
+```bash
+cd ~/mnn_lp/MNN/build
+./llm_demo ~/mnn_lp/Qwen2.5-Omni-7B-MNN/config.json ~/mnn_lp/text_baseline_prompt.txt
+```
+
+You should see a short textual response describing the tutorial’s goal.
+
+## Checkpoint
+
+You have completed this module when:
+
+- `llm_demo` successfully loads the config and model
+- At least one text prompt returns a stable, sensible response
+- No crashes or runtime errors occur

content/learning-paths/cross-platform/multimodel_mnn_v9/4_mnn_vision.md

@@ -0,0 +1,72 @@
+---
+title: Run a vision retail shelf audit with MNN Omni
+weight: 5
+layout: "learningpathall"
+---
+
+## Scenario
+
+In this module you will implement a **smart retail shelf audit** using a single local image.
+
+The model will:
+
+- Inspect a **pet food aisle** image
+- Estimate **facing coverage** for three shelf levels: **top / middle / bottom**
+- Identify a **priority zone** that appears most sparse
+- Use `NOT_SURE` where the image is ambiguous
+
+
+## Assets
+
+Create an `assets` directory:
+
+```bash
+mkdir -p ~/mnn_lp/assets
+```
+
+Download the tutorial image:
+
+```bash
+curl -L -o ~/mnn_lp/assets/pet_food_aisle.jpg \
+  "https://upload.wikimedia.org/wikipedia/commons/5/57/Pet_Food_Aisle.jpg"
+
+file ~/mnn_lp/assets/pet_food_aisle.jpg
+```
+
+The `file` command confirms that this is a valid JPEG image.
+
+
+## Prompt design
+
+You will design a prompt that:
+
+- Uses `<img>…</img>` to attach the local image
+- Restricts the model to auditing **only the main left shelf**
+- Requests coverage as **high/medium/low** for each level
+- Asks for a single **priority zone** on the left shelf
+- Encodes the output as a **single line** with `;`-separated segments
+
+Create the prompt file (adjust `/home/radxa` to your actual user home if needed):
+
+```bash
+cat > ~/mnn_lp/prompt_picture_coverage.txt <<'EOF'
+<img>/home/radxa/mnn_lp/assets/pet_food_aisle.jpg</img> You are an on-device retail shelf auditing assistant. Audit ONLY the main left shelf (ignore the aisle on the right, hanging toys, and floor items). Do NOT count every item. Estimate facing coverage for top/middle/bottom as high|medium|low and identify the sparsest zone. Output ONE line only using bullet-style segments separated by semicolons: Shelf audit; - Coverage: top=<high|medium|low>, middle=<high|medium|low>, bottom=<high|medium|low>; - Priority zone: <top|middle|bottom>-<left|center|right>; - Reason: <one short sentence>; - Notes: <NOT_SURE if unclear>.
+EOF
+```
+
+Run the vision demo:
+
+```bash
+cd ~/mnn_lp/MNN/build
+./llm_demo ~/mnn_lp/Qwen2.5-Omni-7B-MNN/config.json ~/mnn_lp/prompt_picture_coverage.txt
+```
+
+
+## Checkpoint
+
+You have completed this module when the output:
+
+- Includes coverage estimates for **top / middle / bottom**
+- Identifies a **priority zone** such as `middle-center`
+- Provides a **short reason** that clearly references visible sparsity on the shelf
+- Uses `NOT_SURE` only where the image is genuinely unclear

content/learning-paths/cross-platform/multimodel_mnn_v9/5_mnn_vision_audio.md

@@ -0,0 +1,71 @@
+---
+title: Convert audio restock instructions into tickets with MNN Omni
+weight: 6
+layout: "learningpathall"
+---
+
+## Scenario
+
+In this module you will convert a **spoken restock note** into a structured **restock ticket**.
+
+For example, a store associate might say:
+
+> We need to restock large dog food bags on the bottom-left, four units, before tomorrow afternoon. If large bags are out of stock, use medium bags as a substitute.
+
+Your goal is to:
+
+- Extract **items, quantities, zones, deadlines, and substitution policy**
+- Encode them as a **single-line ticket** with clearly labeled fields
+
+## Assets
+
+Prepare a WAV file under `assets`:
+
+```bash
+file ~/mnn_lp/assets/restock_note.wav
+```
+
+If you only have an MP3 file, convert it to a suitable WAV format:
+
+```bash
+ffmpeg -y -i input.mp3 -ac 1 -ar 16000 -c:a pcm_s16le ~/mnn_lp/assets/restock_note.wav
+```
+
+This produces a **mono, 16 kHz, 16-bit PCM** WAV file, which is commonly used for speech models.
+
+
+## Prompt design
+
+You will ask the model to produce a **readable, single-line ticket** with:
+
+- Item
+- Quantity (or `NOT_SURE` if not spoken)
+- Shelf zone
+- Deadline
+- Substitution policy
+- Notes and confidence
+
+Create the prompt file:
+
+```bash
+cat > ~/mnn_lp/prompt_audio_ticket.txt <<'EOF'
+<audio>/home/radxa/mnn_lp/assets/restock_note.wav</audio> You are a retail store replenishment assistant. Convert the spoken note into a restocking ticket. Output EXACTLY ONE line using bullet-style segments separated by semicolons: Restock ticket; - Location: pet food aisle left shelf; - Task 1: <generic item> | qty <number or NOT_SURE> | zone <top|middle|bottom>-<left|center|right or NOT_SURE>; - Task 2: <generic item> | qty <number or NOT_SURE> | zone <top|middle|bottom>-<left|center|right or NOT_SURE>; - Deadline: <time or NOT_SURE>; - Substitution: <use similar substitute|no substitute|NOT_SURE>; - Notes: <short note>; - Confidence: <high|medium|low>. Rules: do not invent quantities if not in audio.
+EOF
+```
+
+Run the audio demo:
+
+```bash
+cd ~/mnn_lp/MNN/build
+./llm_demo ~/mnn_lp/Qwen2.5-Omni-7B-MNN/config.json ~/mnn_lp/prompt_audio_ticket.txt
+```
+
+
+## Checkpoint
+
+You have completed this module when:
+
+- The ticket’s **quantities and deadline** match what is actually spoken
+- Fields that are **not mentioned** in the audio are filled with `NOT_SURE` rather than invented values
+- The resulting line is easy to parse and understand
+