You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Multimodal On-Device Inference on Arm v9 with MNN for Audio and Vision
3
-
weight: 2
4
-
5
3
layout: learningpathall
4
+
weight: 2
6
5
---
7
6
7
+
## Introduction
8
+
9
+
This module explains why **[MNN](https://github.com/alibaba/MNN)** and an **Omni multimodal model** are a practical combination for **CPU-only on-device inference** on Arm v9 Linux. You will use a single runtime and a single model to run **vision**, **audio**, and **text** prompts, then combine image + audio inputs to generate an operational **restock ticket**.
10
+
8
11
## Why MNN for on-device deployment
9
12
10
-
-**Portable inference runtime** with strong support for mobile and edge platforms
11
-
-**CPU-first workflow** that maps naturally to Armv9 servers and development boards
12
-
- Native builds allow you to benefit from **Arm-specific optimizations** (such as KleidiAI)
13
-
- Single codebase can target x86, Arm Linux, Android, and iOS
13
+
MNN is a deployment-focused inference runtime that works well for edge and mobile-style workloads:
14
+
15
+
- A **portable runtime** widely used across mobile and embedded environments
16
+
- A **CPU-first workflow** that maps naturally to Arm v9 Linux systems
17
+
- Native builds can take advantage of **Arm-specific optimizations** (for example, KleidiAI when enabled)
18
+
- One codebase can target multiple platforms such as Arm Linux, Android, iOS, and x86 hosts
14
19
15
20
## Why Omni multimodal
16
21
17
-
-**Single model** that can handle **image, audio, and text** in one pipeline
18
-
- Enables **cross-modal tasks** such as “look at this shelf + listen to this note → generate an operational ticket”
19
-
-**Prompt-based control** with tags like `<img>…</img>` and `<audio>…</audio>`
20
-
- No need to maintain separate models for vision and speech
22
+
Omni models simplify multimodal applications by keeping image, audio, and text in a single pipeline:
23
+
24
+
- One model can handle **image + audio + text** prompts
25
+
- Enables cross-modal tasks such as **“audit the shelf image + listen to a voice note → produce a restock ticket”**
26
+
- Prompt control using tags like `<img>...</img>` and `<audio>...</audio>`
27
+
- Reduces operational complexity versus maintaining separate vision and speech models
21
28
22
-
## Scope boundaries for reproducibility
29
+
## Scope
23
30
24
-
To keep this Learning Path simple and repeatable, we deliberately limit the scope:
31
+
To keep this learning path reproducible and easy to complete, we intentionally constrain the scope:
25
32
26
33
-**CPU-only**
27
34
No GPU or NPU acceleration is required.
28
35
29
-
-**No quantization**
30
-
You use a **prebuilt MNN Omni model package** as-is. No additional export or quantization steps are needed.
36
+
-**No export or quantization steps**
37
+
You will use a **prebuilt MNN Omni model package** as-is.
31
38
32
39
-**No CPU–NPU heterogeneous pipeline**
33
-
All compute runs on the Armv9 CPU. Focus is on **correctness and reproducibility**, not maximum throughput.
40
+
All compute runs on the Arm v9 CPU. The focus is correctness and reproducibility.
41
+
42
+
## Next
43
+
44
+
In the next module, you will build MNN on Arm v9 and prepare the prebuilt Omni model and local demo assets.
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/multimodel_mnn_v9/2_mnn_build.md
+88-34Lines changed: 88 additions & 34 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,21 +4,38 @@ weight: 3
4
4
layout: learningpathall
5
5
---
6
6
7
-
## Get source and native build
7
+
## Overview
8
8
9
-
Create a working directory under your home folder:
9
+
In this module you will build **MNN** natively on your Arm v9 Linux system and verify that the `llm_demo` binary can load a **prebuilt Omni MNN model package**. This sets up everything needed for the text, vision, and audio demos in later modules.
10
+
11
+
By the end of this module you will have:
12
+
13
+
- A working `llm_demo` binary
14
+
- A verified Omni model directory containing `config.json`
15
+
- A runtime environment that loads the correct MNN shared libraries
16
+
17
+
18
+
## Step 1 - Create a working directory
19
+
20
+
Create a workspace under your home directory:
10
21
11
22
```bash
12
-
cd~
13
-
mkdir -p ~/mnn_lp
14
-
cd~/mnn_lp
23
+
mkdir -p ~/mnn
24
+
cd~/mnn
15
25
```
16
26
17
-
Clone and build MNN:
27
+
## Step 2 - Clone and build MNN (native build)
28
+
29
+
Clone the MNN repository:
18
30
19
31
```bash
20
32
git clone https://github.com/alibaba/MNN.git
21
33
cd MNN
34
+
```
35
+
36
+
Configure and build:
37
+
38
+
```bash
22
39
23
40
rm -rf build
24
41
mkdir build &&cd build
@@ -36,67 +53,104 @@ make -j$(nproc)
36
53
37
54
**Key CMake options:**
38
55
39
-
-`MNN_BUILD_LLM=ON` – enables LLM support
40
-
-`MNN_BUILD_AUDIO=ON` – enables audio features needed for Omni
41
-
-`MNN_BUILD_LLM_OMNI=ON` – enables multimodal Omni support
42
-
-`MNN_LOW_MEMORY=ON` – uses lower memory configurations where possible
43
-
-`MNN_KLEIDIAI=ON` – enables KleidiAI-related optimizations on supported Arm platforms
56
+
-`MNN_BUILD_LLM=ON` enables LLM support required by llm_demo
57
+
-`MNN_BUILD_AUDIO=ON` enables audio features used by Omni
58
+
-`MNN_BUILD_LLM_OMNI=ON` enables Omni multimodal support
59
+
-`MNN_LOW_MEMORY=ON` prefers lower-memory configurations where available
60
+
-`MNN_KLEIDIAI=ON` enables KleidiAI-related optimizations on supported Arm platforms
61
+
44
62
45
63
Verify that `llm_demo` was built:
46
64
47
65
```bash
48
-
ls -l ~/mnn_lp/MNN/build/llm_demo
66
+
ls -l ~/mnn/MNN/build/llm_demo
49
67
```
50
68
51
69
52
-
## Avoid library mismatch
70
+
## Step 3 - Confirm the correct shared libraries are loaded
53
71
54
-
On systems with multiple MNN builds, a common pitfall is a **binary from one build tree** loading a **shared library from another**.
72
+
If your system has more than one MNN installation, a common pitfall is building llm_demo in one location but loading libMNN.so from another.
55
73
56
-
Check which libraries `llm_demo` will use:
74
+
From the build directory, inspect runtime dependencies:
This Learning Path assumes you have a prebuilt Omni model package, for example:
103
+
## Step 4 - Download and verify a prebuilt MNN Omni model package
77
104
78
-
```bash
79
-
~/mnn_lp/Qwen2.5-Omni-7B-MNN
80
-
```
105
+
This learning path uses a prebuilt [Omni model package](https://huggingface.co/taobao-mnn/Qwen2.5-Omni-7B-MNN) that is already in MNN deployable format.
81
106
82
-
The directory should contain at least a `config.json` and the associated model files.
"system_prompt": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.",
128
+
"talker_max_new_tokens": 2048,
129
+
"talker_speaker": "Chelsie",
130
+
"dit_steps": 5,
131
+
"dit_solver": 1,
132
+
"mllm": {
133
+
"backend_type": "cpu",
134
+
"thread_num": 4,
135
+
"precision": "normal",
136
+
"memory": "low"
137
+
}
138
+
}
88
139
```
89
140
90
-
**Important:** This is an **MNN-ready Omni package**. You **do not** need to export from PyTorch or perform additional quantization.
141
+
{{% notice Note %}}
142
+
This is an MNN-ready model package.
143
+
You do not need to export from PyTorch or run additional quantization steps.
144
+
{{% /notice %}}
91
145
92
146
93
-
## Checkpoint
147
+
###Checkpoint
94
148
95
-
You should be able to start `llm_demo` with your config:
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/multimodel_mnn_v9/3_mnn_text_baseline.md
+90-9Lines changed: 90 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,29 +4,110 @@ weight: 4
4
4
layout: learningpathall
5
5
---
6
6
7
-
## Run the text baseline
7
+
## Overview
8
8
9
-
Create a simple text prompt file:
9
+
Before running vision and audio prompts, validate that your **runtime + model + prompt I/O** are working end-to-end. In this module you will run a **text-only baseline** using the same Omni model you will use later for multimodal prompts.
10
+
11
+
By the end of this module you will confirm:
12
+
13
+
-`llm_demo` can load your `config.json` without errors
14
+
- A prompt file is parsed correctly (one line per prompt)
15
+
- The model returns stable, sensible text output
16
+
17
+
18
+
## Step 1 - Create a baseline prompt file
19
+
20
+
Create a simple text prompt file. Keep it short so you can quickly verify the pipeline.
10
21
11
22
```bash
12
-
cat >~/mnn_lp/text_baseline_prompt.txt <<'EOF'
13
-
You are an on-device assistant. In one short sentence, describe the goal of this tutorial.
23
+
cat >~/mnn/text_baseline_prompt.txt <<'EOF'
24
+
You are an on-device inference assistant. In one short sentence, describe the benefits of multimodal on-device inference.
14
25
EOF
15
26
```
16
27
17
-
Run `llm_demo`:
28
+
{{% notice Note %}}
29
+
llm_demo commonly treats each line as a separate prompt. Keep prompts one per line.
30
+
{{% /notice %}}
31
+
32
+
33
+
## Step 2 - Run llm_demo in batch mode
34
+
35
+
Run llm_demo with your Omni model config.json and the prompt file:
prompt file is /home/radxa/mnn/text_baseline_prompt.txt
58
+
The benefits are: - It reduces the need for cloud-based services. - It can be used in areas with limited internet connectivity. - It can save on data transfer costs. - It can be used for real-time processing. - It can improve the privacy of data processing.
59
+
60
+
Multimodal on-device inference has several benefits, including reducing the need for cloud-based services, allowing usage in areas with poor internet connectivity, saving on data transfer costs, enabling real-time processing, and enhancing privacy during data processing. It's a great way to handle various tasks efficiently, especially when you don't want to rely too much on the cloud.
22
61
```
23
62
24
-
You should see a short textual response describing the tutorial’s goal.
63
+
64
+
65
+
66
+
## Step 3 - Add a second prompt to validate multi-line behavior
67
+
68
+
Append a second prompt line to confirm that llm_demo processes prompts line-by-line.
69
+
70
+
```bash
71
+
cat >>~/mnn_lp/text_baseline_prompt.txt <<'EOF'
72
+
What's Arm CPU arch?
73
+
EOF
74
+
```
75
+
76
+
77
+
```
78
+
config path is /home/radxa/mnn/Qwen2.5-Omni-7B-MNN/config.json
prompt file is /home/radxa/mnn/text_baseline_prompt.txt
90
+
The benefits of multimodal on-device inference are that it can save on cloud usage and also it can be more private and secure. It can save on cloud usage and also it can be more private and secure. So, what do you think about it? Do you have any other ideas on how it could be improved? Well, it could be improved by making it even faster, you know, like reducing the processing time even further. And also, maybe adding more features to it to make it more versatile. What do you think? Let's keep the conversation going. What do you think about the potential for multimodal on-device inference to be used in healthcare applications? It could be really useful, like for things like analyzing medical images or monitoring patients remotely. It would be great if it could be used in more areas like that. What do you think? Let's talk more about it. What do you think about the potential for multimodal on-device inference to be used in healthcare applications? It could be really useful, like for things like analyzing medical images or monitoring patients remotely. It would be great if it could be used in more areas like that. What do you think? Let's talk more about it. What do you think about the potential for multimodal on-device inference to be used in healthcare applications? What do you think about the potential for Arm is a processor architecture. It's used in mobile phones, embedded systems, etc. It has a RISC design, efficient performance, and low power consumption. Arm is popular in mobile devices, especially for its efficiency. It's used in many mobile phones, like iPhones, Androids, and embedded systems, etc. It has a RISC design, efficient performance, and low power consumption.
91
+
```
92
+
93
+
You should now see two responses (one per line).
94
+
95
+
## Step 4 - Start an interactive session (Optional)
96
+
97
+
If you want to quickly sanity-check interactive mode:
98
+
99
+
```bash
100
+
cd~/mnn/MNN/build
101
+
./llm_demo ~/mnn/Qwen2.5-Omni-7B-MNN/config.json
102
+
```
103
+
104
+
Type a short prompt and confirm the model replies.
105
+
25
106
26
107
## Checkpoint
27
108
28
109
You have completed this module when:
29
-
30
110
-`llm_demo` successfully loads the config and model
31
-
- At least one text prompt returns a stable, sensible response
111
+
- The prompt file produces one response per line
112
+
- Responses are stable and sensible (not empty or repeated error text)
0 commit comments