You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Multimodal On-Device Inference on Arm v9 with MNN for Audio and Vision
2
+
title: Run multimodal inference with MNN on Armv9
3
3
layout: learningpathall
4
4
weight: 2
5
5
---
6
6
7
7
## Introduction
8
8
9
-
This module explains why **[MNN](https://github.com/alibaba/MNN)**and an **Omni multimodal model**are a practical combination for **CPU-only on-device inference** on Arm v9 Linux. You will use a single runtime and a single model to run **vision**, **audio**, and **text** prompts, then combine image + audio inputs to generate an operational **restock ticket**.
9
+
This section introduces the software stack used throughout this Learning Path. You will use **[MNN](https://github.com/alibaba/MNN)**to run a prebuilt **Omni multimodal model**on an Armv9 Linux system using only the CPU.
10
10
11
-
## Why MNN for on-device deployment
11
+
By the end of this section, you will understand why this combination is a practical starting point for reproducible multimodal inference on Armv9.
12
12
13
-
MNN is a deployment-focused inference runtime that works well for edge and mobile-style workloads:
13
+
## Why use MNN on Armv9
14
14
15
-
- A **portable runtime** widely used across mobile and embedded environments
16
-
- A **CPU-first workflow** that maps naturally to Arm v9 Linux systems
17
-
- Native builds can take advantage of **Arm-specific optimizations** (for example, KleidiAI when enabled)
18
-
- One codebase can target multiple platforms such as Arm Linux, Android, iOS, and x86 hosts
15
+
MNN is a lightweight inference engine designed for deployment across mobile, embedded, and edge platforms. It is a good fit for this Learning Path for four reasons:
19
16
20
-
## Why Omni multimodal
17
+
- It provides a **portable runtime** that can be built and used across different device classes.
18
+
- It supports a **CPU-first deployment flow**, which maps well to Armv9 development systems.
19
+
- Native builds can take advantage of **Arm-specific optimizations** when they are enabled in the build.
20
+
- The same runtime approach can be reused across **Arm Linux, Android, iOS, and x86-based development hosts**.
21
21
22
-
Omni models simplify multimodal applications by keeping image, audio, and text in a single pipeline:
22
+
For this Learning Path, MNN gives you a practical way to validate multimodal inference on Armv9 without introducing extra framework complexity.
23
23
24
-
- One model can handle **image + audio + text** prompts
25
-
- Enables cross-modal tasks such as **“audit the shelf image + listen to a voice note → produce a restock ticket”**
26
-
- Prompt control using tags like `<img>...</img>` and `<audio>...</audio>`
27
-
- Reduces operational complexity versus maintaining separate vision and speech models
24
+
## Why use an Omni multimodal model
28
25
29
-
## Scope
26
+
An Omni model combines **text, image, and audio** understanding in a single inference pipeline. This makes it useful for building compact edge applications that need to reason over more than one input type.
30
27
31
-
To keep this learning path reproducible and easy to complete, we intentionally constrain the scope:
28
+
In this Learning Path, you use the model to:
32
29
33
-
-**CPU-only**
34
-
No GPU or NPU acceleration is required.
30
+
- process **text-only prompts**
31
+
- describe **image inputs**
32
+
- interpret **audio inputs**
33
+
- combine **image and audio context** to generate a structured **restock ticket**
35
34
36
-
-**No export or quantization steps**
37
-
You will use a **prebuilt MNN Omni model package** as-is.
35
+
This single-model approach keeps the workflow easier to follow than maintaining separate models for vision and speech tasks.
38
36
39
-
-**No CPU–NPU heterogeneous pipeline**
40
-
All compute runs on the Arm v9 CPU. The focus is correctness and reproducibility.
37
+
## Scope of this Learning Path
41
38
42
-
## Next
39
+
To keep the workflow reproducible, this Learning Path uses a deliberately narrow scope:
43
40
44
-
In the next module, you will build MNN on Arm v9 and prepare the prebuilt Omni model and local demo assets.
41
+
-**CPU-only execution**
42
+
All inference runs on the Armv9 CPU.
43
+
44
+
-**Prebuilt model assets**
45
+
You use a prepared MNN Omni model package instead of exporting or converting models.
46
+
47
+
-**No heterogeneous scheduling**
48
+
This example does not use GPU, NPU, or split CPU-accelerator execution.
49
+
50
+
This scope keeps the focus on setup, validation, and multimodal application flow.
51
+
52
+
## Next steps
53
+
54
+
In the next section, you will build MNN on Armv9 and prepare the model files and local assets used in the remaining examples.
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/multimodel_mnn_v9/2_mnn_build.md
+42-39Lines changed: 42 additions & 39 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,27 +4,28 @@ weight: 3
4
4
layout: learningpathall
5
5
---
6
6
7
-
## Overview
7
+
## Introduction
8
8
9
9
In this module you will build **MNN** natively on your Arm v9 Linux system and verify that the `llm_demo` binary can load a **prebuilt Omni MNN model package**. This sets up everything needed for the text, vision, and audio demos in later modules.
10
10
11
-
By the end of this module you will have:
11
+
This creates the software environment used in the next modules, where you run text, vision, and audio-enabled multimodal inference on Armv9.
12
12
13
-
- A working `llm_demo` binary
14
-
- A verified Omni model directory containing `config.json`
15
-
- A runtime environment that loads the correct MNN shared libraries
13
+
At the end of this module, you will have:
16
14
15
+
- a working `llm_demo` binary
16
+
- a validated model directory that includes `config.json`
17
+
- a runtime environment that resolves the correct MNN shared libraries
17
18
18
-
## Step 1 - Create a working directory
19
+
## Create a workspace
19
20
20
-
Create a workspace under your home directory:
21
+
Create a working directory under your home folder:
Configure and build MNN with LLM, audio, and Omni support enabled:
37
38
38
39
```bash
39
40
@@ -51,72 +52,74 @@ cmake .. \
51
52
make -j$(nproc)
52
53
```
53
54
54
-
**Key CMake options:**
55
+
The most important CMake options are:
55
56
56
-
-`MNN_BUILD_LLM=ON`enables LLM support required by llm_demo
57
-
-`MNN_BUILD_AUDIO=ON`enables audio features used by Omni
58
-
-`MNN_BUILD_LLM_OMNI=ON`enables Omni multimodal support
59
-
-`MNN_LOW_MEMORY=ON`prefers lower-memory configurations where available
60
-
-`MNN_KLEIDIAI=ON`enables KleidiAI-related optimizations on supported Arm platforms
57
+
-`MNN_BUILD_LLM=ON`to enable LLM support required by `llm_demo`
58
+
-`MNN_BUILD_AUDIO=ON`to enable the audio components used by the Omni model
59
+
-`MNN_BUILD_LLM_OMNI=ON`to enable multimodal Omni support
60
+
-`MNN_LOW_MEMORY=ON`to prefer lower-memory runtime settings where available
61
+
-`MNN_KLEIDIAI=ON`to enable KleidiAI optimizations on supported Arm platforms
61
62
62
-
63
-
Verify that `llm_demo` was built:
63
+
Verify that the `llm_demo` binary was created:
64
64
65
65
```bash
66
66
ls -l ~/mnn/MNN/build/llm_demo
67
67
```
68
68
69
+
## Verify shared library resolution
69
70
70
-
## Step 3 - Confirm the correct shared libraries are loaded
71
-
72
-
If your system has more than one MNN installation, a common pitfall is building llm_demo in one location but loading libMNN.so from another.
71
+
If your system already has another MNN installation, `llm_demo` can load a different `libMNN.so` than the one you just built. This is a common source of runtime errors.
73
72
74
-
From the build directory, inspect runtime dependencies:
73
+
From the build directory, inspect the runtime dependencies:
## Step 4 - Download and verify a prebuilt MNN Omni model package
103
+
## Download a prebuilt Omni model package
104
104
105
105
This learning path uses a prebuilt [Omni model package](https://huggingface.co/taobao-mnn/Qwen2.5-Omni-7B-MNN) that is already in MNN deployable format.
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/multimodel_mnn_v9/3_mnn_text_baseline.md
+37-31Lines changed: 37 additions & 31 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,23 +1,22 @@
1
1
---
2
-
title: Run a text baseline with an Omni model on MNN
2
+
title: Validate text-only inference with an Omni model on Armv9
3
3
weight: 4
4
4
layout: learningpathall
5
5
---
6
6
7
-
## Overview
7
+
## Introduction
8
8
9
-
Before running vision and audio prompts, validate that your **runtime + model + prompt I/O** are working end-to-end. In this module you will run a**text-only baseline**using the same Omni model you will use later for multimodal prompts.
9
+
Before you add image and audio inputs, validate the basic inference path with a text-only prompt. In this module, you use the same**Qwen2.5-Omni-7B-MNN**package from the previous section and confirm that `llm_demo` can run prompts from a file on your Armv9 system.
10
10
11
-
By the end of this module you will confirm:
11
+
At the end of this module, you will have verified:
12
12
13
-
-`llm_demo` can load your `config.json` without errors
14
-
-A prompt file is parsed correctly (one line per prompt)
15
-
-The model returns stable, sensible text output
13
+
-the model configuration loads correctly
14
+
- prompt input from a text file is parsed as expected
15
+
-the runtime returns usable text output without crashes or library errors
16
16
17
+
## Create a text baseline prompt
17
18
18
-
## Step 1 - Create a baseline prompt file
19
-
20
-
Create a simple text prompt file. Keep it short so you can quickly verify the pipeline.
19
+
Create a short prompt file in your workspace:
21
20
22
21
```bash
23
22
cat >~/mnn/text_baseline_prompt.txt <<'EOF'
@@ -26,23 +25,23 @@ EOF
26
25
```
27
26
28
27
{{% notice Note %}}
29
-
llm_demo commonly treats each line as a separate prompt. Keep prompts one per line.
28
+
`llm_demo` commonly treats each line in the file as a separate prompt. Keep each prompt on a single line.
30
29
{{% /notice %}}
31
30
31
+
## Run `llm_demo` with the prompt file
32
32
33
-
## Step 2 - Run llm_demo in batch mode
34
-
35
-
Run llm_demo with your Omni model config.json and the prompt file:
33
+
Run `llm_demo` from the MNN build directory and pass both the model configuration and prompt file:
0 commit comments