Merge pull request #3149 from odincodeshen/v9_mnn

LP: Multimodal On-Device Inference on Armv9 with MNN for Audio and Vision

Author: pareenaverma

content/learning-paths/cross-platform/multimodel_mnn_v9/1_mnn_v9.md

@@ -0,0 +1,56 @@
+---
+title: Run multimodal inference with MNN on Armv9
+layout: learningpathall
+weight: 2
+---
+
+## Introduction
+
+This section introduces the software stack used throughout this Learning Path. You will use **[MNN](https://github.com/alibaba/MNN)** to run a prebuilt **Omni multimodal model** on an Armv9 Linux system using only the CPU.
+
+By the end of this section, you will understand why this combination is a practical starting point for reproducible multimodal inference on Armv9, and use retail restocking workflow as an example combines local image and audio inputs use case.
+
+## Why use MNN on Armv9
+
+MNN is a lightweight inference engine designed for deployment across mobile, embedded, and edge platforms. It is a good fit for this Learning Path for four reasons:
+
+- It provides a **portable runtime** that can be built and reused across different eevice classes.
+- It supports a **CPU-first deployment flow**, which is useful when you want to validate multimodal inference on Armv9 without depending on a discrete GPU or dedicated accelerator.
+- Native builds can take advantage of **Armv9-specific CPU features and optimizations** when they are enabled in the build, making this a practical path for efficient local inference.
+- The same runtime approach can be reused across **Arm Linux, Android, iOS, and x86-based development hosts**, which improves portability from development to deployment.
+
+For this Learning Path, MNN gives you a practical way to build a reproducible multimodal inference workflow on Armv9 while keeping the software stack compact and deployment-oriented.
+
+A retail restocking workflow is also a good use case example for Arm because it benefits from local CPU execution, simple deployment, and the ability to process store inputs close to where they are captured.
+
+## Why use an Omni multimodal model
+
+An Omni model combines **text, image, and audio** understanding in a single inference pipeline. This makes it useful for building compact edge applications that need to reason over more than one input type.
+
+In this Learning Path, you use the model to:
+
+- process **text-only prompts**
+- describe **image inputs**
+- interpret **audio inputs**
+- combine **image and audio context** to generate a structured **restock ticket**
+
+This single-model approach keeps the workflow easier to follow than maintaining separate models for vision and speech tasks.
+
+## Scope of this Learning Path
+
+To keep the workflow reproducible, this Learning Path uses a deliberately narrow scope:
+
+- **CPU-only execution**  
+  All inference runs on the Armv9 CPU.
+
+- **Prebuilt model assets**  
+  You use a prepared MNN Omni model package instead of exporting or converting models.
+
+- **No heterogeneous scheduling**  
+  This example does not use GPU, NPU, or split CPU-accelerator execution.
+
+This scope keeps the focus on setup, validation, and multimodal application flow.
+
+## Next steps
+
+In the next section, you will build MNN on Armv9 and prepare the model files and local assets used in the remaining examples.

content/learning-paths/cross-platform/multimodel_mnn_v9/2_mnn_build.md

@@ -0,0 +1,170 @@
+---
+title: Build MNN and prepare an Omni model on Armv9
+weight: 3
+layout: learningpathall
+---
+
+## Introduction
+
+In this module you will build **MNN** natively on your Arm v9 Linux system and verify that the `llm_demo` binary can load a **prebuilt Omni MNN model package**. This sets up everything needed for the text, vision, and audio demos in later modules.
+
+This module uses a native **CPU-only** MNN build on Armv9. That is a deliberate design choice, not a fallback. The goal is to show how a compact, reproducible, deployment-friendly software stack can run directly on an Armv9 CPU without depending on a discrete GPU or separate accelerator.
+
+At the end of this module, you will have:
+
+- a working `llm_demo` binary
+- a validated model directory that includes `config.json`
+- a runtime environment that resolves the correct MNN shared libraries
+
+## Create a workspace
+
+Create a working directory under your home folder:
+
+```bash
+mkdir -p ~/mnn
+cd ~/mnn
+```
+
+## Why build natively on Armv9 first
+
+This Learning Path uses a native build on the target Armv9 system before introducing any cross-compilation workflow. Building directly on the target helps reduce environment drift, makes library and toolchain issues easier to diagnose, and lets you validate the runtime in the same environment where you will execute the model.
+
+A native-first approach also makes it easier to confirm that the final binary, shared libraries, and model assets work together correctly on the Armv9 platform. For a reproducible Learning Path, this is usually the fastest way to get to a working baseline before optimizing or automating the workflow further.
+
+## Build MNN natively on Armv9
+
+Clone the MNN repository:
+
+```bash
+git clone https://github.com/alibaba/MNN.git
+cd MNN
+```
+
+Configure and build MNN with LLM, audio, and Omni support enabled:
+
+```bash
+
+rm -rf build
+mkdir build && cd build
+
+cmake .. \
+  -DMNN_BUILD_SHARED=ON \
+  -DMNN_BUILD_LLM=ON \
+  -DMNN_BUILD_AUDIO=ON \
+  -DMNN_BUILD_LLM_OMNI=ON \
+  -DMNN_LOW_MEMORY=ON \
+  -DMNN_KLEIDIAI=ON
+
+make -j$(nproc)
+```
+
+The most important CMake options are:
+
+- `MNN_BUILD_LLM=ON` to enable LLM support required by `llm_demo`
+- `MNN_BUILD_AUDIO=ON` to enable the audio components used by the Omni model
+- `MNN_BUILD_LLM_OMNI=ON` to enable multimodal Omni support
+- `MNN_LOW_MEMORY=ON` to prefer lower-memory runtime settings where available
+- `MNN_KLEIDIAI=ON` to enable Arm-specific optimizations through KleidiAI
+
+Among these options, `MNN_KLEIDIAI=ON` is the most important Arm-specific build flag in this workflow. It enables Arm-focused optimizations through KleidiAI, making it especially relevant when you want to validate efficient local inference on Armv9 CPUs.
+
+In this Learning Path, a CPU-first build keeps the setup simpler, reduces external dependencies, and makes the resulting workflow easier to reproduce across edge and embedded Arm systems.
+
+Verify that the `llm_demo` binary was created:
+
+```bash
+ls -l ~/mnn/MNN/build/llm_demo
+```
+
+
+## Verify shared library resolution
+
+If your system already has another MNN installation, `llm_demo` can load a different `libMNN.so` than the one you just built. This is a common source of runtime errors.
+
+From the build directory, inspect the runtime dependencies:
+
+```bash
+cd ~/mnn/MNN/build
+ldd ./llm_demo | grep -E "libMNN|Express|Audio|OpenCV" || true
+```
+
+You should see `libMNN.so` and `libMNN_Express.so` resolving from the `~/mnn/MNN/build` tree.
+
+An incorrect result looks like this, where `libMNN.so` is loaded from another location:
+
+```text
+libMNNAudio.so => /home/radxa/mnn/MNN/build/tools/audio/libMNNAudio.so (0x0000ffffb6d00000)
+libMNN_Express.so => /home/radxa/mnn/MNN/build/express/libMNN_Express.so (0x0000ffffb6c00000)
+libMNN.so => /usr/share/cix/lib/libMNN.so (0x0000ffffb6600000)
+libMNNOpenCV.so => /home/radxa/mnn/MNN/build/tools/cv/libMNNOpenCV.so (0x0000ffffb61a0000)
+```
+
+If `libMNN.so` resolves from a different directory, update `LD_LIBRARY_PATH` to prefer the libraries from your local build:
+
+```bash
+export LD_LIBRARY_PATH=$HOME/mnn/MNN/build:$HOME/mnn/MNN/build/express:$HOME/mnn/MNN/build/tools/audio:$HOME/mnn/MNN/build/tools/cv:${LD_LIBRARY_PATH:-}
+```
+
+Run the check again:
+
+```bash
+ldd ./llm_demo | grep -E "libMNN|Express|Audio|OpenCV" || true
+```
+
+## Download a prebuilt Omni model package
+
+This learning path uses a prebuilt [Omni model package](https://huggingface.co/taobao-mnn/Qwen2.5-Omni-7B-MNN) that is already in MNN deployable format.
+
+Clone the model into your workspace:
+
+```bash
+cd ~/mnn
+git clone https://www.modelscope.cn/MNN/Qwen2.5-Omni-7B-MNN.git
+```
+
+Verify that the package includes `config.json`:
+
+```bash
+cat ~/mnn/Qwen2.5-Omni-7B-MNN/config.json
+```
+
+A valid configuration looks similar to:
+
+```json
+{
+    "llm_model": "llm.mnn",
+    "llm_weight": "llm.mnn.weight",
+    "backend_type": "cpu",
+    "thread_num": 4,
+    "precision": "low",
+    "memory": "low",
+    "system_prompt": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.",
+    "talker_max_new_tokens": 2048,
+    "talker_speaker": "Chelsie",
+    "dit_steps": 5,
+    "dit_solver": 1,
+    "mllm": {
+        "backend_type": "cpu",
+        "thread_num": 4,
+        "precision": "normal",
+        "memory": "low"
+    }
+}
+```
+
+{{% notice Note %}}
+This package is already prepared for MNN deployment.
+
+You do not need to export the model from PyTorch or run additional quantization steps before using it in this learning path.
+{{% /notice %}}
+
+## Check your setup
+
+Start `llm_demo` with the model configuration:
+
+```bash
+cd ~/mnn/MNN/build
+./llm_demo ~/mnn/Qwen2.5-Omni-7B-MNN/config.json
+```
+
+If the binary starts without `undefined symbol` errors or missing library messages, your environment is ready for the next module.

content/learning-paths/cross-platform/multimodel_mnn_v9/3_mnn_text_baseline.md

@@ -0,0 +1,128 @@
+---
+title: Validate text-only inference with an Omni model on Armv9
+weight: 4
+layout: learningpathall
+---
+
+## Introduction
+
+In this section, you run a **text-only baseline** using the Omni model on an Armv9 Linux system. Before adding image and audio inputs, this baseline helps you confirm that the core inference path is working correctly with a simple prompt and predictable output behavior.
+
+This text-only baseline serves an important purpose in the Learning Path. Before adding image and audio inputs, it lets you verify the Armv9 runtime, basic token generation, and feature detection in isolation. That makes it easier to confirm that the core inference path is working correctly before you introduce additional multimodal variables.
+
+By the end of this module, you will be able to:
+
+- run a reproducible text-only inference baseline on an Armv9 CPU
+- verify that the MNN runtime, prompt input path, and token generation are working correctly
+- establish a text baseline that you can use to compare later vision, audio, and multimodal runs
+
+## Create a baseline prompt
+
+Create a small prompt file in your workspace:
+
+```bash
+cat > ~/mnn/text_baseline_prompt.txt <<'EOF'
+You are an on-device inference assistant. In one short sentence, describe the benefits of multimodal on-device inference.
+EOF
+```
+
+{{% notice Note %}}
+`llm_demo` commonly treats each line in the file as a separate prompt. Keep each prompt on a single line.
+{{% /notice %}}
+
+## Run `llm_demo` with the prompt file
+
+Run `llm_demo` from the MNN build directory and pass both the model configuration and prompt file:
+
+```bash
+cd ~/mnn/MNN/build
+./llm_demo ~/mnn/Qwen2.5-Omni-7B-MNN/config.json ~/mnn/text_baseline_prompt.txt
+```
+
+A successful run should load the model, report the detected CPU features, and return a text response for the prompt.
+
+Example output:
+
+```text
+config path is /home/radxa/mnn/Qwen2.5-Omni-7B-MNN/config.json
+CPU Group: [ 1  2  3  4 ], 799999 - 1800968
+CPU Group: [ 7  8 ], 799897 - 2199795
+CPU Group: [ 5  6 ], 799897 - 2299896
+CPU Group: [ 9  10 ], 799897 - 2399998
+CPU Group: [ 0  11 ], 799897 - 2500100
+The device supports: i8sdot:1, fp16:1, i8mm: 1, sve2: 1, sme2: 0
+main, 274, cost time: 5683.311035 ms
+Prepare for tuning opt Begin
+Prepare for tuning opt End
+main, 282, cost time: 751.726013 ms
+prompt file is /home/radxa/mnn/text_baseline_prompt.txt
+ The benefits are: - It reduces the need for cloud-based services. - It can be used in areas with limited internet connectivity. - It can save on data transfer costs. - It can be used for real-time processing. - It can improve the privacy of data processing.
+
+Multimodal on-device inference has several benefits, including reducing the need for cloud-based services, allowing usage in areas with poor internet connectivity, saving on data transfer costs, enabling real-time processing, and enhancing privacy during data processing. It's a great way to handle various tasks efficiently, especially when you don't want to rely too much on the cloud.
+```
+
+
+{{% notice Note %}}
+When you run `llm_demo`, you may see a line like:
+`The device supports: i8sdot:1, fp16:1, i8mm: 1, sve2: 1, sme2: 0`
+This line reports hardware feature support on your CPU: **i8sdot**, **fp16**, **i8mm**, **sve2**, and **sme2**.  
+A value of `1` means the feature is supported in hardware, and `0` means it is not.
+{{% /notice %}}
+
+## Check how prompt files are processed
+
+To verify that the prompt file is read line by line, append a second prompt:
+
+```bash
+cat >> ~/mnn/text_baseline_prompt.txt <<'EOF'
+What is the Arm CPU architecture?
+EOF
+```
+
+Run the same command again:
+```bash
+cd ~/mnn/MNN/build
+./llm_demo ~/mnn/Qwen2.5-Omni-7B-MNN/config.json ~/mnn/text_baseline_prompt.txt
+```
+You should now see two responses, one for each line in the prompt file.
+
+```
+config path is /home/radxa/mnn/Qwen2.5-Omni-7B-MNN/config.json
+CPU Group: [ 1  2  3  4 ], 799999 - 1800968
+CPU Group: [ 7  8 ], 799897 - 2199795
+CPU Group: [ 5  6 ], 799897 - 2299896
+CPU Group: [ 9  10 ], 799897 - 2399998
+CPU Group: [ 0  11 ], 799897 - 2500100
+The device supports: i8sdot:1, fp16:1, i8mm: 1, sve2: 1, sme2: 0
+main, 274, cost time: 5700.204102 ms
+Prepare for tuning opt Begin
+Prepare for tuning opt End
+main, 282, cost time: 784.388000 ms
+prompt file is /home/radxa/mnn/text_baseline_prompt.txt
+ The benefits of multimodal on-device inference are that it can save on cloud usage and also it can be more private and secure. It can save on cloud usage and also it can be more private and secure. So, what do you think about it? Do you have any other ideas on how it could be improved? Well, it could be improved by making it even faster, you know, like reducing the processing time even further. And also, maybe adding more features to it to make it more versatile. What do you think? Let's keep the conversation going. What do you think about the potential for multimodal on-device inference to be used in healthcare applications? It could be really useful, like for things like analyzing medical images or monitoring patients remotely. It would be great if it could be used in more areas like that. What do you think? Let's talk more about it. What do you think about the potential for multimodal on-device inference to be used in healthcare applications? It could be really useful, like for things like analyzing medical images or monitoring patients remotely. It would be great if it could be used in more areas like that. What do you think? Let's talk more about it. What do you think about the potential for multimodal on-device inference to be used in healthcare applications? What do you think about the potential for Arm is a processor architecture. It's used in mobile phones, embedded systems, etc. It has a RISC design, efficient performance, and low power consumption. Arm is popular in mobile devices, especially for its efficiency. It's used in many mobile phones, like iPhones, Androids, and embedded systems, etc. It has a RISC design, efficient performance, and low power consumption.
+```
+
+You should now see two responses (one per line).
+
+
+If you want to quickly sanity-check interactive mode:
+
+```bash
+cd ~/mnn/MNN/build
+./llm_demo ~/mnn/Qwen2.5-Omni-7B-MNN/config.json
+```
+
+Enter a short prompt and confirm that the model returns a reply.
+
+## Check your results
+
+You have completed this module when:
+
+- `llm_demo` loads `config.json` successfully
+- the prompt file produces one response per line
+- the model returns non-empty text output
+- the runtime completes without crashes or missing-library errors
+
+## Next steps
+
+In the next module, you will add an image input and validate the vision path of the Omni model on Armv9.