Skip to content

Commit 5ca7cc9

Browse files
committed
Rewrite the content, add AI use case to explain the idea.
1 parent 3eece60 commit 5ca7cc9

7 files changed

Lines changed: 56 additions & 27 deletions

File tree

content/learning-paths/cross-platform/multimodel_mnn_v9/1_mnn_v9.md

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,18 +8,20 @@ weight: 2
88

99
This section introduces the software stack used throughout this Learning Path. You will use **[MNN](https://github.com/alibaba/MNN)** to run a prebuilt **Omni multimodal model** on an Armv9 Linux system using only the CPU.
1010

11-
By the end of this section, you will understand why this combination is a practical starting point for reproducible multimodal inference on Armv9.
11+
By the end of this section, you will understand why this combination is a practical starting point for reproducible multimodal inference on Armv9, and use retail restocking workflow as an example combines local image and audio inputs use case.
1212

1313
## Why use MNN on Armv9
1414

1515
MNN is a lightweight inference engine designed for deployment across mobile, embedded, and edge platforms. It is a good fit for this Learning Path for four reasons:
1616

17-
- It provides a **portable runtime** that can be built and used across different device classes.
18-
- It supports a **CPU-first deployment flow**, which maps well to Armv9 development systems.
19-
- Native builds can take advantage of **Arm-specific optimizations** when they are enabled in the build.
20-
- The same runtime approach can be reused across **Arm Linux, Android, iOS, and x86-based development hosts**.
17+
- It provides a **portable runtime** that can be built and reused across different eevice classes.
18+
- It supports a **CPU-first deployment flow**, which is useful when you want to validate multimodal inference on Armv9 without depending on a discrete GPU or dedicated accelerator.
19+
- Native builds can take advantage of **Armv9-specific CPU features and optimizations** when they are enabled in the build, making this a practical path for efficient local inference.
20+
- The same runtime approach can be reused across **Arm Linux, Android, iOS, and x86-based development hosts**, which improves portability from development to deployment.
2121

22-
For this Learning Path, MNN gives you a practical way to validate multimodal inference on Armv9 without introducing extra framework complexity.
22+
For this Learning Path, MNN gives you a practical way to build a reproducible multimodal inference workflow on Armv9 while keeping the software stack compact and deployment-oriented.
23+
24+
A retail restocking workflow is also a good use case example for Arm because it benefits from local CPU execution, simple deployment, and the ability to process store inputs close to where they are captured.
2325

2426
## Why use an Omni multimodal model
2527

@@ -51,4 +53,4 @@ This scope keeps the focus on setup, validation, and multimodal application flow
5153

5254
## Next steps
5355

54-
In the next section, you will build MNN on Armv9 and prepare the model files and local assets used in the remaining examples.
56+
In the next section, you will build MNN on Armv9 and prepare the model files and local assets used in the remaining examples.

content/learning-paths/cross-platform/multimodel_mnn_v9/2_mnn_build.md

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ layout: learningpathall
88

99
In this module you will build **MNN** natively on your Arm v9 Linux system and verify that the `llm_demo` binary can load a **prebuilt Omni MNN model package**. This sets up everything needed for the text, vision, and audio demos in later modules.
1010

11-
This creates the software environment used in the next modules, where you run text, vision, and audio-enabled multimodal inference on Armv9.
11+
This module uses a native **CPU-only** MNN build on Armv9. That is a deliberate design choice, not a fallback. The goal is to show how a compact, reproducible, deployment-friendly software stack can run directly on an Armv9 CPU without depending on a discrete GPU or separate accelerator.
1212

1313
At the end of this module, you will have:
1414

@@ -25,6 +25,12 @@ mkdir -p ~/mnn
2525
cd ~/mnn
2626
```
2727

28+
## Why build natively on Armv9 first
29+
30+
This Learning Path uses a native build on the target Armv9 system before introducing any cross-compilation workflow. Building directly on the target helps reduce environment drift, makes library and toolchain issues easier to diagnose, and lets you validate the runtime in the same environment where you will execute the model.
31+
32+
A native-first approach also makes it easier to confirm that the final binary, shared libraries, and model assets work together correctly on the Armv9 platform. For a reproducible Learning Path, this is usually the fastest way to get to a working baseline before optimizing or automating the workflow further.
33+
2834
## Build MNN natively on Armv9
2935

3036
Clone the MNN repository:
@@ -58,14 +64,19 @@ The most important CMake options are:
5864
- `MNN_BUILD_AUDIO=ON` to enable the audio components used by the Omni model
5965
- `MNN_BUILD_LLM_OMNI=ON` to enable multimodal Omni support
6066
- `MNN_LOW_MEMORY=ON` to prefer lower-memory runtime settings where available
61-
- `MNN_KLEIDIAI=ON` to enable KleidiAI optimizations on supported Arm platforms
67+
- `MNN_KLEIDIAI=ON` to enable Arm-specific optimizations through KleidiAI
68+
69+
Among these options, `MNN_KLEIDIAI=ON` is the most important Arm-specific build flag in this workflow. It enables Arm-focused optimizations through KleidiAI, making it especially relevant when you want to validate efficient local inference on Armv9 CPUs.
70+
71+
In this Learning Path, a CPU-first build keeps the setup simpler, reduces external dependencies, and makes the resulting workflow easier to reproduce across edge and embedded Arm systems.
6272

6373
Verify that the `llm_demo` binary was created:
6474

6575
```bash
6676
ls -l ~/mnn/MNN/build/llm_demo
6777
```
6878

79+
6980
## Verify shared library resolution
7081

7182
If your system already has another MNN installation, `llm_demo` can load a different `libMNN.so` than the one you just built. This is a common source of runtime errors.

content/learning-paths/cross-platform/multimodel_mnn_v9/3_mnn_text_baseline.md

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,17 +6,19 @@ layout: learningpathall
66

77
## Introduction
88

9-
Before you add image and audio inputs, validate the basic inference path with a text-only prompt. In this module, you use the same **Qwen2.5-Omni-7B-MNN** package from the previous section and confirm that `llm_demo` can run prompts from a file on your Armv9 system.
9+
In this section, you run a **text-only baseline** using the Omni model on an Armv9 Linux system. Before adding image and audio inputs, this baseline helps you confirm that the core inference path is working correctly with a simple prompt and predictable output behavior.
1010

11-
At the end of this module, you will have verified:
11+
This text-only baseline serves an important purpose in the Learning Path. Before adding image and audio inputs, it lets you verify the Armv9 runtime, basic token generation, and feature detection in isolation. That makes it easier to confirm that the core inference path is working correctly before you introduce additional multimodal variables.
1212

13-
- the model configuration loads correctly
14-
- prompt input from a text file is parsed as expected
15-
- the runtime returns usable text output without crashes or library errors
13+
By the end of this module, you will be able to:
1614

17-
## Create a text baseline prompt
15+
- run a reproducible text-only inference baseline on an Armv9 CPU
16+
- verify that the MNN runtime, prompt input path, and token generation are working correctly
17+
- establish a text baseline that you can use to compare later vision, audio, and multimodal runs
1818

19-
Create a short prompt file in your workspace:
19+
## Create a baseline prompt
20+
21+
Create a small prompt file in your workspace:
2022

2123
```bash
2224
cat > ~/mnn/text_baseline_prompt.txt <<'EOF'

content/learning-paths/cross-platform/multimodel_mnn_v9/4_mnn_vision.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ layout: learningpathall
66

77
## Introduction
88

9-
In this module, you use a local image and an ***MNN Omni multimodal model*** to run a simple retail shelf audit on an ***Armv9*** system.
9+
This section shows how Armv9 can run lightweight operational vision reasoning locally without requiring cloud round trips, making it a practical fit for store-side restocking workflows.
1010

1111
The goal is not to produce SKU-level counting. Instead, you generate an operational signal that is useful for store staff:
1212

@@ -17,6 +17,8 @@ The goal is not to produce SKU-level counting. Instead, you generate an operatio
1717

1818
This keeps the task practical for on-device multimodal inference. In many retail workflows, deciding **where to restock first** is more useful than attempting perfect item counting from a single photo.
1919

20+
By the end of this section, you will produce a structured shelf audit signal from a local image and verify that it can guide a restocking decision
21+
2022
## Prepare the image asset
2123

2224
Create a local directory for assets:
@@ -93,6 +95,12 @@ If you have any other questions about this shelf auditing, feel free to let me k
9395

9496
The exact wording can vary, but the response should still follow the requested structure.
9597

98+
## Why a coarse coverage estimate is enough for edge workflows
99+
100+
In many edge retail workflows, you do not need a perfectly precise inventory measurement to make a useful decision. A coarse estimate such as `High`, `Medium`, or `Low` coverage is often enough to identify which shelf area needs attention first and to generate a practical restocking signal.
101+
102+
This keeps the prompt and output simple, reduces post-processing requirements, and makes the result easier to validate during early on-device prototyping. For this Learning Path, the goal is not to build a full inventory counting system, but to show how local vision reasoning on Armv9 can produce an operationally useful input for a restocking workflow.
103+
96104
## Verify the result
97105

98106
Check that the output meets these conditions:
@@ -108,4 +116,4 @@ If the model returns extra conversational text, tighten the prompt further by re
108116

109117
You have now validated image-based multimodal inference with MNN Omni on Armv9.
110118

111-
In the next module, you extend this workflow by running an audio-based restock instruction demo using an `<audio>...</audio>` prompt.
119+
In the next section, you extend this workflow by running an audio-based restock instruction demo using an `<audio>...</audio>` prompt.

content/learning-paths/cross-platform/multimodel_mnn_v9/5_mnn_vision_audio.md

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,10 @@ The spoken content can be similar to:
5757
This is the pet food aisle, left shelf. Please restock the bottom left large pet food bags, add 10 bags. Also restock the middle left canned food, add 24 cans. Finish before 3 PM today. If something is out of stock, use a similar substitute.
5858
```
5959

60+
In store inspection scenarios, unreliable connectivity, or environments where teams prefer not to upload audio in real time, local speech understanding on Armv9 has clear deployment value.
61+
62+
In this module, the audio input is not just an audio prompt. It acts as on-device spoken task capture for store operations, turning a short spoken restocking note into structured operational input.
63+
6064
## Create the audio prompt
6165

6266
Create a prompt file that attaches the WAV file and asks the model to return a single structured line.
@@ -120,15 +124,15 @@ The exact wording can vary. What matters is that the response stays close to the
120124

121125
## Verify the result
122126

123-
Check that:
127+
A successful run should show that:
124128

125-
- the quantities and deadline match the spoken note
126-
- any missing field is returned as `NOT_SURE`
127-
- the output remains on a single structured line
128-
- the ticket is readable enough for a simple downstream workflow
129+
- the model can process the local audio file on Armv9
130+
- the response keeps the requested ticket structure
131+
- explicit spoken values such as quantity or deadline appear in the output
132+
- missing or unclear fields are marked as `NOT_SURE` rather than invented
129133

130-
If the model adds extra commentary after the ticket, tighten the prompt and rerun the command. If quantities or zones are wrong, verify the audio clarity and confirm that the WAV file was converted correctly.
134+
At this stage, the goal is not to evaluate full speech recognition quality. The goal is to validate that local audio understanding can extract operational task information that you can combine with other inputs later.
131135

132136
## Next steps
133137

134-
In the next module, you will combine multimodal inference patterns into a practical restock workflow on Armv9.
138+
In the next section, you will combine multimodal inference patterns into a practical restock workflow on Armv9.

content/learning-paths/cross-platform/multimodel_mnn_v9/6_mnn_restock_ticket.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@ layout: learningpathall
66

77
## Create the single-shot multimodal prompt (image + audio)
88

9-
In this final module, you combine image and audio inputs into a single multimodal prompt and generate one operational restock ticket on an Armv9 system.
9+
In this final section, you combine image and audio inputs into a single multimodal prompt and generate one operational restock ticket on an Armv9 system.
10+
11+
This section demonstrates end-to-end multimodal task synthesis on an Armv9 CPU. Instead of testing a single modality in isolation, you combine image and audio inputs to produce one structured restocking ticket that can support a store action.
1012

1113
You will combine:
1214
- A **shelf image** to estimate shelf coverage and select a priority zone

content/learning-paths/cross-platform/multimodel_mnn_v9/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Multimodal On-Device Inference on Armv9 with MNN for Audio and Vision
2+
title: Build a Multimodal Retail Restocking Assistant on Armv9 With MNN
33

44
minutes_to_complete: 60
55

0 commit comments

Comments
 (0)