Skip to content

Commit 67a96b6

Browse files
NinaARMNinaARM
committed
updates:
* new section on benchmark voice assistant after kleidiAI section * rename pages and adjust weights * more info in performance benchmarking table
1 parent 5fe8dd4 commit 67a96b6

4 files changed

Lines changed: 76 additions & 36 deletions

File tree

‎content/learning-paths/mobile-graphics-and-gaming/voice-assistant/5-kleidiai.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,4 @@ KleidiAI simplifies development by abstracting away low-level optimization: deve
3333

3434
As newer versions of the architecture become available, KleidiAI becomes even more powerful: simply updating the library allows applications like the multimodal Voice Assistant to take advantage of the latest architectural improvements such as SME2, without requiring any code changes. This means better performance on newer devices with no additional effort from developers.
3535

36+
Now that you can build the Voice Assistant with and without KleidiAI, you can test out the benchmarking functionality it provides.

‎content/learning-paths/mobile-graphics-and-gaming/voice-assistant/6-benchmark.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
3+
title: Benchmark Voice Assistant
4+
5+
weight: 8
6+
7+
### FIXED, DO NOT MODIFY
8+
9+
layout: learningpathall
10+
11+
---
12+
13+
## Benchmarking
14+
15+
The Voice Assistant application also provides a benchmark mode so you can easily test out the performance of an LLM model with a sample number of input and output tokens.
16+
17+
![welcome image alt-text#center](voice_assistant_welcome.png "Welcome Screen")
18+
19+
Tap **Benchmark** to navigate to benchmark screen.
20+
21+
![Benchmark image alt-text#center](voice_assistant_benchmark_1.png "Benchmark Screen")
22+
23+
## Benchmark controls
24+
25+
You can use application controls to enable extra functionality or gather performance data.
26+
27+
|Setting|Default|Description|
28+
|---|---|---|
29+
|Input tokens|128|Number of prompt (input) tokens fed to the model before generation starts.|
30+
|Output tokens|128|Number of new tokens the model should generate after the prompt.|
31+
|Threads|4|Number of CPU threads used for inference.|
32+
|Iterations|5|Number of measured benchmark runs to collect stable, averaged measurements.|
33+
|Warmup|1|Number of warmup iterations which are not counted in benchmarking, these eliminate one-time overheads before measuring.
34+
35+
To deep dive into more specific performance, you can build the Voice Assistant modules individually and run benchmarks on your Android device.
36+
37+

content/learning-paths/mobile-graphics-and-gaming/voice-assistant/6-performance.md renamed to content/learning-paths/mobile-graphics-and-gaming/voice-assistant/7-performance.md

Lines changed: 35 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -2,36 +2,14 @@
22

33
title: Performance
44

5-
weight: 8
5+
weight: 9
66

77
### FIXED, DO NOT MODIFY
88

99
layout: learningpathall
1010

1111
---
1212

13-
## Benchmarking
14-
15-
The Voice Assistant application also provides a benchmark mode so you can easily test out the performance of an LLM model with a sample number of input and output tokens.
16-
17-
![welcome image alt-text#center](voice_assistant_welcome.png "Welcome Screen")
18-
19-
Tap **Benchmark** to navigate to benchmark screen.
20-
21-
![Benchmark image alt-text#center](voice_assistant_benchmark_1.png "Benchmark Screen")
22-
23-
## Benchmark controls
24-
25-
You can use application controls to enable extra functionality or gather performance data.
26-
27-
|Setting|Default|Description|
28-
|---|---|---|
29-
|Input tokens|128|Number of prompt (input) tokens fed to the model before generation starts.|
30-
|Output tokens|128|Number of new tokens the model should generate after the prompt.|
31-
|Threads|4|Number of CPU threads used for inference.|
32-
|Iterations|5|Number of measured benchmark runs to collect stable, averaged measurements.|
33-
|Warmup|1|Number of warmup iterations which are not counted in benchmarking, these eliminate one-time overheads before measuring.|
34-
3513
## Benchmarking LLM on Android phone
3614

3715
You can also benchmark the LLM functionality on Android phone outside of RTVA application. For this, you can use the Large Language Models repository:
@@ -96,16 +74,25 @@ You can now run the executable in ADB shell, providing the path to libraries and
9674
```sh
9775
adb shell
9876
cd /data/local/tmp/benchmark_test/
99-
LD_LIBRARY_PATH=./ ./arm-llm-bench-cli -m mnn/llama-3.2-1b/ -i 128 -o 64 -t 1 -n 5 -w 1
77+
LD_LIBRARY_PATH=./ ./arm-llm-bench-cli -m mnn/llama-3.2-1b/ -i 128 -o 128 -t 1 -n 5 -w 1
10078
```
79+
80+
As you see in the output, the flags used by executable are listed below:
81+
* `-m` : path to the specific model or a directory with model and configuration files
82+
* `-i` : number input tokens to use
83+
* `-o` : number output tokens to generate
84+
* `-t` : number of threads to use
85+
* `-n` : number of iterations used for benchmarking
86+
* `-w` : number of warmup iterations, not included in benchmarking
87+
10188
```output
10289
10390
=== ARM LLM Benchmark ===
10491
10592
Parameters:
10693
model_path : mnn/llama-3.2-1b/
10794
num_input_tokens : 128
108-
num_output_tokens : 64
95+
num_output_tokens : 128
10996
num_threads : 1
11097
num_iterations : 5
11198
num_warmup : 1
@@ -115,29 +102,42 @@ Parameters:
115102
116103
| Framework | Threads | Test | Performance |
117104
| ------------------ | ------- | ------ | -------------------------- |
118-
| mnn | 1 | pp128 | 192.788 ± 5.658 (t/s) |
119-
| mnn | 1 | tg64 | 30.530 ± 3.186 (t/s) |
120-
| mnn | 1 | TTFT | 697.401 ± 29.332 (ms) |
121-
| mnn | 1 | Total | 2782.626 ± 211.955 (ms) |
105+
| mnn | 1 | pp128 | 196.446 ± 0.377 (t/s) |
106+
| mnn | 1 | tg128 | 27.222 ± 0.369 (t/s) |
107+
| mnn | 1 | TTFT | 687.931 ± 2.279 (ms) |
108+
| mnn | 1 | Total | 5354.526 ± 63.163 (ms) |
122109
123110
```
124111

125-
To get benchmark numbers with use of SME kernels, you can rerun the full Benchmarking LLM on Android phone section without the -DMNN_SME2=OFF as SME kernels are enabled by default.
112+
To get benchmark numbers with use of SME kernels, you can rerun the full Benchmarking LLM on Android phone section without using `MNN_SME` flag as follows, which will leave SME instructions enabled by default:
113+
114+
```
115+
cmake --preset=x-android-aarch64 -B build/ -DBUILD_BENCHMARK=ON -DLLM_FRAMEWORK=mnn
116+
cmake --build ./build
117+
```
126118

127119

128120
## Example performance with a Vivo X300 Android phone
129121

130122
The table table shows the measurements taken on a Vivo X300 Android phone:
131123

132-
| LLM Framework | Model | Without SME2 | With SME2 | Uplift |
133-
|-------------------|-----------------------|----------------|-----------|----------|
134-
| mnn | llama-3.2-1B | 187.06 | 334.57 | 78.87 % |
135-
| | qwen25vl-3b | 73.49 | 124.56 | 69.49 % |
124+
| LLM Framework | Model | Threads | Without SME2 | With SME2 | Uplift |
125+
|-------------------|-----------------------|---------|----------------|-----------|----------|
126+
| mnn | llama-3.2-1B | 1 | 196 | 339 | 72.96 % |
127+
| | | 2 | 275 | 396 | 44.00 % |
128+
| | qwen25vl-3b | 1 | 73.49 | 124.56 | 69.49 % |
129+
| | | 2 | | | |
130+
| llama.cpp | llama-3.2-1B | 1 | 148 | 173 | 16.89 % |
131+
| | | 2 | 124 | 191 | 54.03 % |
132+
| | qwen-2-VL | 1 | 113 | 146 | 29.20 % |
133+
| | | 2 | 92 | 139 | 51.09 % |
134+
| | phi-2 | 1 | 58 | 77 | 32.76 % |
135+
| | | 2 | 46 | 60 | 30.43 % |
136136

137137

138138
{{% notice Note %}}
139139
The Android system enforces throttling, so your own results may vary slightly.
140140
{{% /notice %}}
141141

142-
These measurements show how fast the model processes (encodes) 128 input tokens when running on a single CPU thread. As the results illustrate, SME2 delivers a significant performance boost even when using just one CPU core on an Android phone, meaning faster processing without needing to involve multiple CPU cores.
142+
These measurements show how fast the model processes (encodes) 128 input tokens when running on a single CPU thread. As the results illustrate, SME2 delivers a significant performance boost even when using just one or two CPU cores on an Android phone, meaning faster processing without needing to involve multiple CPU cores.
143143

‎content/learning-paths/mobile-graphics-and-gaming/voice-assistant/_index.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,9 @@ learning_objectives:
1212
- Optimize performance of multimodal Voice Assistant using KleidiAI and SME2.
1313

1414
prerequisites:
15-
- An Android phone that supports the i8mm Arm architecture feature (8-bit integer matrix multiplication). This Learning Path was tested on a Google Pixel 8 Pro.
15+
- An Android phone that supports the i8mm Arm architecture feature (8-bit integer matrix multiplication).
16+
- An Android phone with support for SME (Scalable Matrix Extension) instructions, required for SME performance checking
17+
- This Learning Path was tested on a Vivo X300 Pro.
1618
- A development machine with [Android Studio](https://developer.android.com/studio) installed.
1719

1820
author:

0 commit comments

Comments
 (0)