Skip to content

Commit 09cc4a9

Browse files
authored
Merge pull request #2772 from NinaARM/feature/voice-assistant-benchmarking
Feature/voice assistant benchmarking
2 parents d33b941 + 3f99522 commit 09cc4a9

13 files changed

Lines changed: 203 additions & 9 deletions

content/learning-paths/mobile-graphics-and-gaming/voice-assistant/2-overview.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,14 +72,15 @@ You can build this pipeline for various platforms and independently benchmark th
7272
|Platform|Details|
7373
|---|---|
7474
|Linux|x86_64 - KleidiAI is disabled by default, aarch64 - KleidiAI is enabled by default.|
75-
|Android|Cross-compile for an Android device, ensure the Android NDK path is set and correct toolchain file is provided. KleidiAI enabled by default.|
75+
|Android|Cross-compile for an Android device, ensure the Android NDK path is set and correct toolchain file is provided. KleidiAI enabled by default. SME kernels can be used if available on device.|
7676
|macOS|Native or cross-compilation for a Mac device. KleidiAI and SME kernels can be used if available on device.|
7777

7878
Currently, this module provides a thin C++ layer as well as JNI bindings for developers targeting Android based applications, supported backends are:
7979
|Framework|Dependency|Input modalities supported|Output modalities supported|Neural Network|
8080
|---|---|---|---|---|
81-
|llama.cpp|https://github.com/ggml-org/llama.cpp|`image`, `text`|`text`|phi-2,Qwen2-VL-2B-Instruct|
81+
|llama.cpp|https://github.com/ggml-org/llama.cpp|`image`, `text`|`text`|phi-2, qwen-2-VL, llama-3.2-1B|
8282
|onnxruntime-genai|https://github.com/microsoft/onnxruntime-genai|`text`|`text`|phi-4-mini-instruct-onnx|
83+
|mnn|https://github.com/alibaba/MNN|`image`, `text`|`text`|qwen-2.5-VL, llama-3.2-1B|
8384
|mediapipe|https://github.com/google-ai-edge/mediapipe|`text`|`text`|gemma-2b-it-cpu-int4|
8485

8586

content/learning-paths/mobile-graphics-and-gaming/voice-assistant/4-run.md

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,21 +22,31 @@ In the graphic below, a Google Pixel 8 Pro phone is connected to the USB cable:
2222

2323
## Launch the Voice Assistant
2424

25-
The app starts with this welcome screen:
25+
The app starts with this welcome screen, you can choose between the Chat and Benchmark mode:
2626

27-
![welcome image alt-text#center](voice_assistant_view1.png "Welcome Screen")
27+
![welcome image alt-text#center](voice_assistant_welcome.png "Welcome Screen")
2828

29-
Tap **Press to talk** at the bottom of the screen to begin speaking your request.
29+
Tap **Chat** to check the voice assistant pipeline in action.
30+
31+
## Voice Assistant Chat
32+
33+
In the chat mode, tap **Press to talk** at the bottom of the screen to begin speaking your request.
34+
35+
![Chat image alt-text#center](voice_assistant_view1.png "Chat Screen")
3036

3137
## Voice Assistant controls
3238

33-
You can use application controls to enable extra functionality or gather performance data.
39+
You can use application controls to enable extra functionality or gather performance data.
3440

3541
|Button|Control name|Description|
3642
|---|---|---|
37-
|1|Performance counters|Performance counters are hidden by default, click this to show speech recognition time, LLM encode and decode rate.|
43+
|1|Back to welcome screen|Go back to welcome screen to select mode - chat or benchmark.|
3844
|2|Speech generation|Speech generation is disabled by default, click this to use Android Text-to-Speech and get audible answers.|
3945
|3|Reset conversation|By default, the application keeps context so you can follow-up questions, click this to reset voice assistant conversation history.|
46+
|4|Memory used|This metric shows memory used by this application as well as memory available on the device.|
47+
|5|Device thermal status|This metric shows current heat level of the device and whether the system is applying performance throttling to prevent overheating.|
48+
|6|User performance metrics|Performance metrics for user's query including time to transcript the query (STT - Speech-to-Text module) and time for LLM to encode the query, measured in tokens per second.|
49+
|7|Voice assistant metrics|Performance metrics for voice assistant's reply - decode performance measured in tokens per second.|
4050

4151
Click the icon circled in red in the top left corner to show or hide these metrics:
4252

@@ -58,7 +68,7 @@ Choose the image, and add image for voice assistant:
5868

5969
![add image alt-text#center](add_image.png "Add image to the question")
6070

61-
You can now ask questions related to this image, the large language model will you the image and text for multimodal question answering.
71+
You can now ask questions related to this image, the large language model will use both the image and text for multimodal question answering.
6272

6373
![ask question image alt-text#center](voice_assistant_multimodal_2.png "Add image to the question")
6474

content/learning-paths/mobile-graphics-and-gaming/voice-assistant/5-kleidiai.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,4 @@ KleidiAI simplifies development by abstracting away low-level optimization: deve
3333

3434
As newer versions of the architecture become available, KleidiAI becomes even more powerful: simply updating the library allows applications like the multimodal Voice Assistant to take advantage of the latest architectural improvements such as SME2, without requiring any code changes. This means better performance on newer devices with no additional effort from developers.
3535

36+
Now that you can build the Voice Assistant with and without KleidiAI, you can test out the benchmarking functionality it provides.
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
3+
title: Benchmark Voice Assistant
4+
5+
weight: 8
6+
7+
### FIXED, DO NOT MODIFY
8+
9+
layout: learningpathall
10+
11+
---
12+
13+
## Benchmarking
14+
15+
The Voice Assistant application also provides a benchmark mode so you can easily test out the performance of an LLM model with a sample number of input and output tokens.
16+
17+
![welcome image alt-text#center](voice_assistant_welcome.png "Welcome Screen")
18+
19+
Tap **Benchmark** to navigate to benchmark screen.
20+
21+
![Benchmark image alt-text#center](voice_assistant_benchmark_1.png "Benchmark Screen")
22+
23+
## Benchmark controls
24+
25+
You can use application controls to enable extra functionality or gather performance data.
26+
27+
|Setting|Default|Description|
28+
|---|---|---|
29+
|Input tokens|128|Number of prompt (input) tokens fed to the model before generation starts.|
30+
|Output tokens|128|Number of new tokens the model should generate after the prompt.|
31+
|Threads|4|Number of CPU threads used for inference.|
32+
|Iterations|5|Number of measured benchmark runs to collect stable, averaged measurements.|
33+
|Warmup|1|Number of warmup iterations which are not counted in benchmarking, these eliminate one-time overheads before measuring.
34+
35+
To deep dive into more specific performance, you can build the Voice Assistant modules individually and run benchmarks on your Android device.
36+
37+
Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
3+
title: Performance
4+
5+
weight: 9
6+
7+
### FIXED, DO NOT MODIFY
8+
9+
layout: learningpathall
10+
11+
---
12+
13+
## Benchmarking LLM on Android phone
14+
15+
You can also benchmark the LLM functionality on Android phone outside of RTVA application. For this, you can use the Large Language Models repository:
16+
17+
```
18+
https://gitlab.arm.com/kleidi/kleidi-examples/large-language-models
19+
```
20+
21+
and build for your chosen LLM backend, ensure that `NDK_PATH` is set properly. SME kernels are enabled by default, so let's first build with SME disabled:
22+
23+
```
24+
cmake --preset=x-android-aarch64 -B build/ -DBUILD_BENCHMARK=ON -DLLM_FRAMEWORK=mnn -DMNN_SME2=OFF
25+
cmake --build ./build
26+
```
27+
28+
{{% notice %}}
29+
For troubleshooting any build issues, refer to [large-language-models README](https://gitlab.arm.com/kleidi/kleidi-examples/large-language-models/-/blob/main/README.md?ref_type=heads)
30+
{{% /notice %}}
31+
32+
### Phone setup
33+
34+
Now that you have all the libraries and executables needed, you can create a benchmarking directory and push the needed libraries to the phone:
35+
36+
```sh
37+
adb shell mkdir /data/local/tmp/benchmark_test/
38+
adb push build/lib/* /data/local/tmp/benchmark_test/
39+
```
40+
```output
41+
build/lib/archive/: 9 files pushed. 140.0 MB/s (36970298 bytes in 0.252s)
42+
build/lib/libMNN.so: 1 file pushed. 139.5 MB/s (4973176 bytes in 0.034s)
43+
build/lib/libarm-llm-jni.so: 1 file pushed. 153.8 MB/s (3832152 bytes in 0.024s)
44+
11 files pushed. 137.0 MB/s (45775626 bytes in 0.319s)
45+
```
46+
47+
This will copy the executables you can run:
48+
```sh
49+
adb push build/bin/* /data/local/tmp/benchmark_test/
50+
```
51+
```output
52+
build/bin/arm-llm-bench-cli: 1 file pushed. 134.3 MB/s (3415344 bytes in 0.024s)
53+
build/bin/llm-cpp-tests: 1 file pushed. 157.7 MB/s (17783848 bytes in 0.108s)
54+
build/bin/llm_bench: 1 file pushed. 22.6 MB/s (85688 bytes in 0.004s)
55+
build/bin/llm_demo: 1 file pushed. 12.6 MB/s (34656 bytes in 0.003s)
56+
4 files pushed. 141.7 MB/s (21319536 bytes in 0.143s)
57+
```
58+
Finally, copy the models to benchmark:
59+
```sh
60+
adb push resources_downloaded/models/mnn/ /data/local/tmp/benchmark_test/
61+
```
62+
63+
### Benchmarking the models
64+
65+
To make sure the screen stays on and the CPU is not throttled use the following commands:
66+
67+
```sh
68+
adb shell svc power stayon true
69+
adb shell dumpsys deviceidle disable
70+
```
71+
72+
You can now run the executable in ADB shell, providing the path to libraries and the number of iterations to benchmark:
73+
74+
```sh
75+
adb shell
76+
cd /data/local/tmp/benchmark_test/
77+
LD_LIBRARY_PATH=./ ./arm-llm-bench-cli -m mnn/llama-3.2-1b/ -i 128 -o 128 -t 1 -n 5 -w 1
78+
```
79+
80+
As you see in the output, the flags used by executable are listed below:
81+
* `-m` : path to the specific model or a directory with model and configuration files
82+
* `-i` : number input tokens to use
83+
* `-o` : number output tokens to generate
84+
* `-t` : number of threads to use
85+
* `-n` : number of iterations used for benchmarking
86+
* `-w` : number of warmup iterations, not included in benchmarking
87+
88+
```output
89+
90+
=== ARM LLM Benchmark ===
91+
92+
Parameters:
93+
model_path : mnn/llama-3.2-1b/
94+
num_input_tokens : 128
95+
num_output_tokens : 128
96+
num_threads : 1
97+
num_iterations : 5
98+
num_warmup : 1
99+
100+
101+
======= Results =========
102+
103+
| Framework | Threads | Test | Performance |
104+
| ------------------ | ------- | ------ | -------------------------- |
105+
| mnn | 1 | pp128 | 196.446 ± 0.377 (t/s) |
106+
| mnn | 1 | tg128 | 27.222 ± 0.369 (t/s) |
107+
| mnn | 1 | TTFT | 687.931 ± 2.279 (ms) |
108+
| mnn | 1 | Total | 5354.526 ± 63.163 (ms) |
109+
110+
```
111+
112+
To get benchmark numbers with use of SME kernels, you can rerun the full "Benchmarking LLM on Android phone" section without setting `MNN_SME2` to `OFF`. Omitting the `MNN_SME2` flag enables SME instructions by default.:
113+
114+
```
115+
cmake --preset=x-android-aarch64 -B build/ -DBUILD_BENCHMARK=ON -DLLM_FRAMEWORK=mnn
116+
cmake --build ./build
117+
```
118+
119+
120+
## Example performance with a Vivo X300 Android phone
121+
122+
The table table shows the measurements taken on a Vivo X300 Android phone:
123+
124+
| LLM Framework | Model | Threads | Without SME2 | With SME2 | Uplift |
125+
|-------------------|-----------------------|---------|----------------|-----------|----------|
126+
| mnn | qwen25vl-3b | 1 | 85 | 134 | 57.65 % |
127+
| | | 2 | 95 | 140 | 47.37 % |
128+
| | llama-3.2-1B | 1 | 196 | 339 | 72.96 % |
129+
| | | 2 | 275 | 396 | 44.00 % |
130+
| llama.cpp | qwen-2-VL | 1 | 113 | 146 | 29.20 % |
131+
| | | 2 | 92 | 139 | 51.09 % |
132+
| | llama-3.2-1B | 1 | 148 | 173 | 16.89 % |
133+
| | | 2 | 124 | 191 | 54.03 % |
134+
| | phi-2 | 1 | 58 | 77 | 32.76 % |
135+
| | | 2 | 46 | 60 | 30.43 % |
136+
137+
138+
{{% notice Note %}}
139+
The Android system enforces throttling, so your own results may vary slightly.
140+
{{% /notice %}}
141+
142+
These measurements show how fast the model processes (encodes) 128 input tokens when running on a single CPU thread. As the results illustrate, SME2 delivers a significant performance boost even when using just one or two CPU cores on an Android phone, meaning faster processing without needing to involve multiple CPU cores.
143+

content/learning-paths/mobile-graphics-and-gaming/voice-assistant/_index.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,9 @@ learning_objectives:
1212
- Optimize performance of multimodal Voice Assistant using KleidiAI and SME2.
1313

1414
prerequisites:
15-
- An Android phone that supports the i8mm Arm architecture feature (8-bit integer matrix multiplication). This Learning Path was tested on a Google Pixel 8 Pro.
15+
- An Android phone that supports the i8mm Arm architecture feature (8-bit integer matrix multiplication).
16+
- An Android phone with support for SME (Scalable Matrix Extension) instructions, required for SME performance checking
17+
- This Learning Path was tested on a Vivo X300 Pro.
1618
- A development machine with [Android Studio](https://developer.android.com/studio) installed.
1719

1820
author:
35.8 KB
Loading
-167 KB
Loading
64.5 KB
Loading
98.1 KB
Loading

0 commit comments

Comments
 (0)