|
| 1 | +--- |
| 2 | + |
| 3 | +title: Performance |
| 4 | + |
| 5 | +weight: 9 |
| 6 | + |
| 7 | +### FIXED, DO NOT MODIFY |
| 8 | + |
| 9 | +layout: learningpathall |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +## Benchmarking LLM on Android phone |
| 14 | + |
| 15 | +You can also benchmark the LLM functionality on Android phone outside of RTVA application. For this, you can use the Large Language Models repository: |
| 16 | + |
| 17 | +``` |
| 18 | +https://gitlab.arm.com/kleidi/kleidi-examples/large-language-models |
| 19 | +``` |
| 20 | + |
| 21 | +and build for your chosen LLM backend, ensure that `NDK_PATH` is set properly. SME kernels are enabled by default, so let's first build with SME disabled: |
| 22 | + |
| 23 | +``` |
| 24 | +cmake --preset=x-android-aarch64 -B build/ -DBUILD_BENCHMARK=ON -DLLM_FRAMEWORK=mnn -DMNN_SME2=OFF |
| 25 | +cmake --build ./build |
| 26 | +``` |
| 27 | + |
| 28 | +{{% notice %}} |
| 29 | +For troubleshooting any build issues, refer to [large-language-models README](https://gitlab.arm.com/kleidi/kleidi-examples/large-language-models/-/blob/main/README.md?ref_type=heads) |
| 30 | +{{% /notice %}} |
| 31 | + |
| 32 | +### Phone setup |
| 33 | + |
| 34 | +Now that you have all the libraries and executables needed, you can create a benchmarking directory and push the needed libraries to the phone: |
| 35 | + |
| 36 | +```sh |
| 37 | +adb shell mkdir /data/local/tmp/benchmark_test/ |
| 38 | +adb push build/lib/* /data/local/tmp/benchmark_test/ |
| 39 | +``` |
| 40 | +```output |
| 41 | +build/lib/archive/: 9 files pushed. 140.0 MB/s (36970298 bytes in 0.252s) |
| 42 | +build/lib/libMNN.so: 1 file pushed. 139.5 MB/s (4973176 bytes in 0.034s) |
| 43 | +build/lib/libarm-llm-jni.so: 1 file pushed. 153.8 MB/s (3832152 bytes in 0.024s) |
| 44 | +11 files pushed. 137.0 MB/s (45775626 bytes in 0.319s) |
| 45 | +``` |
| 46 | + |
| 47 | +This will copy the executables you can run: |
| 48 | +```sh |
| 49 | +adb push build/bin/* /data/local/tmp/benchmark_test/ |
| 50 | +``` |
| 51 | +```output |
| 52 | +build/bin/arm-llm-bench-cli: 1 file pushed. 134.3 MB/s (3415344 bytes in 0.024s) |
| 53 | +build/bin/llm-cpp-tests: 1 file pushed. 157.7 MB/s (17783848 bytes in 0.108s) |
| 54 | +build/bin/llm_bench: 1 file pushed. 22.6 MB/s (85688 bytes in 0.004s) |
| 55 | +build/bin/llm_demo: 1 file pushed. 12.6 MB/s (34656 bytes in 0.003s) |
| 56 | +4 files pushed. 141.7 MB/s (21319536 bytes in 0.143s) |
| 57 | +``` |
| 58 | +Finally, copy the models to benchmark: |
| 59 | +```sh |
| 60 | +adb push resources_downloaded/models/mnn/ /data/local/tmp/benchmark_test/ |
| 61 | +``` |
| 62 | + |
| 63 | +### Benchmarking the models |
| 64 | + |
| 65 | +To make sure the screen stays on and the CPU is not throttled use the following commands: |
| 66 | + |
| 67 | +```sh |
| 68 | +adb shell svc power stayon true |
| 69 | +adb shell dumpsys deviceidle disable |
| 70 | +``` |
| 71 | + |
| 72 | +You can now run the executable in ADB shell, providing the path to libraries and the number of iterations to benchmark: |
| 73 | + |
| 74 | +```sh |
| 75 | +adb shell |
| 76 | +cd /data/local/tmp/benchmark_test/ |
| 77 | +LD_LIBRARY_PATH=./ ./arm-llm-bench-cli -m mnn/llama-3.2-1b/ -i 128 -o 128 -t 1 -n 5 -w 1 |
| 78 | +``` |
| 79 | + |
| 80 | +As you see in the output, the flags used by executable are listed below: |
| 81 | +* `-m` : path to the specific model or a directory with model and configuration files |
| 82 | +* `-i` : number input tokens to use |
| 83 | +* `-o` : number output tokens to generate |
| 84 | +* `-t` : number of threads to use |
| 85 | +* `-n` : number of iterations used for benchmarking |
| 86 | +* `-w` : number of warmup iterations, not included in benchmarking |
| 87 | + |
| 88 | +```output |
| 89 | +
|
| 90 | +=== ARM LLM Benchmark === |
| 91 | +
|
| 92 | +Parameters: |
| 93 | + model_path : mnn/llama-3.2-1b/ |
| 94 | + num_input_tokens : 128 |
| 95 | + num_output_tokens : 128 |
| 96 | + num_threads : 1 |
| 97 | + num_iterations : 5 |
| 98 | + num_warmup : 1 |
| 99 | +
|
| 100 | +
|
| 101 | +======= Results ========= |
| 102 | +
|
| 103 | +| Framework | Threads | Test | Performance | |
| 104 | +| ------------------ | ------- | ------ | -------------------------- | |
| 105 | +| mnn | 1 | pp128 | 196.446 ± 0.377 (t/s) | |
| 106 | +| mnn | 1 | tg128 | 27.222 ± 0.369 (t/s) | |
| 107 | +| mnn | 1 | TTFT | 687.931 ± 2.279 (ms) | |
| 108 | +| mnn | 1 | Total | 5354.526 ± 63.163 (ms) | |
| 109 | +
|
| 110 | +``` |
| 111 | + |
| 112 | +To get benchmark numbers with use of SME kernels, you can rerun the full "Benchmarking LLM on Android phone" section without setting `MNN_SME2` to `OFF`. Omitting the `MNN_SME2` flag enables SME instructions by default.: |
| 113 | + |
| 114 | +``` |
| 115 | +cmake --preset=x-android-aarch64 -B build/ -DBUILD_BENCHMARK=ON -DLLM_FRAMEWORK=mnn |
| 116 | +cmake --build ./build |
| 117 | +``` |
| 118 | + |
| 119 | + |
| 120 | +## Example performance with a Vivo X300 Android phone |
| 121 | + |
| 122 | +The table table shows the measurements taken on a Vivo X300 Android phone: |
| 123 | + |
| 124 | +| LLM Framework | Model | Threads | Without SME2 | With SME2 | Uplift | |
| 125 | +|-------------------|-----------------------|---------|----------------|-----------|----------| |
| 126 | +| mnn | qwen25vl-3b | 1 | 85 | 134 | 57.65 % | |
| 127 | +| | | 2 | 95 | 140 | 47.37 % | |
| 128 | +| | llama-3.2-1B | 1 | 196 | 339 | 72.96 % | |
| 129 | +| | | 2 | 275 | 396 | 44.00 % | |
| 130 | +| llama.cpp | qwen-2-VL | 1 | 113 | 146 | 29.20 % | |
| 131 | +| | | 2 | 92 | 139 | 51.09 % | |
| 132 | +| | llama-3.2-1B | 1 | 148 | 173 | 16.89 % | |
| 133 | +| | | 2 | 124 | 191 | 54.03 % | |
| 134 | +| | phi-2 | 1 | 58 | 77 | 32.76 % | |
| 135 | +| | | 2 | 46 | 60 | 30.43 % | |
| 136 | + |
| 137 | + |
| 138 | +{{% notice Note %}} |
| 139 | +The Android system enforces throttling, so your own results may vary slightly. |
| 140 | +{{% /notice %}} |
| 141 | + |
| 142 | +These measurements show how fast the model processes (encodes) 128 input tokens when running on a single CPU thread. As the results illustrate, SME2 delivers a significant performance boost even when using just one or two CPU cores on an Android phone, meaning faster processing without needing to involve multiple CPU cores. |
| 143 | + |
0 commit comments