|
| 1 | +--- |
| 2 | + |
| 3 | +title: Performance with Streamline |
| 4 | + |
| 5 | +weight: 10 |
| 6 | + |
| 7 | +### FIXED, DO NOT MODIFY |
| 8 | + |
| 9 | +layout: learningpathall |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +## Arm Performance Studio |
| 14 | + |
| 15 | +[Arm Performance Studio](https://developer.arm.com/Tools%20and%20Software/Arm%20Performance%20Studio) is a performance analysis tool suite for developers to performance test their applications on devices with Arm CPUs. You can use |
| 16 | +[Streamline](https://developer.arm.com/Tools%20and%20Software/Streamline%20Performance%20Analyzer) to capture a performance profile that shows all the performance counter activity from the device. Generate an easy-to-read performance summary from an annotated Streamline capture, and get actionable advice about where you should optimize. |
| 17 | +## Download and Install Arm Performance Studio |
| 18 | + |
| 19 | +Arm Performance Studio is supported on Windows, Linux, and macOS hosts. Get the [Arm Performance Studio installation package](https://developer.arm.com/Tools%20and%20Software/Arm%20Performance%20Studio#Downloads). |
| 20 | + |
| 21 | +Refer to the [Arm Performance Studio install guide](/install-guides/ams/) for installation instructions. |
| 22 | + |
| 23 | +## Launch the tools |
| 24 | + |
| 25 | +To open the tools, launch the Performance Studio Hub: |
| 26 | + |
| 27 | +- On Windows, search for Performance Studio. |
| 28 | +- On macOS and Linux, open the Performance Studio application file from the install directory. |
| 29 | + |
| 30 | + |
| 31 | + |
| 32 | +## Profiling LLM with Arm Streamline |
| 33 | + |
| 34 | +You are now all set to profile the LLM functionality on Android phone with Arm Streamline. For this, you can use the Large Language Models repository again: |
| 35 | + |
| 36 | +``` |
| 37 | +https://github.com/Arm-Examples/LLM-Runner |
| 38 | +``` |
| 39 | + |
| 40 | +and build for your chosen LLM backend, ensure that `NDK_PATH` is set properly. You can use `ENABLE_STREAMLINE` flag to add streamline annotations: |
| 41 | + |
| 42 | +``` |
| 43 | +cmake --preset=x-android-aarch64 -B build -DLLM_FRAMEWORK=mnn -DBUILD_BENCHMARK=ON -DENABLE_STREAMLINE=ON |
| 44 | +cmake --build ./build |
| 45 | +``` |
| 46 | + |
| 47 | +{{% notice %}} |
| 48 | +For troubleshooting any build issues, refer to [large-language-models README](https://github.com/Arm-Examples/LLM-Runner/blob/main/README.md) |
| 49 | +{{% /notice %}} |
| 50 | + |
| 51 | +### Phone setup |
| 52 | + |
| 53 | +Now that you have all the libraries and executables needed, you can create a benchmarking directory and push the needed libraries to the phone: |
| 54 | + |
| 55 | +```sh |
| 56 | +adb shell mkdir /data/local/tmp/benchmark_test/ |
| 57 | +adb push build/lib/* /data/local/tmp/benchmark_test/ |
| 58 | +``` |
| 59 | +```output |
| 60 | +build/lib/archive/: 9 files pushed. 140.0 MB/s (36970298 bytes in 0.252s) |
| 61 | +build/lib/libMNN.so: 1 file pushed. 139.5 MB/s (4973176 bytes in 0.034s) |
| 62 | +build/lib/libllm-jni.so: 1 file pushed. 153.8 MB/s (3832152 bytes in 0.024s) |
| 63 | +11 files pushed. 137.0 MB/s (45775626 bytes in 0.319s) |
| 64 | +``` |
| 65 | + |
| 66 | +This will copy the executables you can run: |
| 67 | +```sh |
| 68 | +adb push build/bin/* /data/local/tmp/benchmark_test/ |
| 69 | +``` |
| 70 | +```output |
| 71 | +build/bin/llm-bench-cli: 1 file pushed. 134.3 MB/s (3415344 bytes in 0.024s) |
| 72 | +build/bin/llm-cpp-tests: 1 file pushed. 157.7 MB/s (17783848 bytes in 0.108s) |
| 73 | +build/bin/llm_bench: 1 file pushed. 22.6 MB/s (85688 bytes in 0.004s) |
| 74 | +build/bin/llm_demo: 1 file pushed. 12.6 MB/s (34656 bytes in 0.003s) |
| 75 | +4 files pushed. 141.7 MB/s (21319536 bytes in 0.143s) |
| 76 | +``` |
| 77 | +Finally, copy the models to benchmark: |
| 78 | +```sh |
| 79 | +adb push resources_downloaded/models/mnn/ /data/local/tmp/benchmark_test/ |
| 80 | +``` |
| 81 | + |
| 82 | +### Benchmarking the models |
| 83 | + |
| 84 | +To make sure the screen stays on and the CPU is not throttled use the following commands: |
| 85 | + |
| 86 | +```sh |
| 87 | +adb shell svc power stayon true |
| 88 | +adb shell dumpsys deviceidle disable |
| 89 | +``` |
| 90 | + |
| 91 | +Copy the `gatord` binary from your Arm Performance Studio installation to your android device: |
| 92 | + |
| 93 | +```sh |
| 94 | +adb push ./streamline/bin/android/arm64/gatord /data/local/tmp/ |
| 95 | +``` |
| 96 | + |
| 97 | +Start the gator daemon on your android device and point to the benchmarking executable you profile, `--wait-process` flag will ensure you can start the gator daemon and wait for a process matching the specified command to launch before starting capture: |
| 98 | + |
| 99 | +```bash |
| 100 | +adb shell |
| 101 | +cd /data/local/tmp |
| 102 | +./gatord --allow-command --wait-process llm-bench-cli |
| 103 | +``` |
| 104 | + |
| 105 | +In Streamline tool, you can use the following to configure the capture: |
| 106 | +1. Select TCP target and ensure there's an existing target available |
| 107 | +2. Use select counters button to select the counters you would like to see in the Streamline capture |
| 108 | +3. Use capture settings button to add more information to your capture |
| 109 | + |
| 110 | + |
| 111 | + |
| 112 | + |
| 113 | +Add counters from a template with the button shown below, select `SME2 Basic Utilization` and save: |
| 114 | + |
| 115 | + |
| 116 | + |
| 117 | +In Capture Settings, add previously built libraries from `build/lib/` and executables from `build/bin/` to get a more detailed analysis: |
| 118 | + |
| 119 | + |
| 120 | + |
| 121 | +You can now click `Start Capture` in Streamline and click `Save` which opens up a Capture file and waits for the process. |
| 122 | + |
| 123 | +On your target device, run the executable in ADB shell, providing the path to libraries and the number of iterations to benchmark: |
| 124 | + |
| 125 | +```sh |
| 126 | +adb shell |
| 127 | +cd /data/local/tmp/benchmark_test/ |
| 128 | +LD_LIBRARY_PATH=./ ./llm-bench-cli -m mnn/llama-3.2-1b/ -i 128 -o 128 -t 1 -n 5 -w 1 |
| 129 | +``` |
| 130 | + |
| 131 | +You will start to see the capture being populated once the benchmarking application starts: |
| 132 | + |
| 133 | + |
| 134 | + |
| 135 | +Once the benchmarking application finishes, `Cancel` image download in streamline, it will take some time for capture to be analysed. |
| 136 | + |
| 137 | +You can now look at the `Timeline` tab to see SME kernels utilization throughout the execution of your benchmarking application: |
| 138 | + |
| 139 | + |
| 140 | + |
| 141 | +You can also look in `Log` tab to see the Streamline annotations you have enabled in LLM-Runner build: |
| 142 | + |
| 143 | + |
| 144 | + |
| 145 | +By double-clicking on one of the annotations like `LLM::NextToken`, you can look at the timeline and SME/CPU instruction ratio for this specific part of LLM execution: |
| 146 | + |
| 147 | + |
| 148 | + |
| 149 | +{{% notice Note %}} |
| 150 | +The Android system enforces throttling, so your own results may vary slightly. |
| 151 | +{{% /notice %}} |
| 152 | + |
| 153 | +These measurements show how fast the model processes (encodes) 128 input tokens when running on a single CPU thread. As the results illustrate, SME2 delivers a significant performance boost even when using just one or two CPU cores on an Android phone, meaning faster processing without needing to involve multiple CPU cores. |
| 154 | + |
| 155 | +### What can Arm Streamline show? |
| 156 | + |
| 157 | +To summarize, with use of Arm Streamline you can firstly confirm that SME2 kernels are being used by your chosen framework. By looking at the timeline, you can match SME2 activity to the benchmarking phases phases. For multithreaded code, compare threads. Good signs: SME2 activity across worker threads during the compute phase. Bad signs: one active thread, gaps, or long non-SME2 regions. |
| 158 | + |
| 159 | + |
0 commit comments