You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Then run the following from the root of the ONNX Runtime repository (the `onnxruntime.git/` directory from the previous command):
26
+
Build ONNX Runtime with KleidiAI support enabled. The build script configures cross-compilation for Android arm64-v8a, enables shared library output, and activates KleidiAI integration through the `onnxruntime_USE_KLEIDIAI=ON` flag. Run the following from the root of the ONNX Runtime repository (the `onnxruntime.git/` directory from the previous command):
- The flag `onnxruntime_USE_KLEIDIAI=ON` triggers the inclusion of Arm KleidiAI kernels into the MLAS library.
33
-
- The build directory is `build/` by default. This can be overriden with the `--build_dir <path_to_your_build_directory>` commande line option to `build.sh`
33
+
- The build directory is `build/` by default. This can be overridden with the `--build_dir <path_to_your_build_directory>` command line option to `build.sh`.
34
+
{{% /notice %}}
35
+
36
+
## Profile model performance with onnxruntime_perf_test
34
37
35
-
## Profiling Performance with onnxruntime_perf_test
36
38
Once the build is complete, you will find the `libonnxruntime.so` shared library and `onnxruntime_perf_test` binary in your build directory.
37
39
38
40
`onnxruntime_perf_test` is essential for measuring latency and identifying bottlenecks of an ONNX model (named `<your_model>.onnx` hereafter). Note that `onnxruntime_perf_test` expects the ONNX model to come with some ancilliary files organized in some directory tree (input data for example).
39
41
40
-
### Step 1: Push files to Android Device
42
+
## Push files to Android device
43
+
44
+
Transfer the benchmark binary and shared library to your Android device:
The `onnxruntime_perf_test` tool allows you to simulate inference and gather statistics. For example,
51
+
## Run the performance test
52
+
53
+
The `onnxruntime_perf_test` tool simulates inference and gathers statistics. Run a benchmark with 20 iterations:
48
54
```bash
49
55
# Execute on the device
50
56
adb shell "/data/local/tmp/onnxruntime_perf_test -e cpu -m times -r 20 -s -Z -x 1 /data/local/tmp/<your_model>/<your_model>.onnx"
51
57
```
58
+
### Command options explained
52
59
53
-
The command example set the arguments of the application as,
54
-
-`-e cpu` specifies the provider as cpu provider
55
-
-`-m times` specifies the test mode as “times”
56
-
-`-r 20` specifies the repeated times as 20
57
-
-`-Z` disallows thread from spinning during runs to reduce cpu usage
58
-
-`-s` shows statistics result
59
-
-`-x 1` sets the number of threads used to parallelize the execution within nodes as 1
60
+
The benchmark command uses several flags to control execution:
60
61
61
-
You can try other arguments setting if you would like to.
62
+
-`-e cpu`: Use the CPU execution provider
63
+
-`-m times`: Run in timing mode to measure latency
64
+
-`-r 20`: Repeat the test 20 times for consistent results
65
+
-`-Z`: Prevent thread spinning to reduce CPU usage
66
+
-`-s`: Display statistics after the run
67
+
-`-x 1`: Use a single thread for parallel execution within nodes
62
68
63
-
### Step 3: Deep Dive into Operator Profiling
64
-
To see exactly how many milliseconds are spent on each operator, use the profiling flag `-p`.
69
+
You can adjust these settings based on your performance testing needs.
70
+
71
+
## Deep dive into operator profiling
72
+
73
+
To see exactly how many milliseconds each operator consumes, use the profiling flag `-p`. This generates a JSON trace file:
65
74
```bash
66
75
adb shell "/data/local/tmp/onnxruntime_perf_test -p /data/local/tmp/profile.json -e cpu -m times -r 5 -s -Z -x 1 /data/local/tmp/<your model>/<your_model>.onnx"
67
76
adb pull /data/local/tmp/profile.json
68
77
```
69
78
70
-
The argument `-p` enables performance profiling during the benchmark run. When you provide this flag followed by a filename, ONNX Runtime will generate a JSON file containing a detailed trace of the model execution.
71
-
You can view the results by opening [perfetto tool](https://ui.perfetto.dev/), and loading the generated JSON file. This allows you to see a visual timeline of which operations took the most time.
72
-
You also can convert the JSON file to a CSV sheet by creating a python script.
79
+
The `-p`flag enables performance profiling during the benchmark run. When you provide this flag followed by a filename, ONNX Runtime generates a JSON file containing a detailed trace of model execution.
80
+
81
+
You can view the results by opening [perfetto tool](https://ui.perfetto.dev/) and loading the generated JSON file. This shows a visual timeline of which operations took the most time. You can also convert the JSON file to a CSV sheet by creating a Python script.
Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/kleidiai_integration.md
+20-13Lines changed: 20 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,21 +6,22 @@ weight: 3
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Integration of KleidiAI to ONNX runtime MLAS
10
-
ONNX runtime is built with KleidiAI support:
11
-
1. Detection: At runtime, MLAS checks the CPU capabilities for SME2 support.
12
-
2. Dispatch: If SME2 is detected, MLAS overrides its default kernels. For example, a Gemm (General Matrix Multiplication) operation that would normally use standard vector instructions (such as NEON) is dispatched to a KleidiAI SME2 micro-kernel.
9
+
## Integration of KleidiAI to ONNX Runtime MLAS
10
+
ONNX Runtime automatically detects and uses KleidiAI when SME2 support is available:
13
11
14
-
Currently, KleidiAI in MLAS provides `ArmKleidiAI::MlasConv`, `ArmKleidiAI::MlasGemmBatch` and `ArmKleidiAI::MlasDynamicQGemmBatch` kernels.
12
+
- Detection: MLAS checks the CPU capabilities for SME2 support at runtime.
13
+
- Dispatch: when SME2 is detected, MLAS replaces its default kernels with KleidiAI micro-kernels. For example, a Gemm operation that normally uses NEON instructions dispatches to a KleidiAI SME2 micro-kernel instead.
14
+
15
+
Currently, KleidiAI in MLAS provides `ArmKleidiAI::MlasConv`, `ArmKleidiAI::MlasGemmBatch`, and `ArmKleidiAI::MlasDynamicQGemmBatch` kernels.
15
16
16
17
### The ArmKleidiAI::MlasConv kernel
17
-
Usually, 2D fp32 convolution operators with batch_size=1 and multiple filters (filter kernel is equal or greater than (3,3)) are dispatched to the `ArmKleidiAI::MlasConv` kernel.
18
+
ORT dispatches 2D fp32 (32-bit floating point) convolution operators with batch_size=1 and multiple filters (filter kernel equal to or greater than (3,3)) to the `ArmKleidiAI::MlasConv` kernel.
18
19
19
20
For example, the figure below shows a (7,7) Conv node.
20
21
21
-
![Diagram illustrating an example of 7x7 Conv alt-text#center](images/conv_nodes_7x7.jpg"An example of (7,7) Conv node")
22
+
![Diagram showing a 7x7 convolution node with input tensor of size 112x112x64, filter size 7x7x64, and output tensor size 56x56x64 with stride 2#center](images/conv_nodes_7x7.jpg"An example of a (7,7) Conv node")
22
23
23
-
`ArmKleidiAI::MlasConv` kernel makes use of KleidiAI’s indirect matrix multiplication (imatmul) micro kernel to accelerate the convolution.
24
+
`ArmKleidiAI::MlasConv` kernel uses KleidiAI's indirect matrix multiplication (imatmul) micro kernel to accelerate the convolution.
It performs a batched fp32 matrix multiplication (GEMM or GemV) operation using KleidiAI matmul micro kernels. fp32 Conv operators with (1,1) filter kernels also uses this kernel.
50
+
This kernel performs a batched fp32 matrix multiplication (GEMM or GemV) operation using KleidiAI matmul micro kernels. fp32 Conv operators with (1,1) filter kernels also use this kernel.
50
51
51
52
For example, the figure below shows a (1,1) Conv node.
52
53
53
-
![Diagram illustrating an example of 1x1 Conv alt-text#center](images/conv_nodes_1x1.jpg"An example of (1,1) FusedConv node")
54
+
![Diagram showing a 1x1 convolution node with input tensor of size 56x56x64, filter size 1x1x64, and output tensor size 56x56x256 fused with BatchNormalization and Relu operations alt-txt#center](images/conv_nodes_1x1.jpg"An example of a (1,1) FusedConv node")
54
55
55
56
The function calls of fp32 Conv operators with (1,1) filter kernels are shown below.
![Diagram illustrating an example of Gemm node alt-text#center](images/gemm_node.jpg"An example of Gemm node")
83
+
![Diagram showing a Gemm (General Matrix Multiplication) node with input matrices A and B, performing matrix multiplication with optional bias addition alt-txt#center](images/gemm_node.jpg"An example of a Gemm node")
83
84
84
85
The function calls of fp32 Gemm operators are shown below.
This kernel is for matrix multiplication with float output of dynamic quantized A and symmetric quantized B.
105
-
It uses KleidiAI `kai_kernel_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa` micro kernel.
105
+
This kernel performs matrix multiplication with float output of dynamically quantized A and symmetrically quantized B.
106
+
It uses the KleidiAI `kai_kernel_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa` micro kernel.
107
+
108
+
## What you've accomplished and what's next
109
+
110
+
You now understand how KleidiAI integrates into ONNX Runtime's MLAS backend and which operators benefit from SME2 acceleration. You've learned about the three main kernel types: `MlasConv` for convolution operations, `MlasGemmBatch` for matrix multiplications and 1x1 convolutions, and `MlasDynamicQGemmBatch` for quantized operations. You've also seen the complete function call stacks showing how ONNX Runtime dispatches to KleidiAI kernels.
111
+
112
+
Next, you'll build ONNX Runtime with KleidiAI support enabled and prepare the benchmark tools for profiling on Android.
0 commit comments