You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Then run the following from the root of the ONNX Runtime repository (the `onnxruntime.git/` directory from the previous command):
26
+
Build ONNX Runtime with KleidiAI support enabled. The build script configures cross-compilation for Android arm64-v8a, enables shared library output, and activates KleidiAI integration through the `onnxruntime_USE_KLEIDIAI=ON` flag. Run the following from the root of the ONNX Runtime repository (the `onnxruntime.git/` directory from the previous command):
- The flag `onnxruntime_USE_KLEIDIAI=ON` triggers the inclusion of Arm KleidiAI kernels into the MLAS library.
33
33
- The build directory is `build/` by default. This can be overriden with the `--build_dir <path_to_your_build_directory>` commande line option to `build.sh`
34
34
35
-
## Profiling Performance with onnxruntime_perf_test
35
+
## Profile model performance with onnxruntime_perf_test
36
+
36
37
Once the build is complete, you will find the `libonnxruntime.so` shared library and `onnxruntime_perf_test` binary in your build directory.
37
38
38
39
`onnxruntime_perf_test` is essential for measuring latency and identifying bottlenecks of an ONNX model (named `<your_model>.onnx` hereafter). Note that `onnxruntime_perf_test` expects the ONNX model to come with some ancilliary files organized in some directory tree (input data for example).
39
40
40
-
### Step 1: Push files to Android Device
41
+
## Push files to Android device
42
+
43
+
Transfer the benchmark binary and shared library to your Android device:
The `onnxruntime_perf_test` tool allows you to simulate inference and gather statistics. For example,
50
+
## Run the performance test
51
+
52
+
The `onnxruntime_perf_test` tool simulates inference and gathers statistics. Run a benchmark with 20 iterations:
48
53
```bash
49
54
# Execute on the device
50
55
adb shell "/data/local/tmp/onnxruntime_perf_test -e cpu -m times -r 20 -s -Z -x 1 /data/local/tmp/<your_model>/<your_model>.onnx"
@@ -60,13 +65,14 @@ The command example set the arguments of the application as,
60
65
61
66
You can try other arguments setting if you would like to.
62
67
63
-
### Step 3: Deep Dive into Operator Profiling
64
-
To see exactly how many milliseconds are spent on each operator, use the profiling flag `-p`.
68
+
## Deep dive into operator profiling
69
+
70
+
To see exactly how many milliseconds each operator consumes, use the profiling flag `-p`. This generates a JSON trace file:
65
71
```bash
66
72
adb shell "/data/local/tmp/onnxruntime_perf_test -p /data/local/tmp/profile.json -e cpu -m times -r 5 -s -Z -x 1 /data/local/tmp/<your model>/<your_model>.onnx"
67
73
adb pull /data/local/tmp/profile.json
68
74
```
69
75
70
-
The argument `-p` enables performance profiling during the benchmark run. When you provide this flag followed by a filename, ONNX Runtime will generate a JSON file containing a detailed trace of the model execution.
71
-
You can view the results by opening [perfetto tool](https://ui.perfetto.dev/), and loading the generated JSON file. This allows you to see a visual timeline of which operations took the most time.
72
-
You also can convert the JSON file to a CSV sheet by creating a python script.
76
+
The `-p`flag enables performance profiling during the benchmark run. When you provide this flag followed by a filename, ONNX Runtime generates a JSON file containing a detailed trace of model execution.
77
+
78
+
You can view the results by opening [perfetto tool](https://ui.perfetto.dev/) and loading the generated JSON file. This shows a visual timeline of which operations took the most time. You can also convert the JSON file to a CSV sheet by creating a Python script.
Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/kleidiai_integration.md
+18-12Lines changed: 18 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,21 +6,21 @@ weight: 3
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Integration of KleidiAI to ONNX runtime MLAS
10
-
ONNX runtime is built with KleidiAI support:
9
+
## Integration of KleidiAI to ONNX Runtime MLAS
10
+
ONNX Runtime builds with KleidiAI support:
11
11
1. Detection: At runtime, MLAS checks the CPU capabilities for SME2 support.
12
-
2. Dispatch: If SME2 is detected, MLAS overrides its default kernels. For example, a Gemm (General Matrix Multiplication) operation that would normally use standard vector instructions (such as NEON) is dispatched to a KleidiAI SME2 micro-kernel.
12
+
2. Dispatch: If SME2 is detected, MLAS overrides its default kernels. For example, a Gemm (General Matrix Multiplication) operation that would normally use standard vector instructions (such as NEON) dispatches to a KleidiAI SME2 micro-kernel.
13
13
14
-
Currently, KleidiAI in MLAS provides `ArmKleidiAI::MlasConv`, `ArmKleidiAI::MlasGemmBatch` and `ArmKleidiAI::MlasDynamicQGemmBatch` kernels.
14
+
Currently, KleidiAI in MLAS provides `ArmKleidiAI::MlasConv`, `ArmKleidiAI::MlasGemmBatch`, and `ArmKleidiAI::MlasDynamicQGemmBatch` kernels.
15
15
16
16
### The ArmKleidiAI::MlasConv kernel
17
-
Usually, 2D fp32 convolution operators with batch_size=1 and multiple filters (filter kernel is equal or greater than (3,3)) are dispatched to the `ArmKleidiAI::MlasConv` kernel.
17
+
ORT dispatches 2D fp32 (32-bit floating point) convolution operators with batch_size=1 and multiple filters (filter kernel equal to or greater than (3,3)) to the `ArmKleidiAI::MlasConv` kernel.
18
18
19
19
For example, the figure below shows a (7,7) Conv node.
20
20
21
-
![Diagram illustrating an example of 7x7 Conv alt-text#center](images/conv_nodes_7x7.jpg"An example of (7,7) Conv node")
21
+
![Diagram showing a 7x7 convolution node with input tensor of size 112x112x64, filter size 7x7x64, and output tensor size 56x56x64 with stride 2#center](images/conv_nodes_7x7.jpg"An example of (7,7) Conv node")
22
22
23
-
`ArmKleidiAI::MlasConv` kernel makes use of KleidiAI’s indirect matrix multiplication (imatmul) micro kernel to accelerate the convolution.
23
+
`ArmKleidiAI::MlasConv` kernel uses KleidiAI's indirect matrix multiplication (imatmul) micro kernel to accelerate the convolution.
It performs a batched fp32 matrix multiplication (GEMM or GemV) operation using KleidiAI matmul micro kernels. fp32 Conv operators with (1,1) filter kernels also uses this kernel.
49
+
This kernel performs a batched fp32 matrix multiplication (GEMM or GemV) operation using KleidiAI matmul micro kernels. fp32 Conv operators with (1,1) filter kernels also use this kernel.
50
50
51
51
For example, the figure below shows a (1,1) Conv node.
52
52
53
-
![Diagram illustrating an example of 1x1 Conv alt-text#center](images/conv_nodes_1x1.jpg"An example of (1,1) FusedConv node")
53
+
![Diagram showing a 1x1 convolution node with input tensor of size 56x56x64, filter size 1x1x64, and output tensor size 56x56x256 fused with BatchNormalization and Relu operations alt-txt#center](images/conv_nodes_1x1.jpg"An example of (1,1) FusedConv node")
54
54
55
55
The function calls of fp32 Conv operators with (1,1) filter kernels are shown below.
![Diagram illustrating an example of Gemm node alt-text#center](images/gemm_node.jpg"An example of Gemm node")
82
+
![Diagram showing a Gemm (General Matrix Multiplication) node with input matrices A and B, performing matrix multiplication with optional bias addition alt-txt#center](images/gemm_node.jpg"An example of Gemm node")
83
83
84
84
The function calls of fp32 Gemm operators are shown below.
This kernel is for matrix multiplication with float output of dynamic quantized A and symmetric quantized B.
105
-
It uses KleidiAI `kai_kernel_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa` micro kernel.
104
+
This kernel performs matrix multiplication with float output of dynamically quantized A and symmetrically quantized B.
105
+
It uses the KleidiAI `kai_kernel_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa` micro kernel.
106
+
107
+
## What you've accomplished and what's next
108
+
109
+
You now understand how KleidiAI integrates into ONNX Runtime's MLAS backend and which operators benefit from SME2 acceleration. You've learned about the three main kernel types: `MlasConv` for convolution operations, `MlasGemmBatch` for matrix multiplications and 1x1 convolutions, and `MlasDynamicQGemmBatch` for quantized operations. You've also seen the complete function call stacks showing how ONNX Runtime dispatches to KleidiAI kernels.
110
+
111
+
Next, you'll build ONNX Runtime with KleidiAI support enabled and prepare the benchmark tools for profiling on Android.
With the rise of on-device AI, squeezing performance from CPUs has become critical. Arm’s Scalable Matrix Extension 2 (SME2) represents a leap forward, offering significant speedups for matrix-heavy workloads like Transformers and CNNs.
11
-
This learning path will walk you through the technical steps to integrate KleidiAI—Arm's specialized micro-kernel library with SME2 support—into ONNX Runtime (ORT) and profile its performance using onnxruntime_perf_test on Android devices.
9
+
## ONNX Runtime overview
10
+
With the rise of on-device AI, squeezing performance from CPUs has become critical. Arm's Scalable Matrix Extension 2 (SME2) represents a leap forward, offering significant speedups for matrix-heavy workloads like Transformers and CNNs.
11
+
This Learning Path walks you through the technical steps to integrate KleidiAI—Arm's specialized micro-kernel library with SME2 support—into ONNX Runtime (ORT) and profile its performance using onnxruntime_perf_test on Android devices.
12
12
13
-
### Understanding the ONNX Runtime Software Stack
14
-
Firstly, let us look at the internal architecture of the ONNX Runtime.
ONNX Runtime's internal architecture consists of four main components that work together to execute AI models efficiently:
15
+
![Diagram showing the four main components of ONNX Runtime: In-Memory Graph at the top, followed by Graph Partitioner, Graph Runner, and Execution Provider at the bottom, with data flow indicated between layers alt-txt#center](images/ort_overview.jpg"The ONNX Runtime overview")
16
16
17
17
#### 1. In-Memory Graph
18
18
When loading an ONNX model, ORT parses the protobuf file and creates an In-Memory Graph. This is a live representation of the model’s structure, consisting of:
@@ -24,8 +24,8 @@ During this stage, ORT performs Graph Optimizations like constant folding and no
24
24
The Graph Partitioner decides which part of the model runs on which hardware. It analyzes the computational graph and matches nodes to the registered Execution Providers.
25
25
It clusters adjacent nodes assigned to the same EP into "Subgraphs".
26
26
#### 3. Graph Runner
27
-
Once the graph is partitioned, the Graph Runner is responsible for the actual execution of the operators in the correct order. It manages the flow of data (Tensors) between nodes.
28
-
In ORT, parallelism is split into two distinct levels to maximize hardware utilization: Intra-op (inside an operator/node, splitting a single heavy operation/node into smaller chunks) and Inter-op (between different operators, running multiple independent operators at the same time).
27
+
Once the graph is partitioned, the Graph Runner executes the operators in the correct order. It manages the flow of data (Tensors) between nodes.
28
+
In ORT, parallelism splits into two distinct levels to maximize hardware utilization: Intra-op (inside an operator/node, splitting a single heavy operation/node into smaller chunks) and Inter-op (between different operators, running multiple independent operators at the same time).
29
29
30
30
#### 4. Execution Provider (EP)
31
31
An Execution Provider is the abstraction layer that interfaces with specific hardware or libraries.
@@ -37,11 +37,17 @@ Examples:
37
37
38
38
If a specialized EP doesn't support a specific operator, ORT automatically falls back to the CPU provider.
39
39
40
-
Default CPU provider uses Microsoft Linear Algebra Subprogram (MLAS). MLAS is a minimal version of BLAS library which implements an optimized version of linear algebra operations such as general matrix multiply (GEMM) in low-level languages with various processor support. For aarch64, MLAS already utilizes dotprod, i8mm, fp16, bf16 vector instructions for acceleration.
40
+
The default CPU provider uses Microsoft Linear Algebra Subprograms (MLAS). MLAS is a minimal version of BLAS library that implements an optimized version of linear algebra operations such as general matrix multiply (GEMM) in low-level languages with various processor support. For aarch64, MLAS already uses dotprod, i8mm, fp16, and bf16 vector instructions for acceleration.
41
41
42
-
The KleidiAI-optimized MLAS can delegate high-performance matrix operations to KleidiAI micro kernels. KleidiAI provides micro-kernels specifically tuned for SME2, allowing ORT to instantly leverage the latest hardware features.
42
+
The KleidiAI-optimized MLAS can delegate high-performance matrix operations to KleidiAI micro kernels. KleidiAI provides micro-kernels specifically tuned for SME2, allowing ORT to leverage the latest hardware features.
43
43
44
-
This learning path focuses on Arm CPU Execution Provider.
44
+
This Learning Path focuses on Arm CPU Execution Provider.
45
+
46
+
## What you've accomplished and what's next
47
+
48
+
You now understand how ONNX Runtime processes models through its layered architecture—from the in-memory graph to execution providers. You've learned how the Graph Partitioner assigns operations to hardware, how the Graph Runner orchestrates execution, and how Execution Providers like the CPU provider use optimized kernels. You also know that MLAS serves as the default CPU backend and that KleidiAI can optimize it with SME2-specific kernels.
49
+
50
+
Next, you'll explore how KleidiAI integrates into MLAS and which specific operators benefit from SME2 acceleration.
0 commit comments