You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- The flag `onnxruntime_USE_KLEIDIAI=ON` triggers the inclusion of Arm KleidiAI kernels into the MLAS library.
33
-
- The build directory is `build/` by default. This can be overriden with the `--build_dir <path_to_your_build_directory>` commande line option to `build.sh`
33
+
- The build directory is `build/` by default. This can be overridden with the `--build_dir <path_to_your_build_directory>` command line option to `build.sh`.
34
+
{{% /notice %}}
34
35
35
36
## Profile model performance with onnxruntime_perf_test
36
37
@@ -54,16 +55,18 @@ The `onnxruntime_perf_test` tool simulates inference and gathers statistics. Run
54
55
# Execute on the device
55
56
adb shell "/data/local/tmp/onnxruntime_perf_test -e cpu -m times -r 20 -s -Z -x 1 /data/local/tmp/<your_model>/<your_model>.onnx"
56
57
```
58
+
### Command options explained
57
59
58
-
The command example set the arguments of the application as,
59
-
-`-e cpu` specifies the provider as cpu provider
60
-
-`-m times` specifies the test mode as “times”
61
-
-`-r 20` specifies the repeated times as 20
62
-
-`-Z` disallows thread from spinning during runs to reduce cpu usage
63
-
-`-s` shows statistics result
64
-
-`-x 1` sets the number of threads used to parallelize the execution within nodes as 1
60
+
The benchmark command uses several flags to control execution:
65
61
66
-
You can try other arguments setting if you would like to.
62
+
-`-e cpu`: Use the CPU execution provider
63
+
-`-m times`: Run in timing mode to measure latency
64
+
-`-r 20`: Repeat the test 20 times for consistent results
65
+
-`-Z`: Prevent thread spinning to reduce CPU usage
66
+
-`-s`: Display statistics after the run
67
+
-`-x 1`: Use a single thread for parallel execution within nodes
68
+
69
+
You can adjust these settings based on your performance testing needs.
Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/kleidiai_integration.md
+7-6Lines changed: 7 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,9 +7,10 @@ layout: learningpathall
7
7
---
8
8
9
9
## Integration of KleidiAI to ONNX Runtime MLAS
10
-
ONNX Runtime builds with KleidiAI support:
11
-
1. Detection: At runtime, MLAS checks the CPU capabilities for SME2 support.
12
-
2. Dispatch: If SME2 is detected, MLAS overrides its default kernels. For example, a Gemm (General Matrix Multiplication) operation that would normally use standard vector instructions (such as NEON) dispatches to a KleidiAI SME2 micro-kernel.
10
+
ONNX Runtime automatically detects and uses KleidiAI when SME2 support is available:
11
+
12
+
- Detection: MLAS checks the CPU capabilities for SME2 support at runtime.
13
+
- Dispatch: when SME2 is detected, MLAS replaces its default kernels with KleidiAI micro-kernels. For example, a Gemm operation that normally uses NEON instructions dispatches to a KleidiAI SME2 micro-kernel instead.
13
14
14
15
Currently, KleidiAI in MLAS provides `ArmKleidiAI::MlasConv`, `ArmKleidiAI::MlasGemmBatch`, and `ArmKleidiAI::MlasDynamicQGemmBatch` kernels.
15
16
@@ -18,7 +19,7 @@ ORT dispatches 2D fp32 (32-bit floating point) convolution operators with batch_
18
19
19
20
For example, the figure below shows a (7,7) Conv node.
20
21
21
-
 Conv node")
22
+
 Conv node")
22
23
23
24
`ArmKleidiAI::MlasConv` kernel uses KleidiAI's indirect matrix multiplication (imatmul) micro kernel to accelerate the convolution.
24
25
@@ -50,7 +51,7 @@ This kernel performs a batched fp32 matrix multiplication (GEMM or GemV) operati
50
51
51
52
For example, the figure below shows a (1,1) Conv node.
52
53
53
-
 FusedConv node")
54
+
 FusedConv node")
54
55
55
56
The function calls of fp32 Conv operators with (1,1) filter kernels are shown below.

83
+

83
84
84
85
The function calls of fp32 Gemm operators are shown below.
Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/overview.md
+28-16Lines changed: 28 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,32 +8,44 @@ layout: learningpathall
8
8
9
9
## ONNX Runtime overview
10
10
With the rise of on-device AI, squeezing performance from CPUs has become critical. Arm's Scalable Matrix Extension 2 (SME2) represents a leap forward, offering significant speedups for matrix-heavy workloads like Transformers and CNNs.
11
-
This Learning Path walks you through the technical steps to integrate KleidiAI—Arm's specialized micro-kernel library with SME2 support—into ONNX Runtime (ORT) and profile its performance using onnxruntime_perf_test on Android devices.
11
+
This Learning Path walks you through the technical steps to integrate KleidiAI-Arm's specialized micro-kernel library with SME2 support-into ONNX Runtime (ORT) and profile its performance using onnxruntime_perf_test on Android devices.
12
12
13
-
###Understanding the ONNX Runtime software stack
13
+
## Understanding the ONNX Runtime software stack
14
14
ONNX Runtime's internal architecture consists of four main components that work together to execute AI models efficiently:
15
15

16
16
17
-
#### 1. In-Memory Graph
18
-
When loading an ONNX model, ORT parses the protobuf file and creates an In-Memory Graph. This is a live representation of the model’s structure, consisting of:
- Edges: Representing the flow of data (tensors) between those operations.
17
+
### In-Memory Graph
18
+
When ORT loads an ONNX model, it parses the protobuf file and builds an in-memory representation of the model's structure. This graph consists of:
19
+
20
+
- Nodes: operations like MatMul, Conv, and Add
21
+
- Edges: tensor data flowing between operations
21
22
22
23
During this stage, ORT performs Graph Optimizations like constant folding and node fusion.
23
-
#### 2. Graph Partitioner
24
+
### Graph Partitioner
24
25
The Graph Partitioner decides which part of the model runs on which hardware. It analyzes the computational graph and matches nodes to the registered Execution Providers.
25
26
It clusters adjacent nodes assigned to the same EP into "Subgraphs".
26
-
#### 3. Graph Runner
27
+
### Graph Runner
27
28
Once the graph is partitioned, the Graph Runner executes the operators in the correct order. It manages the flow of data (Tensors) between nodes.
28
29
In ORT, parallelism splits into two distinct levels to maximize hardware utilization: Intra-op (inside an operator/node, splitting a single heavy operation/node into smaller chunks) and Inter-op (between different operators, running multiple independent operators at the same time).
30
+
### Execution Provider (EP)
31
+
32
+
An Execution Provider is the abstraction layer that interfaces with specific hardware or libraries. Each EP provides optimized math functions (called "Kernels") for specific operators.
33
+
34
+
ORT supports multiple Execution Providers across different hardware types:
35
+
36
+
**CPU-based providers:**
37
+
- Default CPU provider
38
+
- Intel DNNL
39
+
- XNNPACK
40
+
41
+
**GPU-based providers:**
42
+
- NVIDIA CUDA/TensorRT
43
+
- AMD MIGraphX
44
+
- DirectML
29
45
30
-
#### 4. Execution Provider (EP)
31
-
An Execution Provider is the abstraction layer that interfaces with specific hardware or libraries.
32
-
Each EP provides a set of "Kernels" (optimized math functions) for specific operators.
33
-
Examples:
34
-
- CPU: Default CPU, Intel DNNL, XNNPACK etc.
35
-
- GPU: NVIDIA CUDA/TensorRT, AMD MIGraphX, DirectML etc.
36
-
- Others: NPU, Qualcomm QNN etc.
46
+
**Specialized accelerators:**
47
+
- NPU
48
+
- Qualcomm QNN
37
49
38
50
If a specialized EP doesn't support a specific operator, ORT automatically falls back to the CPU provider.
39
51
@@ -45,7 +57,7 @@ This Learning Path focuses on Arm CPU Execution Provider.
45
57
46
58
## What you've accomplished and what's next
47
59
48
-
You now understand how ONNX Runtime processes models through its layered architecture—from the in-memory graph to execution providers. You've learned how the Graph Partitioner assigns operations to hardware, how the Graph Runner orchestrates execution, and how Execution Providers like the CPU provider use optimized kernels. You also know that MLAS serves as the default CPU backend and that KleidiAI can optimize it with SME2-specific kernels.
60
+
You now understand how ONNX Runtime processes models through its layered architecture-from the in-memory graph to execution providers. You've learned how the Graph Partitioner assigns operations to hardware, how the Graph Runner orchestrates execution, and how Execution Providers like the CPU provider use optimized kernels. You also know that MLAS serves as the default CPU backend and that KleidiAI can optimize it with SME2-specific kernels.
49
61
50
62
Next, you'll explore how KleidiAI integrates into MLAS and which specific operators benefit from SME2 acceleration.
Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example.md
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,12 +1,12 @@
1
1
---
2
-
title: Profiling – Use Resnet50v2 fp32 model as an example
2
+
title: Profile ONNX model performance
3
3
weight: 5
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Profile an ONNX model – Using Resnet50v2 as an example
9
+
## Profile ONNX model performance using ResNet-50 v2
10
10
11
11
Resnet50v2 serves as the example model in this Learning Path. Download the model package containing the ONNX model and its input data from the [ONNX model repository](https://github.com/onnx/models/tree/main/validated/vision/classification/resnet/model), then transfer it to your Android device:
12
12
@@ -76,11 +76,12 @@ P95 Latency: 0.34682 s
76
76
P99 Latency: 0.34682 s
77
77
P999 Latency: 0.34682 s
78
78
```
79
+
79
80
## Performance analysis
80
81
81
-
#### Visualize profiling data with perfetto
82
+
#### Visualize profiling data with Perfetto
82
83
83
-
You can use [perfetto tool](https://ui.perfetto.dev/) to view the two JSON profile files.
84
+
You can use [Perfetto](https://ui.perfetto.dev/) to view the two JSON profile files.
84
85
85
86
The figure below is a screenshot of the Non-KleidiAI version of the JSON profile file. The selected part (one `model_run/SequentialExecutor`) in the figure includes information of one inference execution.
0 commit comments