Skip to content

Commit b55bde1

Browse files
Refactor documentation for ONNX Runtime learning path: enhance clarity, improve structure, and correct formatting in multiple markdown files.
1 parent 9a9c4ae commit b55bde1

4 files changed

Lines changed: 58 additions & 41 deletions

File tree

content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/build_ort.md

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@ layout: learningpathall
88

99
## Build ONNX Runtime with KleidiAI and SME2 for Android
1010

11-
To run this on an Android device, cross-compile ORT using the Android NDK.
11+
To run this on an Android device, you need to cross-compile ONNX Runtime using the Android NDK.
1212

13-
Prerequisites:
14-
- Android NDK: Version r26b or newer (r27+ recommended for latest SME2 toolchain support).
15-
- CMake & Ninja: Ensure these are in your system PATH.
13+
Before you begin, verify that you have:
14+
- Android NDK version r26b or newer (r27 or later is recommended for the latest SME2 toolchain support)
15+
- CMake and Ninja installed and available in your system PATH
1616

17-
### Build onnxruntime
17+
## Build onnxruntime
1818

1919
First, clone the [ONNX Runtime](https://github.com/microsoft/onnxruntime). This Learning Path uses version `v1.23.2`:
2020
```bash
@@ -28,9 +28,10 @@ Build ONNX Runtime with KleidiAI support enabled. The build script configures cr
2828
./build.sh --android --android_sdk_path $ANDROID_NDK_HOME --android_ndk_path $ANDROID_NDK_HOME --android_abi arm64-v8a --android_api 27 --config RelWithDebInfo --build_shared_lib --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON --cmake_generator Ninja --parallel
2929
```
3030

31-
Notes:
31+
{{% notice Note %}}
3232
- The flag `onnxruntime_USE_KLEIDIAI=ON` triggers the inclusion of Arm KleidiAI kernels into the MLAS library.
33-
- The build directory is `build/` by default. This can be overriden with the `--build_dir <path_to_your_build_directory>` commande line option to `build.sh`
33+
- The build directory is `build/` by default. This can be overridden with the `--build_dir <path_to_your_build_directory>` command line option to `build.sh`.
34+
{{% /notice %}}
3435

3536
## Profile model performance with onnxruntime_perf_test
3637

@@ -54,16 +55,18 @@ The `onnxruntime_perf_test` tool simulates inference and gathers statistics. Run
5455
# Execute on the device
5556
adb shell "/data/local/tmp/onnxruntime_perf_test -e cpu -m times -r 20 -s -Z -x 1 /data/local/tmp/<your_model>/<your_model>.onnx"
5657
```
58+
### Command options explained
5759

58-
The command example set the arguments of the application as,
59-
- `-e cpu` specifies the provider as cpu provider
60-
- `-m times` specifies the test mode as “times”
61-
- `-r 20` specifies the repeated times as 20
62-
- `-Z` disallows thread from spinning during runs to reduce cpu usage
63-
- `-s` shows statistics result
64-
- `-x 1` sets the number of threads used to parallelize the execution within nodes as 1
60+
The benchmark command uses several flags to control execution:
6561

66-
You can try other arguments setting if you would like to.
62+
- `-e cpu`: Use the CPU execution provider
63+
- `-m times`: Run in timing mode to measure latency
64+
- `-r 20`: Repeat the test 20 times for consistent results
65+
- `-Z`: Prevent thread spinning to reduce CPU usage
66+
- `-s`: Display statistics after the run
67+
- `-x 1`: Use a single thread for parallel execution within nodes
68+
69+
You can adjust these settings based on your performance testing needs.
6770

6871
## Deep dive into operator profiling
6972

content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/kleidiai_integration.md

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,10 @@ layout: learningpathall
77
---
88

99
## Integration of KleidiAI to ONNX Runtime MLAS
10-
ONNX Runtime builds with KleidiAI support:
11-
1. Detection: At runtime, MLAS checks the CPU capabilities for SME2 support.
12-
2. Dispatch: If SME2 is detected, MLAS overrides its default kernels. For example, a Gemm (General Matrix Multiplication) operation that would normally use standard vector instructions (such as NEON) dispatches to a KleidiAI SME2 micro-kernel.
10+
ONNX Runtime automatically detects and uses KleidiAI when SME2 support is available:
11+
12+
- Detection: MLAS checks the CPU capabilities for SME2 support at runtime.
13+
- Dispatch: when SME2 is detected, MLAS replaces its default kernels with KleidiAI micro-kernels. For example, a Gemm operation that normally uses NEON instructions dispatches to a KleidiAI SME2 micro-kernel instead.
1314

1415
Currently, KleidiAI in MLAS provides `ArmKleidiAI::MlasConv`, `ArmKleidiAI::MlasGemmBatch`, and `ArmKleidiAI::MlasDynamicQGemmBatch` kernels.
1516

@@ -18,7 +19,7 @@ ORT dispatches 2D fp32 (32-bit floating point) convolution operators with batch_
1819

1920
For example, the figure below shows a (7,7) Conv node.
2021

21-
![Diagram showing a 7x7 convolution node with input tensor of size 112x112x64, filter size 7x7x64, and output tensor size 56x56x64 with stride 2#center](images/conv_nodes_7x7.jpg "An example of (7,7) Conv node")
22+
![Diagram showing a 7x7 convolution node with input tensor of size 112x112x64, filter size 7x7x64, and output tensor size 56x56x64 with stride 2#center](images/conv_nodes_7x7.jpg "An example of a (7,7) Conv node")
2223

2324
`ArmKleidiAI::MlasConv` kernel uses KleidiAI's indirect matrix multiplication (imatmul) micro kernel to accelerate the convolution.
2425

@@ -50,7 +51,7 @@ This kernel performs a batched fp32 matrix multiplication (GEMM or GemV) operati
5051

5152
For example, the figure below shows a (1,1) Conv node.
5253

53-
![Diagram showing a 1x1 convolution node with input tensor of size 56x56x64, filter size 1x1x64, and output tensor size 56x56x256 fused with BatchNormalization and Relu operations alt-txt#center](images/conv_nodes_1x1.jpg "An example of (1,1) FusedConv node")
54+
![Diagram showing a 1x1 convolution node with input tensor of size 56x56x64, filter size 1x1x64, and output tensor size 56x56x256 fused with BatchNormalization and Relu operations alt-txt#center](images/conv_nodes_1x1.jpg "An example of a (1,1) FusedConv node")
5455

5556
The function calls of fp32 Conv operators with (1,1) filter kernels are shown below.
5657

@@ -79,7 +80,7 @@ onnxruntime::InferenceSession::Run
7980

8081
For example, the figure below shows a Gemm node.
8182

82-
![Diagram showing a Gemm (General Matrix Multiplication) node with input matrices A and B, performing matrix multiplication with optional bias addition alt-txt#center](images/gemm_node.jpg "An example of Gemm node")
83+
![Diagram showing a Gemm (General Matrix Multiplication) node with input matrices A and B, performing matrix multiplication with optional bias addition alt-txt#center](images/gemm_node.jpg "An example of a Gemm node")
8384

8485
The function calls of fp32 Gemm operators are shown below.
8586
```text

content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/overview.md

Lines changed: 28 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -8,32 +8,44 @@ layout: learningpathall
88

99
## ONNX Runtime overview
1010
With the rise of on-device AI, squeezing performance from CPUs has become critical. Arm's Scalable Matrix Extension 2 (SME2) represents a leap forward, offering significant speedups for matrix-heavy workloads like Transformers and CNNs.
11-
This Learning Path walks you through the technical steps to integrate KleidiAIArm's specialized micro-kernel library with SME2 supportinto ONNX Runtime (ORT) and profile its performance using onnxruntime_perf_test on Android devices.
11+
This Learning Path walks you through the technical steps to integrate KleidiAI-Arm's specialized micro-kernel library with SME2 support-into ONNX Runtime (ORT) and profile its performance using onnxruntime_perf_test on Android devices.
1212

13-
### Understanding the ONNX Runtime software stack
13+
## Understanding the ONNX Runtime software stack
1414
ONNX Runtime's internal architecture consists of four main components that work together to execute AI models efficiently:
1515
![Diagram showing the four main components of ONNX Runtime: In-Memory Graph at the top, followed by Graph Partitioner, Graph Runner, and Execution Provider at the bottom, with data flow indicated between layers alt-txt#center](images/ort_overview.jpg "The ONNX Runtime overview")
1616

17-
#### 1. In-Memory Graph
18-
When loading an ONNX model, ORT parses the protobuf file and creates an In-Memory Graph. This is a live representation of the model’s structure, consisting of:
19-
- Nodes: Representing operations (e.g., MatMul, Conv, Add).
20-
- Edges: Representing the flow of data (tensors) between those operations.
17+
### In-Memory Graph
18+
When ORT loads an ONNX model, it parses the protobuf file and builds an in-memory representation of the model's structure. This graph consists of:
19+
20+
- Nodes: operations like MatMul, Conv, and Add
21+
- Edges: tensor data flowing between operations
2122

2223
During this stage, ORT performs Graph Optimizations like constant folding and node fusion.
23-
#### 2. Graph Partitioner
24+
### Graph Partitioner
2425
The Graph Partitioner decides which part of the model runs on which hardware. It analyzes the computational graph and matches nodes to the registered Execution Providers.
2526
It clusters adjacent nodes assigned to the same EP into "Subgraphs".
26-
#### 3. Graph Runner
27+
### Graph Runner
2728
Once the graph is partitioned, the Graph Runner executes the operators in the correct order. It manages the flow of data (Tensors) between nodes.
2829
In ORT, parallelism splits into two distinct levels to maximize hardware utilization: Intra-op (inside an operator/node, splitting a single heavy operation/node into smaller chunks) and Inter-op (between different operators, running multiple independent operators at the same time).
30+
### Execution Provider (EP)
31+
32+
An Execution Provider is the abstraction layer that interfaces with specific hardware or libraries. Each EP provides optimized math functions (called "Kernels") for specific operators.
33+
34+
ORT supports multiple Execution Providers across different hardware types:
35+
36+
**CPU-based providers:**
37+
- Default CPU provider
38+
- Intel DNNL
39+
- XNNPACK
40+
41+
**GPU-based providers:**
42+
- NVIDIA CUDA/TensorRT
43+
- AMD MIGraphX
44+
- DirectML
2945

30-
#### 4. Execution Provider (EP)
31-
An Execution Provider is the abstraction layer that interfaces with specific hardware or libraries.
32-
Each EP provides a set of "Kernels" (optimized math functions) for specific operators.
33-
Examples:
34-
- CPU: Default CPU, Intel DNNL, XNNPACK etc.
35-
- GPU: NVIDIA CUDA/TensorRT, AMD MIGraphX, DirectML etc.
36-
- Others: NPU, Qualcomm QNN etc.
46+
**Specialized accelerators:**
47+
- NPU
48+
- Qualcomm QNN
3749

3850
If a specialized EP doesn't support a specific operator, ORT automatically falls back to the CPU provider.
3951

@@ -45,7 +57,7 @@ This Learning Path focuses on Arm CPU Execution Provider.
4557

4658
## What you've accomplished and what's next
4759

48-
You now understand how ONNX Runtime processes models through its layered architecturefrom the in-memory graph to execution providers. You've learned how the Graph Partitioner assigns operations to hardware, how the Graph Runner orchestrates execution, and how Execution Providers like the CPU provider use optimized kernels. You also know that MLAS serves as the default CPU backend and that KleidiAI can optimize it with SME2-specific kernels.
60+
You now understand how ONNX Runtime processes models through its layered architecture-from the in-memory graph to execution providers. You've learned how the Graph Partitioner assigns operations to hardware, how the Graph Runner orchestrates execution, and how Execution Providers like the CPU provider use optimized kernels. You also know that MLAS serves as the default CPU backend and that KleidiAI can optimize it with SME2-specific kernels.
4961

5062
Next, you'll explore how KleidiAI integrates into MLAS and which specific operators benefit from SME2 acceleration.
5163

content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
---
2-
title: Profiling – Use Resnet50v2 fp32 model as an example
2+
title: Profile ONNX model performance
33
weight: 5
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9-
## Profile an ONNX model – Using Resnet50v2 as an example
9+
## Profile ONNX model performance using ResNet-50 v2
1010

1111
Resnet50v2 serves as the example model in this Learning Path. Download the model package containing the ONNX model and its input data from the [ONNX model repository](https://github.com/onnx/models/tree/main/validated/vision/classification/resnet/model), then transfer it to your Android device:
1212

@@ -76,11 +76,12 @@ P95 Latency: 0.34682 s
7676
P99 Latency: 0.34682 s
7777
P999 Latency: 0.34682 s
7878
```
79+
7980
## Performance analysis
8081

81-
#### Visualize profiling data with perfetto
82+
#### Visualize profiling data with Perfetto
8283

83-
You can use [perfetto tool](https://ui.perfetto.dev/) to view the two JSON profile files.
84+
You can use [Perfetto](https://ui.perfetto.dev/) to view the two JSON profile files.
8485

8586
The figure below is a screenshot of the Non-KleidiAI version of the JSON profile file. The selected part (one `model_run/SequentialExecutor`) in the figure includes information of one inference execution.
8687

0 commit comments

Comments
 (0)