You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/litert-sme/1-litert-kleidiai-sme2.md
+21-15Lines changed: 21 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,41 +6,47 @@ weight: 2
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Understand the LiteRT software stack and SME2 integration
9
+
## Inside the LiteRT software stack
10
10
11
-
LiteRT (Lightweight Runtime, formerly TensorFlow Lite) is a runtime for on-device AI on Arm platforms. The default CPU acceleration library used by LiteRT is XNNPACK (an open-source library providing highly optimized neural-network operators).
11
+
LiteRT (Lightweight Runtime, formerly TensorFlow Lite) is a runtime for on-device AI on Arm platforms. The default CPU acceleration library used by LiteRT is XNNPACK.
12
12
13
-
XNNPACK is an open-source library that provides highly optimized implementations of neural-network operators. It continuously integrates the KleidiAI library to use new CPU features such as Scalable Matrix Extension version 2 (SME2).
13
+
XNNPACK is an open-source library that provides highly optimized implementations of neural-network operators. It continuously integrates the KleidiAI library to use new CPU features such as Scalable Matrix Extension 2 (SME2).
14
14
15
15
KleidiAI is a library developed by Arm that offers performance-critical micro-kernels using Arm architecture features, such as SME2.
16
16
17
17
The software stack for LiteRT is shown below.
18
18
19
-
![Diagram showing the software stack for on-device AI on Arm platforms. The stack is organized in layers from top to bottom: LiteRT at the top, followed by XNNPACK, then KleidiAI, and SME2 at the bottom. Arrows indicate the flow of execution from LiteRT through XNNPACK to KleidiAI and SME2. The diagram includes the following text labels: LiteRT, XNNPACK, KleidiAI, SME2. The environment is technical and structured, emphasizing the integration of Arm-optimized libraries for efficient AI inference. alt-text#center](./litert-sw-stack.png"LiteRT, XNNPACK, KleidiAI and SME2")
19
+
![Diagram showing the software stack for on-device AI on Arm platforms. The stack is organized in layers from top to bottom: LiteRT at the top, followed by XNNPACK, then KleidiAI, and SME2 at the bottom. Arrows indicate the flow of execution from LiteRT through XNNPACK to KleidiAI and SME2. The diagram includes the following text labels: LiteRT, XNNPACK, KleidiAI, SME2. The environment is technical and structured, emphasizing the integration of Arm-optimized libraries for efficient AI inference. alt-text#center](./litert-sw-stack.png"LiteRT, XNNPACK, KleidiAI, and SME2 software stack")
20
20
21
21
## How KleidiAI works in LiteRT
22
22
23
-
To understand how KleidiAI SME2 micro-kernels work in LiteRT, consider a LiteRT model with one Fully Connected operator using the FP32 data type.
23
+
To understand how KleidiAI SME2 micro-kernels work in LiteRT, think about a LiteRT model with one fully connected operator using the FP32 data type. The following diagrams illustrate the execution workflow of XNNPACK’s implementation compared with the workflow when KleidiAI SME2 is enabled in XNNPACK.
24
24
25
-
The following diagrams illustrate the execution workflow of XNNPACK’s implementation compared with the workflow when KleidiAI SME2 is enabled in XNNPACK.
25
+
### LiteRT → XNNPACKworkflow
26
26
27
-
## LiteRT → XNNPACK workflow
27
+
![Diagram showing the workflow for a fully connected operator in LiteRT using XNNPACK. The diagram depicts the flow from LiteRT to XNNPACK, highlighting the use of NEON instructions for matrix multiplication and weight packing on Arm platforms. The technical environment emphasizes operator traversal, hardware detection, and parallel computation. alt-text #center](./litert-xnnpack-workflow.png"LiteRT, XNNPACK workflow")
28
+
A fully connected operator multiplies two matrices: the input activations (LHS) and the weights (RHS).
28
29
29
-
![Diagram showing the workflow for a Fully Connected operator in LiteRT using XNNPACK. The diagram depicts the flow from LiteRT to XNNPACK, highlighting the use of NEON instructions for matrix multiplication and weight packing on Arm platforms. The technical environment emphasizes operator traversal, hardware detection, and parallel computation. alt-text #center](./litert-xnnpack-workflow.png"LiteRT, XNNPACK workflow")
30
-
A Fully Connected operator works by multiplying two matrices together.
30
+
When LiteRT loads a model, it reads the operators and builds a computation graph. If you select the CPU as the accelerator, LiteRT uses XNNPACK by default.
31
31
32
-
When LiteRT loads a model, it reads the operators and builds a computation graph. If you choose the CPU as the accelerator, LiteRT uses XNNPACK by default.
32
+
XNNPACK scans the computation graph and looks for operators it can optimize. It packs the weight matrix to prepare for efficient computation. On Arm platforms, XNNPACK uses NEON instructions to speed up this packing and the matrix multiplication.
33
33
34
-
XNNPACK scans the computation graph and tries to replace operators with its own optimized versions. At this stage, XNNPACK packs the weight matrix. On Arm platforms, it uses NEON instructions to make this process faster. XNNPACK includes different implementations for different hardware. At runtime, it checks the hardware and picks the best micro-kernel.
34
+
At runtime, XNNPACK checks the hardware and chooses the best available micro-kernel. During inference, it splits the matrices into smaller tiles and runs the multiplications in parallel across multiple threads, using NEON instructions for faster processing.
35
35
36
-
During inference, XNNPACK multiplies the activation matrix (left-hand side, LHS) by the packed weight matrix (right-hand side, RHS). It splits the matrices into smaller tiles and runs the multiplications in parallel using multiple threads. NEON instructions help speed up these calculations.
36
+
### LiteRT → XNNPACK → KleidiAI workflow
37
37
38
-
## LiteRT → XNNPACK → KleidiAI workflow
39
-
40
-
![Diagram showing the workflow for a Fully Connected operator in LiteRT using XNNPACK and KleidiAI with SME2. The diagram illustrates the flow from LiteRT to XNNPACK, then to KleidiAI, highlighting SME2 micro-kernel integration for matrix multiplication and packing. The technical context emphasizes runtime hardware detection and optimized operator execution. alt-text #center](./litert-xnnpack-kleidiai-workflow.png"LiteRT, XNNPACK, KleidiAI workflow")
38
+
![Diagram showing the workflow for a fully connected operator in LiteRT using XNNPACK and KleidiAI with SME2. The diagram illustrates the flow from LiteRT to XNNPACK, then to KleidiAI, highlighting SME2 micro-kernel integration for matrix multiplication and packing. The technical context emphasizes runtime hardware detection and optimized operator execution. alt-text #center](./litert-xnnpack-kleidiai-workflow.png"LiteRT, XNNPACK, KleidiAI workflow")
41
39
42
40
When KleidiAI and SME2 are enabled at build time, the KleidiAI SME2 micro-kernels are compiled into XNNPACK.
43
41
44
42
During the model loading stage, when XNNPACK optimizes the subgraph, it checks the operator’s data type to determine whether a KleidiAI implementation is available. If KleidiAI supports it, XNNPACK bypasses its own default implementation. As a result, RHS packing is performed using the KleidiAI SME packing micro-kernel. Because KleidiAI typically requires packing of the LHS, a flag is also set during this stage.
45
43
46
44
During model inference, the LHS packing micro-kernel is invoked. After the LHS is packed, XNNPACK performs the matrix multiplication. At this point, the KleidiAI SME micro-kernel is used to compute the matrix product.
45
+
46
+
## What you've accomplished and what's next
47
+
48
+
In this section, you explored how LiteRT leverages XNNPACK and KleidiAI to accelerate fully connected operators on Arm platforms. You learned how XNNPACK uses NEON instructions for efficient matrix operations and how enabling KleidiAI with SME2 further optimizes performance by introducing specialized micro-kernels for packing and matrix multiplication.
49
+
50
+
You have completed the overview of LiteRT, XNNPACK, KleidiAI, and SME2 integration. Next, you’ll dive deeper into building and benchmarking models with these technologies.
To demonstrate SME2 acceleration on Android, you will construct simple single-layer models (for example, Fully Connected) using Keras and convert them into LiteRT (`.tflite`) format. This allows you to benchmark isolated operators and directly observe SME2 improvements.
51
+
To demonstrate SME2 acceleration on Android, you will construct simple single-layer models (for example, fully connected) using Keras and convert them into LiteRT (`.tflite`) format. This allows you to benchmark isolated operators and directly observe SME2 improvements.
53
52
54
53
Install the TensorFlow package dependency for your script:
55
54
@@ -122,8 +121,6 @@ This method applies FP16 quantization to a model with FP32 operators. In practic
122
121
123
122
KleidiAI provides FP16 packing micro-kernels for both the activations and weights matrix, as well as FP16 matrix multiplication micro-kernels.
124
123
125
-
---
126
-
127
124
**Post-training INT8 dynamic range quantization**
128
125
129
126
```python
@@ -138,8 +135,6 @@ This quantization method optimizes operators with large parameter sizes by quant
138
135
139
136
KleidiAI provides micro-kernels that dynamically quantize activations to INT8 at runtime, as well as packing micro-kernels for the weights matrix and INT8 matrix multiplication micro-kernels that produce FP32 outputs.
140
137
141
-
---
142
-
143
138
**Post-training INT8 static quantization**
144
139
145
140
```python
@@ -163,6 +158,13 @@ This quantization method quantizes both the activations and the weights to INT8.
163
158
164
159
KleidiAI provides INT8 packing micro-kernels for both the activations and weights matrix, as well as INT8 matrix multiplication micro-kernels.
165
160
166
-
---
161
+
## What you've accomplishee and what's next
162
+
You have now created several LiteRT models with different quantization options, ready for benchmarking on your Arm-based Android device. You have:
163
+
164
+
- Built a simple Keras model and converted it to LiteRT (`.tflite`) format.
165
+
- Generated models with different quantization types: FP32, FP16, INT8 dynamic, and INT8 static.
166
+
- Learned how each quantization method affects model size, performance, and compatibility with KleidiAI SME2 micro-kernels.
167
+
168
+
Now that you have created and converted your models, you can benchmark them on your Android device to measure the performance gains from SME2 acceleration. Consider experimenting with additional layers such as Conv2D, BatchMatMul, or TransposeConv to further explore SME2 support for different operators. You can also apply more quantization and optimization techniques to enhance model efficiency. Finally, integrate your optimized models into your Android applications to leverage Arm SME2 acceleration in real-world use cases.
167
169
168
-
You have now created several LiteRT models with different quantization options, ready for benchmarking on your Arm-based Android device.
170
+
By following these steps, you can maximize the performance of your machine learning models on Arm-based devices using LiteRT and KleidiAI SME2.
@@ -113,7 +117,7 @@ You will see the time spent on model initialization, warm-up, and inference, as
113
117
Because the model contains only a single fully connected layer, the node type `Fully Connected (NC, PF32) GEMM` shows the average execution time and its percentage of total inference time.
114
118
115
119
{{% notice Note %}}
116
-
To verify that KleidiAI SME2 micro-kernels are invoked for the Fully Connected operator during inference, you can use `simpleperf record -g -- <workload>` to capture the calling graph. For `benchmark_model`, build with the `-c dbg` option.
120
+
To verify that KleidiAI SME2 micro-kernels are invoked for the FullyConnected operator during inference, run `simpleperf record -g -- <workload>` to capture the calling graph. If you're using the `benchmark_model`, build it with the `-c dbg` option.
117
121
{{% /notice %}}
118
122
119
123
## Measure the performance impact of KleidiAI SME2 micro-kernels
@@ -217,3 +221,10 @@ The letter “P” in the node type indicates a KleidiAI implementation.
217
221
For example, `Convolution (NHWC, PQS8, QS8, QC8W)` represents a Conv2D operator computed by a KleidiAI micro-kernel, where the tensor is in NHWC layout, the input is packed INT8 quantized, the weights are per-channel INT8 quantized, and the output is INT8 quantized.
218
222
219
223
By comparing `benchmark_model` runs with and without KleidiAI and SME2, and inspecting the profiled node types (PF32, PF16, QP8, PQS8), you can confirm that LiteRT is dispatching to SME2-optimized KleidiAI micro-kernels and quantify their performance impact on your Android device.
224
+
225
+
226
+
## What you've accomplished and what's next
227
+
228
+
In this section, you learned how to benchmark LiteRT models on SME2-capable Android devices, verify SME2 support, and interpret performance results with and without KleidiAI SME2 micro-kernels. You also discovered how to identify which micro-kernels are used during inference by examining node type names in the profiler output.
229
+
230
+
You are now ready to further analyze performance, experiment with different models, or explore advanced profiling and optimization techniques for on-device AI with LiteRT, XNNPACK, and KleidiAI. Continue to the next section to deepen your understanding or apply these skills to your own projects.
Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/litert-sme/_index.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ title: Accelerate LiteRT Models on Android with KleidiAI and SME2
3
3
4
4
minutes_to_complete: 30
5
5
6
-
who_is_this_for: This is an advanced topic for developers looking to leverage Arm's Scalable Matrix Extension version 2 (SME2) instructions to accelerate LiteRT model inference on Android.
6
+
who_is_this_for: This is an advanced topic for developers looking to leverage Arm's Scalable Matrix Extension 2 (SME2) instructions to accelerate LiteRT model inference on Android.
7
7
8
8
learning_objectives:
9
9
- Understand how KleidiAI integrates with LiteRT
@@ -13,7 +13,7 @@ learning_objectives:
13
13
14
14
prerequisites:
15
15
- An Arm64 Linux development machine
16
-
- An Android device that supports Arm SME2 architecture features - see the list of [devices with SME2 support](/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/#devices)
16
+
- An Android device that supports Arm SME2 architecture features - see this [list of devices with SME2 support](/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/#devices)
0 commit comments