You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Understand LiteRT, XNNPACK, KleidiAI, and SME2
2
+
title: Explore LiteRT, XNNPACK, KleidiAI, and SME2
3
3
weight: 2
4
4
5
5
### FIXED, DO NOT MODIFY
@@ -8,42 +8,40 @@ layout: learningpathall
8
8
9
9
## LiteRT, XNNPACK, KleidiAI, and SME2
10
10
11
-
LiteRT (Lite Runtime, formerly TensorFlow Lite) is a runtime for on-device AI on Arm platforms. The default CPU acceleration library used by LiteRT is XNNPACK (an open-source library providing highly optimized neural-network operators).
11
+
LiteRT (Lightweight Runtime, formerly TensorFlow Lite) is a runtime for on-device AI on Arm platforms. The default CPU acceleration library used by LiteRT is XNNPACK (an open-source library providing highly optimized neural-network operators).
12
12
13
-
XNNPACK is an open-source library that provides highly optimized implementations of neural-network operators. It continuously integrates KleidiAI library to leverage new CPU features such as SME2.
13
+
XNNPACK is an open-source library that provides highly optimized implementations of neural-network operators. It continuously integrates the KleidiAI library to use new CPU features such as Scalable Matrix Extension version 2 (SME2).
14
14
15
-
KleidiAI is a library developed by Arm that offers performance-critical micro-kernels leveraging Arm architecture features, such as SME2.
15
+
KleidiAI is a library developed by Arm that offers performance-critical micro-kernels using Arm architecture features, such as SME2.
16
16
17
17
The software stack for LiteRT is shown below.
18
18
19
-

19
+

20
20
21
+
## How KleidiAI works in LiteRT
21
22
22
-
## Understand how KleidiAI works in LiteRT
23
+
To understand how KleidiAI SME2 micro-kernels work in LiteRT, consider a LiteRT model with one Fully Connected operator using the FP32 data type.
23
24
24
-
To understand how KleidiAI SME2 micro-kernel works in LiteRT, a LiteRT model with one Fully Connected operator with FP32 datatype is used as an example.
25
-
26
-
The following illustrates the execution workflow of XNNPACK’s implementation compared with the workflow when KleidiAI SME2 is enabled in XNNPACK.
25
+
The following diagrams illustrate the execution workflow of XNNPACK’s implementation compared with the workflow when KleidiAI SME2 is enabled in XNNPACK.
A Fully Connected operator can be essentially implemented as a matrix multiplication.
29
+

33
30
34
-
When LiteRT loads a model, it parses the operators and create a computation graph. If the CPU is selected as the accelerator, LiteRT uses XNNPACK by default.
31
+
A Fully Connected operator is essentially implemented as a matrix multiplication.
35
32
36
-
XNNPACK traverses the operators in the graph and tries to replace them with its own implementations. During this stage, XNNPACK performs the necessary packing of the weight matrix. To speed up the packing process, XNNPACK uses NEON instructions for Arm platform. XNNPACK provides different implementations for different hardware platforms. At runtime, it detects the hardware capabilities and selects the appropriate micro-kernel.
33
+
When LiteRT loads a model, it parses the operators and creates a computation graph. If the CPU is selected as the accelerator, LiteRT uses XNNPACK by default.
37
34
38
-
During model inference, XNNPACK performs matrix multiplication on the activation matrix (the left-hand side matrix, LHS) and the repacked weight matrix (the right-hand side matrix, RHS). In this stage, XNNPACK applies tiling strategies to the matrices and performs parallel multiplication across the resulting tiles using multiple threads. To accelerate the computation, XNNPACK uses NEON instructions.
35
+
XNNPACK traverses the operators in the graph and tries to replace them with its own implementations. During this stage, XNNPACK packs the weight matrix, using NEON instructions for Arm platforms to speed up the process. XNNPACK provides different implementations for different hardware platforms. At runtime, it detects hardware capabilities and selects the appropriate micro-kernel.
39
36
37
+
During model inference, XNNPACK performs matrix multiplication on the activation matrix (left-hand side, LHS) and the repacked weight matrix (right-hand side, RHS). XNNPACK applies tiling strategies to the matrices and performs parallel multiplication across the resulting tiles using multiple threads. To accelerate computation, XNNPACK uses NEON instructions.

44
42
45
-
When KleidiAI and SME2 are enabled at building stage, the KleidiAI SME2 micro-kernels are compiled into the XNNPACK.
43
+
When KleidiAI and SME2 are enabled at build time, the KleidiAI SME2 micro-kernels are compiled into XNNPACK.
46
44
47
-
During the model loading stage, when XNNPACK optimizes the subgraph, it checks the operator’s data type to determine whether a KleidiAI implementation is available. If KleidiAI supports it, XNNPACK bypasses its own default implementation. As a result, RHS packing is performed using the KleidiAI SME packing micro-kernel. In addition, because KleidiAI typically requires packing of the LHS, a flag is also set during this stage.
45
+
During the model loading stage, when XNNPACK optimizes the subgraph, it checks the operator’s data type to determine whether a KleidiAI implementation is available. If KleidiAI supports it, XNNPACK bypasses its own default implementation. As a result, RHS packing is performed using the KleidiAI SME packing micro-kernel. Because KleidiAI typically requires packing of the LHS, a flag is also set during this stage.
48
46
49
47
During model inference, the LHS packing micro-kernel is invoked. After the LHS is packed, XNNPACK performs the matrix multiplication. At this point, the KleidiAI SME micro-kernel is used to compute the matrix product.
Because LiteRT integrates KleidiAI through XNNPACK, you must build LiteRT from source to enable SME2 micro-kernels.
25
25
26
-
Next, set up your Android build environment using Docker on your Linux developement machine.
27
-
Google provides a Dockerfile that installs the toolchain needed for TFLite/LiteRT Android builds.
26
+
Because LiteRT integrates KleidiAI through XNNPACK (an open-source library providing highly optimized neural-network operators), you must build LiteRT from source to enable SME2 micro-kernels.
27
+
28
+
Next, set up your Android build environment using Docker on your Linux development machine. Google provides a Dockerfile that installs the toolchain needed for TensorFlow Lite (TFLite)/LiteRT Android builds.
The Docker image includes Bazel, NDK, CMake, toolchains, and Python required for cross-compiling Android binaries.
38
41
39
-
You will now install Android SDK/NDK Components inside the Container.
40
-
Launch the docker container:
42
+
The Docker image includes Bazel, Android Native Development Kit (NDK), CMake, toolchains, and Python required for cross-compiling Android binaries.
43
+
44
+
Now, install Android Software Development Kit (SDK) and NDK components inside the container.
45
+
46
+
Launch the Docker container:
47
+
41
48
```bash
42
49
docker run -it -v $PWD:/host_dir tflite-builder bash
43
50
```
51
+
44
52
Install Android platform tools:
45
-
```bash
53
+
54
+
```bash
46
55
sdkmanager \
47
56
"build-tools;${ANDROID_BUILD_TOOLS_VERSION}" \
48
57
"platform-tools" \
49
58
"platforms;android-${ANDROID_API_LEVEL}"
50
59
```
51
60
52
-
Configure LiteRT Build Options inside your running container:
61
+
Configure LiteRT build options inside your running container:
53
62
54
-
```bash
63
+
```bash
55
64
cd /host_dir/LiteRT
56
65
./configure
57
66
```
58
67
59
-
Use default values for all the prompts except when prompted:
68
+
Use default values for all prompts except when asked:
69
+
60
70
```output
61
71
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]
62
72
```
63
-
Type in `y`.
64
73
65
-
LiteRT's configuration script will detect SDK + NDK paths, set toolchain versions, configure Android ABI (arm64-v8a) and initialize Bazel workspace rules.
74
+
Type `y`and press Enter.
66
75
67
-
Now, you can build the benchmark tool with KleidiAI + SME2 Enabled.
76
+
LiteRT's configuration script will detect SDK and NDK paths, set toolchain versions, configure the Android Application Binary Interface (ABI) to arm64-v8a, and initialize Bazel workspace rules.
77
+
78
+
Now, you can build the benchmark tool with KleidiAI and SME2 enabled.
68
79
69
80
Enable XNNPACK, quantization paths, and SME2 acceleration:
This build enables the KleidiAI and SME2 micro-kernels integrated into XNNPACK and produces an Android binary under:
101
+
This build enables the KleidiAI and SME2 micro-kernels integrated into XNNPACK and produces an Android binary at:
88
102
89
103
```output
90
104
bazel-bin/litert/tools/benchmark_model
91
105
```
92
106
93
-
### Build the LiteRT benchmark tool without KleidiAI (Baseline Comparison)
107
+
---
108
+
109
+
## Build the LiteRT benchmark tool without KleidiAI (baseline comparison)
110
+
111
+
To compare the performance of the KleidiAI SME2 implementation against XNNPACK’s original implementation, build another version of the LiteRT benchmark tool without KleidiAI and SME2 enabled.
94
112
95
-
To compare the performance of KleidiAI SME2 implementation against XNNPACK’s original implementation, you can build another version of LiteRT benchmark tool without KleidiAI and SME2 enabled.
113
+
Set the build options to disable SME2 and KleidiAI:
You have now built both versions of the LiteRT benchmark tool. You are ready to benchmark and compare SME2-accelerated and baseline performance on your Arm-based Android device.
0 commit comments