Skip to content

Commit ddb25be

Browse files
Update Litert SME learning path content
1 parent 71c8cbe commit ddb25be

4 files changed

Lines changed: 148 additions & 109 deletions

File tree

Lines changed: 16 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Understand LiteRT, XNNPACK, KleidiAI, and SME2
2+
title: Explore LiteRT, XNNPACK, KleidiAI, and SME2
33
weight: 2
44

55
### FIXED, DO NOT MODIFY
@@ -8,42 +8,40 @@ layout: learningpathall
88

99
## LiteRT, XNNPACK, KleidiAI, and SME2
1010

11-
LiteRT (Lite Runtime, formerly TensorFlow Lite) is a runtime for on-device AI on Arm platforms. The default CPU acceleration library used by LiteRT is XNNPACK (an open-source library providing highly optimized neural-network operators).
11+
LiteRT (Lightweight Runtime, formerly TensorFlow Lite) is a runtime for on-device AI on Arm platforms. The default CPU acceleration library used by LiteRT is XNNPACK (an open-source library providing highly optimized neural-network operators).
1212

13-
XNNPACK is an open-source library that provides highly optimized implementations of neural-network operators. It continuously integrates KleidiAI library to leverage new CPU features such as SME2.
13+
XNNPACK is an open-source library that provides highly optimized implementations of neural-network operators. It continuously integrates the KleidiAI library to use new CPU features such as Scalable Matrix Extension version 2 (SME2).
1414

15-
KleidiAI is a library developed by Arm that offers performance-critical micro-kernels leveraging Arm architecture features, such as SME2.
15+
KleidiAI is a library developed by Arm that offers performance-critical micro-kernels using Arm architecture features, such as SME2.
1616

1717
The software stack for LiteRT is shown below.
1818

19-
![LiteRT, XNNPACK, KleidiAI and SME2#center](./litert-sw-stack.png "LiteRT, XNNPACK, KleidiAI and SME2")
19+
![Diagram showing the software stack for on-device AI on Arm platforms. The stack is organized in layers from top to bottom: LiteRT at the top, followed by XNNPACK, then KleidiAI, and SME2 at the bottom. Arrows indicate the flow of execution from LiteRT through XNNPACK to KleidiAI and SME2. The diagram includes the following text labels: LiteRT, XNNPACK, KleidiAI, SME2. The environment is technical and structured, emphasizing the integration of Arm-optimized libraries for efficient AI inference. #center](./litert-sw-stack.png "LiteRT, XNNPACK, KleidiAI and SME2")
2020

21+
## How KleidiAI works in LiteRT
2122

22-
## Understand how KleidiAI works in LiteRT
23+
To understand how KleidiAI SME2 micro-kernels work in LiteRT, consider a LiteRT model with one Fully Connected operator using the FP32 data type.
2324

24-
To understand how KleidiAI SME2 micro-kernel works in LiteRT, a LiteRT model with one Fully Connected operator with FP32 datatype is used as an example.
25-
26-
The following illustrates the execution workflow of XNNPACK’s implementation compared with the workflow when KleidiAI SME2 is enabled in XNNPACK.
25+
The following diagrams illustrate the execution workflow of XNNPACK’s implementation compared with the workflow when KleidiAI SME2 is enabled in XNNPACK.
2726

2827
### LiteRT → XNNPACK workflow
2928

30-
![LiteRT, XNNPACK workflow#center](./litert-xnnpack-workflow.png "LiteRT, XNNPACK workflow")
31-
32-
A Fully Connected operator can be essentially implemented as a matrix multiplication.
29+
![Diagram showing the workflow for a Fully Connected operator in LiteRT using XNNPACK. The diagram depicts the flow from LiteRT to XNNPACK, highlighting the use of NEON instructions for matrix multiplication and weight packing on Arm platforms. The technical environment emphasizes operator traversal, hardware detection, and parallel computation. #center](./litert-xnnpack-workflow.png "LiteRT, XNNPACK workflow")
3330

34-
When LiteRT loads a model, it parses the operators and create a computation graph. If the CPU is selected as the accelerator, LiteRT uses XNNPACK by default.
31+
A Fully Connected operator is essentially implemented as a matrix multiplication.
3532

36-
XNNPACK traverses the operators in the graph and tries to replace them with its own implementations. During this stage, XNNPACK performs the necessary packing of the weight matrix. To speed up the packing process, XNNPACK uses NEON instructions for Arm platform. XNNPACK provides different implementations for different hardware platforms. At runtime, it detects the hardware capabilities and selects the appropriate micro-kernel.
33+
When LiteRT loads a model, it parses the operators and creates a computation graph. If the CPU is selected as the accelerator, LiteRT uses XNNPACK by default.
3734

38-
During model inference, XNNPACK performs matrix multiplication on the activation matrix (the left-hand side matrix, LHS) and the repacked weight matrix (the right-hand side matrix, RHS). In this stage, XNNPACK applies tiling strategies to the matrices and performs parallel multiplication across the resulting tiles using multiple threads. To accelerate the computation, XNNPACK uses NEON instructions.
35+
XNNPACK traverses the operators in the graph and tries to replace them with its own implementations. During this stage, XNNPACK packs the weight matrix, using NEON instructions for Arm platforms to speed up the process. XNNPACK provides different implementations for different hardware platforms. At runtime, it detects hardware capabilities and selects the appropriate micro-kernel.
3936

37+
During model inference, XNNPACK performs matrix multiplication on the activation matrix (left-hand side, LHS) and the repacked weight matrix (right-hand side, RHS). XNNPACK applies tiling strategies to the matrices and performs parallel multiplication across the resulting tiles using multiple threads. To accelerate computation, XNNPACK uses NEON instructions.
4038

4139
### LiteRT → XNNPACK → KleidiAI workflow
4240

43-
![LiteRT, XNNPACK, KleidiAI workflow#center](./litert-xnnpack-kleidiai-workflow.png "LiteRT, XNNPACK, KleidiAI workflow")
41+
![Diagram showing the workflow for a Fully Connected operator in LiteRT using XNNPACK and KleidiAI with SME2. The diagram illustrates the flow from LiteRT to XNNPACK, then to KleidiAI, highlighting SME2 micro-kernel integration for matrix multiplication and packing. The technical context emphasizes runtime hardware detection and optimized operator execution. alt-text #center](./litert-xnnpack-kleidiai-workflow.png "LiteRT, XNNPACK, KleidiAI workflow")
4442

45-
When KleidiAI and SME2 are enabled at building stage, the KleidiAI SME2 micro-kernels are compiled into the XNNPACK.
43+
When KleidiAI and SME2 are enabled at build time, the KleidiAI SME2 micro-kernels are compiled into XNNPACK.
4644

47-
During the model loading stage, when XNNPACK optimizes the subgraph, it checks the operator’s data type to determine whether a KleidiAI implementation is available. If KleidiAI supports it, XNNPACK bypasses its own default implementation. As a result, RHS packing is performed using the KleidiAI SME packing micro-kernel. In addition, because KleidiAI typically requires packing of the LHS, a flag is also set during this stage.
45+
During the model loading stage, when XNNPACK optimizes the subgraph, it checks the operator’s data type to determine whether a KleidiAI implementation is available. If KleidiAI supports it, XNNPACK bypasses its own default implementation. As a result, RHS packing is performed using the KleidiAI SME packing micro-kernel. Because KleidiAI typically requires packing of the LHS, a flag is also set during this stage.
4846

4947
During model inference, the LHS packing micro-kernel is invoked. After the LHS is packed, XNNPACK performs the matrix multiplication. At this point, the KleidiAI SME micro-kernel is used to compute the matrix product.

content/learning-paths/mobile-graphics-and-gaming/litert-sme/2-build-tool.md

Lines changed: 53 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -6,68 +6,80 @@ weight: 3
66
layout: learningpathall
77
---
88

9-
### Build the LiteRT benchmark tool with KleidiAI and SME2 enabled
9+
## Build the LiteRT benchmark tool with KleidiAI and SME2 enabled
1010

11-
LiteRT provides provides a standalone performance measurement utility called `benchmark_model` for evaluating the performance of LiteRT models.
11+
LiteRT provides a standalone performance measurement utility called `benchmark_model` for evaluating the performance of LiteRT models.
1212

1313
In this section, you will build two versions of the benchmark tool:
14-
* With KleidiAI + SME/SME2 enabled → uses Arm-optimized micro-kernels
15-
* Without KleidiAI + SME/SME2 → baseline performance (NEON/SVE2 fallback)
16-
This comparison clearly demonstrates the gains provided by SME2 acceleration.
14+
- With KleidiAI and Scalable Matrix Extension version 2 (SME2) enabled, which uses Arm-optimized micro-kernels
15+
- Without KleidiAI and SME2, which provides baseline performance using NEON or SVE2 fallback
1716

18-
First, clone the LiteRT repository.
17+
This comparison demonstrates the performance gains provided by SME2 acceleration.
1918

20-
``` bash
19+
First, clone the LiteRT repository:
20+
21+
```bash
2122
cd $WORKSPACE
2223
git clone https://github.com/google-ai-edge/LiteRT.git
2324
```
24-
Because LiteRT integrates KleidiAI through XNNPACK, you must build LiteRT from source to enable SME2 micro-kernels.
2525

26-
Next, set up your Android build environment using Docker on your Linux developement machine.
27-
Google provides a Dockerfile that installs the toolchain needed for TFLite/LiteRT Android builds.
26+
Because LiteRT integrates KleidiAI through XNNPACK (an open-source library providing highly optimized neural-network operators), you must build LiteRT from source to enable SME2 micro-kernels.
27+
28+
Next, set up your Android build environment using Docker on your Linux development machine. Google provides a Dockerfile that installs the toolchain needed for TensorFlow Lite (TFLite)/LiteRT Android builds.
2829

2930
Download the Dockerfile:
30-
``` bash
31+
32+
```bash
3133
wget https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/tools/tflite-android.Dockerfile
3234
```
35+
3336
Build the Docker image:
37+
3438
```bash
3539
docker build . -t tflite-builder -f tflite-android.Dockerfile
3640
```
37-
The Docker image includes Bazel, NDK, CMake, toolchains, and Python required for cross-compiling Android binaries.
3841

39-
You will now install Android SDK/NDK Components inside the Container.
40-
Launch the docker container:
42+
The Docker image includes Bazel, Android Native Development Kit (NDK), CMake, toolchains, and Python required for cross-compiling Android binaries.
43+
44+
Now, install Android Software Development Kit (SDK) and NDK components inside the container.
45+
46+
Launch the Docker container:
47+
4148
```bash
4249
docker run -it -v $PWD:/host_dir tflite-builder bash
4350
```
51+
4452
Install Android platform tools:
45-
``` bash
53+
54+
```bash
4655
sdkmanager \
4756
"build-tools;${ANDROID_BUILD_TOOLS_VERSION}" \
4857
"platform-tools" \
4958
"platforms;android-${ANDROID_API_LEVEL}"
5059
```
5160

52-
Configure LiteRT Build Options inside your running container:
61+
Configure LiteRT build options inside your running container:
5362

54-
``` bash
63+
```bash
5564
cd /host_dir/LiteRT
5665
./configure
5766
```
5867

59-
Use default values for all the prompts except when prompted:
68+
Use default values for all prompts except when asked:
69+
6070
```output
6171
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]
6272
```
63-
Type in `y`.
6473

65-
LiteRT's configuration script will detect SDK + NDK paths, set toolchain versions, configure Android ABI (arm64-v8a) and initialize Bazel workspace rules.
74+
Type `y` and press Enter.
6675

67-
Now, you can build the benchmark tool with KleidiAI + SME2 Enabled.
76+
LiteRT's configuration script will detect SDK and NDK paths, set toolchain versions, configure the Android Application Binary Interface (ABI) to arm64-v8a, and initialize Bazel workspace rules.
77+
78+
Now, you can build the benchmark tool with KleidiAI and SME2 enabled.
6879

6980
Enable XNNPACK, quantization paths, and SME2 acceleration:
70-
``` bash
81+
82+
```bash
7183
export BENCHMARK_TOOL_PATH="litert/tools:benchmark_model"
7284
export XNNPACK_OPTIONS="--define tflite_with_xnnpack=true \
7385
--define=tflite_with_xnnpack_qs8=true \
@@ -77,24 +89,30 @@ export XNNPACK_OPTIONS="--define tflite_with_xnnpack=true \
7789
--define=xnn_enable_arm_sme2=true \
7890
--define=xnn_enable_kleidiai=true"
7991
```
92+
8093
Build for Android:
94+
8195
```bash
8296
bazel build -c opt --config=android_arm64 \
8397
${XNNPACK_OPTIONS} "${BENCHMARK_TOOL_PATH}" \
8498
--repo_env=HERMETIC_PYTHON_VERSION=3.12
8599
```
86100

87-
This build enables the KleidiAI and SME2 micro-kernels integrated into XNNPACK and produces an Android binary under:
101+
This build enables the KleidiAI and SME2 micro-kernels integrated into XNNPACK and produces an Android binary at:
88102

89103
```output
90104
bazel-bin/litert/tools/benchmark_model
91105
```
92106

93-
### Build the LiteRT benchmark tool without KleidiAI (Baseline Comparison)
107+
---
108+
109+
## Build the LiteRT benchmark tool without KleidiAI (baseline comparison)
110+
111+
To compare the performance of the KleidiAI SME2 implementation against XNNPACK’s original implementation, build another version of the LiteRT benchmark tool without KleidiAI and SME2 enabled.
94112

95-
To compare the performance of KleidiAI SME2 implementation against XNNPACK’s original implementation, you can build another version of LiteRT benchmark tool without KleidiAI and SME2 enabled.
113+
Set the build options to disable SME2 and KleidiAI:
96114

97-
``` bash
115+
```bash
98116
export BENCHMARK_TOOL_PATH="litert/tools:benchmark_model"
99117
export XNNPACK_OPTIONS="--define tflite_with_xnnpack=true \
100118
--define=tflite_with_xnnpack_qs8=true \
@@ -106,15 +124,22 @@ export XNNPACK_OPTIONS="--define tflite_with_xnnpack=true \
106124
```
107125

108126
Then rebuild:
127+
109128
```bash
110129
bazel build -c opt --config=android_arm64 \
111130
${XNNPACK_OPTIONS} "${BENCHMARK_TOOL_PATH}" \
112131
--repo_env=HERMETIC_PYTHON_VERSION=3.12
113132
```
114-
This build of the `benchmark_model` disables all SME2 micro-kernels and forces fallback to XNNPACK’s NEON/SVE2 kernels.
115-
You can then use ADB to push the benchmark tool to your Android device.
133+
134+
This build of the `benchmark_model` disables all SME2 micro-kernels and forces fallback to XNNPACK’s NEON or SVE2 kernels.
135+
136+
You can then use Android Debug Bridge (ADB) to push the benchmark tool to your Android device:
116137

117138
```bash
118139
adb push bazel-bin/litert/tools/benchmark_model /data/local/tmp/
119140
adb shell chmod +x /data/local/tmp/benchmark_model
120141
```
142+
143+
---
144+
145+
You have now built both versions of the LiteRT benchmark tool. You are ready to benchmark and compare SME2-accelerated and baseline performance on your Arm-based Android device.

0 commit comments

Comments
 (0)