You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/litert-sme/1-litert-kleidiai-sme2.md
+9-10Lines changed: 9 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ weight: 2
6
6
layout: learningpathall
7
7
---
8
8
9
-
## LiteRT, XNNPACK, KleidiAI, and SME2
9
+
## Understand the LiteRT software stack and SME2 integration
10
10
11
11
LiteRT (Lightweight Runtime, formerly TensorFlow Lite) is a runtime for on-device AI on Arm platforms. The default CPU acceleration library used by LiteRT is XNNPACK (an open-source library providing highly optimized neural-network operators).
12
12
@@ -16,27 +16,26 @@ KleidiAI is a library developed by Arm that offers performance-critical micro-ke
16
16
17
17
The software stack for LiteRT is shown below.
18
18
19
-

19
+

20
20
21
21
## How KleidiAI works in LiteRT
22
22
23
23
To understand how KleidiAI SME2 micro-kernels work in LiteRT, consider a LiteRT model with one Fully Connected operator using the FP32 data type.
24
24
25
25
The following diagrams illustrate the execution workflow of XNNPACK’s implementation compared with the workflow when KleidiAI SME2 is enabled in XNNPACK.
26
26
27
-
###LiteRT → XNNPACK workflow
27
+
## LiteRT → XNNPACK workflow
28
28
29
-

29
+

30
+
A Fully Connected operator works by multiplying two matrices together.
30
31
31
-
A Fully Connected operator is essentially implemented as a matrix multiplication.
32
+
When LiteRT loads a model, it reads the operators and builds a computation graph. If you choose the CPU as the accelerator, LiteRT uses XNNPACK by default.
32
33
33
-
When LiteRT loads a model, it parses the operators and creates a computation graph. If the CPU is selected as the accelerator, LiteRT uses XNNPACK by default.
34
+
XNNPACK scans the computation graph and tries to replace operators with its own optimized versions. At this stage, XNNPACK packs the weight matrix. On Arm platforms, it uses NEON instructions to make this process faster. XNNPACK includes different implementations for different hardware. At runtime, it checks the hardware and picks the best micro-kernel.
34
35
35
-
XNNPACK traverses the operators in the graph and tries to replace them with its own implementations. During this stage, XNNPACK packs the weight matrix, using NEON instructions for Arm platforms to speed up the process. XNNPACK provides different implementations for different hardware platforms. At runtime, it detects hardware capabilities and selects the appropriate micro-kernel.
36
+
During inference, XNNPACK multiplies the activation matrix (left-hand side, LHS) by the packed weight matrix (right-hand side, RHS). It splits the matrices into smaller tiles and runs the multiplications in parallel using multiple threads. NEON instructions help speed up these calculations.
36
37
37
-
During model inference, XNNPACK performs matrix multiplication on the activation matrix (left-hand side, LHS) and the repacked weight matrix (right-hand side, RHS). XNNPACK applies tiling strategies to the matrices and performs parallel multiplication across the resulting tiles using multiple threads. To accelerate computation, XNNPACK uses NEON instructions.
38
-
39
-
### LiteRT → XNNPACK → KleidiAI workflow
38
+
## LiteRT → XNNPACK → KleidiAI workflow
40
39
41
40

Copy file name to clipboardExpand all lines: content/learning-paths/mobile-graphics-and-gaming/litert-sme/2-build-tool.md
-4Lines changed: 0 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -104,8 +104,6 @@ This build enables the KleidiAI and SME2 micro-kernels integrated into XNNPACK a
104
104
bazel-bin/litert/tools/benchmark_model
105
105
```
106
106
107
-
---
108
-
109
107
## Build the LiteRT benchmark tool without KleidiAI (baseline comparison)
110
108
111
109
To compare the performance of the KleidiAI SME2 implementation against XNNPACK’s original implementation, build another version of the LiteRT benchmark tool without KleidiAI and SME2 enabled.
You have now built both versions of the LiteRT benchmark tool. You are ready to benchmark and compare SME2-accelerated and baseline performance on your Arm-based Android device.
0 commit comments