Skip to content

Commit 28becd5

Browse files
authored
Update 4-benchmark.md
1 parent 2bcea79 commit 28becd5

1 file changed

Lines changed: 32 additions & 16 deletions

File tree

  • content/learning-paths/mobile-graphics-and-gaming/litert-sme

content/learning-paths/mobile-graphics-and-gaming/litert-sme/4-benchmark.md

Lines changed: 32 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -5,33 +5,44 @@ weight: 4
55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
8-
9-
### Use the benchmark tool
10-
11-
First, you should check if your Android phone supports SME2. You can check it by the following command.
12-
8+
Once you have:
9+
* A LiteRT model (for example, fc_fp32.tflite), and
10+
* The benchmark_model binary built with and without KleidiAI + SME2,
11+
you can run benchmarks directly on an SME2-capable Android device.
12+
### Verify SME2 Support on the Device
13+
14+
First, you should check if your Android phone supports SME2.
15+
On the device (via adb shell), run:
1316
``` bash
1417
cat /proc/cpuinfo
15-
18+
```
19+
You should see a Features line similar to:
20+
```output
1621
...
1722
processor : 7
1823
BogoMIPS : 2000.00
1924
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti mte ecv afp mte3 sme smei8i32 smef16f32 smeb16f32 smef32f32 wfxt rprfm sme2 smei16i32 smebi32i32 hbc lrcpc3
2025
```
21-
2226
As you can see from the `Features`, the CPU 7 supports the SME2.
2327

24-
Then, you can run the `benchmark_model` tool on the CPU that supports the SME2. One of the example is as follows.
2528

26-
This example uses the `taskset` command to configure the benchmark workload to run on cores 7. It specifies to utilize 1 thread by the option `--num_threads=1`, and running the inferences at least 1000 times by the option `--num_runs=1000`. The CPU is selected. Also, it passes the option `--use_profiler=true` to produce a operator level profiling during inference.
29+
## Run benchmark_model on an SME2 Core
30+
31+
Next, run the benchmark tool and bind execution to a core that supports SME2.
32+
In this example, you will pin to CPU 7, use a single thread, and run enough iterations to get stable timing.
2733

2834
``` bash
2935
taskset 80 ./benchmark_model --graph=./fc_fp32.tflite --num_runs=1000 --num_threads=1 --use_cpu=true --use_profiler=true
36+
```
37+
38+
This example uses the `taskset` command to configure the benchmark workload to run on cores 7. It specifies to utilize 1 thread by the option `--num_threads=1`, and running the inferences at least 1000 times by the option `--num_runs=1000`. The CPU is selected. Also, it passes the option `--use_profiler=true` to produce a operator level profiling during inference.
3039

40+
You should see output similar to:
41+
```output
3142
...
3243
INFO: [litert/runtime/accelerators/auto_registration.cc:148] CPU accelerator registered.
3344
INFO: [litert/runtime/compiled_model.cc:415] Flatbuffer model initialized directly from incoming litert model.
34-
INFO: Initialized TensorFlow Lite runtime.
45+
INFO: Initialized TensorFlow Lite runtime.
3546
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
3647
VERBOSE: Replacing 1 out of 1 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 1 partitions for subgraph 0.
3748
INFO: The input model file size (MB): 3.27774
@@ -90,7 +101,7 @@ Memory (bytes): count=0
90101
5 nodes observed
91102
```
92103

93-
As you can see from the results above, the results include the time spent on model initialization, warm-up, and inference, as well as memory usage. Since the profiler was enabled, the output also reports the execution time of each operator.
104+
From the results above you will see the time spent on model initialization, warm-up, and inference, as well as memory usage. Since the profiler was enabled, the output also reports the execution time of each operator.
94105

95106
Because the model contains only a single fully connected layer, the node type `Fully Connected (NC, PF32) GEMM` shows the average execution time is 0.382 ms, accounting for 93.171% of the total inference time.
96107

@@ -100,13 +111,15 @@ To verify the KleidiAI SME2 micro-kernels are invoked for the Fully Connected op
100111

101112
## Measure the performance impact of KleidiAI SME2 micro-kernels
102113

103-
To compare the performance of the KleidiAI SME2 implementation with the original XNNPACK implementation, you can run the `benchmark_model` tool without KleidiAI enabled and then benchmark again using the same parameters.
114+
To compare the performance of the KleidiAI SME2 implementation with the original XNNPACK implementation, you can run the `benchmark_model` tool without KleidiAI enabled using the same parameters.
104115

105-
One example is as follows.
116+
Run with the same parameters:
106117

107118
``` bash
108119
taskset 80 ./benchmark_model --graph=./fc_fp32.tflite --num_runs=1000 --num_threads=1 --use_cpu=true --use_profiler=true
109-
120+
```
121+
The output should look like:
122+
```output
110123
...
111124
INFO: [litert/runtime/accelerators/auto_registration.cc:148] CPU accelerator registered.
112125
INFO: [litert/runtime/compiled_model.cc:415] Flatbuffer model initialized directly from incoming litert model.
@@ -169,6 +182,9 @@ Memory (bytes): count=0
169182
5 nodes observed
170183
```
171184

185+
From these benchmarking results you should notice significant latency reduction and throughput increase when KleidiAI SME2 micro-kernels are enabled.
186+
187+
### Interpreting Node Type Names for KleidiAI
172188
As you can see from the results, for the same model, the XNNPACK node type name is different. For the non-KleidiAI implementation, the node type is `Fully Connected (NC, F32) GEMM`, whereas for the KleidiAI implementation, it is `Fully Connected (NC, PF32) GEMM`.
173189

174190
For other operators supported by KleidiAI, the per-operator profiling node types differ between the implementations with and without KleidiAI enabled in XNNPACK as follows:
@@ -188,10 +204,10 @@ For other operators supported by KleidiAI, the per-operator profiling node types
188204
| Conv2D | Convolution (NHWC, PQS8, QS8, QC8W) | Convolution (NHWC, QC8) |
189205
| TransposeConv | Deconvolution (NHWC, PQS8, QS8, QC8W) | Deconvolution (NC, QS8, QC8W) |
190206

191-
As you can see from the list, the letter “P” indicates that the node type corresponds to a KleidiAI implementation.
207+
The letter “P” in the node type indicates that it corresponds to a KleidiAI implementation.
192208

193209
For example, in `Convolution (NHWC, PQS8, QS8, QC8W)`, this represents a Conv2D operator computation by KleidiAI micro-kernel, where the tensor is in NHWC layout.
194210

195211
* The input is packed INT8 quantized,
196212
* The weights are per-channel INT8 quantized,
197-
* The output is INT8 quantized.
213+
* The output is INT8 quantized.

0 commit comments

Comments
 (0)