You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As you can see from the `Features`, the CPU 7 supports the SME2.
23
27
24
-
Then, you can run the `benchmark_model` tool on the CPU that supports the SME2. One of the example is as follows.
25
28
26
-
This example uses the `taskset` command to configure the benchmark workload to run on cores 7. It specifies to utilize 1 thread by the option `--num_threads=1`, and running the inferences at least 1000 times by the option `--num_runs=1000`. The CPU is selected. Also, it passes the option `--use_profiler=true` to produce a operator level profiling during inference.
29
+
## Run benchmark_model on an SME2 Core
30
+
31
+
Next, run the benchmark tool and bind execution to a core that supports SME2.
32
+
In this example, you will pin to CPU 7, use a single thread, and run enough iterations to get stable timing.
This example uses the `taskset` command to configure the benchmark workload to run on cores 7. It specifies to utilize 1 thread by the option `--num_threads=1`, and running the inferences at least 1000 times by the option `--num_runs=1000`. The CPU is selected. Also, it passes the option `--use_profiler=true` to produce a operator level profiling during inference.
30
39
40
+
You should see output similar to:
41
+
```output
31
42
...
32
43
INFO: [litert/runtime/accelerators/auto_registration.cc:148] CPU accelerator registered.
33
44
INFO: [litert/runtime/compiled_model.cc:415] Flatbuffer model initialized directly from incoming litert model.
34
-
INFO: Initialized TensorFlow Lite runtime.
45
+
INFO: Initialized TensorFlow Lite runtime.
35
46
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
36
47
VERBOSE: Replacing 1 out of 1 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 1 partitions for subgraph 0.
37
48
INFO: The input model file size (MB): 3.27774
@@ -90,7 +101,7 @@ Memory (bytes): count=0
90
101
5 nodes observed
91
102
```
92
103
93
-
As you can see from the results above, the results include the time spent on model initialization, warm-up, and inference, as well as memory usage. Since the profiler was enabled, the output also reports the execution time of each operator.
104
+
From the results above you will see the time spent on model initialization, warm-up, and inference, as well as memory usage. Since the profiler was enabled, the output also reports the execution time of each operator.
94
105
95
106
Because the model contains only a single fully connected layer, the node type `Fully Connected (NC, PF32) GEMM` shows the average execution time is 0.382 ms, accounting for 93.171% of the total inference time.
96
107
@@ -100,13 +111,15 @@ To verify the KleidiAI SME2 micro-kernels are invoked for the Fully Connected op
100
111
101
112
## Measure the performance impact of KleidiAI SME2 micro-kernels
102
113
103
-
To compare the performance of the KleidiAI SME2 implementation with the original XNNPACK implementation, you can run the `benchmark_model` tool without KleidiAI enabled and then benchmark again using the same parameters.
114
+
To compare the performance of the KleidiAI SME2 implementation with the original XNNPACK implementation, you can run the `benchmark_model` tool without KleidiAI enabled using the same parameters.
INFO: [litert/runtime/accelerators/auto_registration.cc:148] CPU accelerator registered.
112
125
INFO: [litert/runtime/compiled_model.cc:415] Flatbuffer model initialized directly from incoming litert model.
@@ -169,6 +182,9 @@ Memory (bytes): count=0
169
182
5 nodes observed
170
183
```
171
184
185
+
From these benchmarking results you should notice significant latency reduction and throughput increase when KleidiAI SME2 micro-kernels are enabled.
186
+
187
+
### Interpreting Node Type Names for KleidiAI
172
188
As you can see from the results, for the same model, the XNNPACK node type name is different. For the non-KleidiAI implementation, the node type is `Fully Connected (NC, F32) GEMM`, whereas for the KleidiAI implementation, it is `Fully Connected (NC, PF32) GEMM`.
173
189
174
190
For other operators supported by KleidiAI, the per-operator profiling node types differ between the implementations with and without KleidiAI enabled in XNNPACK as follows:
@@ -188,10 +204,10 @@ For other operators supported by KleidiAI, the per-operator profiling node types
As you can see from the list, the letter “P” indicates that the node type corresponds to a KleidiAI implementation.
207
+
The letter “P” in the node type indicates that it corresponds to a KleidiAI implementation.
192
208
193
209
For example, in `Convolution (NHWC, PQS8, QS8, QC8W)`, this represents a Conv2D operator computation by KleidiAI micro-kernel, where the tensor is in NHWC layout.
0 commit comments