Merge pull request #2714 from Arnaud-de-Grandmaison-ARM/performance_onnxruntime_kleidiai_sme2

Some minor tweaks after a first technical review.

Author: jasonrandrews

content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/build_ort.md

@@ -9,48 +9,58 @@ layout: learningpathall
 ## Build ONNX Runtime and benchmark application with KleidiAI and SME2 support for Android
 
 To run this on an Android device, you must cross-compile ORT using the Android NDK.
-Prerequisites
+
+Prerequisites:
 -	Android NDK: Version r26b or newer (r27+ recommended for latest SME2 toolchain support).
 -	CMake & Ninja: Ensure these are in your system PATH.
 
-### Build Command
-Run the following from the root of the ONNX Runtime repository:
+### Build `onnxruntime`
+
+First, clone the [ONNX Runtime](https://github.com/microsoft/onnxruntime).
+
+Then run the following from the root of the ONNX Runtime repository:
 ```bash
-./build.sh --android     --android_sdk_path $ANDROID_NDK_HOME     --android_ndk_path $ANDROID_NDK_HOME     --android_abi arm64-v8a     --android_api 27    --config RelWithDebInfo --build_shared_lib     --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON     --cmake_generator Ninja  --parallel 
+./build.sh --android --android_sdk_path $ANDROID_NDK_HOME --android_ndk_path $ANDROID_NDK_HOME --android_abi arm64-v8a --android_api 27 --config RelWithDebInfo --build_shared_lib --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON --cmake_generator Ninja --parallel
 ```
 
-Note: The flag “onnxruntime_USE_KLEIDIAI=ON” triggers the inclusion of Arm KleidiAI kernels into the MLAS library.
+Note: The flag `onnxruntime_USE_KLEIDIAI=ON` triggers the inclusion of Arm KleidiAI kernels into the MLAS library.
 
 ## Profiling Performance with onnxruntime_perf_test
-Once the build is complete, you will find the libonnxruntime.so shared library and onnxruntime_perf_test binary in your build directory. The onnxruntime_perf_test is essential for measuring latency and identifying bottlenecks.
+Once the build is complete, you will find the `libonnxruntime.so` shared library and `onnxruntime_perf_test` binary in your build directory.
+
+`onnxruntime_perf_test` is essential for measuring latency and identifying bottlenecks of an ONNX model (named `<your_model>.onnx` hereafter).
+
 ### Step 1: Push files to Android Device
 ```bash
 adb push <build_dir>/Android/RelWithDebInfo/onnxruntime_perf_test /data/local/tmp/
 adb push <build_dir>/Android/RelWithDebInfo/libonnxruntime.so  /data/local/tmp/
-adb push your_model.onnx /data/local/tmp/
+adb push <your_model>.onnx /data/local/tmp/
 ```
+
 ### Step 2: Run the Performance Test
-The perf_test tool allows you to simulate inference and gather statistics. For example,
+The `onnxruntime_perf_test` tool allows you to simulate inference and gather statistics. For example,
 ```bash
 # Execute on the device
-adb shell "/data/local/tmp/onnxruntime_perf_test -e cpu -m times  -r 20 -s -Z  -x 1 /data/local/tmp/your_model.onnx"
+adb shell "/data/local/tmp/onnxruntime_perf_test -e cpu -m times  -r 20 -s -Z  -x 1 /data/local/tmp/<your_model>.onnx"
 ```
+
 The command example set the arguments of the application as,
--	“-e cpu” specifies the provider as cpu provider
--	“-m times” specifies the test mode as “times”
--	“-r 20” specifies the repeated times as 20
--	“-Z” disallows thread from spinning during runs to reduce cpu usage
--	“-s” shows statistics result
--	“-x 1” sets the number of threads used to parallelize the execution within nodes as 1
+-	`-e cpu` specifies the provider as cpu provider
+-	`-m times` specifies the test mode as “times”
+-	`-r 20` specifies the repeated times as 20
+-	`-Z` disallows thread from spinning during runs to reduce cpu usage
+-	`-s` shows statistics result
+-	`-x 1` sets the number of threads used to parallelize the execution within nodes as 1
 
 You can try other arguments setting if you would like to.
 
 ### Step 3: Deep Dive into Operator Profiling
-To see exactly how many milliseconds are spent on each operator, use the profiling flag -p.
+To see exactly how many milliseconds are spent on each operator, use the profiling flag `-p`.
 ```bash
-adb shell "/data/local/tmp/onnxruntime_perf_test -p profile.json -e cpu -m times  -r 5 -s -Z  -x 1 /data/local/tmp/your_model.onnx"
+adb shell "/data/local/tmp/onnxruntime_perf_test -p profile.json -e cpu -m times  -r 5 -s -Z  -x 1 /data/local/tmp/<your_model>.onnx"
 adb pull /data/local/tmp/profile.json
 ```
-The argument “-p” enables performance profiling during the benchmark run. When you provide this flag followed by a filename, ONNX Runtime will generate a JSON file containing a detailed trace of the model execution.
-You can view the results by opening [prefetto tool]( https://ui.perfetto.dev/), and loading the generated JSON file. This allows you to see a visual timeline of which operations took the most time.
-You also can convert the JSON file to a CSV sheet by creating a python script.
+
+The argument `-p` enables performance profiling during the benchmark run. When you provide this flag followed by a filename, ONNX Runtime will generate a JSON file containing a detailed trace of the model execution.
+You can view the results by opening [perfetto tool]( https://ui.perfetto.dev/), and loading the generated JSON file. This allows you to see a visual timeline of which operations took the most time.
+You also can convert the JSON file to a CSV sheet by creating a python script.

content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/kleidiai_integration.md

@@ -11,10 +11,10 @@ ONNX runtime is built with KleidiAI support:
 1.	Detection: At runtime, MLAS checks the CPU capabilities for SME2 support.
 2.	Dispatch: If SME2 is detected, MLAS overrides its default kernels. For example, a Gemm (General Matrix Multiplication) operation that would normally use standard vector instructions (such as NEON) is dispatched to a KleidiAI SME2 micro-kernel.
 
-Currently, KleidiAI in MLAS provides ArmKleidiAI::MlasConv, ArmKleidiAI::MlasGemmBatch and ArmKleidiAI::MlasDynamicQGemmBatch kernels.
+Currently, KleidiAI in MLAS provides `ArmKleidiAI::MlasConv`, `ArmKleidiAI::MlasGemmBatch` and `ArmKleidiAI::MlasDynamicQGemmBatch` kernels.
 
 ### The ArmKleidiAI::MlasConv kernel  
-Usually, 2D fp32 convolution operators with batch_size=1 and multiple filters (filter kernel is equal or greater than (3,3)) are dispatched to the ArmKleidiAI::MlasConv kernel. 
+Usually, 2D fp32 convolution operators with batch_size=1 and multiple filters (filter kernel is equal or greater than (3,3)) are dispatched to the `ArmKleidiAI::MlasConv` kernel. 
 
 For example, the figure below shows a (7,7) Conv node.
 
@@ -46,7 +46,7 @@ onnxruntime::InferenceSession::Run
 ```
 
 ### The ArmKleidiAI::MlasGemmBatch kernel
-It performs a batched fp32 matrix multiplication (GEMM or GemV) operation using KleidiAI matmul micro kernels. fp32 Conv operators with (1,1) filter kernels also use this kernel. 
+It performs a batched fp32 matrix multiplication (GEMM or GemV) operation using KleidiAI matmul micro kernels. fp32 Conv operators with (1,1) filter kernels also uses this kernel. 
 
 For example, the figure below shows a (1,1) Conv node.
 
@@ -101,5 +101,5 @@ onnxruntime::InferenceSession::Run
 ```
 
 ### The ArmKleidiAI::MlasDynamicQGemmBatch kernel
-This kernel is for Matmul with float output of dynamic quantized A and symmetric quantized B. 
-It uses KleidiAI *kai_kernel_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa* micro kernel.
+This kernel is for matrix multiplication with float output of dynamic quantized A and symmetric quantized B. 
+It uses KleidiAI *kai_kernel_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa* micro kernel.

content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example.md

@@ -6,22 +6,22 @@ weight: 5
 layout: learningpathall
 ---
 
-## Profile an ONNX model – Use Resnet50v2 as an example 
+## Profile an ONNX model – Using Resnet50v2 as an example 
 The Resnet50v2 fp32 ONNX model can be downloaded from Hugging Face or Modescope.
 
-The Android device that we used is a VIVO X300 phone with MTK D9500 processor, which has Arm C1-Ultra, C1-Premium and C1-Pro CPU cores with SME2 support on it. We chose a C1-Pro CPU core running at 2.0GHz to run the onnxruntime_perf_test benchmark application. You can use any other Android device with SME2 support.
+The Android device that we used is a VIVO X300 phone with MTK D9500 processor, which has Arm C1-Ultra, C1-Premium and C1-Pro CPU cores with SME2 support on it. We chose a C1-Pro CPU core running at 2.0GHz to run the `onnxruntime_perf_test` benchmark application. You can use any other Android device with SME2 support.
 
-To compare the performance of running Resnet50v2 on ORT with SME2 and without SME2 support, we built two versions of ORT, one with SME2 support (set *onnxruntime_USE_KLEIDIAI=ON* when building ORT), the other without SME2 support(*onnxruntime_USE_KLEIDIAI=OFF* when building ORT).
+To compare the performance of running Resnet50v2 on ORT with SME2 and without SME2 support, we built two versions of ORT, one with SME2 support (set `onnxruntime_USE_KLEIDIAI=ON` when building ORT), the other without SME2 support(`onnxruntime_USE_KLEIDIAI=OFF` when building ORT).
 
 Run following command on the device,
 ```bash
-taskset 1 ./onnxruntime_perf_test  -e cpu -r 5 -m times -s -Z  -x 1   ./resnet50v2.onnx  -p "resnet50v2.onnx_1xC1-Pro_profile
+taskset 1 ./onnxruntime_perf_test -e cpu -r 5 -m times -s -Z  -x 1 ./resnet50v2.onnx  -p resnet50v2.onnx_1xC1-Pro_profile
 ```
 
-The *taskset 1* in the command sets the CPU affinity of *onnxruntime_perf_test* benchmark to CPU core 0, which is a C1-Pro CPU core. 
-*-x 1* in the command sets the number of threads used to parallelize the execution within nodes as 1 (single thread).
+The `taskset 1` in the command sets the CPU affinity of `onnxruntime_perf_test` benchmark to CPU core 0, which is a C1-Pro CPU core. 
+`-x 1` in the command sets the number of threads used to parallelize the execution within nodes as 1 (single thread).
 
-Here is output from running onnxruntime_perf_test with ORT with SME2 support as below.
+Here is output from running `onnxruntime_perf_test` with ORT with SME2 support as below.
 ```text
 Setting intra_op_num_threads to 1
 Disabling intra-op thread spinning between runs
@@ -46,7 +46,7 @@ P99 Latency: 0.101519 s
 P999 Latency: 0.101519 s
 ``` 
 
-Here is output from running onnxruntime_perf_test with ORT without SME2 support as below.
+Here is output from running `onnxruntime_perf_test` with ORT without SME2 support as below.
 ```text
 Setting intra_op_num_threads to 1
 Disabling intra-op thread spinning between runs
@@ -70,12 +70,14 @@ P95 Latency: 0.34682 s
 P99 Latency: 0.34682 s
 P999 Latency: 0.34682 s
 ```
-### Performance 
+## Performance analysis
 
-We can use [prefetto tool](https://ui.perfetto.dev/), to view the two JSON profile files.
+### Using [perfetto tool](https://ui.perfetto.dev/)
+
+We can use [perfetto tool](https://ui.perfetto.dev/) to view the two JSON profile files.
 
 The figure below is a screenshot of the view of the Non-KleidiAI version of JSON profile file.  
-The selected part(one model_run/SequentialExecutor) in the figure includes information of one inference execution.
+The selected part(one `model_run/SequentialExecutor`) in the figure includes information of one inference execution.
 
 ![Figure showing profile file of Non-KleidiAI version alt-text#center](images/resnet50v2_no_sme_prefetto.png "prefetto view of Non-KleidiAI version of ORT")
 
@@ -88,6 +90,8 @@ We also convert the two JSON profile files to CSV sheets, then we combine the in
 
 It shows that ORT with KleidiAI (with SME2) kernels uplifts the performance significantly, especially for convolution operators.
 
+### Using Arm Streamline
+
 If we use Arm Streamline tools and PMU counters for further investigation, in the timeline view of Streamline, we can see SME2 floating point Outer Product and Accumulate (MOPA) instruction is used intensively during the inference.
 
 ![Figure showing SME2 instructions and cycles alt-text#center](images/resnet50v2_sme_onnx_streamline_1xgelas_annotation.png "SME2 instructions and cycles shown in Streamline")
@@ -96,7 +100,7 @@ Then we combine the function call view of ORT without KleidiAI and with KleidiAI
 
 ![Figure showing function call percentage of both versions of ORT alt-text#center](images/function_call_compare.png "Function call percentage of both versions of ORT in Streamline ")
 
-It shows that KleidiAI kernels provide a significant performance uplift for convolution operators compared to the default MLSA kernels (*MlasSgemmKernelAdd* and *MlasSgemmKernelZero*).
+It shows that KleidiAI kernels provide a significant performance uplift for convolution operators compared to the default MLSA kernels (`MlasSgemmKernelAdd` and `MlasSgemmKernelZero`).
 
 ## Summary
-By integrating KleidiAI (SME2) into ONNX Runtime, you unlock the massive parallel processing power of Arm SME2. This turns the Arm CPU from a "fallback" into a high-performance AI engine capable of running LLMs and complex vision models locally on devices.
+By enabling KleidiAI (SME2) into ONNX Runtime, you unlock the massive parallel processing power of Arm SME2. This turns the Arm CPU from a "fallback" into a high-performance AI engine capable of running LLMs and complex vision models locally on devices.