You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Identify performance bottlenecks in ExecuTorch models
8
8
9
-
This learning path provides a hands-on, reproducible workflow for analyzing ExecuTorch model performance on Arm-based devices and identifying optimization opportunities after enabling SME2 acceleration.
9
+
This Learning Path provides a hands-on, reproducible workflow for analyzing ExecuTorch model performance on Arm-based devices and identifying optimization opportunities after enabling SME2 acceleration.
10
10
11
-
When SME2 acceleration is enabled, inference latency often improves significantly. Just as importantly, faster compute exposes how execution time is distributed across the rest of the model. Model Inference time is typically spent in several broad operator categories:
12
-
* Matrix compute (for example, convolution and GEMM)
* Data movement (transpose, reshape, layout conversion, memory copies)
15
-
In many models, matrix compute dominates latency, making it the primary bottleneck.
11
+
When SME2 acceleration is enabled, inference latency often improves significantly. More importantly, faster compute exposes how execution time is distributed across the rest of the model.
16
12
17
-
SME2 accelerates CONV and GEMM operations,often by 3–15x,removing the primary compute bottleneck. Once compute is faster, data movement costs become visible and may emerge as the next dominant contributor to latency.
13
+
Model inference time is typically spent in several broad operator categories: matrix compute (convolution and GEMM), non-linear operations (elementwise activations, normalization), and data movement (transpose, reshape, layout conversion, memory copies). In many models, matrix compute dominates latency, making it the primary bottleneck.
18
14
19
-
Key idea:
20
-
End-to-end latency alone tells you that a model is faster, but not why or where time is still spent. Operator-level profiling reveals how execution time shifts across categories when SME2 is enabled, making it clear which operations should be optimized next.
15
+
SME2 accelerates CONV and GEMM operations, often by 3× to 15×, removing the primary compute bottleneck. Once compute is faster, data movement costs become visible and can emerge as the next dominant contributor to latency.
21
16
22
-
## 1. What You Will Build
17
+
End-to-end latency alone tells you that a model is faster, but not why or where time is still spent. Operator-level profiling reveals how execution time shifts across categories when SME2 is enabled, making it clear which operations to optimize next.
23
18
24
-
You will construct a model-agnostic performance analysis pipeline for ExecuTorch models running on Arm-based devices:
19
+
## Build a model-agnostic performance analysis pipeline
25
20
26
-
1. Export any PyTorch model to ExecuTorch `.pte` format
27
-
2. Run the same model with SME2 enabled and disabled for an apples-to-apples comparison
28
-
3. Collect ETDump traces with operator-level timing
29
-
4. Aggregate operators into high-level categories (CONV, GEMM, Data Movement, Elementwise, Other)
30
-
5. Identify where bottlenecks move after SME2 acceleration
21
+
In this Learning Path, you will construct a model-agnostic performance analysis pipeline for ExecuTorch models running on Arm-based devices:
31
22
32
-
Key principle: Once you have a .pte file, the same pipeline and commands apply to any model. Only the export step is model-specific.
23
+
- Export any PyTorch model to ExecuTorch `.pte` format
24
+
- Run the same model with SME2 enabled and disabled for an apples-to-apples comparison
25
+
- Collect ETDump traces with operator-level timing
26
+
- Aggregate operators into high-level categories (CONV, GEMM, Data Movement, Elementwise, Other)
27
+
- Identify where bottlenecks move after SME2 acceleration
33
28
34
-
## 2. Clone the Code Repository
29
+
Once you have a .pte file, the same pipeline and commands apply to any model. Only the export step is model-specific.
35
30
36
-
All profiling and analysis steps in this Learning Path are performed using a single, shared code repository. This repository contains the scripts, configuration, and example models used to export ExecuTorch models, run profiling with SME2 enabled and disabled, and analyze the resulting performance data.
37
-
The repository you will use throughout this Learning Path is [sme-executorch-profiling](https://github.com/ArmDeveloperEcosystem/sme-executorch-profiling). The repository includes:
38
-
* Example models (EdgeTAM image segmentation and a video-focused segmentation model)
39
-
* Predefined ExecuTorch runners
40
-
* Scripts for profiling, trace collection, and analysis
31
+
## Clone the profiling repository
32
+
33
+
All profiling and analysis steps in this Learning Path use a single, shared code repository. This repository contains the scripts, configuration, and example models for exporting ExecuTorch models, running profiling with SME2 enabled and disabled, and analyzing the resulting performance data.
34
+
35
+
You will use the [sme-executorch-profiling repository](https://github.com/ArmDeveloperEcosystem/sme-executorch-profiling) throughout this Learning Path. The repository includes an example model (small convolutional neural network designed for testing and validation purposes), predefined ExecuTorch runners, and scripts for profiling, trace collection, and analysis.
41
36
42
37
Clone the performance analysis kit repository:
43
38
@@ -50,62 +45,68 @@ cd executorch_sme2_kit
50
45
51
46
This creates a self-contained workspace. Your Python virtual environment, ExecuTorch build outputs, models, and profiling runs will all live under this directory.
52
47
53
-
## 3. Execution Stack Overview: ExecuTorch, XNNPACK, Arm KleidiAI, and SME2
48
+
## Understand the ExecuTorch execution stack with XNNPACK and KleidiAI
49
+
50
+
Before running the pipeline, it helps to understand how the execution stack is composed. The profiling results you generate reflect behavior across multiple layers. The diagram below summarizes the CPU execution stack used in this workflow.
51
+
![Execution stack diagram showing PyTorch model export to ExecuTorch with XNNPACK backend delegation and KleidiAI kernels using SME2 acceleration#center](images/sme2_stack_01062026.png"The execution stack: a model is defined in PyTorch, exported and run by ExecuTorch, and CPU compute is delegated to XNNPACK as the backend.")
54
52
55
-
Before running the pipeline, it helps to understand how the execution stack is composed, since the profiling results reflect behavior across multiple layers. The diagram below summarizes the CPU execution stack used in this workflow.
56
-
![pipeline-execution #center](images/sme2_stack_01062026.png"The execution stack: A model is defined in PyTorch, exported and run by ExecuTorch, and CPU compute is delegated to XNNPACK as the backend.")
53
+
### PyTorch to ExecuTorch export
57
54
58
-
**PyTorch to ExecuTorch export**: Models are defined using standard PyTorch APIs and exported to a .pte (Portable ExecuTorch Executable) format. During export, backend delegation is specified, in this case XNNPACK, indicating which operators should be executed by the backend at runtime.
55
+
Models are defined using standard PyTorch APIs and exported to a `.pte` (Portable ExecuTorch Executable) format. During export, you specify backend delegation—in this case, XNNPACK—which determines which operators the backend executes at runtime.
59
56
60
-
**ExecuTorch runtime and delegation**: At runtime, ExecuTorch schedules operators and delegates supported operations (such as Conv2d and Linear) to XNNPACK. XNNPACK, in turn, uses Arm KleidiAI kernels, which exploit SME2 acceleration on supported hardware. This delegation is transparent to the model author.
57
+
### ExecuTorch runtime and delegation
61
58
62
-
**Why operator-level profiling matters**: ExecuTorch's ETDump captures timing for each operator in the execution graph. This makes backend behavior visible, which operators are delegated, which kernels are used, and how much time each operation consumes. Aggregating operators into categories allows you to see where SME2 delivers gains and where non-compute costs dominate.
59
+
At runtime, ExecuTorch schedules operators and delegates supported operations (such as `Conv2d` and `Linear`) to XNNPACK. XNNPACK uses Arm KleidiAI kernels, which use SME2 acceleration on supported hardware. The delegation is transparent to you as the model author.
63
60
64
-
##4. Quickstart: Run the Pipeline
61
+
### Why operator-level profiling matters
65
62
66
-
This learning path supports profiling on both:
67
-
* Android – representative of real-world edge ML deployment
68
-
* macOS (Apple Silicon) – convenient for developer learning and experimentation
69
-
70
-
The workflow is identical on both platforms; only the runner binaries differ.
63
+
ExecuTorch's ETDump captures timing for each operator in the execution graph. This makes backend behavior visible: which operators are delegated, which kernels are used, and how much time each operation consumes. Aggregating operators into categories shows where SME2 delivers gains and where non-compute costs dominate.
71
64
72
-
Android runs provide the most representative performance results because they reflect real device constraints such as memory bandwidth, thermal behavior, and platform-specific scheduling. macOS is included to make it easier to learn the workflow and validate the pipeline before running on target devices.
65
+
## Run the profiling pipeline
66
+
67
+
This Learning Path supports profiling on both Android and macOS (Apple Silicon). Android represents real-world edge ML deployment, while macOS provides convenient developer learning and experimentation. The workflow is identical on both platforms; only the runner binaries differ.
68
+
69
+
Android runs provide the most representative performance results because they reflect real device constraints such as memory bandwidth, thermal behavior, and platform-specific scheduling. macOS is included to simplify learning the workflow and validating the pipeline before running on target devices.
73
70
74
71
The steps below walk through the full workflow end to end: setting up the environment, building ExecuTorch runners, running a profiling pass, and generating analysis artifacts.
75
72
76
-
Step 1: Set up the environment and ExecuTorch
73
+
###Set up the environment and ExecuTorch
77
74
This step creates a Python virtual environment, clones and builds ExecuTorch, and installs all required dependencies.
78
75
79
76
```bash
80
77
bash model_profiling/scripts/setup_repo.sh
81
78
```
82
-
Step 2: Build SME2-enabled and SME2-disabled runners
83
-
Next, build the ExecuTorch runner binaries used for profiling. Both SME2-enabled and SME2-disabled runners are built so that you can perform a direct comparison later.
79
+
### Build SME2-enabled and SME2-disabled runners
80
+
81
+
Build the ExecuTorch runner binaries used for profiling. You need both SME2-enabled and SME2-disabled runners to compare performance directly.
82
+
84
83
On macOS, the runners are built automatically.
85
-
On Android, this step requires the ANDROID_NDK environment variable to be set and a compatible NDK installed.
84
+
85
+
On Android, set the `ANDROID_NDK` environment variable and install a compatible NDK before running this step.
86
86
87
87
```bash
88
88
bash model_profiling/scripts/build_runners.sh
89
89
```
90
+
### Run a smoke test end to end
91
+
92
+
This step runs a complete smoke test of the pipeline. It exports a model, runs it through ExecuTorch, collects ETDump traces, and confirms that profiling data is generated correctly.
90
93
91
-
Step 3: Run a smoke test end to end
92
-
This step performs a complete smoke test of the pipeline: exporting a model, running it through ExecuTorch, collecting ETDump traces, and validating that profiling data is generated correctly.
93
-
On macOS, the test runs locally.
94
-
On Android, a compatible device must be connected via adb.
94
+
On macOS, the test runs locally. On Android, connect a compatible device via adb before running this command.
95
95
96
96
```bash
97
97
python model_profiling/scripts/run_quick_test.py
98
98
```
99
+
### Inspect and analyze results
99
100
100
-
Step 4: Inspect and analyze results
101
-
By default, the pipeline automatically analyzes the collected traces and generates CSV and JSON summaries. You do not need to run analysis manually unless you want to repeat or customize it.
102
-
If needed, you can rerun the analysis script directly and point it at a specific run directory.
101
+
The pipeline automatically analyzes collected traces and generates CSV and JSON summaries. You don't need to run analysis manually.
102
+
103
+
To rerun analysis or customize it, point the analysis script at a specific run directory:
After this step completes, you will have operator-level timing data and aggregated operator-category breakdowns for SME2-on and SME2-off runs.
107
+
After this step completes, you have operator-level timing data and aggregated operator-category breakdowns for SME2-on and SME2-off runs.
107
108
108
-
## 5. What You Will Produce: Artifacts
109
+
## Review ETDump traces and profiling artifacts
109
110
110
111
Running the pipeline generates a consistent set of artifacts that capture both raw performance data and derived analysis results.
111
112
The most important artifacts are the ETDump trace files, which contain operator-level timing information collected during execution. All higher-level summaries are derived from these traces.
@@ -128,23 +129,27 @@ The generated artifacts are listed below:
128
129
129
130
Although multiple file formats are generated, the .etdump files are the authoritative data source. All analysis scripts operate on these traces.
130
131
131
-
## 6. Expected Results: Case Study Insights
132
+
## Interpret SME2 performance results
132
133
133
134
After analyzing your artifacts, you'll see two key insights: end-to-end latency improvements and the bottleneck shift. The case study below shows results from SqueezeSAM, an interactive image segmentation model, running on an SME2-enabled Android device. The performance analysis kit includes EdgeTAM's image segmentation module as the example model, which is a more recent video-focused segmentation model that demonstrates advanced model onboarding patterns.
134
135
135
136
**End-to-end latency**: With SME2 enabled, FP16 inference improves by 3.9× (from 1,163 ms to 298 ms on a single CPU core), making on-device execution viable for interactive use cases. INT8 also sees substantial speedups (1.83×), demonstrating that SME2 accelerates both quantized and floating-point models.
136
137
137
-
SqueezeSAM on SME2-enabled Android device Results:
138
-
![end-latency #center](images/squeeze_sam_latency_comparison.png"End-to-end latency comparison with SME2 on vs off")
138
+
SqueezeSAM on SME2-enabled Android device results:
139
+
![Bar chart comparing SqueezeSAM end-to-end latency with SME2 enabled versus disabled, showing FP16 improvement of 3.9× and INT8 improvement of 1.83×#center](images/squeeze_sam_latency_comparison.png"End-to-end latency comparison with SME2 on vs off")
139
140
140
141
**The bottleneck shift**: After SME2 accelerates CONV and GEMM operations, data movement operations (transpose, reshape, layout conversions) become the dominant cost. This is expected, as SME2 reveals the next optimization frontier. The operator-category breakdown makes it obvious where to focus next.
141
142
142
-
![end-latency #center](images/combined_operator_breakdown_stacked.png"Operator-category breakdown showing data movement becoming dominant after SME2")
143
+
![Stacked bar chart showing operator-category breakdown with SME2 off versus on, demonstrating CONV and GEMM operations shrinking while data movement operations become the dominant cost#center](images/combined_operator_breakdown_stacked.png"Operator-category breakdown showing data movement becoming dominant after SME2")
143
144
144
145
The operator-category breakdown shows CONV/GEMM shrink while data movement becomes the dominant cost after SME2 acceleration. It makes it clear where further optimization effort should be focused once compute is no longer the limiting factor.
145
146
146
-
## 7. Where to go next
147
+
## What you've accomplished and what's next
148
+
149
+
You now understand the complete profiling workflow: how SME2 acceleration changes the performance profile of ExecuTorch models, what operator-level profiling reveals about bottleneck shifts, and how to interpret the results to guide optimization decisions.
150
+
151
+
The following sections walk through each stage of the pipeline in detail, from environment setup to model onboarding and automated workflows:
147
152
148
-
- Continue to [Setup and pipeline](/learning-paths/cross-platform/sme-executorch-profiling/02-setup-and-pipeline) to understand the environment setup and runner build process in more detail.
149
-
- Continue to [Model onboarding and performance analysis](learning-paths/cross-platform/sme-executorch-profiling/03-model-onboarding-and-profiling) to export additional models and analyze their performance.
150
-
- Continue to [Agent skills](learning-paths/cross-platform/sme-executorch-profiling/04-agent-skills) to explore how this workflow can be automated using AI-assisted tooling.
153
+
- Continue to [Setup and pipeline](/learning-paths/cross-platform/sme-executorch-profiling/02-setup-and-pipeline/) to understand the environment setup and runner build process in more detail.
154
+
- Continue to [Model onboarding and performance analysis](/learning-paths/cross-platform/sme-executorch-profiling/03-model-onboarding-and-profiling/) to export additional models and analyze their performance.
155
+
- Continue to [Agent skills](/learning-paths/cross-platform/sme-executorch-profiling/04-agent-skills/) to explore how this workflow can be automated using AI-assisted tooling.
0 commit comments