Skip to content

Commit f630aff

Browse files
pareenavermapareenaverma
authored
Merge pull request #3413 from matt-cossins/model_explorer
[LP] Visualize Model Artifacts with Model Explorer + ExecuTorch Extension
2 parents 63d1c30 + db7f4a2 commit f630aff

30 files changed

Lines changed: 1246 additions & 0 deletions

‎content/learning-paths/cross-platform/explore-model-artifacts-with-model-explorer/1-what-is-model-explorer.md‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
---
2+
title: "What is Model Explorer and what will you learn?"
3+
4+
weight: 2
5+
6+
### FIXED, DO NOT MODIFY
7+
layout: "learningpathall"
8+
---
9+
10+
## Model Explorer and Arm extensions
11+
12+
[Model Explorer](https://ai.google.dev/edge/model-explorer) is an open-source, web-based graph visualizer and debugger from Google AI Edge. It provides a hierarchical view of model graphs, lets you expand and collapse layers, search for nodes, inspect metadata, highlight inputs and outputs, compare models, and add overlays to graph nodes. The default version, [contained in this repository](https://github.com/google-ai-edge/model-explorer/) supports TFLite, TF, TFJS, MLIR, and PyTorch (Exported Program) formats.
13+
14+
Model Explorer uses adapters and data providers to load formats beyond the built-in model types. The [ExecuTorch extension for Model Explorer](https://github.com/arm/executorch-extension-model-explorer) combines support for `.pte`, `.etrecord`, and `.etdp` files. Standalone `.tosa` and `.vgf` files use separate TOSA and VGF adapters. You install all three packages together in the next section.
15+
16+
## What you will do
17+
18+
This learning path is about model artifacts, not model training or export. You start from small pre-generated files and use each to gain an understanding of when you would use Model Explorer, and the insights you can gain.
19+
20+
The artifacts covered in this learning path are:
21+
22+
| Artifact | Model Explorer support | Workflow layer | Use it to inspect |
23+
| --- | --- | --- | --- |
24+
| `.pte` | PTE adapter in the ExecuTorch extension | ExecuTorch program | Delegate regions, backend partitioning, work outside delegates, and the deployed ExecuTorch graph |
25+
| `.tosa` | Separate TOSA adapter | Compiler/backend intermediate representation | Lowered operators, tensor shapes, quantized types, graph splits, and missed optimization opportunities |
26+
| `.vgf` | Separate VGF adapter | Graph artifact for the ML extensions for Vulkan | Inputs, outputs, constants, tensor metadata, graph connectivity, and SPIR-V graph modules |
27+
| `.etrecord` | ETRecord adapter in the ExecuTorch extension | Export-time profiling context | Graph structure, debug handles, operator names, and delegate metadata used to map runtime events back to graph nodes |
28+
| `.etdp` | ETDump data provider in the ExecuTorch extension | Runtime trace overlay | Timing data from a specific execution |
29+
30+
This Learning Path focuses on Arm adapters and extensions for Model Explorer, but the artifacts repository also includes `.tflite` and `.pt2` files. You can optionally try these files because Model Explorer supports them without an additional adapter.
31+
32+
{{% notice Note %}}
33+
Model Explorer visualizes the specific artifact you generated or received. Small differences in the target the model has been delegated to, could result in a very different model graph. For example, delegating the same model to an Ethos-U55, may produce a very different model graph from delegating to an Ethos-U85.
34+
{{% /notice %}}
35+
36+
## Understand the model artifact flow
37+
38+
Not every graph in this learning path needs to start with PyTorch and ExecuTorch. Let's run through what is specific to the ExecuTorch flow, and what is applicable more broadly.
39+
40+
PTE is ExecuTorch-specific. A `.pte` file is the serialized ExecuTorch program loaded by the ExecuTorch runtime. Baseline portable-kernel, XNNPACK, Cortex-M, Ethos-U, and VGF-backend examples are all ExecuTorch deployment artifacts.
41+
42+
The portable-kernel and XNNPACK routes are Cortex-A CPU routes. Cortex-M uses a separate ExecuTorch flow that applies CMSIS-NN-oriented passes for supported quantized operators.
43+
44+
For Ethos-U, the ExecuTorch backend uses the Ethos-U Vela compiler to compile TOSA flatbuffers into an Ethos-U command stream that is packaged into the final `.pte`.
45+
46+
TOSA is not inherently ExecuTorch-specific. TOSA is an intermediate representation (IR) that can sit between a model frontend and an Arm backend compiler or converter. ExecuTorch can lower supported graph partitions to TOSA, but another framework could also provide its own TOSA exporter and generate `.tosa` files.
47+
48+
```output
49+
PyTorch model
50+
|
51+
v
52+
ExecuTorch export
53+
|
54+
+-- Cortex-A portable kernels -> baseline .pte
55+
|
56+
+-- Cortex-A XNNPACK delegate -> XNNPACK .pte
57+
|
58+
+-- Cortex-M CMSIS-NN passes -> Cortex-M .pte
59+
|
60+
+-- Lower to TOSA -------------+
61+
|
62+
|
63+
|
64+
Alternative (non PT/ET) v
65+
frontend with TOSA export ---> TOSA (.tosa)
66+
|
67+
|
68+
+-- Ethos-U Vela
69+
| -> command stream
70+
| -> Ethos-U .pte
71+
|
72+
+-- ML SDK Model Converter
73+
-> VGF payload
74+
-> VGF-backend .pte
75+
-> standalone .vgf
76+
for workflows using the ML extensions for Vulkan
77+
```
78+
79+
For VGF, the ExecuTorch Arm VGF backend uses the [Arm ML SDK Model Converter](https://github.com/arm/ai-ml-sdk-model-converter) to produce a VGF backend payload from TOSA. But the Arm ML SDK Model Converter could be used to convert `.tosa` files generated from a different flow, so VGF is also not specific to just ExecuTorch.
80+
81+
When ExecuTorch is used for VGF, a `.pte` is emitted as well. Use that VGF-backend `.pte` when you want to run through ExecuTorch. Use the standalone `.vgf` when you want to inspect or integrate the VGF artifact used with the ML extensions for Vulkan, such as in a neural graphics workflow.
82+
83+
If you have used the [Arm Neural Graphics Model Gym](https://github.com/arm/neural-graphics-model-gym), then under the hood you have been using ExecuTorch to export your neural graphics model to VGF. If you are interested in learning more, try out the [Fine-tune neural graphics using Model Gym](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/model-training-gym/#:~:text=Upon%20completion%20of%20this%20Learning,and%20train%20neural%20graphics%20models) learning path, which briefly introduces Model Explorer.
84+
85+
ETRecord and ETDump are additional, ExecuTorch-specific, artifacts. ETRecord is generated at export time and preserves the graph context needed for profiling attribution. ETDump is generated at runtime and records what actually happened when a `.pte` ran. Together, they let Model Explorer move from static inspection to runtime overlays: you can connect the graph structures you saw in the `.pte`, `.tosa`, and `.vgf` sections to operator and delegate events measured during execution.
86+
87+
## Terminology
88+
89+
A helpful glossary of different terms is provided below:
90+
91+
| Term | Meaning in this learning path |
92+
| --- | --- |
93+
| Model | The neural network you want to deploy, before or after transformation by export and compiler tools. |
94+
| Framework | The software used to define or train the model, such as PyTorch. |
95+
| Export | The step that turns a framework model into a deployable or compiler-friendly representation. |
96+
| Artifact | A file produced by export, lowering, compilation, or conversion, such as `.pte`, `.tosa`, or `.vgf`. |
97+
| Runtime | The software on the target system that loads and executes a deployable artifact. ExecuTorch is the runtime for `.pte` files. |
98+
| Compiler | A tool that transforms an intermediate representation into a lower-level target representation, such as Vela compiling TOSA for Ethos-U. |
99+
| Lower | To transform a model or graph from a higher-level representation into a lower-level representation closer to a target backend. |
100+
| Convert | To change one artifact format into another, such as TOSA to VGF. |
101+
| Intermediate representation | A representation between the original model and the final target artifact. TOSA is the main intermediate representation in this learning path. |
102+
| Delegate | A backend-specific execution path that handles supported parts of a graph. Unsupported parts can remain outside the delegate, and can run on a CPU path only when the deployed runtime includes compatible kernels. |
103+
| Kernel | The code that executes a model operator for a specific runtime or backend, such as a portable ExecuTorch kernel or a CMSIS-NN kernel. |
104+
| Flatbuffer | A compact binary serialization format. TOSA flatbuffers store TOSA graphs; `.pte` files use a FlatBuffer-based ExecuTorch program format. |
105+
| SPIR-V | Standard Portable Intermediate Representation - Vulkan. SPIR-V modules inside VGF files describe the data graph used by the ML extensions for Vulkan. |
106+
| ETRecord | An ExecuTorch export-time debug artifact that preserves graph and delegate metadata for profiling attribution. |
107+
| ETDump | An ExecuTorch runtime trace artifact that can record operator, delegate, backend, timing, and cycle-count events from execution. The first Model Explorer overlay used in this Learning Path focuses on timing data. |
108+
109+
## What models will I use?
110+
111+
The hands-on sections will use a variety of pre-provided model artifacts. These are provided purely for educational purposes.
112+
113+
```output
114+
ml-model-artifacts/
115+
├── pte/
116+
│ ├── mv2_cortex_m.pte
117+
│ ├── opt125m_cortex_a_portable.pte
118+
│ ├── opt125m_cortex_a_xnnpack.pte
119+
│ ├── mv2_fp32_ethos_u85.pte
120+
│ ├── mv2_int8_ethos_u85.pte
121+
│ ├── mv2_lrn_int8_ethos_u85.pte
122+
│ └── small_upscaler_ptq_vgf.pte
123+
├── tosa/
124+
│ ├── mv2_fp32.tosa
125+
│ ├── mv2_int8.tosa
126+
│ ├── mv2_lrn_int8_1.tosa
127+
│ ├── mv2_lrn_int8_2.tosa
128+
│ ├── small_upscaler_ptq.tosa
129+
│ └── small_upscaler_qat.tosa
130+
├── vgf/
131+
│ └── small_upscaler_ptq.vgf
132+
├── etrecord/
133+
│ ├── opt125m_portable.etrecord
134+
│ ├── opt125m_xnnpack.etrecord
135+
│ ├── mobilenetv2_fp32_ethosu.etrecord
136+
│ ├── mobilenetv2_int8_ethosu.etrecord
137+
│ └── mobilenetv2_lrn_int8_ethosu.etrecord
138+
└── etdump/
139+
├── opt125m_portable.etdp
140+
├── opt125m_xnnpack.etdp
141+
├── mobilenetv2_fp32_ethosu.etdp
142+
├── mobilenetv2_int8_ethosu.etdp
143+
└── mobilenetv2_lrn_int8_ethosu.etdp
144+
```
145+
146+
## What you have learned
147+
148+
You have learned how the combined ExecuTorch extension and the separate TOSA and VGF adapters add artifact formats to Model Explorer. You have also seen how `.pte`, `.tosa`, `.vgf`, `.etrecord`, and `.etdp` files support Cortex-A, Cortex-M, Ethos-U, the ML extensions for Vulkan, and ExecuTorch profiling workflows.
149+
150+
Next, you will install Model Explorer and the Arm extensions, then open the first `.pte` artifact.

‎content/learning-paths/cross-platform/explore-model-artifacts-with-model-explorer/2-install-and-open-pte.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
title: "Install Model Explorer extensions and view a Cortex-M model graph"
3+
4+
weight: 3
5+
6+
### FIXED, DO NOT MODIFY
7+
layout: "learningpathall"
8+
---
9+
10+
## Clone the repo of example models
11+
12+
In this section, you install Model Explorer and the Arm extensions in a clean Python virtual environment. You then confirm the installation by opening a Cortex-M `.pte` file. First, clone the repository of example models used throughout this Learning Path.
13+
14+
Use a machine capable of displaying a browser e.g, a laptop.
15+
16+
This repository uses Git LFS for model artifacts. Install and configure Git LFS for your operating system. You only need to run `git lfs install` once for your user account.
17+
18+
{{< tabpane code=true >}}
19+
{{< tab header="Linux" language="bash">}}
20+
sudo apt update
21+
sudo apt install -y git-lfs
22+
git lfs install
23+
{{< /tab >}}
24+
{{< tab header="macOS" language="bash">}}
25+
brew install git-lfs
26+
git lfs install
27+
{{< /tab >}}
28+
{{< tab header="Windows PowerShell" language="powershell">}}
29+
winget install -e --id GitHub.GitLFS
30+
git lfs install
31+
{{< /tab >}}
32+
{{< /tabpane >}}
33+
34+
Clone the artifacts repository, then use `git lfs pull` to download the model files:
35+
36+
```bash
37+
git clone https://github.com/arm-education/ml-model-artifacts.git
38+
cd ml-model-artifacts
39+
git lfs pull
40+
```
41+
42+
## Create a virtual environment
43+
44+
Use a separate environment to avoid dependency conflicts with any ExecuTorch build, notebook, or application environment you already use:
45+
46+
Use Python 3.10, 3.11, or 3.12 for this environment.
47+
48+
If you use WSL on Windows, follow the Linux/macOS commands.
49+
50+
{{< tabpane code=true >}}
51+
{{< tab header="Linux/macOS" language="bash">}}
52+
python3 -m venv model_explorer_env
53+
source model_explorer_env/bin/activate
54+
python -m pip install --upgrade pip
55+
{{< /tab >}}
56+
{{< tab header="Windows PowerShell" language="powershell">}}
57+
py -m venv model_explorer_env
58+
.\model_explorer_env\Scripts\Activate.ps1
59+
python -m pip install --upgrade pip
60+
{{< /tab >}}
61+
{{< /tabpane >}}
62+
63+
{{% notice Tip %}}
64+
On Windows, if PowerShell blocks `Activate.ps1`, allow local activation scripts for your user account:
65+
66+
```powershell
67+
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
68+
```
69+
{{% /notice %}}
70+
71+
## Install Model Explorer and the Arm extensions
72+
73+
Install the combined ExecuTorch extension with the separate TOSA and VGF adapters:
74+
75+
```bash
76+
python -m pip install executorch-extension-model-explorer tosa-adapter-model-explorer vgf-adapter-model-explorer
77+
```
78+
79+
The ExecuTorch extension provides the PTE adapter, ETRecord adapter, and ETDump profiling data provider. The separate TOSA and VGF adapters open standalone `.tosa` and `.vgf` files.
80+
81+
{{% notice Note %}}
82+
For component development or focused debugging, you can install the ExecuTorch components separately as `pte-adapter-model-explorer`, `etrecord-adapter-model-explorer`, and `etdump-data-provider-model-explorer`. For this Learning Path, use the combined `executorch-extension-model-explorer` package.
83+
{{% /notice %}}
84+
85+
## Launch Model Explorer
86+
87+
Launch Model Explorer with all extensions used in this Learning Path. The combined ExecuTorch extension opens `.pte` and `.etrecord` files and adds `.etdp` profiling data. The separate TOSA and VGF extensions open `.tosa` and `.vgf` files.
88+
89+
This will launch a webpage in your browser.
90+
91+
{{< tabpane code=true >}}
92+
{{< tab header="All Learning Path extensions" language="bash">}}
93+
model-explorer --extensions=executorch_extension_model_explorer,tosa_adapter_model_explorer,vgf_adapter_model_explorer
94+
{{< /tab >}}
95+
{{< tab header="ExecuTorch extension" language="bash">}}
96+
model-explorer --extensions=executorch_extension_model_explorer
97+
{{< /tab >}}
98+
{{< tab header="No adapters" language="bash">}}
99+
model-explorer
100+
{{< /tab >}}
101+
{{< /tabpane >}}
102+
103+
{{% notice Tip %}}
104+
Use `CTRL + C` to stop Model Explorer.
105+
{{% /notice %}}
106+
107+
{{% notice Note %}}
108+
If you have a specific interest in one model format, such as `.pte`, `.tosa`, or `.vgf`, or in a particular target (e.g., Cortex-M, Cortex-A, Ethos-U, Neural Graphics) you can skip to the appropriate section.
109+
{{% /notice %}}
110+
111+
## Open the Cortex-M PTE
112+
113+
We will start with a `.pte` generated for the Cortex-M backend. This `.pte` was generated for the MobileNetV2 model, a typical Convolutional Neural Network (CNN) used in embedded ML.
114+
115+
The ExecuTorch Cortex-M backend prepares models for Arm Cortex-M microcontrollers, where memory and compute resources are much more limited than on application-class CPUs. It rewrites supported quantized operators so they can use CMSIS-NN, an Arm library of optimized neural network kernels for Cortex-M processors. CMSIS-NN exists to make common ML operations such as convolutions, fully connected layers, activations, and quantization-related operations run efficiently on small embedded CPUs. Treat this first `.pte` as a good way to learn the Model Explorer interface while seeing how an ExecuTorch graph can reflect Cortex-M-specific lowering.
116+
117+
{{% notice Note %}}
118+
The Cortex-M backend is a work-in-progress proof of concept. It is not intended for production use, and APIs may change without notice. However, the `.pte` is pre-generated for you in the provided repo. If you would like to find out more about the Cortex-M backend, use the [Cortex-M Backend Documentation](https://docs.pytorch.org/executorch/1.2/backends/arm-cortex-m/arm-cortex-m-overview.html), which also links to a Jupyter Notebook.
119+
{{% /notice %}}
120+
121+
In the Model Explorer UI, open:
122+
123+
```output
124+
ml-model-artifacts/pte/mv2_cortex_m.pte
125+
```
126+
127+
Your view in browser should appear as follows:
128+
129+
![Screenshot of Model Explorer with Arm Adapters and a loaded Cortex-M PTE.#center](model_explorer.png "Model Explorer with Arm Adapters")
130+
131+
Click `View selected models` and your view should appear as below:
132+
133+
![Screenshot of top-level view in Model Explorer of a loaded Cortex-M PTE.#center](cortex_m_top.png "Typical top-level graph view in Model Explorer")
134+
135+
A right-hand bar tells you the graph info, including the `op node count` and the `layer count`. The `op node count` is the number of operator nodes in the graph. The `layer count` is the number of hierarchical graph components represented in the current view, not necessarily the number of neural network layers in the original model.
136+
137+
Double click the `forward` layer to see the various operators comprising the layer. Click a specific operator, e.g., `cortex_m::quantize_per_tensor` to see various attributes, as well as inputs and outputs, in the right-hand bar.
138+
139+
![Screenshot of examining a specific Cortex-M operator in Model Explorer.#center](cortex_m_inspect.png "Inspecting specific operators with Model Explorer")
140+
141+
When you inspect a `.pte` for the first time, focus on the higher-level graph information first:
142+
143+
- **Operator names** show the work the ExecuTorch program will perform.
144+
- **Inputs and outputs** show how tensors flow through the graph.
145+
- **Tensor shapes and types** help you check whether the model was exported and quantized as expected.
146+
- **Hierarchical layers** let you expand or collapse parts of the graph so you can move between an overview and individual operators.
147+
- **Delegate or backend-specific names** show where a backend flow has changed the graph. In this Cortex-M example, names such as `cortex_m::...` indicate operators affected by the Cortex-M backend flow.
148+
149+
You might also see lower-level `.pte` execution fields in the node attributes. These fields come from the serialized ExecuTorch program:
150+
151+
| Field | What it means |
152+
| --- | --- |
153+
| `instruction type: KernelCall` | This instruction calls an ExecuTorch operator kernel. A kernel is the code that executes a model operator for a runtime or backend. |
154+
| `instr_args_type: 1` | The internal FlatBuffer type tag for the instruction arguments. In this case, `1` identifies the arguments as a `KernelCall`. |
155+
| `op_index` | An index into the `.pte` operator table. It tells ExecuTorch which operator this instruction calls. |
156+
| `args: [471, 473, 474, ...]` | Indexes into the `.pte` values table. These entries identify the tensors or other values used as inputs and outputs by the instruction. |
157+
158+
You do not need to memorize these internal fields. Use them as clues when you want to connect a visible graph node to the underlying ExecuTorch program structure.
159+
160+
Click the `eye symbol` in the top left bar to select data to view on nodes and edges. Select the data you want to see - e.g., `op node id` and `op node attributes` to include more data within the graph itself.
161+
162+
## What you have learned
163+
164+
You have installed Model Explorer, launched it with the Arm extensions, and opened your first `.pte` artifact. You have also learned how to inspect the graph overview, expand into the `forward` layer, read operator metadata, and interpret common low-level `.pte` fields such as `KernelCall`, `op_index`, and `args`.
165+
166+
Next, you will keep the same browser tab open and compare portable and XNNPACK `.pte` artifacts to see how backend delegation changes the graph.

0 commit comments

Comments
 (0)