Skip to content

Commit 7745852

Browse files
Update overview.md to enhance clarity on ONNX Runtime architecture and SME2 integration
1 parent 5952d11 commit 7745852

1 file changed

Lines changed: 17 additions & 11 deletions

File tree

  • content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2

content/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/overview.md

Lines changed: 17 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,40 @@
11
---
2-
title: ONNX Runtime overview
2+
title: ONNX Runtime architecture with SME2 acceleration
33
weight: 2
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9-
## ONNX Runtime overview
9+
## Integrating KleidiAI micro-kernels into ONNX Runtime
1010
With the rise of on-device AI, squeezing performance from CPUs has become critical. Arm's Scalable Matrix Extension 2 (SME2) represents a leap forward, offering significant speedups for matrix-heavy workloads like Transformers and CNNs.
11+
1112
This Learning Path walks you through the technical steps to integrate KleidiAI-Arm's specialized micro-kernel library with SME2 support-into ONNX Runtime (ORT) and profile its performance using onnxruntime_perf_test on Android devices.
1213

13-
## Understanding the ONNX Runtime software stack
14+
## How does ONNX Runtime process AI models?
15+
1416
ONNX Runtime's internal architecture consists of four main components that work together to execute AI models efficiently:
17+
18+
- In-Memory Graph: represents the model structure as nodes (operations) and edges (data flows)
19+
- Graph Partitioner: assigns operations to appropriate hardware accelerators
20+
- Graph Runner: orchestrates execution and manages data flow between operations
21+
- Execution Provider: hardware-specific backends that run the actual computations
22+
1523
![Diagram showing the four main components of ONNX Runtime: In-Memory Graph at the top, followed by Graph Partitioner, Graph Runner, and Execution Provider at the bottom, with data flow indicated between layers alt-txt#center](images/ort_overview.jpg "The ONNX Runtime overview")
1624

17-
### In-Memory Graph
25+
### How does ONNX Runtime represent models in memory?
1826
When ORT loads an ONNX model, it parses the protobuf file and builds an in-memory representation of the model's structure. This graph consists of:
1927

2028
- Nodes: operations like MatMul, Conv, and Add
2129
- Edges: tensor data flowing between operations
2230

2331
During this stage, ORT performs Graph Optimizations like constant folding and node fusion.
24-
### Graph Partitioner
25-
The Graph Partitioner decides which part of the model runs on which hardware. It analyzes the computational graph and matches nodes to the registered Execution Providers.
26-
It clusters adjacent nodes assigned to the same EP into "Subgraphs".
27-
### Graph Runner
28-
Once the graph is partitioned, the Graph Runner executes the operators in the correct order. It manages the flow of data (Tensors) between nodes.
29-
In ORT, parallelism splits into two distinct levels to maximize hardware utilization: Intra-op (inside an operator/node, splitting a single heavy operation/node into smaller chunks) and Inter-op (between different operators, running multiple independent operators at the same time).
30-
### Execution Provider (EP)
32+
### How does ONNX Runtime assign operations to hardware?
33+
The Graph Partitioner decides which part of the model runs on which hardware. It analyzes the computational graph and matches nodes to the registered Execution Providers. It clusters adjacent nodes assigned to the same EP into "Subgraphs".
34+
### How does ONNX Runtime execute operations?
35+
Once the graph is partitioned, the Graph Runner executes the operators in the correct order. It manages the flow of data (Tensors) between nodes. In ORT, parallelism splits into two distinct levels to maximize hardware utilization: Intra-op (inside an operator/node, splitting a single heavy operation/node into smaller chunks) and Inter-op (between different operators, running multiple independent operators at the same time).
3136

37+
### What are ONNX Runtime execution providers?
3238
An Execution Provider is the abstraction layer that interfaces with specific hardware or libraries. Each EP provides optimized math functions (called "Kernels") for specific operators.
3339

3440
ORT supports multiple Execution Providers across different hardware types:

0 commit comments

Comments
 (0)