|
1 | 1 | --- |
2 | | -title: ONNX Runtime overview |
| 2 | +title: ONNX Runtime architecture with SME2 acceleration |
3 | 3 | weight: 2 |
4 | 4 |
|
5 | 5 | ### FIXED, DO NOT MODIFY |
6 | 6 | layout: learningpathall |
7 | 7 | --- |
8 | 8 |
|
9 | | -## ONNX Runtime overview |
| 9 | +## Integrating KleidiAI micro-kernels into ONNX Runtime |
10 | 10 | With the rise of on-device AI, squeezing performance from CPUs has become critical. Arm's Scalable Matrix Extension 2 (SME2) represents a leap forward, offering significant speedups for matrix-heavy workloads like Transformers and CNNs. |
| 11 | + |
11 | 12 | This Learning Path walks you through the technical steps to integrate KleidiAI-Arm's specialized micro-kernel library with SME2 support-into ONNX Runtime (ORT) and profile its performance using onnxruntime_perf_test on Android devices. |
12 | 13 |
|
13 | | -## Understanding the ONNX Runtime software stack |
| 14 | +## How does ONNX Runtime process AI models? |
| 15 | + |
14 | 16 | ONNX Runtime's internal architecture consists of four main components that work together to execute AI models efficiently: |
| 17 | + |
| 18 | +- In-Memory Graph: represents the model structure as nodes (operations) and edges (data flows) |
| 19 | +- Graph Partitioner: assigns operations to appropriate hardware accelerators |
| 20 | +- Graph Runner: orchestrates execution and manages data flow between operations |
| 21 | +- Execution Provider: hardware-specific backends that run the actual computations |
| 22 | + |
15 | 23 |  |
16 | 24 |
|
17 | | -### In-Memory Graph |
| 25 | +### How does ONNX Runtime represent models in memory? |
18 | 26 | When ORT loads an ONNX model, it parses the protobuf file and builds an in-memory representation of the model's structure. This graph consists of: |
19 | 27 |
|
20 | 28 | - Nodes: operations like MatMul, Conv, and Add |
21 | 29 | - Edges: tensor data flowing between operations |
22 | 30 |
|
23 | 31 | During this stage, ORT performs Graph Optimizations like constant folding and node fusion. |
24 | | -### Graph Partitioner |
25 | | -The Graph Partitioner decides which part of the model runs on which hardware. It analyzes the computational graph and matches nodes to the registered Execution Providers. |
26 | | -It clusters adjacent nodes assigned to the same EP into "Subgraphs". |
27 | | -### Graph Runner |
28 | | -Once the graph is partitioned, the Graph Runner executes the operators in the correct order. It manages the flow of data (Tensors) between nodes. |
29 | | -In ORT, parallelism splits into two distinct levels to maximize hardware utilization: Intra-op (inside an operator/node, splitting a single heavy operation/node into smaller chunks) and Inter-op (between different operators, running multiple independent operators at the same time). |
30 | | -### Execution Provider (EP) |
| 32 | +### How does ONNX Runtime assign operations to hardware? |
| 33 | +The Graph Partitioner decides which part of the model runs on which hardware. It analyzes the computational graph and matches nodes to the registered Execution Providers. It clusters adjacent nodes assigned to the same EP into "Subgraphs". |
| 34 | +### How does ONNX Runtime execute operations? |
| 35 | +Once the graph is partitioned, the Graph Runner executes the operators in the correct order. It manages the flow of data (Tensors) between nodes. In ORT, parallelism splits into two distinct levels to maximize hardware utilization: Intra-op (inside an operator/node, splitting a single heavy operation/node into smaller chunks) and Inter-op (between different operators, running multiple independent operators at the same time). |
31 | 36 |
|
| 37 | +### What are ONNX Runtime execution providers? |
32 | 38 | An Execution Provider is the abstraction layer that interfaces with specific hardware or libraries. Each EP provides optimized math functions (called "Kernels") for specific operators. |
33 | 39 |
|
34 | 40 | ORT supports multiple Execution Providers across different hardware types: |
|
0 commit comments