You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/ernie_moe_v9/2_llamacpp_installation.md
+4-2Lines changed: 4 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ weight: 3
4
4
layout: "learningpathall"
5
5
---
6
6
7
-
## Set up llama.cpp on an Armv9 development board
7
+
## Overview
8
8
9
9
In the previous section, you learned how Mixture of Experts (MoE) models reduce resource consumption by activating only a fraction of parameters.
10
10
Now, you'll walk through how to prepare your environment to deploy ERNIE-4.5 models on an Armv9 platform using llama.cpp.
@@ -121,4 +121,6 @@ This confirms the GGUF model is successfully loaded, the llama.cpp build functio
121
121
122
122
The prompt "Please introduce Mixture of Experts in Chinese." was chosen because it tests two important capabilities. First, it verifies bilingual capability since the instruction is in English but the answer is expected in Chinese, confirming that ERNIE-4.5's multilingual support works correctly. Second, it engages MoE behavior by having the model explain a technical concept in Chinese, which likely activates different experts within the model during inference. Even though routing isn't explicitly logged, the richness and precision of the output suggest that MoE routing functions as designed. This type of prompt increases the likelihood of multiple experts being activated simultaneously, including language generation, machine learning knowledge, and Chinese translation experts.
123
123
124
-
You've now successfully set up llama.cpp on Armv9 hardware, deployed the ERNIE-4.5 model, and confirmed that multilingual inference works correctly. With this foundation in place, you're ready to explore expert routing behavior and benchmark performance optimizations in the next sections.
124
+
## What you've accomplished and what's next
125
+
126
+
You've successfully set up llama.cpp on Armv9 hardware, deployed the ERNIE-4.5 model, and confirmed that multilingual inference works correctly. With this foundation in place, you're ready to explore expert routing behavior and compare model variants in the next section.
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/ernie_moe_v9/3_erine_moe.md
+12-7Lines changed: 12 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -41,7 +41,7 @@ Here is the PT variant:
41
41
--jinja
42
42
```
43
43
44
-
The answer looks like this:
44
+
The output is similar to:
45
45
46
46
```output
47
47
Assistant: 1. **Top Motivation**: Achieving visible results and maintaining progress through efficient recovery nutrition.
@@ -65,7 +65,7 @@ Here is the Thinking variant:
65
65
--jinja
66
66
```
67
67
68
-
You see a more comprehensive answer like this:
68
+
The output is similar to:
69
69
70
70
```output
71
71
assistant
@@ -161,18 +161,23 @@ Run inference with the same prompt and monitor the console for lines like this:
161
161
---[DEBUG]--- entering build_moe_ffn at layer 27 with 64 experts (use 64)
162
162
```
163
163
164
-
This reveals how many experts (for example, 6) and how many tokens (for example, 16) were routed at that layer.
164
+
This output shows that each layer has 64 total experts available. The actual number of experts activated per token (typically 6 for ERNIE-4.5) is determined by the router during inference and isn't directly visible in this debug output.
165
165
166
166
{{% notice Note %}}
167
-
You can also trace the function `llm_graph_context::build_moe_ffn()` in `src/llama-graph.cpp` to see how expert selection works.
167
+
You can also trace the function `llm_graph_context::build_moe_ffn()` in `src/llama-graph.cpp` to see how expert selection works at a deeper level.
168
168
{{% /notice %}}
169
169
170
-
Remove the print statement from `src/models/ernie4-5-moe.cpp` before moving to the next section.
170
+
Remove the print statement from `src/models/ernie4-5-moe.cpp`and rebuild llama.cpp before moving to the next section.
171
171
172
172
As you review the debug output, observe whether the number of active experts changes between the PT and Thinking models. Look for patterns in routing, such as different token batches routing to differing expert sets. You can also correlate routing behavior with output differences, as deeper routing variety might align with more detailed responses.
173
173
174
-
## Summary
174
+
## What you've accomplished and what's next
175
175
176
-
This task highlights the advantage of MoE fine-tuning. Even with the same architecture, thoughtful tuning can significantly change a model's reasoning behavior. The Thinking model is better suited for applications that need analytical depth, making it ideal for edge AI scenarios like customer profiling or real-time recommendations.
176
+
In this section, you:
177
+
- Compared ERNIE-4.5 PT and Thinking model outputs on the same task
178
+
- Observed how fine-tuning affects reasoning depth and response structure
179
+
- Learned how to add debug instrumentation to examine MoE expert routing
180
+
181
+
This comparison highlights the advantage of MoE fine-tuning. Even with the same architecture, thoughtful tuning can significantly change a model's reasoning behavior. The Thinking model is better suited for applications that need analytical depth, making it ideal for edge AI scenarios like customer profiling or real-time recommendations.
177
182
178
183
In the next section, you switch focus from model behavior to system-level performance by compiling with Armv9 instruction sets and measuring the impact on inference speed.
Copy file name to clipboardExpand all lines: content/learning-paths/cross-platform/ernie_moe_v9/4_v9_optimization.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -114,8 +114,8 @@ After running benchmarks with and without Armv9-specific instructions, the resul
114
114
115
115
Vectorized kernels (i8mm, dotprod, SVE) drastically improve inference throughput. The pp512 test shows the most significant acceleration with a 2.57× improvement. Other patterns like tg128 and pp128+tg128 also achieve measurable gains. These results demonstrate the broad benefit of hardware-aware builds and show that Armv9 optimization enables practical real-time inference for 21 B models on edge-class hardware.
116
116
117
-
### Summary
117
+
##What you've accomplished
118
118
119
-
Throughout this Learning Path, you deployed a 21 B parameter Chinese MoE model on edge-class Armv9 hardware. You started by understanding how MoE reduces memory usage by activating only a small subset of parameters per token. After setting up llama.cpp on a Radxa O6 board, you compared ERNIE-4.5 Thinking and PT model behaviors while examining expert routing logic with debug instrumentation. Finally, you applied Armv9 hardware optimizations and achieved over 2.5× speed improvements in token throughput.
119
+
Throughout this Learning Path, you deployed a 21-billion-parameter Chinese MoE model on edge-class Armv9 hardware. You started by understanding how MoE reduces memory usage by activating only a small subset of parameters per token. After setting up llama.cpp on a Radxa O6 board, you compared ERNIE-4.5 Thinking and PT model behaviors while examining expert routing logic with debug instrumentation. Finally, you applied Armv9 hardware optimizations and achieved over 2.5× speed improvements in token throughput.
120
120
121
121
You can now deploy, profile, and tune Chinese LLMs for efficient inference on modern Arm CPUs.
0 commit comments