You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This directory contains the C++ code for the LoRA demo.
3
+
This directory contains a C++ example demonstrating program-data separation with LoRA adapters in ExecuTorch.
4
4
5
5
You'll learn how to:
6
6
1. Export LoRA PTE files that share a single foundation weight file.
7
-
2. Load and run multiple LoRA PTE files at the same, and notice that the runtime memory increases by the LoRA adapter size (small) and not the foundation weight size (large), because the foundation weights are shared.
7
+
2. Load and run multiple LoRA PTE files at the same time, sharing the foundation weights to reduce runtime memory.
8
8
9
-
Note:
10
-
- Weight-sharing is supported with the XNNPACK backend.
11
-
- Quantization (outside of embedding quantization) is currently not supported when weight-sharing.
12
-
- There are many ways to fine-tune LoRA adapters. We will go through a few examples to create a demo.
9
+
This example uses **Qwen3-0.6B** with a math-focused LoRA adapter. The approach works for LoRA adapters trained on the same base model.
-[Finetune lora adapters from scratch with unsloth and Llama](#finetune-from-scratch-with-unsloth-and-llama)
17
-
-[Install executorch](#install-executorch)
18
-
-[Export lora models](#export-models)
19
-
-[Run lora models](#install-runtime-dependencies)
20
-
-[Demo video](#demo-video)
21
19
22
-
## Size savings
23
-
24
-
Size results will vary depending on the model and LoRA config. For this demo, we save ~5GB of disk space by storing weights in a separate, sharable file and ~5GB runtime memory by sharing weights at runtime through the XNNPACK weight cache. Detailed results are below.
25
-
26
-
### XNNPACK weight sharing
20
+
## LoRA adapters
27
21
28
-
The XNNPACK backend is a singleton. Weight sharing is implemented via the XNNPACK weight cache. At delegate init time, XNNPACK checks the weight cache for the weights it needs. If they don't exist, XNNPACK will fetch weights from the NamedDataMap (the API that exposes weights in a PTD file), pack them, store them in the weight cache and free the original. This means we won't keep around multiple copies of the same weights.
22
+
This example uses a pre-trained LoRA adapter from HuggingFace:
[Unsloth](https://unsloth.ai/) provides a [colab notebook](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/datasets-guide#synthetic-dataset-notebook) that showcases how to generate data using the Meta Synthetic Data Kit, and then fine-tune it to create a LoRA adapter.
26
+
If you want to train your own LoRA adapters, [Unsloth](https://unsloth.ai/) provides tools and notebooks for fine-tuning. The key files needed from training are:
27
+
-`adapter_config.json`
28
+
-`adapter_model.safetensors`
32
29
33
-
For this demo, we trained on two datasets:
34
-
1. executorch/docs/source/: an adapter with domain knowledge of executorch. This used Meta Synthetic Data Kit to generate qa pairs based on the documentation.
35
-
2. Recent Nobel prize winners (2024-2025): an adapter with knowledge beyond the cutoff date of Llama-3-2-1B. This data was taken from [Wikipedia](https://en.wikipedia.org/wiki/List_of_Nobel_laureates), and formatted into the chat template for training.
30
+
## Install ExecuTorch
36
31
37
-
The training notebook takes a few shortcuts to reduce the latency/compute. You can change these settings for better results.
38
-
1. When generating data, play around with the chunk sizes and overlap to see what works best for your dataset.
39
-
2. At the training step, the notebook uses max_steps=60 to speed things up. Setting num_train_epochs=1 (or greater) for a full run and max_steps=None has better results.
32
+
[Install from source](https://docs.pytorch.org/executorch/stable/using-executorch-building-from-source.html#install-executorch-pip-package-from-source):
40
33
41
-
Unsloth will output the adapter artifacts to the specified directory (in the colab notebook, 'lora_model/'). You will see a few files like such:
42
34
```bash
43
-
-rw-r--r-- 1 lfq users 1092 Oct 15 11:01 adapter_config.json
44
-
-rw-r--r-- 1 lfq users 45118424 Oct 15 11:01 adapter_model.safetensors
45
-
-rw-r--r-- 1 lfq users 3827 Oct 15 11:01 chat_template.jinja
46
-
-rw-r--r-- 1 lfq users 5268 Oct 15 11:01 README.md
47
-
-rw-r--r-- 1 lfq users 454 Oct 15 11:01 special_tokens_map.json
48
-
-rw-r--r-- 1 lfq users 50642 Oct 15 11:01 tokenizer_config.json
49
-
-rw-r--r-- 1 lfq users 17209920 Oct 15 11:01 tokenizer.json
50
-
```
51
-
52
-
The files we want are:
53
-
- adapter_config.json
54
-
- adapter_model.safetensors
55
-
56
-
## Install executorch
57
-
[Install from source](https://docs.pytorch.org/executorch/stable/using-executorch-building-from-source.html#install-executorch-pip-package-from-source).
# Or export quantized models; 8da4w and embedding quantization
62
+
bash export_lora.sh -q
113
63
```
114
64
115
-
Expect to see two files: '<model_name>.pte' and 'foundation.ptd'. Run the command again to generate more adapter PTE files. You only need to keep one `foundation.ptd` file.
116
-
117
-
You can also run `~/executorch-examples/program-data-separation/export_lora.sh`. This will export the dummy lora model and the base Llama-3-2-1B model PTE files.
65
+
This script will:
66
+
1. Download the Qwen3-0.6B base model and LoRA adapter from HuggingFace
67
+
2. Export a non-LoRA model with program-data separation
68
+
3. Export a LoRA model with program-data separation
118
69
119
-
Example files, trained on executorch/docs/source/ and recent Nobel prize winners.
70
+
After running, you'll see files in the `models/` directory:
-rw-r--r-- 1 lfq users 792K Dec 15 16:55 qwen3_06B_lora.pte # LoRA program
75
+
-rw-r--r-- 1 lfq users 2.3G Dec 15 16:55 qwen3_06B.ptd # Base (foundation) weights
76
+
-rw-r--r-- 1 lfq users 561K Dec 15 16:55 qwen3_06B.pte # Base model program
125
77
```
126
78
127
-
Notice the adapter PTE files are about the same size as the `adapter_model.safetensors`/`adapter_model.pt` files generated during training. The PTE contains the adapter weights (which are not shared) and the program.
79
+
For quantized versions, expect file sizes like:
80
+
```bash
81
+
models/
82
+
-rw-r--r-- 1 lfq users 39M Dec 16 17:11 qwen3_06B_lora_q.ptd
83
+
-rw-r--r-- 1 lfq users 855K Dec 16 17:11 qwen3_06B_lora_q.pte
84
+
-rw-r--r-- 1 lfq users 473M Dec 16 17:11 qwen3_06B_q.ptd
85
+
-rw-r--r-- 1 lfq users 621K Dec 16 16:47 qwen3_06B_q.pte
86
+
```
128
87
129
88
## Install runtime dependencies
130
-
The ExecuTorch repository is configured as a git submodule at `~/executorch-examples/program-data-separation/cpp/executorch`. To initialize it:
89
+
90
+
The ExecuTorch repository is configured as a git submodule at `~/executorch-examples/program-data-separation/cpp/executorch`. To initialize it:
Passing in the `DOWNLOADED_PATH` as the tokenizer directory will invoke the HFTokenizer, and parse additional tokenizers files: `tokenizer_config.json` and `special_tokens_map.json`. `special_tokens_map.json` tells us which bos/eos token to use, especially if there are multiple.
173
135
174
-
`apply_chat_template` formats the prompt according to the LLAMA chat template.
136
+
The `--tokenizer_path` should point to the directory containing the tokenizer files. The `--apply_chat_template` flag formats the prompt using the Qwen chat template.
137
+
138
+
This example runs two models:
139
+
1.**model1** (LoRA): The math-focused LoRA adapter, which should give accurate math answers
140
+
2.**model2** (base): The base Qwen model without LoRA
141
+
142
+
The foundation weights are shared between both models via the XNNPACK weight cache, reducing memory usage.
175
143
176
-
Sample output:
144
+
Sample output, lora model:
177
145
```
178
-
I 00:00:00.538779 executorch:main.cpp:133] Generating with model et.pte..
146
+
I 00:00:01.256382 executorch:main.cpp:145] Generating with model ../../models/qwen3_06B_lora_q.pte...
179
147
...
180
-
I 00:00:06.999737 executorch:text_llm_runner.cpp:182] RSS after prompt prefill: 6941.296875 MiB (0 if unsupported)
181
-
I don't have information on the winners of the Nobel Prize in Physics in 2025.<|eot_id|>
148
+
I 00:00:02.433530 executorch:text_llm_runner.cpp:92] RSS after loading model: 1393.593750 MiB (0 if unsupported)
182
149
...
183
-
I 00:00:11.635379 executorch:main.cpp:141] Generating with model nobel.pte...
184
-
...
185
-
I 00:00:14.109447 executorch:text_llm_runner.cpp:182] RSS after prompt prefill: 8041.632812 MiB (0 if unsupported)
186
-
John Clarke, Michel H. Devoret, John M. Martinis<|eot_id|>
150
+
ToI 00:00:03.012924 executorch:text_llm_runner.cpp:188] RSS after prompt prefill: 1468.085938 MiB (0 if unsupported)
151
+
calculate 15% of 80, we can multiply 80 by 15/100.
152
+
So, 80 * 15/100 = 12.
153
+
The answer is 12.<|im_end|>
154
+
I 00:00:22.764386 executorch:text_token_generator.h:130]
155
+
Reached to the end of generation
187
156
```
188
-
We can see that the ExecuTorch-trained adapter model does not have knowledge of the recent Nobel Prize winners, as neither the base model or adapter was trained on it. Meanwhile, the Nobel-prize adapter model can answer well.
189
-
190
-
There is about ~1.1GB memory increase between running the two models.
191
-
Most of that (about ~1GB) comes from embeddings that are not lowered to XNNPACK (and currently are not shared). This can be alleviated by quantizing the embeddings by adding the config `quantization.embedding_quantize=\'4,32\'` to the export command.
192
-
~50MB comes from the adapter model, which is not shared.
--prompt="Help me get started with ExecuTorch in 3 steps" \
205
-
--apply_chat_template
206
-
```
207
-
208
-
Sample output:
158
+
Sample output, base model:
209
159
```
160
+
I 00:00:22.764502 executorch:main.cpp:153] Generating with model ../../models/qwen3_06B_qq.pte...
161
+
I 00:00:23.136561 executorch:text_llm_runner.cpp:92] RSS after loading model: 2227.761719 MiB (0 if unsupported)
210
162
...
211
-
I 00:00:00.554048 executorch:main.cpp:133] Generating with model et.pte...
212
-
...
213
-
Here are 3 steps to get started with ExecuTorch:
163
+
<think>I 00:00:23.504387 executorch:text_llm_runner.cpp:188] RSS after prompt prefill: 2227.761719 MiB (0 if unsupported)
214
164
215
-
Step 1: Install ExecuTorch dependencies. This includes installing Python 3.8+ library, PyTorch library, and the ExecuTorch runtime.
165
+
Okay, so I need to calculate 15% of 80. Let me think about how to approach this. Hmm, percentages can sometimes be tricky, but I remember that percentages are essentially fractions with 100 as the denominator. So, 15% is the same as 15/100 or 0.15 in decimal form.
216
166
217
-
Step 2: Set up a Python environment with pip and a virtual environment (e.g., conda) to isolate ExecuTorch dependencies.
167
+
Alright, so if I have 80 and multiply it by 15%, then I can do that step by step. Let me write it out
168
+
I 00:00:49.814044 executorch:text_llm_runner.cpp:214] RSS after finishing text generation: 2227.761719 MiB (0 if unsupported)
169
+
```
170
+
We can see the base model is less capable than the lora model at mathematics.
218
171
219
-
Step 3: Clone the Execu
220
-
I 00:00:27.243400 executorch:text_llm_runner.cpp:206] RSS after finishing text generation: 6940.410156 MiB (0 if unsupported)
221
-
...
222
-
I 00:00:27.243504 executorch:main.cpp:141] Generating with model nobel.pte...
223
-
...
224
-
Here are the 3 steps to get started with Excetorch:
172
+
## Size savings
173
+
174
+
Size savings will vary depending on the model and LoRA configuration. By storing foundation weights in a separate, sharable PTD file, you can:
175
+
- Save disk space by avoiding duplicate weights across multiple LoRA models
176
+
- Save runtime memory by sharing weights through the XNNPACK weight cache
225
177
226
-
**Step 1: Install Node.js and npm**
178
+
### XNNPACK weight sharing
227
179
228
-
Excetorch is a JavaScript compiler, so you'll need Node.js and npm (the Node Package Manager) installed on your computer. You can download Node.js from the official website and npm from the npm website. Follow the installation instructions for your operating system.
180
+
The XNNPACK backend implements weight sharing via a global, singleton weight cache. At delegate init time, XNNPACK fetches weights from the weight cache. If they don't exist, weights are retrieved from the NamedDataMap (the API exposing weights in the PTD file), packs them, stores them in the weight cache, and frees the original. Because the backend is a singleton, these weights can be shared by completely separate runners, as in this example.
229
181
230
-
**Step 2: Install Excetorch**
182
+
### Runtime memory usage
183
+
See the runtime memory usage below.
231
184
185
+
- WS = weight sharing. This can be configured in `build_example.sh` by setting `-DEXECUTORCH_XNNPACK_ENABLE_WEIGHT_CACHE=TRUE/FALSE`.
186
+
- Model 1 = LoRA model.
187
+
- Model 2 = base model.
232
188
233
-
I 00:00:50.189743 executorch:text_llm_runner.cpp:206] RSS after finishing text generation: 8039.152344 MiB (0 if unsupported)
234
-
```
189
+
| Configuration | WS Enabled - Model 1 | WS Enabled - Model 2 (Total) | WS Disabled - Model 1 | WS Disabled - Model 2 (Total) |
The ExecuTorch-trained adapter model has domain knowledge of ExecuTorch codebase, whereas the Nobel-prize trained adapter model does not.
237
195
238
196
## Demo video
197
+
This demo video contains two lora adapters, with one trained on ExecuTorch documentation and another on recent Nobel Prize Winners, using Llama3 1B as the base model.
0 commit comments