Skip to content

Commit 613fc03

Browse files
authored
Merge pull request #2395 from parichaydas/FTLLM_New
newLP
2 parents fcebbd7 + 992a223 commit 613fc03

10 files changed

Lines changed: 577 additions & 0 deletions

File tree

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
title: Customer Support Chatbot with Llama and ExecuTorch on Arm-Based Mobile Devices (with Agentic AI Capabilities)
3+
4+
draft: true
5+
cascade:
6+
draft: true
7+
8+
minutes_to_complete: 60
9+
10+
who_is_this_for: This learning plan is designed for developers with basic knowledge of Python, Mobile development, and machine learning concepts.It guides you through creating an on-device customer support chatbot using Meta's Llama models deployed via PyTorch's ExecuTorch runtime.The focus is on Arm-based Android devices.The chatbot will handle common customer queries (e.g., product info, troubleshooting) with low latency, privacy (no cloud dependency), and optimized performance.Incorporates agentic AI capabilities, transforming the chatbot from reactive (simple Q&A) to proactive and autonomous. Agentic AI enables the bot to plan multi-step actions, use external tools,reason over user intent, and adapt responses dynamically. This is achieved by extending the core LLM with tool-calling mechanisms and multi-agent orchestration.
11+
12+
learning_objectives:
13+
- Explain the architecture and capabilities of Llama models (e.g., Llama 3.2 1B/3B) for mobile use.
14+
- Master the process of quantizing LLMs (e.g., 4-bit PTQ) to reduce model size and enable efficient inference on resource-constrained mobile devices.
15+
- Gain proficiency in using ExecuTorch to export PyTorch models to .pte format for on-device deployment.
16+
- Learn to leverage Arm-specific optimizations (e.g., XNNPACK, KleidiAI) to achieve 2-3x faster inference on Arm-based Android devices.
17+
- Implement real-time inference with Llama models, enabling seamless customer support interactions (e.g., handling FAQs, troubleshooting).
18+
19+
prerequisites:
20+
- Basic Understanding of Machine Learning & Deep Learning (Familiarity with concepts like supervised learning, neural networks, transfer learning and Understanding of model training, validation, & overfitting concepts).
21+
- Familiarity with Deep Learning Frameworks (Experience with PyTorch for building, training neural networks and Knowledge of Hugging Face Transformers for working with pre-trained LLMs.
22+
- An Arm-powered smartphone with the i8mm feature running Android, with 16GB of RAM.
23+
- A USB cable to connect your smartphone to your development machine.
24+
- An AWS Graviton4 r8g.16xlarge instance to test Arm performance optimizations, or any [Arm based instance](/learning-paths/servers-and-cloud-computing/csp/) from a cloud service provider or an on-premise Arm server or Arm based laptop.
25+
- Android Debug Bridge (adb) installed on your device. Follow the steps in [adb](https://developer.android.com/tools/adb) to install Android SDK Platform Tools. The adb tool is included in this package.
26+
- Java 17 JDK. Follow the steps in [Java 17 JDK](https://www.oracle.com/java/technologies/javase/jdk17-archive-downloads.html) to download and install JDK for host.
27+
- Android Studio. Follow the steps in [Android Studio](https://developer.android.com/studio) to download and install Android Studio for host.
28+
- Python 3.10.
29+
30+
author: Parichay Das
31+
32+
### Tags
33+
skilllevels: Introductory
34+
subjects: ML
35+
armips:
36+
- Neoverse
37+
38+
tools_software_languages:
39+
- LLM
40+
- GenAI
41+
- Python
42+
- PyTorch
43+
- ExecuTorch
44+
operatingsystems:
45+
- Linux
46+
- Windows
47+
- Android
48+
49+
50+
further_reading:
51+
- resource:
52+
title: Hugging Face Documentation
53+
link: https://huggingface.co/docs
54+
type: documentation
55+
- resource:
56+
title: PyTorch Documentation
57+
link: https://pytorch.org/docs/stable/index.html
58+
type: documentation
59+
- resource:
60+
title: Android
61+
link: https://www.android.com/
62+
type: website
63+
64+
65+
### FIXED, DO NOT MODIFY
66+
# ================================================================================
67+
weight: 1 # _index.md always has weight of 1 to order correctly
68+
layout: "learningpathall" # All files under learning paths have this same wrapper
69+
learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
70+
---
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
# ================================================================================
3+
# FIXED, DO NOT MODIFY THIS FILE
4+
# ================================================================================
5+
weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation.
6+
title: "Next Steps" # Always the same, html page title.
7+
layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing.
8+
---
Loading
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
title: Overview
3+
weight: 2
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Understanding Llama: Meta’s Large Language Model
10+
Llama is a family of large language models trained using publicly available datasets. These models demonstrate strong performance across a range of natural language processing (NLP) tasks, including language translation, question answering, and text summarization.
11+
12+
In addition to their analytical capabilities, Llama models can generate human-like, coherent, and contextually relevant text, making them highly effective for applications that rely on natural language generation. Consequently, they serve as powerful tools in areas such as chatbots, virtual assistants, and language translation, as well as in creative and content-driven domains where producing natural and engaging text is essential.
13+
14+
Please note that the models are subject to the [acceptable use policy](https://github.com/meta-llama/llama/blob/main/USE_POLICY.md) and this [responsible use guide](https://github.com/meta-llama/llama/blob/main/RESPONSIBLE_USE_GUIDE.md) .
15+
16+
17+
18+
## Quantization
19+
A practical approach to make models fit within smartphone memory constraints is through 4-bit groupwise per-token dynamic quantization of all linear layers. In this technique, dynamic quantization is applied to activations—meaning the quantization parameters are computed at runtime based on the observed minimum and maximum activation values. Meanwhile, the model weights are statically quantized, where each channel is quantized in groups using 4-bit signed integers. This method significantly reduces memory usage while maintaining model performance for on-device inference.
20+
21+
This method ensures efficient memory usage while maintaining model performance on resource-constrained devices.
22+
23+
For further information, refer to [torchao: PyTorch Architecture Optimization](https://github.com/pytorch-labs/ao/).
24+
25+
The table below evaluates WikiText perplexity using [LM Eval](https://github.com/EleutherAI/lm-evaluation-harness).
26+
27+
The results are for two different groupsizes, with max_seq_len 2048, and 1000 samples:
28+
29+
|Model | Baseline (FP32) | Groupwise 4-bit (128) | Groupwise 4-bit (256)
30+
|--------|-----------------| ---------------------- | ---------------
31+
|Llama 2 7B | 9.2 | 10.2 | 10.7
32+
|Llama 3 8B | 7.9 | 9.4 | 9.7
33+
34+
Note that groupsize less than 128 was not enabled in this example, since the model was still too large. This is because current efforts have focused on enabling FP32, and support for FP16 is under way.
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
---
2+
title: Environment Setup
3+
weight: 3
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Android NDK and Android Studio - Environment Setup
10+
11+
#### Plartform Required
12+
- An AWS Graviton4 r8g.16xlarge instance to test Arm performance optimizations, or any [Arm based instance](/learning-paths/servers-and-cloud-computing/csp/) from a cloud service provider or an on-premise Arm server or Arm based laptop.
13+
- An Arm-powered smartphone with the i8mm feature running Android, with 16GB of RAM.
14+
- A USB cable to connect your smartphone to your development machine.
15+
16+
The installation and configuration of Android Studio can be accomplished through the following steps:
17+
1. Download and install the latest version of [Android Studio](https://developer.android.com/studio).
18+
2. Launch Android Studio and access the Settings dialog.
19+
3. Navigate to Languages & Frameworks → Android SDK.
20+
4. Under the SDK Platforms tab, ensure that Android 14.0 (“UpsideDownCake”) is selected.
21+
22+
Next, proceed to install the required version of the Android NDK by first setting up the Android Command Line Tools.
23+
Linux:
24+
```bash
25+
curl https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip -o commandlinetools.zip
26+
unzip commandlinetools.zip
27+
./commandlinetools/bin/sdkmanager --install "ndk;26.1.10909697"
28+
```
29+
Install the NDK in the same directory where Android Studio has installed the SDK, which is typically located at ~/Library/Android/sdk by default. Then, configure the necessary environment variables as follows:
30+
```bash
31+
export ANDROID_HOME="$(realpath ~/Library/Android/sdk)"
32+
export PATH=$ANDROID_HOME/cmdline-tools/bin/:$PATH
33+
sdkmanager --sdk_root="${ANDROID_HOME}" --install "ndk;28.0.12433566"
34+
export ANDROID_NDK=$ANDROID_HOME/ndk/28.0.12433566/
35+
```
36+
37+
#### Install Java 17 JDK
38+
1. Open the Java SE 17 Archive [Downloads](https://www.oracle.com/java/technologies/javase/jdk17-archive-downloads.html) page in your browser.
39+
2. Select an appropriate download for your development machine operating system.
40+
41+
#### Install Git and cmake
42+
```bash
43+
sudo apt-get install git cmake
44+
```
45+
46+
#### Install Python 3.10
47+
```bash
48+
sudo apt-get install python3.10
49+
```
50+
51+
#### Set up ExecuTorch
52+
ExecuTorch is an end-to-end framework designed to facilitate on-device inference across a wide range of mobile and edge platforms, including wearables, embedded systems, and microcontrollers. As a component of the PyTorch Edge ecosystem, it streamlines the efficient deployment of PyTorch models on edge devices. For further details, refer to the [ExecuTorch Overview](https://pytorch.org/executorch/stable/overview/).
53+
54+
It is recommended to create an isolated Python environment to install the ExecuTorch dependencies. Instructions are available for setting up either a Python virtual environment or a Conda virtual environment—you only need to choose one of these options.
55+
56+
##### Install Required Tools ( Python environment setup)
57+
```python
58+
python3 -m venv exec_env
59+
source exec_env/bin/activate
60+
pip install torch torchvision torchaudio
61+
pip install executorch
62+
```
63+
##### Clone Required Repositories
64+
```bash
65+
git clone https://github.com/pytorch/executorch.git
66+
git clone https://github.com/pytorch/text.git
67+
```
68+
##### Download Pretrained Model (Llama 3.1 Instruct)
69+
Download the quantized model weights optimized for mobile deployment from either the Meta AI Hub or Hugging Face.
70+
```
71+
git lfs install
72+
git clone https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
73+
```
74+
75+
##### Verify Arm SDK Path
76+
```
77+
ANDROID_SDK_ROOT=/Users/<you>/Library/Android/sdk
78+
ANDROID_NDK_HOME=$ANDROID_SDK_ROOT/ndk/26.1.10909125
79+
```
Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
---
2+
title: Model Preparation and Conversion
3+
weight: 4
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
To begin working with Llama 3, the pre-trained model parameters can be accessed through Meta’s Llama Downloads page. Users are required to request access by submitting their details and reviewing and accepting the Responsible Use Guide. Upon approval, a license and a download link—valid for 24 hours—are provided. For this exercise, the Llama 3.2 1B Instruct model is utilized; however, the same procedures can be applied to other available variants with only minor modifications.
10+
11+
Convert the model into an ExecuTorch-compatible format optimized for Arm devices
12+
## Script the Model
13+
14+
```python
15+
import torch
16+
from transformers import AutoModelForCausalLM
17+
18+
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", torch_dtype=torch.float16)
19+
scripted_model = torch.jit.script(model)
20+
scripted_model.save("llama_exec.pt")
21+
22+
```
23+
24+
Install the llama-stack package from pip.
25+
```python
26+
pip install llama-stack
27+
```
28+
29+
Run the command to download, and paste the download link from the email when prompted.
30+
```python
31+
llama model download --source meta --model-id Llama3.2-1B-Instruct
32+
```
33+
34+
When the download is finished, the installation path is printed as output.
35+
```python
36+
Successfully downloaded model to /<path-to-home>/.llama/checkpoints/Llama3.2-1B-Instruct
37+
```
38+
Verify by viewing the downloaded files under this path:
39+
```
40+
ls $HOME/.llama/checkpoints/Llama3.2-1B-Instruct
41+
checklist.chk consolidated.00.pth params.json tokenizer.model
42+
43+
```
44+
45+
Export the model and generate a .pte file by running the appropriate Python command. This command will export the model and save the resulting file in your current working directory.
46+
```python
47+
python3 -m examples.models.llama.export_llama \
48+
--checkpoint $HOME/.llama/checkpoints/Llama3.2-1B-Instruct/consolidated.00.pth \
49+
--params $HOME/.llama/checkpoints/Llama3.2-1B-Instruct/params.json \
50+
-kv --use_sdpa_with_kv_cache -X --xnnpack-extended-ops -qmode 8da4w \
51+
--group_size 64 -d fp32 \
52+
--metadata '{"get_bos_id":128000, "get_eos_ids":[128009, 128001, 128006, 128007]}' \
53+
--embedding-quantize 4,32 \
54+
--output_name="llama3_1B_kv_sdpa_xnn_qe_4_64_1024_embedding_4bit.pte" \
55+
--max_seq_length 1024 \
56+
--max_context_length 1024
57+
```
58+
59+
Because Llama 3 has a larger vocabulary size, it is recommended to quantize the embeddings using the parameter --embedding-quantize 4,32. This helps to further optimize memory usage and reduce the overall model size.
60+
61+
62+
###### Load a pre-fine-tuned model (from Hugging Face)
63+
- Example: meta-llama/Llama-3-8B-Instruct or a customer-support fine-tuned variant
64+
65+
###### Model Optimization for ARM (Understanding Quantization)
66+
- Reduces model precision (e.g., 32-bit → 8-bit)
67+
- Decreases memory footprint (~4x reduction)
68+
- Speeds up inference on CPU
69+
- Minimal accuracy loss for most tasks
70+
71+
###### Apply Dynamic Quantization
72+
- Create optimize_model.py
73+
74+
```python
75+
import torch
76+
from transformers import AutoModelForCausalLM, AutoTokenizer
77+
from torch.quantization import quantize_dynamic
78+
import time
79+
import os
80+
81+
def load_base_model(model_name):
82+
"""Load the base model"""
83+
print(f"Loading base model: {model_name}")
84+
85+
tokenizer = AutoTokenizer.from_pretrained(model_name)
86+
tokenizer.pad_token = tokenizer.eos_token
87+
88+
model = AutoModelForCausalLM.from_pretrained(
89+
model_name,
90+
torch_dtype=torch.float32,
91+
device_map=None,
92+
low_cpu_mem_usage=True
93+
)
94+
model.eval()
95+
96+
return model, tokenizer
97+
98+
def apply_quantization(model):
99+
"""Apply dynamic quantization"""
100+
print("Applying dynamic quantization...")
101+
102+
quantized_model = quantize_dynamic(
103+
model,
104+
{torch.nn.Linear}, # Quantize linear layers
105+
dtype=torch.qint8
106+
)
107+
108+
return quantized_model
109+
110+
def test_model(model, tokenizer, prompt):
111+
"""Test model with a sample prompt"""
112+
inputs = tokenizer(prompt, return_tensors="pt")
113+
114+
start_time = time.time()
115+
with torch.no_grad():
116+
outputs = model.generate(
117+
inputs.input_ids,
118+
max_new_tokens=100,
119+
do_sample=False,
120+
pad_token_id=tokenizer.eos_token_id
121+
)
122+
inference_time = time.time() - start_time
123+
124+
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
125+
126+
return response, inference_time
127+
128+
def main():
129+
model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
130+
131+
# Load base model
132+
base_model, tokenizer = load_base_model(model_name)
133+
134+
# Test base model
135+
test_prompt = "How do I track my order?"
136+
print("\nTesting base model...")
137+
response, base_time = test_model(base_model, tokenizer, test_prompt)
138+
print(f"Base model inference time: {base_time:.2f}s")
139+
140+
# Apply quantization
141+
quantized_model = apply_quantization(base_model)
142+
143+
# Test quantized model
144+
print("\nTesting quantized model...")
145+
response, quant_time = test_model(quantized_model, tokenizer, test_prompt)
146+
print(f"Quantized model inference time: {quant_time:.2f}s")
147+
print(f"Speedup: {base_time / quant_time:.2f}x")
148+
149+
# Save quantized model
150+
save_dir = "./models/quantized_llama3"
151+
os.makedirs(save_dir, exist_ok=True)
152+
153+
torch.save(quantized_model.state_dict(), f"{save_dir}/model.pt")
154+
tokenizer.save_pretrained(save_dir)
155+
156+
print(f"\nQuantized model saved to: {save_dir}")
157+
158+
if __name__ == "__main__":
159+
main()
160+
161+
```
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
title: Building the Chatbot Logic
3+
4+
### FIXED, DO NOT MODIFY
5+
layout: learningpathall
6+
---
7+
8+
## Conversation Framework (Python prototype)
9+
```python
10+
from transformers import AutoTokenizer
11+
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
12+
13+
def generate_response(model, query, context):
14+
prompt = f"### Context:\n{context}\n### User Query:\n{query}\n### Assistant Response:"
15+
inputs = tokenizer(prompt, return_tensors="pt")
16+
outputs = model.generate(**inputs, max_new_tokens=200)
17+
return tokenizer.decode(outputs[0], skip_special_tokens=True)
18+
```
19+
20+
###### Context Memory (Simple JSON Store)
21+
22+
```python
23+
import json
24+
25+
def update_memory(user_id, query, response):
26+
memory = json.load(open("chat_memory.json", "r"))
27+
memory[user_id].append({"query": query, "response": response})
28+
json.dump(memory, open("chat_memory.json", "w"))
29+
30+
```
31+

0 commit comments

Comments
 (0)