You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-1.md
+66-26Lines changed: 66 additions & 26 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,39 +1,46 @@
1
1
---
2
-
title: Convert model to gguf and quantize
2
+
title: Convert model to GGUP and quantize
3
3
weight: 2
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
+
8
9
## Overview
9
-
This example will run on three AWS Graviton4 c8g.16xlarge instances with 64 cores and 128GB of RAM. The instances should have 2TB disk storage, to store downloaded and quantized model weights.
10
10
11
-
You will perform these steps in this Learning Path:
11
+
This example runs on three AWS Graviton4 `c8g.16xlarge` instances. Each instance has 64 cores, 128 GB of RAM, and 2 TB of disk storage to store the downloaded and quantized model weights.
2. Download and build `llama.cpp`, a C++ library for efficient CPU inference of LLaMA and similar large language models on CPUs, optimized for local and embedded environments.
17
+
3. Convert Meta's `safetensors` files to a single GGUF file.
18
+
4. Quantize the 16-bit GGUF weights file to 4-bit weights.
2. Download and build llama.cpp, a C++ library that enables efficient inference of LLaMA and similar large language models on CPUs, optimized for local and embedded environments.
15
-
3. Convert Meta's safetensors files to a single gguf file.
16
-
4. Quantize the 16 bit gguf weights file to 4 bit weights.
17
-
5. Load and run the model.
21
+
{{% notice Note %}}
22
+
The **Reading time** shown on the Introduction page does not include downloading, converting, and quantizing the model. These steps can take more than six hours. If you already have a quantized GGUF file, you can skip the download and quantization.
23
+
{{% /notice %}}
18
24
19
-
{{% notice Note %}}The "reading time" mentioned on the Introduction page doesn't include downloading, converting, and requantizing the model. The process mentioned on this page will take 6+ hours. You may skip the model download and quantization if you have a quantized gguf file ready to use.{{% /notice %}}
25
+
## Set up dependencies
20
26
21
-
## Procedure
22
-
First, ensure you have permissions to access to Meta's [405B parameter llama 3.1 model](https://huggingface.co/meta-llama/Llama-3.1-405B).
27
+
Before you start, make sure you have permission to access Meta's [Llama 3.1 405B parameter model](https://huggingface.co/meta-llama/Llama-3.1-405B).
23
28
24
29
{{% notice Note %}}
25
-
Remember that you will need to replicate the install steps below on each device. Do NOT replicate the download and quantization step, llama.cpp will send the tensors to the cache.
30
+
You must repeat the install steps on each device. However, only run the download and quantization steps once. `llama.cpp` will cache the tensors for reuse across devices.
26
31
{{% /notice %}}
27
32
28
-
##### 1. Generate a virtual environment
33
+
##Create a virtual environment
29
34
30
35
```bash
31
36
apt update
32
37
apt install python3.12-venv
33
38
python3 -m venv myenv
34
39
source myenv/bin/activate
35
40
```
36
-
##### 2. Clone the llama.cpp repo and build dependencies
41
+
42
+
## Clone the llama.cpp repo and build dependencies
43
+
37
44
```bash
38
45
git clone https://github.com/ggerganov/llama.cpp
39
46
apt install -y cmake build-essential
@@ -45,54 +52,87 @@ cd build-rpc
45
52
cmake .. -DGGML_RPC=ON -DLLAMA_BUILD_SERVER=ON
46
53
cmake --build . --config Release
47
54
```
48
-
`llama.cpp` is now built in the `build-rpc/bin` directory.
49
-
Check that `llama.cpp` has built correctly by running the help command:
55
+
56
+
The build output is placed in the `build-rpc/bin` directory.
57
+
58
+
Verify that the build succeeded by running the help command:
59
+
50
60
```bash
51
61
cd build-rpc
52
62
bin/llama-cli -h
53
63
```
54
64
55
-
##### 3. Download the model (on a single instance)
56
-
Install Huggingface Hub in the virtual environment:
65
+
## Download the model (single instance only)
66
+
67
+
Install Hugging Face Hub in your virtual environment:
0 commit comments