Skip to content

Commit 9b62c04

Browse files
- Applied Microsoft Learn style (sentence case, active voice, concise phrasing)
- Capitalized GGUF consistently for correctness - Smoothed instructional flow and reformatted steps - Added "Next steps" section with follow-on guidance - Preserved YAML metadata and structure
1 parent 5591966 commit 9b62c04

1 file changed

Lines changed: 66 additions & 26 deletions

File tree

  • content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp

content/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-1.md

Lines changed: 66 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,46 @@
11
---
2-
title: Convert model to gguf and quantize
2+
title: Convert model to GGUP and quantize
33
weight: 2
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
8+
89
## Overview
9-
This example will run on three AWS Graviton4 c8g.16xlarge instances with 64 cores and 128GB of RAM. The instances should have 2TB disk storage, to store downloaded and quantized model weights.
1010

11-
You will perform these steps in this Learning Path:
11+
This example runs on three AWS Graviton4 `c8g.16xlarge` instances. Each instance has 64 cores, 128 GB of RAM, and 2 TB of disk storage to store the downloaded and quantized model weights.
12+
13+
In this Learning Path, you will:
14+
15+
1. Download Meta's [Llama 3.1 405B parameter model](https://huggingface.co/meta-llama/Llama-3.1-405B).
16+
2. Download and build `llama.cpp`, a C++ library for efficient CPU inference of LLaMA and similar large language models on CPUs, optimized for local and embedded environments.
17+
3. Convert Meta's `safetensors` files to a single GGUF file.
18+
4. Quantize the 16-bit GGUF weights file to 4-bit weights.
19+
5. Load and run the model.
1220

13-
1. Download Meta's [405B parameter llama 3.1 model](https://huggingface.co/meta-llama/Llama-3.1-405B).
14-
2. Download and build llama.cpp, a C++ library that enables efficient inference of LLaMA and similar large language models on CPUs, optimized for local and embedded environments.
15-
3. Convert Meta's safetensors files to a single gguf file.
16-
4. Quantize the 16 bit gguf weights file to 4 bit weights.
17-
5. Load and run the model.
21+
{{% notice Note %}}
22+
The **Reading time** shown on the Introduction page does not include downloading, converting, and quantizing the model. These steps can take more than six hours. If you already have a quantized GGUF file, you can skip the download and quantization.
23+
{{% /notice %}}
1824

19-
{{% notice Note %}}The "reading time" mentioned on the Introduction page doesn't include downloading, converting, and requantizing the model. The process mentioned on this page will take 6+ hours. You may skip the model download and quantization if you have a quantized gguf file ready to use.{{% /notice %}}
25+
## Set up dependencies
2026

21-
## Procedure
22-
First, ensure you have permissions to access to Meta's [405B parameter llama 3.1 model](https://huggingface.co/meta-llama/Llama-3.1-405B).
27+
Before you start, make sure you have permission to access Meta's [Llama 3.1 405B parameter model](https://huggingface.co/meta-llama/Llama-3.1-405B).
2328

2429
{{% notice Note %}}
25-
Remember that you will need to replicate the install steps below on each device. Do NOT replicate the download and quantization step, llama.cpp will send the tensors to the cache.
30+
You must repeat the install steps on each device. However, only run the download and quantization steps once. `llama.cpp` will cache the tensors for reuse across devices.
2631
{{% /notice %}}
2732

28-
##### 1. Generate a virtual environment
33+
## Create a virtual environment
2934

3035
```bash
3136
apt update
3237
apt install python3.12-venv
3338
python3 -m venv myenv
3439
source myenv/bin/activate
3540
```
36-
##### 2. Clone the llama.cpp repo and build dependencies
41+
42+
## Clone the llama.cpp repo and build dependencies
43+
3744
```bash
3845
git clone https://github.com/ggerganov/llama.cpp
3946
apt install -y cmake build-essential
@@ -45,54 +52,87 @@ cd build-rpc
4552
cmake .. -DGGML_RPC=ON -DLLAMA_BUILD_SERVER=ON
4653
cmake --build . --config Release
4754
```
48-
`llama.cpp` is now built in the `build-rpc/bin` directory.
49-
Check that `llama.cpp` has built correctly by running the help command:
55+
56+
The build output is placed in the `build-rpc/bin` directory.
57+
58+
Verify that the build succeeded by running the help command:
59+
5060
```bash
5161
cd build-rpc
5262
bin/llama-cli -h
5363
```
5464

55-
##### 3. Download the model (on a single instance)
56-
Install Huggingface Hub in the virtual environment:
65+
## Download the model (single instance only)
66+
67+
Install Hugging Face Hub in your virtual environment:
68+
5769
```bash
5870
pip3 install huggingface_hub
59-
6071
```
61-
Make a python file and name it download.py:
72+
73+
Create a new Python file named `download.py`:
74+
6275
```bash
6376
vi download.py
6477
```
65-
Write the following code to it:
78+
79+
Add the following code:
80+
6681
```python
6782
import os
6883
from huggingface_hub import snapshot_download
84+
6985
model_id = "meta-llama/Llama-3.1-405B"
7086
local_dir = "llama-hf"
87+
7188
# Create the directory if it doesn't exist
7289
os.makedirs(local_dir, exist_ok=True)
90+
7391
# Download the model snapshot
7492
snapshot_download( repo_id=model_id, local_dir=local_dir,
7593
revision="main",
7694
token="your_hf_token",
7795
allow_patterns=["*.md", "*.json", "*.safetensors"]
7896
)
7997
```
80-
Execute the file:
98+
99+
Run the script:
100+
81101
```bash
82102
python3 download.py
83103
```
84-
##### 4. Convert the model from .safetensors to gguf and quantize (on a single instance)
85-
Following lines installs the files important for conversion to .gguf format.
104+
105+
## Convert and quantize the model (single instance only)
106+
107+
Install the conversion dependencies:
108+
86109
```bash
87110
pip3 install -r llama.cpp/requirements.txt
111+
```
112+
113+
Convert the model:
114+
115+
```bash
88116
python3 llama.cpp/convert_hf_to_gguf.py llama-hf
117+
```
118+
119+
Quantize the model to 4-bit weights:
120+
121+
```bash
89122
cd llama.cpp/build-rpc
90-
bin/llama-quantize ../../llama-hf/llama-3.1-405B-F16.gguf Q4_0
123+
bin/llama-quantize ../../llama-hf/llama-3.1-405B-F16.GGUF Q4_0
91124
```
92-
You may rename the resultant file to model.gguf and use it. There are different quantization options as well, as shown below:
125+
126+
You can rename the output file to `model.GGUF` for easier use.
127+
128+
Check available quantization options:
129+
93130
```bash
94131
bin/llama-quantize -h
95132
```
133+
134+
This command lists supported quantization formats and options. For example:
135+
96136
```output
97137
usage: bin/llama-quantize [--help] [--allow-requantize] [--leave-output-tensor] [--pure] [--imatrix] [--include-weights] [--exclude-weights] [--output-tensor-type]
98138
[--token-embedding-type] [--tensor-type] [--keep-split] [--override-kv] model-f32.gguf [model-quant.gguf] type [nthreads]

0 commit comments

Comments
 (0)