You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/vllm-acceleration/2-quantize-model.md
+14-8Lines changed: 14 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,33 +5,39 @@ weight: 3
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
+
## Accelerating LLMs with 4-bit Quantization
8
9
9
-
You can accelerate many LLMs on Arm CPUs with 4‑bit quantization. In this guide, we use `deepseek-ai/DeepSeek-V2-Lite` as the example model which gets accelerated by the INT4 path in vLLM using Arm KleidiAI microkernels.
10
+
You can accelerate many LLMs on Arm CPUs with 4‑bit quantization. In this section, you’ll quantize the deepseek-ai/DeepSeek-V2-Lite model to 4-bit integer (INT4) weights.
11
+
The quantized model runs efficiently through vLLM’s INT4 inference path, which is accelerated by Arm KleidiAI microkernels.
10
12
11
13
## Install quantization tools
12
14
13
-
Install the vLLM model quantization packages
15
+
Install the quantization dependencies used by vLLM and the llmcompressor toolkit:
14
16
15
17
```bash
16
18
pip install --no-deps compressed-tensors
17
19
pip install llmcompressor
18
20
```
19
-
20
-
Reinstall your locally built vLLM if you rebuilt it:
21
+
* compressed-tensors provides the underlying tensor storage and compression utilities used for quantized model formats.
22
+
* llmcompressor includes quantization, pruning, and weight clustering utilities compatible with Hugging Face Transformers and vLLM runtime formats.
23
+
24
+
If you recently rebuilt vLLM, reinstall your locally built wheel to ensure compatibility with the quantization extensions:
21
25
22
26
```bash
23
27
pip install --no-deps dist/*.whl
24
28
```
25
29
26
-
If your chosen model is gated on Hugging Face, authenticate first:
30
+
Authenticate with Hugging Face (if required):
31
+
32
+
If the model you plan to quantize is gated on Hugging Face (e.g., DeepSeek or proprietary models), log in to authenticate your credentials before downloading model weights:
27
33
28
34
```bash
29
35
huggingface-cli login
30
36
```
31
37
32
-
## INT4 Quantization recipe
38
+
## INT4 Quantization Recipe
33
39
34
-
Save the following as`quantize_vllm_models.py`:
40
+
Using a file editor of your choice, save the following code into a file named`quantize_vllm_models.py`:
35
41
36
42
```python
37
43
import argparse
@@ -124,7 +130,7 @@ if __name__ == "__main__":
124
130
main()
125
131
```
126
132
127
-
This script creates a Arm KleidiAI 4‑bit quantized copy of the vLLM model and saves it to a new directory.
133
+
This script creates a Arm KleidiAI INT4 quantized copy of the vLLM model and saves it to a new directory.
0 commit comments