Skip to content

Commit f2235bf

Browse files
authored
Update 2-quantize-model.md
1 parent 27311af commit f2235bf

1 file changed

Lines changed: 14 additions & 8 deletions

File tree

content/learning-paths/servers-and-cloud-computing/vllm-acceleration/2-quantize-model.md

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,33 +5,39 @@ weight: 3
55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
8+
## Accelerating LLMs with 4-bit Quantization
89

9-
You can accelerate many LLMs on Arm CPUs with 4‑bit quantization. In this guide, we use `deepseek-ai/DeepSeek-V2-Lite` as the example model which gets accelerated by the INT4 path in vLLM using Arm KleidiAI microkernels.
10+
You can accelerate many LLMs on Arm CPUs with 4‑bit quantization. In this section, you’ll quantize the deepseek-ai/DeepSeek-V2-Lite model to 4-bit integer (INT4) weights.
11+
The quantized model runs efficiently through vLLM’s INT4 inference path, which is accelerated by Arm KleidiAI microkernels.
1012

1113
## Install quantization tools
1214

13-
Install the vLLM model quantization packages
15+
Install the quantization dependencies used by vLLM and the llmcompressor toolkit:
1416

1517
```bash
1618
pip install --no-deps compressed-tensors
1719
pip install llmcompressor
1820
```
19-
20-
Reinstall your locally built vLLM if you rebuilt it:
21+
* compressed-tensors provides the underlying tensor storage and compression utilities used for quantized model formats.
22+
* llmcompressor includes quantization, pruning, and weight clustering utilities compatible with Hugging Face Transformers and vLLM runtime formats.
23+
24+
If you recently rebuilt vLLM, reinstall your locally built wheel to ensure compatibility with the quantization extensions:
2125

2226
```bash
2327
pip install --no-deps dist/*.whl
2428
```
2529

26-
If your chosen model is gated on Hugging Face, authenticate first:
30+
Authenticate with Hugging Face (if required):
31+
32+
If the model you plan to quantize is gated on Hugging Face (e.g., DeepSeek or proprietary models), log in to authenticate your credentials before downloading model weights:
2733

2834
```bash
2935
huggingface-cli login
3036
```
3137

32-
## INT4 Quantization recipe
38+
## INT4 Quantization Recipe
3339

34-
Save the following as `quantize_vllm_models.py`:
40+
Using a file editor of your choice, save the following code into a file named `quantize_vllm_models.py`:
3541

3642
```python
3743
import argparse
@@ -124,7 +130,7 @@ if __name__ == "__main__":
124130
main()
125131
```
126132

127-
This script creates a Arm KleidiAI 4‑bit quantized copy of the vLLM model and saves it to a new directory.
133+
This script creates a Arm KleidiAI INT4 quantized copy of the vLLM model and saves it to a new directory.
128134

129135
## Quantize DeepSeek‑V2‑Lite model
130136

0 commit comments

Comments
 (0)