Skip to content

Commit 059f10f

Browse files
authored
Update 2-quantize-model.md
1 parent f2235bf commit 059f10f

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

content/learning-paths/servers-and-cloud-computing/vllm-acceleration/2-quantize-model.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,17 +135,21 @@ This script creates a Arm KleidiAI INT4 quantized copy of the vLLM model and sav
135135
## Quantize DeepSeek‑V2‑Lite model
136136

137137
### Quantization parameter tuning
138+
Quantization parameters determine how the model’s floating-point weights and activations are converted into lower-precision integer formats. Choosing the right combination is essential for balancing model accuracy, memory footprint, and runtime throughput on Arm CPUs.
139+
138140
1. You can choose `minmax` (faster model quantization) or `mse` (more accurate but slower model quantization) method.
139141
2. `channelwise` is a good default for most models.
140142
3. `groupwise` can improve accuracy further; `--groupsize 32` is common.
141143

144+
Execute the following command to quantize the DeepSeek-V2-Lite model:
145+
142146
```bash
143147
# DeepSeek example
144148
python3 quantize_vllm_models.py deepseek-ai/DeepSeek-V2-Lite \
145149
--scheme channelwise --method mse
146150
```
147151

148-
The 4-bit quantized DeepSeek-V2-Lite will be stored the directory:
152+
This will generate an INT4 quantized model directory such as:
149153

150154
```text
151155
DeepSeek-V2-Lite-w4a8dyn-mse-channelwise

0 commit comments

Comments
 (0)