@@ -28,11 +28,11 @@ pip install huggingface_hub hf_xet
2828This command installs:
2929
3030- ` huggingface_hub ` : Python client for downloading models and datasets
31- - ` hf_xet ` : Git extension for fetching large model files stored on Hugging Face
31+ - ` hf_xet ` : Git extension for fetching large model files hosted on Hugging Face
3232
3333These tools include the ` hf ` command-line interface you'll use next.
3434
35- ## Login to the Hugging Face Hub
35+ ## Log in to the Hugging Face Hub
3636
3737``` bash
3838hf auth login
@@ -86,7 +86,7 @@ This command creates a 4-bit quantized version of the model:
8686- ` llama-quantize ` is the quantization tool from Llama.cpp.
8787- ` afm-4-5B-F16.gguf ` is the input GGUF model file in 16-bit precision.
8888- ` Q4_0 ` applies zero-point 4-bit quantization.
89- - This reduces the model size by approximately 45 % (from ~ 15GB to ~ 8GB ).
89+ - This reduces the model size by approximately ~ 70 % (from ~ 15GB to ~ 4.4GB ).
9090- The quantized model will use less memory and run faster, though with a small reduction in accuracy.
9191- The output file will be ` afm-4-5B-Q4_0.gguf ` .
9292
@@ -104,7 +104,7 @@ bin/llama-quantize models/afm-4-5b/afm-4-5B-F16.gguf models/afm-4-5b/afm-4-5B-Q8
104104
105105This command creates an 8-bit quantized version of the model:
106106- ` Q8_0 ` specifies 8-bit quantization with zero-point compression.
107- - This reduces the model size by approximately 70 % (from ~ 15GB to ~ 4.4GB ).
107+ - This reduces the model size by approximately ~ 45 % (from ~ 15GB to ~ 8GB ).
108108- The 8-bit version provides a better balance between memory usage and accuracy than 4-bit quantization.
109109- The output file is named ` afm-4-5B-Q8_0.gguf ` .
110110- Commonly used in production scenarios where memory resources are available.
0 commit comments