Skip to content

Commit da553c4

Browse files
authored
Add the information about 1.25bit STQ1_0 kernel (#298)
1 parent f76d316 commit da553c4

3 files changed

Lines changed: 74 additions & 1 deletion

File tree

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ A more accessible, comprehensive, and efficient toolkit for large model compress
2222
</p>
2323

2424
## 📣Latest News
25+
- [26/05/08] We have released STQ1_0 kernel for 1.25-bit model and given a PR to llama.cpp [PR #22836](https://github.com/ggml-org/llama.cpp/pull/22836) ! If you have any questions or suggestions for STQ_0, welcome to comment under the PR !🔥🔥🔥
2526
- [26/04/29] We have released 2-bit and 1.25-bit versions of Tencent Hy-MT1.5-1.8B Translation Model: [Hy-MT1.5-1.8B-2bit](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-2bit) and [Hy-MT1.5-1.8B-1.25bit](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-1.25bit). Additionally, we have make an [offline translation demo](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-1.25bit/blob/main/Hy-MT-demo.apk) for you to try out. We invite you to give it a spin! 🔥🔥🔥
2627
- [26/04/23] We now support FP8-Static quantization for **Hy3-preview** (MoE A20B).
2728
- [26/03/25] We have released **DAQ**, the quantization algorithm that preserves the knowledge acquired while the update of parameters is relatively small during post-training training.[[Paper]](https://arxiv.org/abs/2603.22324) | [[Docs]](docs/source/features/quantization/daq.md)

README_cn.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
</p>
2323

2424
## 📣最新进展
25+
- [26/05/08] 我们发布了用于 1.25-bit 模型的 STQ1_0 内核,并向 llama.cpp 提交了 [PR #22836](https://github.com/ggml-org/llama.cpp/pull/22836)!如果您对 STQ_0 有任何疑问或建议,欢迎在该 PR 下留言!🔥🔥🔥
2526
- [26/04/29] 我们发布了 2bit 与 1.25bit 腾讯混元翻译模型 [Hy-MT1.5-1.8B-2bit](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-2bit), [Hy-MT1.5-1.8B-1.25bit](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-1.25bit)。并且还制作了 [离线翻译体验 Demo](https://huggingface.co/AngelSlim/Hy-MT1.5-1.8B-1.25bit/blob/main/Hy-MT-demo.apk)。 欢迎体验 🔥🔥🔥
2627
- [26/04/23] 我们支持了 **Hy3-preview**(MoE A20B)模型的 FP8-Static 量化。
2728
- [26/03/25] 我们发布了量化算法DAQ,该方法在后训练参数更新较小时,可保留量化后模型能力 [[论文]](https://arxiv.org/abs/2603.22324) | [[文档]](docs/source/features/quantization/daq.md)

docs/source/models/Hy-MT1.5/hy-mt1.5.md

Lines changed: 72 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,78 @@ Demo device: Snapdragon 7+ Gen 2, 16GB RAM.
100100
:::
101101

102102
## 💻 Deployment
103-
Our llama.cpp kernel (including STQ kernel) is coming soon.
103+
104+
### Clone llama.cpp
105+
106+
```bash
107+
git clone https://github.com/ggml-org/llama.cpp.git
108+
```
109+
110+
### Enter the llama.cpp folder
111+
112+
```bash
113+
cd llama.cpp
114+
```
115+
116+
### Fetch and check out the PR branch
117+
118+
```bash
119+
git fetch origin pull/22836/head:pr-22836-stq_0
120+
git checkout pr-22836-stq_0
121+
```
122+
123+
### Build llama.cpp
124+
125+
```bash
126+
pip install -r requirements.txt
127+
cmake -B build
128+
cmake --build build --config Release
129+
```
130+
131+
### Download the HF model
132+
133+
134+
```bash
135+
pip install huggingface_hub
136+
huggingface-cli download AngelSlim/Hy-MT1.5-1.8B-1.25bit \
137+
--local-dir model_zoo/Hy-MT1.5-1.8B-1.25bit
138+
```
139+
140+
### Convert HF → bf16 GGUF
141+
142+
```bash
143+
python convert_hf_to_gguf.py model_zoo/Hy-MT1.5-1.8B-1.25bit \
144+
--outfile model_zoo/Hy-MT1.5-1.8B-bf16.gguf \
145+
--outtype bf16
146+
```
147+
148+
### Quantize bf16 → STQ1_0
149+
150+
```bash
151+
./build/bin/llama-quantize \
152+
model_zoo/Hy-MT1.5-1.8B-bf16.gguf \
153+
model_zoo/Hy-MT1.5-1.8B-STQ1_0.gguf \
154+
STQ1_0
155+
```
156+
157+
### Run a completion example
158+
159+
The prompt format can be viewed at [HY-MT1.5-1.8B](https://huggingface.co/tencent/HY-MT1.5-1.8B)
160+
161+
```bash
162+
./build/bin/llama-completion \
163+
--model model_zoo/Hy-MT1.5-1.8B-STQ1_0.gguf \
164+
-p "Translate the following segment into Chinese, without additional explanation. Hello " \
165+
--jinja \
166+
-ngl 0 \
167+
-n 64 -st
168+
```
169+
170+
### Run the llama.cpp benchmark
171+
172+
```bash
173+
./build/bin/llama-bench -m model_zoo/Hy-MT1.5-1.8B-STQ1_0.gguf -ngl 0
174+
```
104175

105176
## 📥 Download Links
106177

0 commit comments

Comments
 (0)