Skip to content

Commit e422fff

Browse files
authored
Update readme.md
1 parent 868b93c commit e422fff

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

examples/LLMs/readme.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -482,7 +482,7 @@ wikitext perplexity 92795.3984375
482482

483483
### :rocket: Qwen/Qwen2-7B
484484

485-
The Qwen2-7b model has 28 heads with ``num_key_value_heads=4``. This limits the pruning ratio to be [1/7, 2/7, 3/7, 4/7, 5/7, 6/7] if you want to save and load the pruned model using hugingface transformers, since HF only supports the same in_features and out_features in the ``q_proj`` and ``o_proj``.
485+
The Qwen2-7b model has 28 heads with ``num_key_value_heads=4``. This limits the pruning ratio to be [1/7, 2/7, 3/7, 4/7, 5/7, 6/7] if you want to save and load the pruned model using Huggingface Transformers, since HF only supports ``in_features==out_features`` in the ``q_proj`` and ``o_proj``.
486486

487487
```bash
488488
# 3/7 ~ 0.428571428, this script will craft a 2B model

0 commit comments

Comments
 (0)