Skip to content

Commit 2a8ae7e

Browse files
authored
add qwen3-omni (#121)
1 parent 02eaa65 commit 2a8ae7e

25 files changed

Lines changed: 564 additions & 17 deletions

File tree

README.md

Lines changed: 36 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ A more accessible, comprehensive, and efficient toolkit for large model compress
1717
</p>
1818

1919
## 📣Latest News
20-
- [25/11/03] We have released v0.2. Quantization support for new models, such as `GLM-4.6` and `Qwen3-VL`, open-sources the Eagle3 speculative decoding training framework, and updates the Diffusion model quantization tools.
20+
- [25/11/05] We have released v0.2. Quantization support for new models, such as `GLM-4.6`, `Qwen3-VL` and `Qwen3-Omni`, open-sources the Eagle3 speculative decoding training framework, and updates the Diffusion model quantization tools.
2121
- [25/09/30] We have released **SpecExit**, the reasoning early-exit algorithm: [[Paper]](http://arxiv.org/abs/2509.24248) | [[Docs]](https://angelslim.readthedocs.io/zh-cn/latest/features/speculative_decoding/spec_exit.html) | [[vLLM Code]](https://github.com/vllm-project/vllm/pull/27192)🔥🔥🔥
2222
- [25/09/26] We have released **TEQUILA**, the ternary quantization algorithm [[Paper]](https://arxiv.org/abs/2509.23809) | [[Code]](https://github.com/Tencent/AngelSlim/tree/tequila/TernaryQuant)🔥🔥🔥
2323
- [25/09/24] We now support the PTQ quantification of NVFP4 for the Qwen3 series models. We also opensource [Qwen3-32B-NVFP4](https://huggingface.co/AngelSlim/Qwen3-32B_nvfp4) and [Qwen3-235B-A22B-NVFP4](https://huggingface.co/AngelSlim/Qwen3-235B-A22B_nvfp4) weights.
@@ -171,7 +171,7 @@ A more accessible, comprehensive, and efficient toolkit for large model compress
171171
</td>
172172
<td>
173173
<ul style="padding-left: 0; list-style-position: inside;">
174-
<li>Under Development</li>
174+
<li><a href="https://github.com/Tencent/AngelSlim/blob/main/docs/source/models/qwen3_omni/qwen3_omni_quant.md">FP8-Static/Dynamic</a></li>
175175
</ul>
176176
</td>
177177
<td>
@@ -510,7 +510,40 @@ Benchmark results for Qwen2.5VL series models with `BF16`、`FP8-Static`、`FP8-
510510
511511
</details>
512512
513-
#### 1.5 Other Models
513+
#### 1.5 Qwen-Omni Series Models
514+
515+
**Qwen3-Omni Text to Text Benchmark**
516+
517+
Benchmark results for Qwen3-Omni series models in BF16, FP8-Static, and FP8-Dynamic on aime25, gpqa_diamond, and mmlu_redux are as follows:
518+
519+
<table>
520+
<thead>
521+
<tr><th>Model</th><th>Quantization</th><th>aime25</th><th>gpqa_diamond</th><th>mmlu_redux</th></tr>
522+
</thead>
523+
<tbody>
524+
<tr><td rowspan="3">Qwen3-Omni-30B-A3B-Instruct</td><td>BF16</td><td>73.32</td><td>56.77</td><td>88.09</td></tr>
525+
<tr><td>FP8-Static</td><td>71.33</td><td>56.57</td><td>87.91</td></tr>
526+
<tr><td>FP8-Dynamic</td><td>73.33</td><td>55.15</td><td>88.07</td></tr>
527+
</tbody>
528+
</table>
529+
530+
<details>
531+
<summary>Note</summary>
532+
533+
> - The above evaluation results were obtained by deploying with the vLLM framework and averaging over 5 runs (vLLM only supports the thinker component).
534+
> - The hyperparameters used during evaluation are as follows:
535+
> ```json
536+
>{
537+
> "top_p": 0.95,
538+
> "temperature": 0.6,
539+
> "do_sample": true,
540+
> "max-model-len 65536": 65536
541+
>}
542+
>```
543+
544+
</details>
545+
546+
#### 1.6 Other Models
514547
515548
Other models such as GLM-4.6, Qwen2.5, and Seed-OSS have been evaluated on benchmarks like `CEVAL`, `MMLU`, and `GSM8K` using quantization strategies including `FP8-Static`, `FP8-Dynamic`, `INT4-GPTQ`, and `INT4-AWQ`.
516549

README_cn.md

Lines changed: 36 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
</p>
1818

1919
## 📣最新进展
20-
- [25/11/03] 我们发布V0.2版本,支持了包括GLM-4.6/Qwen3-VL等更多模型的量化,开源投机采样Eagle3训练框架,更新Diffusion模型量化工具。
20+
- [25/11/05] 我们发布V0.2版本,支持了包括GLM-4.6/Qwen3-VL/Qwen3-Omni等更多模型的量化,开源投机采样Eagle3训练框架,更新Diffusion模型量化工具。
2121
- [25/09/30] 我们开源了思考早退新算法 **SpecExit** [[论文]](http://arxiv.org/abs/2509.24248) | [[文档]](https://angelslim.readthedocs.io/zh-cn/latest/features/speculative_decoding/spec_exit.html) | [[vLLM代码]](https://github.com/vllm-project/vllm/pull/27192)🔥🔥🔥
2222
- [25/09/30] 我们发布了三值量化新算法 **Tequila** [[论文]](https://arxiv.org/abs/2509.23809) | [[代码]](https://github.com/Tencent/AngelSlim/tree/tequila/TernaryQuant)。🔥🔥🔥
2323
- [25/09/24] 我们支持了Qwen3系列模型的NVFP4的PTQ量化,我们还开源了[Qwen3-32B-NVFP4](https://huggingface.co/AngelSlim/Qwen3-32B_nvfp4)[Qwen3-235B-A22B-NVFP4](https://huggingface.co/AngelSlim/Qwen3-235B-A22B_nvfp4)权重。
@@ -172,7 +172,7 @@
172172
</td>
173173
<td>
174174
<ul style="padding-left: 0; list-style-position: inside;">
175-
<li>建设中</li>
175+
<li><a href="https://github.com/Tencent/AngelSlim/blob/main/docs/source/models/qwen3_omni/qwen3_omni_quant.md">FP8-Static/Dynamic</a></li>
176176
</ul>
177177
</td>
178178
<td>
@@ -517,7 +517,40 @@ Qwen2.5VL系列模型的`BF16`、`FP8-Static`、`FP8-Dynamic`、`INT4-GPTQ`、`I
517517
518518
</details>
519519
520-
#### 1.5 其他模型
520+
#### 1.5 Qwen-Omni 系列模型
521+
522+
**Qwen3-Omni Text to Text Benchmark**
523+
524+
Qwen3-Omni系列模型的`BF16`、`FP8-Static`、`FP8-Dynamic`在`aime25`、`gpqa_diamond`、`mmlu_redux`上的评测结果如下:
525+
526+
<table>
527+
<thead>
528+
<tr><th>Model</th><th>Quantization</th><th>aime25</th><th>gpqa_diamond</th><th>mmlu_redux</th></tr>
529+
</thead>
530+
<tbody>
531+
<tr><td rowspan="3">Qwen3-Omni-30B-A3B-Instruct</td><td>BF16</td><td>73.32</td><td>56.77</td><td>88.09</td></tr>
532+
<tr><td>FP8-Static</td><td>71.33</td><td>56.57</td><td>87.91</td></tr>
533+
<tr><td>FP8-Dynamic</td><td>73.33</td><td>55.15</td><td>88.07</td></tr>
534+
</tbody>
535+
</table>
536+
537+
<details>
538+
<summary>备注</summary>
539+
540+
> - 以上评测结果使用vllm框架部署测试5次求平均(vllm只支持thinker部分)
541+
> - 评测时使用的超参如下:
542+
> ```json
543+
>{
544+
> "top_p": 0.95,
545+
> "temperature": 0.6,
546+
> "do_sample": true,
547+
> "max-model-len 65536": 65536
548+
>}
549+
>```
550+
551+
</details>
552+
553+
#### 1.6 其他模型
521554
522555
其他模型比如GLM、Qwen2.5、Seed-OSS等模型利用`FP8-Static`、`FP8-Dynamic`、`INT4-GPTQ`、`INT4-AWQ`量化等策略在`CEVAL`、`MMLU`、`GSM8K`上进行了评测。
523556

angelslim/compressor/quant/ptq.py

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414

1515
import json
1616
import os
17+
import warnings
1718

1819
import torch
1920
from safetensors.torch import load_file
@@ -193,9 +194,19 @@ def _convert(self):
193194
)
194195
is not None
195196
):
196-
self.quant_model.act_scales_dict[name] = self.ptq_hook.observer_dict[
197-
sub_layer
198-
].act_observer.scales()
197+
try:
198+
self.quant_model.act_scales_dict[name] = (
199+
self.ptq_hook.observer_dict[sub_layer].act_observer.scales()
200+
)
201+
except ValueError:
202+
self.quant_model.act_scales_dict[name] = torch.tensor(
203+
1.0, device=torch.cuda.current_device()
204+
)
205+
warnings.warn(
206+
f"Not calibrated for {name}. Using default act scale 1.0.",
207+
RuntimeWarning,
208+
stacklevel=2,
209+
)
199210
if (
200211
getattr( # noqa: B009
201212
self.ptq_hook.observer_dict[sub_layer], "kv_cache_observer"

angelslim/data/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,6 @@
66

77
from .dataloader import DataLoaderFactory # noqa: F401
88
from .multimodal_dataset import MultiModalDataset # noqa: F401
9+
from .omni_dataset import OmniDataset # noqa: F401
910
from .text2image_dataset import Text2ImageDataset # noqa: F401
1011
from .text_dataset import TextDataset # noqa: F401

angelslim/data/dataloader.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020

2121
from .base_dataset import BaseDataset
2222
from .multimodal_dataset import MultiModalDataset
23+
from .omni_dataset import OmniDataset
2324
from .text2image_dataset import Text2ImageDataset
2425
from .text_dataset import TextDataset
2526

@@ -39,6 +40,7 @@ def create_data_loader(
3940
data_type: str = "auto",
4041
num_workers: int = 0,
4142
inference_settings: Dict = None,
43+
use_audio_in_video: bool = False,
4244
model_name: str = None,
4345
) -> DataLoader:
4446
"""
@@ -98,6 +100,16 @@ def create_data_loader(
98100
num_samples=num_samples,
99101
inference_settings=inference_settings,
100102
)
103+
elif data_type == "OmniDataset":
104+
dataset = OmniDataset(
105+
processor=processor,
106+
device=device,
107+
max_length=max_length,
108+
num_samples=num_samples,
109+
data_source=data_source,
110+
is_hf_dataset=not os.path.isfile(data_source),
111+
use_audio_in_video=use_audio_in_video,
112+
)
101113
else:
102114
raise ValueError(f"Unsupported data type: {data_type}")
103115

angelslim/data/multimodal_dataset.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,12 @@
1616
import os
1717
from typing import Dict, List, Union
1818

19-
import qwen_vl_utils
2019
from datasets import load_dataset
2120
from PIL import Image
2221
from tqdm import tqdm
2322
from transformers import ProcessorMixin
2423

24+
from ..utils.lazy_imports import qwen_vl_utils
2525
from .base_dataset import BaseDataset
2626

2727

angelslim/data/omni_dataset.py

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Copyright 2025 Tencent Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import json
16+
import os
17+
from pathlib import Path
18+
from typing import Dict, List, Union
19+
20+
from transformers import ProcessorMixin
21+
22+
from ..utils.lazy_imports import qwen_omni_utils
23+
from .base_dataset import BaseDataset
24+
25+
26+
class OmniDataset(BaseDataset):
27+
"""Dataset for multimodal (text + image) data"""
28+
29+
def __init__(
30+
self,
31+
processor: ProcessorMixin,
32+
device: str = "cpu",
33+
max_length: int = 4096,
34+
num_samples: int = -1,
35+
data_source: Union[str, Dict] = None,
36+
is_hf_dataset: bool = False,
37+
use_audio_in_video: bool = False,
38+
):
39+
super().__init__(processor, device, max_length)
40+
self.is_hf_dataset = is_hf_dataset
41+
self.use_audio_in_video = use_audio_in_video
42+
43+
self._load_file_based_dataset(data_source, num_samples)
44+
45+
def _load_file_based_dataset(self, data_path: str, num_samples: int):
46+
"""Load dataset from local file system"""
47+
path_obj = Path(data_path)
48+
data_dir = path_obj.parent
49+
50+
line_count = 0
51+
with open(data_path, "r") as f:
52+
for line in f:
53+
if num_samples > 0 and line_count >= num_samples:
54+
break
55+
data = json.loads(line.strip())
56+
video_path = None
57+
audio_path = None
58+
image_path = None
59+
60+
if "video_path" in data:
61+
video_path = os.path.normpath(
62+
os.path.join(data_dir, data["video_path"])
63+
)
64+
if "audio_path" in data:
65+
audio_path = os.path.normpath(
66+
os.path.join(data_dir, data["audio_path"])
67+
)
68+
if "image_path" in data:
69+
image_path = os.path.normpath(
70+
os.path.join(data_dir, data["image_path"])
71+
)
72+
73+
ms = data.get("messages")
74+
75+
conversation = []
76+
for m in ms:
77+
if m["role"] == "system":
78+
conversation.append(
79+
{
80+
"role": "system",
81+
"content": [{"type": "text", "text": m["content"]}],
82+
}
83+
)
84+
elif m["role"] == "user":
85+
content = []
86+
text_content = m["content"]
87+
text_content = (
88+
text_content.replace("<video>", "")
89+
.replace("<audio>", "")
90+
.replace("<image>", "")
91+
)
92+
content.append({"type": "text", "text": text_content})
93+
if video_path:
94+
content.append({"type": "video", "video": video_path})
95+
if audio_path:
96+
content.append({"type": "audio", "audio": audio_path})
97+
if image_path:
98+
content.append({"type": "image", "image": image_path})
99+
conversation.append(
100+
{
101+
"role": "user",
102+
"content": content,
103+
}
104+
)
105+
self._process_and_append(conversation)
106+
line_count += 1
107+
108+
def _process_and_append(self, messages: List[Dict]):
109+
"""Process messages and append to dataset"""
110+
text = self.processor.apply_chat_template(
111+
messages, tokenize=False, add_generation_prompt=True
112+
)
113+
audios, images, videos = qwen_omni_utils.process_mm_info(
114+
messages, use_audio_in_video=self.use_audio_in_video
115+
)
116+
117+
# Process inputs
118+
inputs = self.processor(
119+
text=text,
120+
images=images,
121+
audios=audios,
122+
videos=videos,
123+
padding=True,
124+
return_tensors="pt",
125+
use_audio_in_video=self.use_audio_in_video,
126+
)
127+
self.data.append(inputs)

angelslim/engine.py

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ def prepare_model(
7474
cache_dir=None,
7575
deploy_backend="vllm",
7676
using_multi_nodes=False,
77+
use_audio_in_video=False,
7778
) -> Any:
7879
"""Load pretrained model and tokenizer
7980
Args:
@@ -117,6 +118,16 @@ def prepare_model(
117118
using_multi_nodes=using_multi_nodes,
118119
)
119120
self.model_path = model_path
121+
elif self.series in ["Omni"]:
122+
if not model:
123+
self.slim_model.from_pretrained(
124+
model_path,
125+
torch_dtype=torch_dtype,
126+
device_map=device_map,
127+
trust_remote_code=trust_remote_code,
128+
use_audio_in_video=use_audio_in_video,
129+
)
130+
self.model_path = model_path
120131
else:
121132
raise ValueError(f"Unsupported series: {self.series}")
122133

@@ -132,6 +143,7 @@ def prepare_data(
132143
num_samples=128,
133144
shuffle=True,
134145
inference_settings=None,
146+
use_audio_in_video=False,
135147
model_name=None,
136148
) -> Optional[Any]:
137149
"""Prepare compression dataset"""
@@ -146,7 +158,7 @@ def prepare_data(
146158
data_type=data_type,
147159
processor=(
148160
self.slim_model.processor
149-
if self.series == "VLM"
161+
if self.series == "VLM" or self.series == "Omni"
150162
else self.slim_model.tokenizer
151163
),
152164
device=self.slim_model.model.device,
@@ -156,6 +168,7 @@ def prepare_data(
156168
num_samples=num_samples,
157169
data_source=data_path,
158170
inference_settings=inference_settings,
171+
use_audio_in_video=use_audio_in_video,
159172
model_name=model_name,
160173
)
161174
self.max_seq_length = max_length
@@ -188,7 +201,7 @@ def prepare_compressor(
188201
f"Compression method '{method_name}' not registered. "
189202
f"Available methods: {CompressorFactory.get_available_compressor()}"
190203
)
191-
if self.series in ["LLM", "VLM"]:
204+
if self.series in ["LLM", "VLM", "Omni"]:
192205
global_config.update(self.model_path, self.max_seq_length)
193206

194207
if default_method:

angelslim/models/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,5 @@
1515
from .diffusion import * # noqa: F401 F403
1616
from .llm import * # noqa: F401 F403
1717
from .model_factory import SlimModelFactory # noqa: F401
18+
from .omni import * # noqa: F401 F403
1819
from .vlm import * # noqa: F401 F403

0 commit comments

Comments
 (0)