|
6 | 6 | - Speech Understanding |
7 | 7 | - Chain-of-Thought |
8 | 8 | - Reinforcement Learning |
9 | | -summary: End-to-end multimodal large language model with audio reasoning capabilities, achieving SOTA performance on multiple benchmarks. |
| 9 | +summary: The world's first industrial-grade end-to-end audio LLM with deep thinking capabilities, achieving SOTA performance across multiple understanding and dialogue tasks. |
10 | 10 | --- |
11 | 11 |
|
12 | | -<div style="position: relative; width: 100%; padding-bottom: 75%; overflow: hidden;"> |
13 | | - <iframe |
14 | | - src="https://www.stepfun.com/research/en/step-audio2" |
15 | | - style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border: none;" |
16 | | - allowfullscreen |
17 | | - loading="lazy"> |
18 | | - </iframe> |
19 | | -</div> |
| 12 | +## Project Resources |
| 13 | + |
| 14 | +- 📄 **Paper**: [arXiv:2507.16632](https://arxiv.org/abs/2507.16632) |
| 15 | +- 💻 **GitHub**: [stepfun-ai/Step-Audio2](https://github.com/stepfun-ai/Step-Audio2) |
| 16 | +- 🌐 **Tech Blog**: [中文](https://www.stepfun.com/research/zh/step-audio2) | [English](https://www.stepfun.com/research/en/step-audio2) |
20 | 17 |
|
21 | 18 | --- |
22 | 19 |
|
23 | | -**项目简介:** |
| 20 | +## Overview |
| 21 | + |
| 22 | +Step-Audio 2 is **the world's first end-to-end audio large language model with deep thinking capabilities designed for industrial applications**. This model innovatively combines a latent space audio encoder with audio reinforcement learning technology. It effectively captures paralinguistic information and speaking style features, and adopts a Chain-of-Thought (CoT) reasoning strategy combined with reinforcement learning optimization. Step-Audio 2 achieves high-performance speech dialogue capabilities across various scenarios. Experimental results demonstrate that the model achieves state-of-the-art (SOTA) performance on multiple understanding and dialogue tasks. |
| 23 | + |
| 24 | +## True End-to-End Architecture: Understanding Beyond Words |
| 25 | + |
| 26 | +Traditional AI voice systems have been criticized for lacking both intelligence and emotional understanding. First, they lack the knowledge base and reasoning capabilities comparable to text-based large models. Second, they sound "robotic" and fail to comprehend subtext, tone, emotions, and laughter—the "unspoken meanings." Step-Audio 2 solves these problems through innovative architectural design, achieving both cognitive and emotional intelligence. |
| 27 | + |
| 28 | + |
| 29 | + |
| 30 | +### Core Technical Features |
| 31 | + |
| 32 | +- **Genuine End-to-End Multimodal Architecture**: Step-Audio 2 breaks through the traditional ASR+LLM+TTS three-stage structure, achieving direct conversion from raw audio input to speech response output. The architecture is more concise with lower latency, and can effectively understand paralinguistic information and non-vocal signals. |
| 33 | + |
| 34 | +- **CoT Reasoning Combined with Reinforcement Learning**: Step-Audio 2 is the first to introduce Chain-of-Thought (CoT) reasoning combined with reinforcement learning optimization in end-to-end speech models. It can perform fine-grained understanding, reasoning, and natural response to paralinguistic and non-speech signals such as emotions, intonation, and music. |
| 35 | + |
| 36 | +- **Audio Knowledge Enhancement**: The model supports external tools including web search, helping to solve hallucination problems and enabling multi-scenario expansion capabilities. |
| 37 | + |
| 38 | +## SOTA Performance |
| 39 | + |
| 40 | +Step-Audio 2 achieves **SOTA results** across multiple key benchmarks, demonstrating outstanding performance in audio understanding, speech recognition, translation, and dialogue scenarios. The overall performance surpasses all open-source end-to-end speech models including Qwen-Omni and Kimi-Audio, and exceeds GPT-4o Audio in most tasks. |
| 41 | + |
| 42 | + |
| 43 | + |
| 44 | +### Key Performance Metrics |
| 45 | + |
| 46 | +- **MMAU (General Multimodal Audio Understanding)**: Ranks **#1** with a score of **78** |
| 47 | + |
| 48 | +- **URO Bench (Spoken Dialogue Capability)**: **#1 in Chinese, #2 in English**, demonstrating excellent dialogue understanding and expression abilities |
24 | 49 |
|
25 | | -Step-Audio 2 is an end-to-end multimodal large language model engineered for industrial applications. This model innovatively integrates a latent space audio encoder with audio reinforcement learning. |
| 50 | +- **Machine Translation Tasks**: |
| 51 | + - CoVoST 2 benchmark: **39.3** |
| 52 | + - CVSS benchmark: **30.9** |
| 53 | + - Significantly outperforms GPT-4o Audio and other open-source speech models |
26 | 54 |
|
27 | | -**关键特性:** |
28 | | -- Genuine end-to-end architecture eliminating traditional ASR+LLM+TTS pipelines |
29 | | -- CoT-reinforcement learning fusion enabling audio reasoning capabilities |
30 | | -- Acoustic knowledge enhancement via web search and audio retrieval |
31 | | -- SOTA performance on MMAU, URO-Bench, and multiple ASR benchmarks |
| 55 | +- **Speech Recognition Tasks**: Achieves first place in multiple languages and dialects |
| 56 | + - Average CER (Character Error Rate) on open-source Chinese test sets: **3.08** |
| 57 | + - Average WER (Word Error Rate) on open-source English test sets: **3.14** |
| 58 | + - Far ahead of other models |
32 | 59 |
|
33 | | -[访问完整页面 →](https://www.stepfun.com/research/en/step-audio2) |
| 60 | + |
34 | 61 |
|
| 62 | +- **Paralinguistic Understanding Tasks**: Ranks **#1** with a score of **83.1** |
0 commit comments