support funaudiochat ST2T mode inference-only in vLLM, speed up around 20x+ in short audios (1-30s) ,and around 50x+ in long audios

[nemoramo]

vllm-project/vllm#33058

[Mousaic]

必须赞一个

[nemoramo]

@lxchtan Can you formally submit a PR to transformers (adding the model FunAudioChat-8B), so as the vLLM modelconfig type can be auto-recognized in the coming transformers version.

[lxchtan]

Thank you for your contribution. I will submit a PR to the transformers library in the next few days.

[nemoramo]

vllm-project/vllm@36d450e already merged in vLLM! Enjoy it.

[BadDeveloper2022]

@nemoramo 感谢分享，我想问下 多轮对话的时候怎么处理？ 文字+音频混合发过去吗

[BadDeveloper2022]

@nemoramo 比如我想实现 ： 一个语音聊天机器人，有上下文的记忆功能，通过VAD切出来之后，AI说话和用户说话的历史记录，怎样给vllm传过去最好，性能最好

[tuyen-tran1]

I would like to ask whether this version of vLLM can be used to run the conversation demo.
I have integrated it into the FunAudio Chat web demo, but the assistant’s responses are incorrect and noticeably less accurate than usual.

[nemoramo]

I would like to ask whether this version of vLLM can be used to run the conversation demo. I have integrated it into the FunAudio Chat web demo, but the assistant’s responses are incorrect and noticeably less accurate than usual.

can you share the comparison between the original version and vllm version. For same sdpa backend, these two versions share nearly same accuracy, but it's not that complete match since the vllm uses its own module. However you may share the way you get this incorrectness, so as i can help.

[DDJ-1Ray]

@nemoramo Hi!

How can I get the audio token with VLLM? It just gives me text token. This is my test code. `def test_with_audio():
url = f"{BASE_URL}/v1/chat/completions"

with open("ck7vv9ag.wav", "rb") as audio_file:
    audio_data = base64.b64encode(audio_file.read()).decode('utf-8')
payload = {
    "model": MODEL_NAME,
    "messages": [{
        "role": "system", "content": SPOKEN_S2M_PROMPT
    },{
        "role": "user",
        "content": [
            {"type": "text", "text": "请转录这段音频"},
            {
                "type": "input_audio",  # input_audio
                "input_audio": {
                    "data": audio_data,
                    "format": "wav"
                }
            }
        ]
        },{"role": "assistant", "content": None}],
    "max_tokens": 200,
    "skip_special_tokens": False,
}
response = requests.post(url, json=payload)
return response.json()

`

[nemoramo]

@nemoramo Hi!

How can I get the audio token with VLLM? It just gives me text token. This is my test code. `def test_with_audio(): url = f"{BASE_URL}/v1/chat/completions"

with open("ck7vv9ag.wav", "rb") as audio_file:
    audio_data = base64.b64encode(audio_file.read()).decode('utf-8')
payload = {
    "model": MODEL_NAME,
    "messages": [{
        "role": "system", "content": SPOKEN_S2M_PROMPT
    },{
        "role": "user",
        "content": [
            {"type": "text", "text": "请转录这段音频"},
            {
                "type": "input_audio",  # input_audio
                "input_audio": {
                    "data": audio_data,
                    "format": "wav"
                }
            }
        ]
        },{"role": "assistant", "content": None}],
    "max_tokens": 200,
    "skip_special_tokens": False,
}
response = requests.post(url, json=payload)
return response.json()

`

Sorry this is a ST2T mode only inference engine. Currently I am not working on TTS branch adapting.

[BadDeveloper2022]

v0.15.0 no releases 33058

[nemoramo]

v0.15.0 no releases 33058

v0.16.0rc0 should be in this tag right now

[nemoramo]

@lxchtan vllm-project/vllm-omni#1748 s2s mode supported in vllm-omni

[nemoramo]

@nemoramo Hi!

How can I get the audio token with VLLM? It just gives me text token. This is my test code. `def test_with_audio(): url = f"{BASE_URL}/v1/chat/completions"

with open("ck7vv9ag.wav", "rb") as audio_file:
    audio_data = base64.b64encode(audio_file.read()).decode('utf-8')
payload = {
    "model": MODEL_NAME,
    "messages": [{
        "role": "system", "content": SPOKEN_S2M_PROMPT
    },{
        "role": "user",
        "content": [
            {"type": "text", "text": "请转录这段音频"},
            {
                "type": "input_audio",  # input_audio
                "input_audio": {
                    "data": audio_data,
                    "format": "wav"
                }
            }
        ]
        },{"role": "assistant", "content": None}],
    "max_tokens": 200,
    "skip_special_tokens": False,
}
response = requests.post(url, json=payload)
return response.json()

`

hi maybe you can check this pr out. vllm-project/vllm-omni#1748