Skip to content

[OpenVINO] Support ResembleAI/chatterbox text-to-speech#1795

Open
openvino-agent wants to merge 1 commit into
huggingface:mainfrom
openvino-agent:worktree-chatterbox
Open

[OpenVINO] Support ResembleAI/chatterbox text-to-speech#1795
openvino-agent wants to merge 1 commit into
huggingface:mainfrom
openvino-agent:worktree-chatterbox

Conversation

@openvino-agent

Copy link
Copy Markdown
Contributor

What does this PR do?

Set up environment

pip install chatterbox-tts
pip install optimum-intel@

Export model

optimum-cli export openvino -m ResembleAI/chatterbox chatterbox

Infer the model:

import os

import soundfile as sf

from optimum.exporters.openvino import main_export
from optimum.intel import OVModelForTextToSpeechSeq2Seq

OV_MODEL_DIR = "./chatterbox"


model = OVModelForTextToSpeechSeq2Seq.from_pretrained(OV_MODEL_DIR)

samples = {
    "en": "Hello, this is the multilingual text-to-speech model with OpenVINO.",
    "fr": "Bonjour, comment ça va? Ceci est le modèle de synthèse vocale multilingue.",
    "de": "Hallo, dies ist das mehrsprachige Text-zu-Sprache-Modell mit OpenVINO.",
}

for language_id, text in samples.items():
    waveform = model.generate(text=text, language_id=language_id)
    out_path = f"test-{language_id}-ov.wav"
    sf.write(out_path, waveform.squeeze().numpy(), model.sampling_rate)
    print(f"[{language_id}] saved {out_path} ({waveform.shape[-1] / model.sampling_rate:.2f}s)")

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Add OpenVINO export and inference support for the ResembleAI Chatterbox TTS
model (both the English and the 23-language multilingual variants) via
OVModelForTextToSpeechSeq2Seq.

Chatterbox is a multi-stage pipeline, decomposed into three exported
submodels plus precomputed assets:
- openvino_t3: the T3 token-to-token model (Llama backbone + speech head),
  exported as a stateful inputs_embeds decoder
- openvino_flow: the whole S3Gen flow (conformer encoder + Euler ODE +
  flow-matching estimator) as a single graph with the diffusion noise as input
- openvino_hifigan: the HiFiGAN vocoder (torch.istft replaced by a traceable
  overlap-add inverse STFT)

The checkpoint ships no AutoConfig-recognizable config.json, so it is detected
by its characteristic files and loaded via library_name="chatterbox".
Multilingual synthesis uses the same language_id argument as the original model.

Tests build a tiny random-weight model at test time to avoid downloading the
~2 GB checkpoint. Docs list the model under supported architectures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment on lines +431 to +434
@unittest.skipUnless(
importlib.util.find_spec("chatterbox") is not None,
"chatterbox-tts package is not installed",
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test will not be launched in any workflow unless the chatterbox-tts package is manually installed, is that intended?

Comment on lines +1306 to +1309
@unittest.skipUnless(
importlib.util.find_spec("chatterbox") is not None,
"chatterbox-tts package is not installed",
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants