Skip to content

Commit 1f1dd97

Browse files
committed
feat(agentkit): add support for mllm & tts
1 parent 8b04761 commit 1f1dd97

17 files changed

Lines changed: 391 additions & 29 deletions

File tree

docs/concepts/vendors.md

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ Used with `agent.with_tts()`. Each TTS vendor produces audio at a specific sampl
7070
| `DeepgramTTS` | Deepgram | `api_key`, `model` | Configurable |
7171
| `GradiumTTS` | Gradium | `api_key` | Configurable |
7272
| `MistralTTS` | Mistral | `api_key` ||
73+
| `TypecastTTS` | Typecast | `api_key`, `voice_id`, `model` ||
7374
| `SarvamTTS` | Sarvam | `api_key` ||
7475
| `XaiTTS` | xAI | `api_key`, `language` | Configurable |
7576

@@ -109,6 +110,8 @@ Used with `agent.with_stt()`.
109110

110111
Use `turn_detection.language` for Agora interaction language; it defaults to `en-US`. STT vendor `language` options are serialized under `asr.params` using each provider's own format. If `with_stt()` is omitted, AgentKit defaults to `AresSTT` for global clients and `FengmingSTT` for `Area.CN` clients. Ares does not take a provider language option; AgentKit uses `turn_detection.language` for REST `asr.language`.
111112

113+
`AresSTT` and `FengmingSTT` accept an optional `keywords: List[str]` hotword list.
114+
112115
| Class | Provider | Required Parameters |
113116
|---|---|---|
114117
| `SpeechmaticsSTT` | Speechmatics | `api_key`, `language` |
@@ -146,12 +149,14 @@ stt = DeepgramSTT(api_key='your-deepgram-key', language='en-US', model='nova-2')
146149

147150
Used with `agent.with_mllm()` for the [MLLM flow](../guides/mllm-flow.md). These handle audio input and output end-to-end.
148151

149-
| Class | Provider | Required Parameters |
150-
|---|---|---|
151-
| `OpenAIRealtime` | OpenAI Realtime | `api_key`; optional `turn_detection` |
152-
| `GeminiLive` | Google Gemini Live API | `api_key`, `model`; optional `turn_detection` |
153-
| `VertexAI` | Vertex AI (Gemini Live) | `model`, `project_id`, `location`, `adc_credentials_string`; optional `turn_detection` |
154-
| `XaiGrok` | xAI Grok (`mllm.vendor`: `xai`) | `api_key`; optional `voice`, `language`, `sample_rate`, `turn_detection` |
152+
| Class | Provider | Area | Required Parameters |
153+
|---|---|---|---|
154+
| `OpenAIRealtime` | OpenAI Realtime | Global | `api_key`; optional `turn_detection` |
155+
| `AzureOpenAIRealtime` | Azure OpenAI Realtime | Global | `api_key`, `url`, `turn_detection`; optional `max_history` |
156+
| `GeminiLive` | Google Gemini Live API | Global | `api_key`, `model`; optional `turn_detection` |
157+
| `VertexAI` | Vertex AI (Gemini Live) | Global | `model`, `project_id`, `location`, `adc_credentials_string`; optional `turn_detection` |
158+
| `XaiGrok` | xAI Grok (`mllm.vendor`: `xai`) | Global | `api_key`; optional `voice`, `language`, `sample_rate`, `turn_detection` |
159+
| `QwenOmni` | Alibaba Cloud Qwen Omni Realtime | CN | `api_key`, `turn_detection`; optional `url` |
155160

156161
<!-- snippet: executable -->
157162
```python
@@ -160,6 +165,12 @@ from agora_agent import OpenAIRealtime
160165
mllm = OpenAIRealtime(api_key='your-openai-key', model='gpt-4o-realtime-preview')
161166
```
162167

168+
Import the CN-only Qwen helper from `agora_agent.cn`:
169+
170+
```python
171+
from agora_agent.cn import QwenOmni
172+
```
173+
163174
## Avatar Vendors
164175

165176
Used with `agent.with_avatar()` in the cascading ASR + LLM + TTS pipeline. Some avatars require specific TTS sample rates — see [Avatar Integration](../guides/avatars.md).

docs/guides/mllm-flow.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
sidebar_position: 2
33
title: MLLM Flow (Multimodal)
4-
description: Use OpenAI Realtime, Gemini Live, Vertex AI, or xAI for end-to-end audio processing.
4+
description: Use supported global or CN realtime multimodal models for end-to-end audio processing.
55
---
66

77
# MLLM Flow (Multimodal)
@@ -11,9 +11,11 @@ The MLLM (Multimodal LLM) flow uses a single model to handle both audio input an
1111
MLLM vendors supported by AgentKit:
1212

1313
- **OpenAI Realtime**`gpt-4o-realtime-preview` and related models
14+
- **Azure OpenAI Realtime** — global Azure-hosted realtime deployments
1415
- **Gemini Live** — direct Google AI API access for audio-native Gemini models
1516
- **Vertex AI** — Gemini Live through Google Cloud Vertex AI
1617
- **xAI Grok** — xAI Realtime API
18+
- **Qwen Omni** — Alibaba Cloud Qwen Omni Realtime for `Area.CN`
1719

1820
## Enable MLLM Mode
1921

@@ -167,7 +169,7 @@ mllm = OpenAIRealtime(
167169
|---|---|---|
168170
| Latency | Lower — single model, no pipeline | Higher — three models in sequence |
169171
| Voice control | Model-dependent | Full vendor choice for TTS |
170-
| Vendor flexibility | Limited to supported MLLM providers (OpenAI Realtime, Gemini Live, Vertex AI, xAI Grok) | Mix and match LLM, TTS, and STT vendors |
172+
| Vendor flexibility | Limited to supported global and CN MLLM providers | Mix and match LLM, TTS, and STT vendors |
171173
| Audio understanding | Model hears tone, pacing, emotion | STT produces text only |
172174

173175
## Next Steps

docs/guides/regional-routing.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,10 +38,10 @@ client = Agora(
3838
Bind `client` into `Agent(client=client, ...)` and construct vendors directly with SDK classes. The bound client selects `CNAgent` or `GlobalAgent` for IDE hints based on `area`, but does not restrict which vendor classes you can configure.
3939
If you omit `with_stt()`, AgentKit uses `FengmingSTT` by default for `Area.CN` clients and `AresSTT` for global clients.
4040

41-
| Client area | STT classes | LLM classes | TTS classes | Avatar classes |
42-
|---|---|---|---|---|
43-
| `Area.US`, `Area.EU`, `Area.AP` | `DeepgramSTT`, `SpeechmaticsSTT`, `MicrosoftSTT`, `OpenAISTT`, `GoogleSTT`, `AmazonSTT`, `AssemblyAISTT`, `AresSTT`, `SarvamSTT`, `XaiSTT` | `OpenAI`, `AzureOpenAI`, `Anthropic`, `Gemini`, `Groq`, `VertexAILLM`, `AmazonBedrock`, `Dify`, `CustomLLM` | `ElevenLabsTTS`, `MicrosoftTTS`, `OpenAITTS`, `CartesiaTTS`, `GoogleTTS`, `AmazonTTS`, `DeepgramTTS`, `GradiumTTS`, `MistralTTS`, `HumeAITTS`, `RimeTTS`, `FishAudioTTS`, `MiniMaxTTS`, `MurfTTS`, `SarvamTTS`, `GenericTTS`, `XaiTTS` | `LiveAvatarAvatar`, `HeyGenAvatar`, `AkoolAvatar`, `AnamAvatar`, `GenericAvatar` |
44-
| `Area.CN` | `FengmingSTT`, `TencentSTT`, `MicrosoftCNSTT`, `XfyunSTT`, `XfyunBigModelSTT`, `XfyunDialectSTT` | `AliyunLLM`, `BytedanceLLM`, `DeepSeekLLM`, `TencentLLM` | `MiniMaxCNTTS`, `TencentTTS`, `BytedanceTTS`, `MicrosoftCNTTS`, `CosyVoiceTTS`, `BytedanceDuplexTTS`, `StepFunTTS`, `GenericTTS` | `SenseTimeAvatar`, `SpatiusAvatar` |
41+
| Client area | STT classes | LLM classes | MLLM classes | TTS classes | Avatar classes |
42+
|---|---|---|---|---|---|
43+
| `Area.US`, `Area.EU`, `Area.AP` | `DeepgramSTT`, `SpeechmaticsSTT`, `MicrosoftSTT`, `OpenAISTT`, `GoogleSTT`, `AmazonSTT`, `AssemblyAISTT`, `AresSTT`, `SarvamSTT`, `XaiSTT` | `OpenAI`, `AzureOpenAI`, `Anthropic`, `Gemini`, `Groq`, `VertexAILLM`, `AmazonBedrock`, `Dify`, `CustomLLM` | `OpenAIRealtime`, `AzureOpenAIRealtime`, `GeminiLive`, `VertexAI`, `XaiGrok` | `ElevenLabsTTS`, `MicrosoftTTS`, `OpenAITTS`, `CartesiaTTS`, `GoogleTTS`, `AmazonTTS`, `DeepgramTTS`, `GradiumTTS`, `MistralTTS`, `TypecastTTS`, `HumeAITTS`, `RimeTTS`, `FishAudioTTS`, `MiniMaxTTS`, `MurfTTS`, `SarvamTTS`, `GenericTTS`, `XaiTTS` | `LiveAvatarAvatar`, `HeyGenAvatar`, `AkoolAvatar`, `AnamAvatar`, `GenericAvatar` |
44+
| `Area.CN` | `FengmingSTT`, `TencentSTT`, `MicrosoftCNSTT`, `XfyunSTT`, `XfyunBigModelSTT`, `XfyunDialectSTT` | `AliyunLLM`, `BytedanceLLM`, `DeepSeekLLM`, `TencentLLM` | `QwenOmni` | `MiniMaxCNTTS`, `TencentTTS`, `BytedanceTTS`, `MicrosoftCNTTS`, `CosyVoiceTTS`, `BytedanceDuplexTTS`, `StepFunTTS`, `GenericTTS` | `SenseTimeAvatar`, `SpatiusAvatar` |
4545

4646
Global client example:
4747

docs/index.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,12 @@ The Agora Conversational AI Python SDK lets you build voice-powered AI agents on
1717

1818
**Cascading flow** uses ASR -> LLM -> TTS and supports the broadest set of vendor combinations.
1919

20-
**MLLM flow** uses a multimodal model such as OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok for end-to-end audio.
20+
**MLLM flow** uses a multimodal model such as OpenAI Realtime, Azure OpenAI Realtime, Gemini Live, Vertex AI, xAI Grok, or Qwen Omni for end-to-end audio.
2121

2222
## Start here
2323

2424
- Start with [Quick Start](./getting-started/quick-start.md). It shows the baseline app-credentials setup and starts a cascading ASR -> LLM -> TTS agent.
25-
- Use [MLLM Flow](./guides/mllm-flow.md) when your agent uses one realtime multimodal model, such as OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok.
25+
- Use [MLLM Flow](./guides/mllm-flow.md) when your agent uses one supported realtime multimodal model.
2626
- Use [Cascading Flow](./guides/cascading-flow.md) for more examples of the default ASR -> LLM -> TTS flow, including provider-specific configuration.
2727

2828
## How the SDK is organized
@@ -46,7 +46,7 @@ The Agora Conversational AI Python SDK lets you build voice-powered AI agents on
4646
| [AgentSession](./concepts/session.md) | Manage the agent lifecycle |
4747
| [Vendors](./concepts/vendors.md) | Browse all LLM, TTS, STT, MLLM, and Avatar providers |
4848
| [Cascading Flow](./guides/cascading-flow.md) | Build an ASR -> LLM -> TTS pipeline |
49-
| [MLLM Flow](./guides/mllm-flow.md) | Use OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok for end-to-end audio |
49+
| [MLLM Flow](./guides/mllm-flow.md) | Use a global or CN realtime multimodal model for end-to-end audio |
5050
| [Avatars](./guides/avatars.md) | Add a digital avatar with LiveAvatar, Akool, Anam, Generic Avatar, SenseTime, or Spatius |
5151
| [Regional Routing](./guides/regional-routing.md) | Route requests to the nearest region |
5252
| [Error Handling](./guides/error-handling.md) | Handle API errors with ApiError |

docs/reference/vendors.md

Lines changed: 60 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -10,17 +10,18 @@ All vendor classes are available from `agora_agent`:
1010

1111
<!-- snippet: fragment -->
1212
```python
13-
from agora_agent import OpenAI, ElevenLabsTTS, DeepgramTTS, DeepgramSTT, OpenAIRealtime, XaiGrok, GenericAvatar
13+
from agora_agent import AzureOpenAIRealtime, DeepgramSTT, OpenAI, TypecastTTS
14+
from agora_agent.cn import QwenOmni
1415
```
1516

1617
## Recommended vendors by area
1718

1819
Construct vendors directly from `agora_agent`, then bind a client with `Agent(client=client, ...)`. The bound client selects `CNAgent` or `GlobalAgent` for IDE hints based on `area`, but does not restrict which vendor classes you can configure.
1920

20-
| Area | STT classes | LLM classes | TTS classes | Avatar classes |
21-
|---|---|---|---|---|
22-
| `Area.US`, `Area.EU`, `Area.AP` | `DeepgramSTT`, `SpeechmaticsSTT`, `MicrosoftSTT`, `OpenAISTT`, `GoogleSTT`, `AmazonSTT`, `AssemblyAISTT`, `AresSTT`, `SarvamSTT`, `XaiSTT` | `OpenAI`, `AzureOpenAI`, `Anthropic`, `Gemini`, `Groq`, `VertexAILLM`, `AmazonBedrock`, `Dify`, `CustomLLM` | `ElevenLabsTTS`, `MicrosoftTTS`, `OpenAITTS`, `CartesiaTTS`, `GoogleTTS`, `AmazonTTS`, `DeepgramTTS`, `GradiumTTS`, `MistralTTS`, `HumeAITTS`, `RimeTTS`, `FishAudioTTS`, `MiniMaxTTS`, `MurfTTS`, `SarvamTTS`, `GenericTTS`, `XaiTTS` | `LiveAvatarAvatar`, `HeyGenAvatar`, `AkoolAvatar`, `AnamAvatar`, `GenericAvatar` |
23-
| `Area.CN` | `FengmingSTT`, `TencentSTT`, `MicrosoftCNSTT`, `XfyunSTT`, `XfyunBigModelSTT`, `XfyunDialectSTT` | `AliyunLLM`, `BytedanceLLM`, `DeepSeekLLM`, `TencentLLM` | `MiniMaxCNTTS`, `TencentTTS`, `BytedanceTTS`, `MicrosoftCNTTS`, `CosyVoiceTTS`, `BytedanceDuplexTTS`, `StepFunTTS`, `GenericTTS` | `SenseTimeAvatar`, `SpatiusAvatar` |
21+
| Area | STT classes | LLM classes | MLLM classes | TTS classes | Avatar classes |
22+
|---|---|---|---|---|---|
23+
| `Area.US`, `Area.EU`, `Area.AP` | `DeepgramSTT`, `SpeechmaticsSTT`, `MicrosoftSTT`, `OpenAISTT`, `GoogleSTT`, `AmazonSTT`, `AssemblyAISTT`, `AresSTT`, `SarvamSTT`, `XaiSTT` | `OpenAI`, `AzureOpenAI`, `Anthropic`, `Gemini`, `Groq`, `VertexAILLM`, `AmazonBedrock`, `Dify`, `CustomLLM` | `OpenAIRealtime`, `AzureOpenAIRealtime`, `GeminiLive`, `VertexAI`, `XaiGrok` | `ElevenLabsTTS`, `MicrosoftTTS`, `OpenAITTS`, `CartesiaTTS`, `GoogleTTS`, `AmazonTTS`, `DeepgramTTS`, `GradiumTTS`, `MistralTTS`, `TypecastTTS`, `HumeAITTS`, `RimeTTS`, `FishAudioTTS`, `MiniMaxTTS`, `MurfTTS`, `SarvamTTS`, `GenericTTS`, `XaiTTS` | `LiveAvatarAvatar`, `HeyGenAvatar`, `AkoolAvatar`, `AnamAvatar`, `GenericAvatar` |
24+
| `Area.CN` | `FengmingSTT`, `TencentSTT`, `MicrosoftCNSTT`, `XfyunSTT`, `XfyunBigModelSTT`, `XfyunDialectSTT` | `AliyunLLM`, `BytedanceLLM`, `DeepSeekLLM`, `TencentLLM` | `QwenOmni` | `MiniMaxCNTTS`, `TencentTTS`, `BytedanceTTS`, `MicrosoftCNTTS`, `CosyVoiceTTS`, `BytedanceDuplexTTS`, `StepFunTTS`, `GenericTTS` | `SenseTimeAvatar`, `SpatiusAvatar` |
2425

2526
Global example:
2627

@@ -411,6 +412,16 @@ AgentKit serializes `credential_mode` at the top level of the Rime TTS configura
411412
| `additional_params` | `Dict[str, Any]` | No | `None` | Additional Mistral TTS parameters, flattened into `params` |
412413
| `skip_patterns` | `List[int]` | No | `None` | Skip patterns |
413414

415+
### `TypecastTTS`
416+
417+
| Parameter | Type | Required | Default | Description |
418+
|---|---|---|---|---|
419+
| `api_key` | `str` | Yes || Typecast API key |
420+
| `voice_id` | `str` | Yes || Typecast voice identifier |
421+
| `model` | `str` | Yes || Typecast TTS model name, for example `ssfm-v30` |
422+
| `additional_params` | `Dict[str, Any]` | No | `None` | Additional Typecast parameters, flattened into `params` |
423+
| `skip_patterns` | `List[int]` | No | `None` | Skip patterns |
424+
414425
### `MurfTTS`
415426

416427
| Parameter | Type | Required | Default | Description |
@@ -550,6 +561,7 @@ For `nova-2` and `nova-3`, omit `api_key` to use Agora-managed credentials. For
550561

551562
| Parameter | Type | Required | Default | Description |
552563
|---|---|---|---|---|
564+
| `keywords` | `List[str]` | No | `None` | Hotwords that improve ASR accuracy |
553565
| `additional_params` | `Dict[str, Any]` | No | `None` | Additional parameters |
554566

555567
### `SarvamSTT`
@@ -700,7 +712,9 @@ All CN TTS vendor classes support `skip_patterns` and `additional_params`.
700712

701713
#### `FengmingSTT`
702714

703-
No constructor parameters. Use `FengmingSTT()`.
715+
| Parameter | Type | Required | Default | Description |
716+
|---|---|---|---|---|
717+
| `keywords` | `List[str]` | No | `None` | Hotwords that improve ASR accuracy |
704718

705719
#### `XfyunSTT`
706720

@@ -789,6 +803,46 @@ No constructor parameters. Use `FengmingSTT()`.
789803
| `params` | `Dict[str, Any]` | No | `None` | Additional parameters |
790804
| `turn_detection` | `MllmTurnDetectionConfig` | No | `None` | MLLM turn detection configuration; overrides top-level `turn_detection` |
791805

806+
### `AzureOpenAIRealtime`
807+
808+
Global Azure OpenAI Realtime vendor (`mllm.vendor`: `"azure"`).
809+
810+
| Parameter | Type | Required | Default | Description |
811+
|---|---|---|---|---|
812+
| `api_key` | `str` | Yes || Azure OpenAI API key |
813+
| `url` | `str` | Yes || Azure OpenAI Realtime WebSocket URL |
814+
| `model` | `str` | No | `None` | Model or deployment name |
815+
| `voice` | `str` | No | `None` | Voice identifier |
816+
| `instructions` | `str` | No | `None` | System instructions |
817+
| `input_audio_transcription` | `Dict[str, Any]` | No | `None` | Audio transcription settings |
818+
| `max_history` | `int` | No | `None` | Number of conversation history messages to cache; Azure-only MLLM field |
819+
| `greeting_message` | `str` | No | `None` | Greeting message |
820+
| `failure_message` | `str` | No | `None` | Message played when the model call fails |
821+
| `output_modalities` | `List[str]` | No | `None` | Output modalities |
822+
| `messages` | `List[Dict]` | No | `None` | Conversation messages |
823+
| `params` | `Dict[str, Any]` | No | `None` | Additional Azure OpenAI parameters |
824+
| `turn_detection` | `MllmTurnDetectionConfig` | Yes || Required MLLM turn detection configuration; overrides top-level `turn_detection` |
825+
826+
### `QwenOmni`
827+
828+
CN Alibaba Cloud Qwen Omni Realtime vendor (`mllm.vendor`: `"qwen_omni"`). Import it with `from agora_agent.cn import QwenOmni`.
829+
830+
| Parameter | Type | Required | Default | Description |
831+
|---|---|---|---|---|
832+
| `api_key` | `str` | Yes || Alibaba Cloud Model Studio API key |
833+
| `url` | `str` | No | `None` | Qwen Omni Realtime WebSocket URL |
834+
| `model` | `str` | No | `None` | Qwen Omni Realtime model name |
835+
| `voice` | `str` | No | `None` | Voice identifier |
836+
| `instructions` | `str` | No | `None` | System instructions |
837+
| `input_audio_transcription` | `Dict[str, Any]` | No | `None` | Audio transcription settings |
838+
| `greeting_message` | `str` | No | `None` | Greeting message |
839+
| `failure_message` | `str` | No | `None` | Message played when the model call fails |
840+
| `input_modalities` | `List[str]` | No | `None` | Input modalities |
841+
| `output_modalities` | `List[str]` | No | `None` | Output modalities |
842+
| `messages` | `List[Dict]` | No | `None` | Conversation messages |
843+
| `params` | `Dict[str, Any]` | No | `None` | Additional Qwen Omni parameters |
844+
| `turn_detection` | `MllmTurnDetectionConfig` | Yes || Required MLLM turn detection configuration; overrides top-level `turn_detection` |
845+
792846
### `GeminiLive`
793847

794848
| Parameter | Type | Required | Default | Description |

src/agora_agent/agentkit/__init__.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,7 @@
156156
AresSTT,
157157
AssemblyAISTT,
158158
AzureOpenAI,
159+
AzureOpenAIRealtime,
159160
BaseAvatar,
160161
BaseLLM,
161162
BaseMLLM,
@@ -195,9 +196,11 @@
195196
OpenAITTS,
196197
GenericTTS,
197198
RimeTTS,
199+
QwenOmni,
198200
SampleRate,
199201
SarvamSTT,
200202
SarvamTTS,
203+
TypecastTTS,
201204
SpeechmaticsSTT,
202205
XaiSTT,
203206
XaiTTS,
@@ -403,6 +406,7 @@
403406
"MiniMaxTTS",
404407
"MiniMaxCNTTS",
405408
"MistralTTS",
409+
"TypecastTTS",
406410
"TencentTTS",
407411
"BytedanceTTS",
408412
"CosyVoiceTTS",
@@ -428,7 +432,9 @@
428432
"XfyunDialectSTT",
429433
"XfyunSTT",
430434
"OpenAIRealtime",
435+
"AzureOpenAIRealtime",
431436
"GeminiLive",
437+
"QwenOmni",
432438
"VertexAI",
433439
"XaiGrok",
434440
"HeyGenAvatar",

src/agora_agent/agentkit/regional_agent.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
import typing
44
from .agent import Agent
5-
from .vendors.base import BaseAvatar, BaseLLM, BaseSTT, BaseTTS
5+
from .vendors.base import BaseAvatar, BaseLLM, BaseMLLM, BaseSTT, BaseTTS
66
from .vendors.cn import (
77
AliyunLLM,
88
BytedanceDuplexTTS,
@@ -14,6 +14,7 @@
1414
MiniMaxTTS as MiniMaxCNTTS,
1515
MicrosoftSTT as MicrosoftCNSTT,
1616
MicrosoftTTS as MicrosoftCNTTS,
17+
QwenOmni,
1718
SenseTimeAvatar,
1819
SpatiusAvatar,
1920
StepFunTTS,
@@ -47,6 +48,7 @@
4748
OpenAI,
4849
VertexAILLM,
4950
)
51+
from .vendors.mllm import AzureOpenAIRealtime, GeminiLive, OpenAIRealtime, VertexAI, XaiGrok
5052
from .vendors.tts import (
5153
AmazonTTS,
5254
CartesiaTTS,
@@ -64,13 +66,15 @@
6466
GenericTTS,
6567
RimeTTS,
6668
SarvamTTS,
69+
TypecastTTS,
6770
XaiTTS,
6871
)
6972
from .vendors.avatar import AkoolAvatar, AnamAvatar, GenericAvatar, HeyGenAvatar, LiveAvatarAvatar
7073

7174
CNSTT = typing.Union[TencentSTT, FengmingSTT, MicrosoftCNSTT, XfyunSTT, XfyunBigModelSTT, XfyunDialectSTT]
7275
CNTTS = typing.Union[MiniMaxCNTTS, TencentTTS, BytedanceTTS, MicrosoftCNTTS, CosyVoiceTTS, BytedanceDuplexTTS, StepFunTTS, GenericTTS]
7376
CNLLM = typing.Union[AliyunLLM, BytedanceLLM, DeepSeekLLM, TencentLLM]
77+
CNMLLM = QwenOmni
7478
CNAvatar = typing.Union[SenseTimeAvatar, SpatiusAvatar]
7579

7680
GlobalSTT = typing.Union[
@@ -103,6 +107,7 @@
103107
DeepgramTTS,
104108
GradiumTTS,
105109
MistralTTS,
110+
TypecastTTS,
106111
]
107112
GlobalLLM = typing.Union[
108113
OpenAI,
@@ -115,6 +120,7 @@
115120
Dify,
116121
CustomLLM,
117122
]
123+
GlobalMLLM = typing.Union[OpenAIRealtime, AzureOpenAIRealtime, GeminiLive, VertexAI, XaiGrok]
118124
GlobalAvatar = typing.Union[AkoolAvatar, LiveAvatarAvatar, AnamAvatar, GenericAvatar, HeyGenAvatar]
119125

120126

@@ -128,6 +134,9 @@ def with_llm(self, vendor: BaseLLM) -> "CNAgent":
128134
def with_tts(self, vendor: BaseTTS) -> "CNAgent":
129135
return typing.cast("CNAgent", super().with_tts(vendor))
130136

137+
def with_mllm(self, vendor: BaseMLLM) -> "CNAgent":
138+
return typing.cast("CNAgent", super().with_mllm(vendor))
139+
131140
def with_avatar(self, vendor: BaseAvatar) -> "CNAgent":
132141
return typing.cast("CNAgent", super().with_avatar(vendor))
133142

@@ -142,6 +151,9 @@ def with_llm(self, vendor: BaseLLM) -> "GlobalAgent":
142151
def with_tts(self, vendor: BaseTTS) -> "GlobalAgent":
143152
return typing.cast("GlobalAgent", super().with_tts(vendor))
144153

154+
def with_mllm(self, vendor: BaseMLLM) -> "GlobalAgent":
155+
return typing.cast("GlobalAgent", super().with_mllm(vendor))
156+
145157
def with_avatar(self, vendor: BaseAvatar) -> "GlobalAgent":
146158
return typing.cast("GlobalAgent", super().with_avatar(vendor))
147159

0 commit comments

Comments
 (0)