You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+11-37Lines changed: 11 additions & 37 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,8 +53,10 @@ from agora_agent.agentkit import (
53
53
Agent,
54
54
DataChannel,
55
55
DeepgramSTT,
56
+
GenericAvatar,
56
57
MiniMaxTTS,
57
58
OpenAI,
59
+
XaiGrok,
58
60
expires_in_hours,
59
61
)
60
62
@@ -190,7 +192,7 @@ If you want to bring your own vendor credentials instead of using Agora-managed
190
192
191
193
## MLLM (Realtime / Multimodal)
192
194
193
-
Use `with_mllm()` for OpenAI Realtime or Gemini Live. No STT, LLM, or TTS vendor is needed when MLLM mode is enabled.
195
+
Use `with_mllm()` for OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok. No STT, LLM, or TTS vendor is needed when MLLM mode is enabled.
194
196
195
197
```python
196
198
from agora_agent.agentkit import Agent, OpenAIRealtime
@@ -216,18 +218,14 @@ A full reference for this library is available [here](https://github.com/AgoraIO
216
218
217
219
## MLLM Flow (Multimodal)
218
220
219
-
For real-time audio processing using OpenAI's Realtime API or Google Gemini Live, use the MLLM (Multimodal Large Language Model) flow instead of the cascading ASR -> LLM -> TTS flow. See the [MLLM Overview](https://docs.agora.io/en/conversational-ai/models/mllm/overview) for more details.
221
+
For real-time audio processing using OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok, use the MLLM (Multimodal Large Language Model) flow instead of the cascading ASR -> LLM -> TTS flow. MLLM mode does not require separate TTS, STT, or LLM vendors. See the [MLLM Overview](https://docs.agora.io/en/conversational-ai/models/mllm/overview) for more details.
For real-time audio processing using OpenAI's Realtime API or Google Gemini Live, use the MLLM (Multimodal Large Language Model) flow instead of the cascading ASR -> LLM -> TTS flow. See the [MLLM Overview](https://docs.agora.io/en/conversational-ai/models/mllm/overview) for more details.
272
+
For real-time audio processing using OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok, use the MLLM (Multimodal Large Language Model) flow instead of the cascading ASR -> LLM -> TTS flow. MLLM mode does not require separate TTS, STT, or LLM vendors. See the [MLLM Overview](https://docs.agora.io/en/conversational-ai/models/mllm/overview) for more details.
Copy file name to clipboardExpand all lines: changelog.md
+35-4Lines changed: 35 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,6 +4,37 @@ All notable changes to this project will be documented in this file.
4
4
5
5
The format is based on [Keep a Changelog](https://keepachangelog.com/).
6
6
7
+
## [v1.5.0] — 2026-05-20
8
+
9
+
### Added
10
+
11
+
-**`XaiGrok`** — New MLLM wrapper for xAI Grok (`mllm.vendor`: `"xai"`), including Realtime API URL, voice, language, sample rate, modalities, messages, and MLLM turn detection support.
12
+
-**`GenericAvatar`** — New generic avatar wrapper (`vendor: "generic"`) for custom avatar providers.
13
+
-**Avatar token enrichment** — `AgentSession.start()` now fills missing generic avatar `agora_appid` and `agora_channel` from the session and generates missing avatar `agora_token` values for HeyGen, LiveAvatar, and Generic avatars using each avatar's `agora_uid`.
14
+
-**Turn pagination** — `AgentSession.get_turns()` and `AsyncAgentSession.get_turns()` now accept `page_index` and `page_size`. New `get_all_turns()` helpers fetch and combine all pages.
15
+
-**Greeting interruption control** — LLM vendor `greeting_configs` now accepts the typed `LlmGreetingConfigs` shape, including v2.7 `interruptable`.
16
+
-**Type alias parity** — Added public aliases for v2.7 generated types such as `LlmConfig`, `TtsConfig`, `SttConfig`, `MllmConfig`, `AvatarConfig`, `AgentConfigUpdate`, `ConversationTurns`, `ConversationHistory`, `SessionInfo`, `Labels`, `SpeakPriority`, and `FillerWordsContentSelectionRule`.
17
+
18
+
### Changed
19
+
20
+
-**`XaiGrok` is the primary xAI MLLM class** — Matches the product name ([xAI Grok](https://docs.agora.io/en/conversational-ai/models/mllm/xai)) and the TypeScript/Go SDKs. `XaiRealtime` remains as a deprecated backward-compatible alias.
21
+
-**Package version** — Bumped to `v1.5.0` to match the Fern-generated SDK headers.
22
+
-**RTM data channel default** — When `advanced_features.enable_rtm=True`, AgentKit now defaults `parameters.data_channel` to `"rtm"` unless the caller explicitly sets a data channel.
23
+
-**Agent-level LLM overrides** — In the standard ASR + LLM + TTS pipeline, agent-level `greeting`, `failure_message`, and `max_history` now override vendor defaults, matching the TypeScript SDK. In MLLM mode, agent-level `greeting` and `failure_message` fill only missing fields.
24
+
-**MLLM core alignment** — MLLM wrappers no longer expose or emit unsupported `predefined_tools` or `max_history` fields because they are not present in the generated v2.7 core `mllm` type.
25
+
-**MLLM without TTS** — MLLM sessions no longer require separate TTS, STT, or LLM vendor configuration.
26
+
-**Avatar pipeline support** — Avatar vendors are now explicitly limited to the cascading ASR + LLM + TTS pipeline. Combining `with_avatar()` with `with_mllm()` is rejected at `Agent.to_properties()` and `AgentSession.start()` (matching the TypeScript SDK), with a disabled avatar (`enable=False`) still permitted alongside MLLM.
27
+
-**VertexAI parity** — `VertexAI.to_config()` now spreads `additional_params` first so explicit `model`, `project_id`, `location`, and `adc_credentials_string` fields always win, matching the TypeScript and Gemini Live wrappers.
28
+
-**Pagination guard parity** — `AgentSession.get_all_turns()` and `AsyncAgentSession.get_all_turns()` now raise `RuntimeError` if the server's pagination metadata is missing (`page_index`/`total_pages`/`is_last_page`) or if the next page does not advance, matching the TypeScript SDK.
29
+
30
+
### Migration notes
31
+
32
+
-**`XaiRealtime` → `XaiGrok`** — Import and use `XaiGrok` for xAI MLLM (`mllm.vendor`: `"xai"`). `XaiRealtime` still works but emits `DeprecationWarning`. Future xAI cascading vendors will use `XaiSTT` and `XaiTTS`, not `XaiRealtime`.
33
+
-**`think()` default** — The server default for `on_listening_action` changed from `inject` to `interrupt` in API v2.7. Pass `on_listening_action="inject"` explicitly to preserve the old behavior.
34
+
-**Turn analytics pagination** — Sessions with more than 50 turns must request additional pages via `get_turns(page_index=..., page_size=...)` or use `get_all_turns()`.
35
+
-**Error reasons** — API v2.7 adds status codes `401`, `429`, and `500`; `InvalidRequest` is split into `InvalidRequestBody`, `MissingRequiredField`, and `InvalidFieldValue`, with new reasons such as `ServiceNotEnabled`, `AccountSuspended`, and `ResourceAllocationFailed`.
36
+
-**Event `112`** — Webhook event `112 turns finished` can be used as an alternative batch delivery path for post-session turn data.
37
+
7
38
## [v1.4.1] — 2026-05-18
8
39
9
40
### Fixed
@@ -58,8 +89,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
58
89
### Fixed
59
90
60
91
-**`AresSTT`** — Removed redundant `language` key from the `params` dict. Language is now emitted only at the top level. `params` is only included when `additional_params` is provided.
61
-
-**`OpenAIRealtime` / `VertexAI` (MLLM)** — Agent-level `greeting`, `failure_message`, and `max_history` overrides are now correctly applied when the agent is in MLLM mode. Previously these values were silently dropped.
62
-
-**`VertexAI` (MLLM)** — `messages` is now correctly placed inside `params` (required by the Gemini Live API). Previously it was emitted at the top level and silently ignored.
92
+
-**`OpenAIRealtime` / `VertexAI` (MLLM)** — Agent-level `greeting`and `failure_message` defaults are now correctly applied when missing in MLLM mode. Previously these values were silently dropped.
93
+
-**`VertexAI` (MLLM)** — `messages` is emitted at the MLLM top level, matching the generated core SDK contract.
63
94
64
95
### Changed
65
96
@@ -72,8 +103,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
72
103
-**`OpenAITTS`** — New optional parameters: `response_format` (str, e.g. `"pcm"`) and `speed` (float).
73
104
-**`CartesiaTTS`** — `voice_id` user-facing field is preserved; voice is serialized to the required nested object format automatically.
74
105
-**`RimeTTS`** — New optional parameters: `lang` (str), `sampling_rate` (int, serialized as `samplingRate`), `speed_alpha` (float, serialized as `speedAlpha`).
Used with `agent.with_avatar()`. Avatars require specific TTS sample rates — see [Avatar Integration](../guides/avatars.md).
113
+
Used with `agent.with_avatar()` in the cascading ASR + LLM + TTS pipeline. Some avatars require specific TTS sample rates — see [Avatar Integration](../guides/avatars.md).
Copy file name to clipboardExpand all lines: docs/guides/avatars.md
+38-13Lines changed: 38 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,32 +1,43 @@
1
1
---
2
2
sidebar_position: 3
3
3
title: Avatar Integration
4
-
description: Add a digital avatar (HeyGen or Akool) to your Conversational AI agent.
4
+
description: Add a digital avatar to your Conversational AI agent.
5
5
---
6
6
7
7
# Avatar Integration
8
8
9
-
You can attach a digital avatar to your voice agent so that users see a visual representation of the AI speaking. Two avatar providers are supported:
9
+
You can attach a digital avatar to your voice agent so that users see a visual representation of the AI speaking.
10
+
11
+
Avatars are currently supported only with the cascading ASR + LLM + TTS pipeline. MLLM sessions handle audio end-to-end and do not support avatars at this time.
The agent and avatar join the same RTC channel with separate UIDs. The agent token is scoped to `agent_uid`; `avatar.params.agora_token` is scoped to the avatar `agora_uid`.
24
+
25
+
When using `AgentSession.start()`, `agora_token` is optional for LiveAvatar, HeyGen, and Generic avatars. If omitted, AgentKit generates it with the same ConvoAI token path as the agent, using the avatar UID. You can still pass `agora_token` explicitly.
15
26
16
27
## Sample Rate Constraint
17
28
18
-
Each avatar vendor requires a specific TTS sample rate. The SDK validates this when you call `with_avatar()` — if the TTS sample rate does not match, a `ValueError` is raised immediately:
29
+
Each avatar vendor requires a specific TTS sample rate. The SDK validates this when you add TTS or avatar configuration and again when the session starts. If the TTS sample rate does not match, a `ValueError` is raised:
19
30
20
31
```
21
32
ValueError: Avatar requires TTS sample rate of 24000 Hz, but TTS is configured with 16000 Hz. Please update your TTS sample_rate to 24000.
22
33
```
23
34
24
-
This validation happens at build time (when chaining methods), not at runtime when the session starts. Python raises this as a `ValueError` — there is no compile-time check as in statically typed languages.
35
+
Python raises this as a `ValueError` — there is no compile-time check as in statically typed languages.
25
36
26
-
Additionally, if the TTS `sample_rate`is not explicitly set (returns `None`), the SDK issues a warning:
37
+
Additionally, if the TTS sample rate is not explicitly available, the SDK issues a warning through the session warning callback:
27
38
28
39
```
29
-
UserWarning: Avatar requires TTS sample rate of 24000 Hz, but TTS sample_rate is not explicitly set. Please ensure your TTS provider is configured for 24000 Hz.
40
+
Warning: LiveAvatar avatar detected but TTS sample_rate is not explicitly set. LiveAvatar requires 24,000 Hz. Please ensure your TTS provider is configured for 24kHz.
30
41
```
31
42
32
43
## HeyGen Avatar (24 kHz)
@@ -58,7 +69,7 @@ agent = (
58
69
api_key='your-heygen-key',
59
70
quality='medium',
60
71
agora_uid='2',
61
-
avatar_name='your-avatar-name',
72
+
avatar_id='your-avatar-id',
62
73
))
63
74
)
64
75
@@ -68,6 +79,21 @@ session.say('Hello! I am your visual assistant.')
68
79
session.stop()
69
80
```
70
81
82
+
## Generic Avatar
83
+
84
+
`GenericAvatar` supports custom avatar providers. `agora_appid`, `agora_channel`, and `agora_token` are optional when using `AgentSession.start()`.
85
+
86
+
```python
87
+
from agora_agent.agentkit.vendors import GenericAvatar
0 commit comments