Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
244d8bc
Refactor ToolContext to parity class taking a list of Tool | Toolset …
toubatbrian May 28, 2026
6b85bfe
feat(agents): add Toolset support to ToolContext and AgentActivity (#…
toubatbrian Jun 1, 2026
57c1634
Merge branch 'main' into 1.5.0
toubatbrian Jun 1, 2026
07648b0
Merge branch 'main' into 1.5.0
toubatbrian Jun 8, 2026
a24fb0b
fix(agents): thread FlushSentinel through Agent.create llmNode types
toubatbrian Jun 8, 2026
c8b56bd
feat(agents): add beta end call tool (#1474)
rosetta-livekit-bot[bot] Jun 8, 2026
f089da4
arden end-call shutdown and tool guards
toubatbrian Jun 8, 2026
0daf4ef
fix(elevenlabs): end server vad turns (#1745)
rosetta-livekit-bot[bot] Jun 10, 2026
d423f3f
Add Inworld delivery mode inference TTS option (#1749)
rosetta-livekit-bot[bot] Jun 10, 2026
9d86bc5
Don't retain recorded events when recording is disabled (#1750)
toubatbrian Jun 10, 2026
03429b8
ci: speed up type checking (#1742)
rosetta-livekit-bot[bot] Jun 11, 2026
ec765ea
fix(voice): scope forwardAudio playback-started listener to its own s…
chenghao-mou Jun 11, 2026
416871a
feat(barge-in): add default threshold support and drop http transport…
chenghao-mou Jun 11, 2026
eae6074
feat(voice): add agent instruction updates (#1783)
rosetta-livekit-bot[bot] Jun 12, 2026
882491c
Merge remote-tracking branch 'origin/main' into 1.5.0
toubatbrian Jun 13, 2026
b579216
test: use fake timers in manual-abort Task test to deflake CI
toubatbrian Jun 13, 2026
2cd85da
fix(elevenlabs): stop recognize from mutating instance language (#1789)
rosetta-livekit-bot[bot] Jun 15, 2026
d9f56db
fix(phonic): add client header to conversations (#1781)
rosetta-livekit-bot[bot] Jun 15, 2026
20fcec7
fix(inference): remove stale Cartesia STT model type (#1794)
rosetta-livekit-bot[bot] Jun 16, 2026
2404611
feat(llm): list tools for unknown functions
rosetta-livekit-bot[bot] Jun 16, 2026
2765bf0
fix(voice): skip stale end-of-turn metrics (#1803)
rosetta-livekit-bot[bot] Jun 16, 2026
73a575f
Merge branch 'main' into 1.5.0
toubatbrian Jun 17, 2026
2aa44aa
Async Toolsets (#1736)
toubatbrian Jun 17, 2026
d5456f0
fix formatting and build
toubatbrian Jun 17, 2026
000f05a
feat(llm): list tools for unknown functions (#1800)
ShayneP Jun 17, 2026
2998c52
Add object tool syntax compatibility (#1819)
toubatbrian Jun 18, 2026
d130e27
feat(llm): add LLMStream.collect() to await full response (#1568)
toubatbrian Jun 18, 2026
039dd45
feat(testing): add withMockTools utility for mocking agent tools (#1549)
toubatbrian Jun 18, 2026
762a167
Update turbo.json
toubatbrian Jun 18, 2026
8a91a0b
Add scoped filler support to RunContext (#1818)
toubatbrian Jun 18, 2026
2a64eae
fix(workflows): normalize two-digit years in GetDOBTask
rosetta-livekit-bot[bot] Jun 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/agent-update-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@livekit/agents': patch
'@livekit/agents-plugin-openai': patch
---

Add `Agent.updateInstructions()` to update an agent's instructions mid-session (parity with Python). The change propagates the new instructions through the active `AgentActivity`, records an `AgentConfigUpdate` in the chat and session history, and syncs the realtime/stateless contexts. For OpenAI realtime, per-response instructions now preserve the session-level instructions instead of replacing them.
6 changes: 6 additions & 0 deletions .changeset/async-toolsets.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@livekit/agents': minor
---

Port async tool execution semantics from Python: tools can release their turn with `ctx.update()`,
`AsyncToolset` controls session/activity scope, and cancellable tools expose task-management helpers.
5 changes: 5 additions & 0 deletions .changeset/bargein-default-threshold-drop-http.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': patch
---

Adaptive interruption detection now omits the threshold from `session.create` unless the user explicitly overrides it, letting the gateway apply its fetched default (surfaced via `default_threshold` on `session.created`). The HTTP transport has been dropped — detection always connects over WebSocket and always requires LiveKit credentials, and its base URL now defaults from `LIVEKIT_INFERENCE_URL` instead of `LIVEKIT_REMOTE_EOT_URL`. Inference requests also send an `X-LiveKit-Worker-Token` header when `LIVEKIT_WORKER_TOKEN` is set (hosted agents); a token supplied via the `--worker-token` CLI flag is now re-exported into the environment so forked job subprocesses inherit it and include the header. The `X-LiveKit-Agent-Id` header is now only attached once the room is connected to avoid leaking an unset local-participant SID. The interruption WebSocket is now closed deterministically on stream teardown (including error and cancel paths) instead of only on graceful completion — previously an orphaned socket leaked per session/activity and accumulated for the worker's lifetime. Mid-session threshold/duration changes via `updateOptions` now reconnect the WebSocket in place rather than closing it and letting the next send error the stream — so option changes no longer consume a failover retry (previously enough updates in a session could exhaust the retry budget and stop interruption detection).
5 changes: 5 additions & 0 deletions .changeset/cold-avocados-behave.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@livekit/agents": patch
---

Add Agent.create method
5 changes: 5 additions & 0 deletions .changeset/funky-mugs-stare.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@livekit/agents": patch
---

Add scoped filler support to RunContext
5 changes: 5 additions & 0 deletions .changeset/gemini-provider-tools.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents-plugin-google': minor
---

Add Gemini provider tools for Google Search, Google Maps, URL context, File Search, code execution, and Vertex RAG retrieval, and serialize them from `ToolContext` for Google LLM and realtime sessions.
5 changes: 5 additions & 0 deletions .changeset/honest-swans-drum.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@livekit/agents": patch
---

Don't retain recorded events when recording is disabled
5 changes: 5 additions & 0 deletions .changeset/inworld-delivery-mode.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': patch
---

Add Inworld `delivery_mode` to inference TTS model options.
16 changes: 16 additions & 0 deletions .changeset/list-syntax-toolcontext.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
'@livekit/agents': minor
---

**BREAKING**: `Agent({ tools })` and `agent.updateTools()` now accept a flat list `(FunctionTool | ProviderTool | Toolset)[]` instead of a `Record<string, FunctionTool>` map, and `llm.tool({ ... })` requires a `name` field. `ToolContext` is now a Python-parity class with `functionTools` / `providerTools` / `toolsets` accessors, plus `flatten()`, `hasTool(id)`, `getFunctionTool(id)`, `updateTools()`, `copy()`, and `equals()`. To match the Python reference, registering two **different** function-tool instances under the same `name` now throws `duplicate function name: <name>` instead of silently overriding the earlier entry; passing the **same instance** twice is a no-op. `agent.toolCtx` returns a defensive copy so callers can no longer mutate the agent's internal state. `LLM.chat({ toolCtx })` accepts either a `ToolContext` instance or a raw `(FunctionTool | ProviderTool | Toolset)[]` array (`ToolCtxInput`) and normalizes it internally, so callers don't have to construct a `ToolContext` themselves.

Tools also expose an `id: string` field on the base `Tool` interface (parity with Python's `Tool.id` property): for `FunctionTool` it mirrors `name`, for `ProviderTool` it is the provider tool id. `ToolContext` keys and equality now use `tool.id` consistently.

**BREAKING**: Provider tools are now modeled to match Python's `ProviderTool`:

- `ProviderDefinedTool` is renamed to `ProviderTool`, and `isProviderDefinedTool` is renamed to `isProviderTool`.
- `ProviderTool` is now an **abstract class** (Python parity). Plugins must subclass it (`class WebSearch extends ProviderTool { ... }`) to attach provider-specific fields and serializers; bare `new ProviderTool(...)` is rejected at compile time.
- The `tool({ id })` factory overload is removed; `tool({ ... })` only creates function tools now. Construct provider tools by instantiating a `ProviderTool` subclass.
- The `ToolType` literal for provider tools is renamed from `'provider-defined'` to `'provider'`.

`Toolset` now carries a `TOOLSET_SYMBOL` marker and is detected via a new `isToolset()` guard (consistent with `isFunctionTool` / `isProviderTool`). Existing `instanceof Toolset` checks still work, but symbol-based detection is preferred for cross-realm safety.
5 changes: 5 additions & 0 deletions .changeset/list-unknown-tools.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': patch
---

List available tools in unknown-function error output.
5 changes: 5 additions & 0 deletions .changeset/llm-stream-collect.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': patch
---

Add `LLMStream.collect()` for awaiting the full response of a chat stream as a single object (text, tool calls, usage, extra).
5 changes: 5 additions & 0 deletions .changeset/mock-tools-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': patch
---

feat(testing): add `mockTools` utility to override an Agent's tool implementations within an async context, mirroring the Python `mock_tools` API
5 changes: 5 additions & 0 deletions .changeset/object-tools-compat.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': patch
---

Add object-map tool syntax compatibility.
5 changes: 5 additions & 0 deletions .changeset/odd-languages-speak.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents-plugin-elevenlabs': patch
---

Avoid persisting per-call STT language overrides on ElevenLabs STT instances.
5 changes: 5 additions & 0 deletions .changeset/openai-provider-tools.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents-plugin-openai': minor
---

Add OpenAI Responses provider tools for web search, file search, and code interpreter.
5 changes: 5 additions & 0 deletions .changeset/phonic-client-header.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents-plugin-phonic': patch
---

Add LiveKit Agents JS client header to Phonic conversation sockets.
5 changes: 5 additions & 0 deletions .changeset/port-dob-task.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': minor
---

Add beta GetDOBTask with two-digit year normalization.
5 changes: 5 additions & 0 deletions .changeset/port-end-call-tool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': minor
---

Add beta EndCallTool for ending calls from agent tools
5 changes: 5 additions & 0 deletions .changeset/quick-meals-breathe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': patch
---

Adds base `Toolset` support: a stateful container for a group of tools with `setup()` / `aclose()` lifecycle hooks. Toolsets can be passed directly into `Agent({ tools: [...] })` alongside individual function tools; their tools are flattened into the agent's `ToolContext` and the runtime drives `setup()` on activity start, `aclose()` on close, and a setup/close diff when `agent.updateTools()` adds or removes Toolsets mid-session. Per-toolset `setup()` errors are logged but do not abort the activity. The `IGNORE_ON_ENTER` flag is also respected for function tools nested inside a Toolset. Every LLM and realtime plugin tool builder iterates `ToolContext.flatten()` so toolset-contributed tools are correctly advertised. Also exports `ToolCalledEvent` / `ToolCompletedEvent` payload types.
15 changes: 15 additions & 0 deletions .changeset/scope-forward-audio-playback-started.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
'@livekit/agents': patch
---

fix(voice): scope forwardAudio's playback-started listener to its own segment

When a speech is interrupted, the scheduling loop immediately authorizes the next
speech, so the new segment's `forwardAudio` registers its `playback_started`
listener on the shared audio output while the interrupted segment is still
emitting events during teardown. The stray event resolved the new segment's
`firstFrameFut` before its first frame was captured, which skipped resampler
creation and pushed an unresampled frame straight to the `AudioSource`
(`RtcError: sample_rate and num_channels don't match`) and corrupted playback
bookkeeping. The listener now only resolves `firstFrameFut` after the segment has
captured its own first frame.
10 changes: 10 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ jobs:
run: pnpm format:check
- name: Throws transformer
run: pnpm throws:check
- name: Type check
run: pnpm typecheck
build:
name: Build
runs-on: ubuntu-latest
Expand All @@ -48,5 +50,13 @@ jobs:
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile --ignore-scripts
- name: Cache turbo
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: .turbo
key: turbo-${{ runner.os }}-node20-${{ hashFiles('pnpm-lock.yaml') }}-${{ github.run_id }}
restore-keys: |
turbo-${{ runner.os }}-node20-${{ hashFiles('pnpm-lock.yaml') }}-
turbo-${{ runner.os }}-node20-
- name: Build
run: pnpm build
8 changes: 8 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,14 @@ jobs:
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile --ignore-scripts
- name: Cache turbo
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: .turbo
key: turbo-${{ runner.os }}-node20-${{ hashFiles('pnpm-lock.yaml') }}-${{ github.run_id }}
restore-keys: |
turbo-${{ runner.os }}-node20-${{ hashFiles('pnpm-lock.yaml') }}-
turbo-${{ runner.os }}-node20-
- name: Build
run: pnpm build
- name: Check which tests to run
Expand Down
1 change: 1 addition & 0 deletions agents/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
"clean": "rm -rf dist",
"clean:build": "pnpm clean && pnpm build",
"lint": "eslint -f unix \"src/**/*.ts\"",
"typecheck": "tsc -p tsconfig.typecheck.json",
"api:check": "api-extractor run --typescript-compiler-folder ../node_modules/typescript",
"api:update": "api-extractor run --local --typescript-compiler-folder ../node_modules/typescript --verbose",
"throws:check": "throws-check src/**/*.ts"
Expand Down
11 changes: 11 additions & 0 deletions agents/src/beta/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,11 @@
//
// SPDX-License-Identifier: Apache-2.0
export {
GetDOBTask,
type GetDOBResult,
type GetDOBTaskOptions,
TaskGroup,
type TimeOfBirth,
type TaskCompletedEvent,
type TaskGroupOptions,
type TaskGroupResult,
Expand All @@ -12,3 +16,10 @@ export {
type WarmTransferTaskOptions,
} from './workflows/index.js';
export { Instructions } from '../llm/index.js';
export {
END_CALL_DESCRIPTION,
createEndCallTool,
type EndCallToolCalledEvent,
type EndCallToolCompletedEvent,
type EndCallToolOptions,
} from './tools/index.js';
181 changes: 181 additions & 0 deletions agents/src/beta/tools/end_call.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
// SPDX-FileCopyrightText: 2026 LiveKit, Inc.
//
// SPDX-License-Identifier: Apache-2.0
import { type EventEmitter, once } from 'node:events';
import { setTimeout as waitFor } from 'node:timers/promises';
import { getJobContext } from '../../job.js';
import {
RealtimeModel,
type ToolCalledEvent,
type ToolCompletedEvent,
Toolset,
tool,
} from '../../llm/index.js';
import { log } from '../../log.js';
import type { AgentSession, AgentSessionCallbacks } from '../../voice/agent_session.js';
import { AgentSessionEventTypes } from '../../voice/events.js';
import type { UnknownUserData } from '../../voice/run_context.js';

/** How long to wait for the agent's goodbye reply to play out before forcing shutdown. */
const END_CALL_REPLY_TIMEOUT = 5000;

/** Typed wrapper around `events.once`; abort resolves to `undefined`, other errors propagate. */
function onceEvent<E extends keyof AgentSessionCallbacks>(
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- callbacks don't depend on UserData
session: AgentSession<any>,
event: E,
options?: { signal?: AbortSignal },
): Promise<Parameters<AgentSessionCallbacks[E]>[0] | undefined> {
return (
once(session as unknown as EventEmitter, event, options) as Promise<
Parameters<AgentSessionCallbacks[E]>
>
).then(
([payload]) => payload,
(err) => {
if (options?.signal?.aborted) return undefined;
throw err;
},
);
}

export const END_CALL_DESCRIPTION = `
Ends the current call and disconnects immediately.

Call when:
- The user clearly indicates they are done (e.g., "that's all, bye").

Do not call when:
- The user asks to pause, hold, or transfer.
- Intent is unclear.

This is the final action the agent can take.
Once called, no further interaction is possible with the user.
Don't generate any other text or response when the tool is called.
`;

export type EndCallToolCalledEvent<UserData = UnknownUserData> = ToolCalledEvent<UserData>;

export type EndCallToolCompletedEvent<UserData = UnknownUserData> = ToolCompletedEvent<UserData>;

export type EndCallToolOptions<UserData = UnknownUserData> = {
/** Additional description to add to the end call tool. */
extraDescription?: string;
/**
* Whether to delete the room when the user ends the call.
* Deleting the room disconnects all remote users, including SIP callers.
*/
deleteRoom?: boolean;
/** Tool output to the LLM for generating the tool response. */
endInstructions?: string | null;
/** Callback to call when the tool is called. */
onToolCalled?: (event: EndCallToolCalledEvent<UserData>) => Promise<void> | void;
/** Callback to call when the tool is completed. */
onToolCompleted?: (event: EndCallToolCompletedEvent<UserData>) => Promise<void> | void;
};

/**
* Allows the agent to end the call and disconnect from the room.
*/
export function createEndCallTool<UserData = UnknownUserData>({
extraDescription = '',
deleteRoom = true,
endInstructions = 'say goodbye to the user',
onToolCalled,
onToolCompleted,
}: EndCallToolOptions<UserData> = {}): Toolset {
// For a realtime LLM that generates the goodbye reply itself, wait for that reply to play out
// (bounded by END_CALL_REPLY_TIMEOUT) before shutting down. `signal` is aborted when the call
// ends or the toolset is torn down, which cancels whichever of the two races is still pending.
const delayedSessionShutdown = async (
session: AgentSession<UserData>,
signal: AbortSignal,
): Promise<void> => {
const speech = onceEvent(session, AgentSessionEventTypes.SpeechCreated, { signal }).then(
(event) => event?.speechHandle,
);
const timeout = waitFor(END_CALL_REPLY_TIMEOUT, 'timeout' as const, { signal }).catch(
() => undefined,
);

const winner = await Promise.race([speech, timeout]);
if (signal.aborted) return; // session already closed or toolset torn down

if (winner === 'timeout') {
log().warn('tool reply timed out, shutting down session');
session.shutdown();
} else if (winner) {
await winner.waitForPlayout();
session.shutdown();
}
};

return Toolset.create({
id: 'end_call',
tools: [
tool<UserData>({
name: 'end_call',
description: `${END_CALL_DESCRIPTION}\n${extraDescription}`,
execute: async (_args, { ctx, abortSignal }) => {
log().debug('end_call tool called');
const session = ctx.session;
const llm = session.currentAgent.getActivityOrThrow().llm;

// Lifetime of this invocation: aborts when the session closes, and also when the tool
// call itself is aborted. All listeners/timers below are scoped to it.
const controller = new AbortController();
const signal = abortSignal
? AbortSignal.any([abortSignal, controller.signal])
: controller.signal;

void onceEvent(session, AgentSessionEventTypes.Close, { signal })
.then((event) => {
if (!event) return; // signal aborted before close fired
controller.abort(); // stop the delayed-shutdown race

const jobCtx = getJobContext(false);
if (!jobCtx) return;

if (deleteRoom) {
jobCtx.addShutdownCallback(async () => {
log().info('deleting the room because the user ended the call');
await jobCtx.deleteRoom();
});
}

jobCtx.shutdown(String(event.reason));
})
.catch((error) => log().error({ error }, 'error during end call shutdown'));

ctx.speechHandle.addDoneCallback(() => {
if (!(llm instanceof RealtimeModel) || !llm.capabilities.autoToolReplyGeneration) {
session.shutdown();
return;
}

void delayedSessionShutdown(session, signal).catch((error) =>
log().error({ error }, 'error during delayed session shutdown'),
);
});

if (onToolCalled) {
await onToolCalled({ ctx, arguments: {} });
}

const completedEvent = {
ctx,
output:
endInstructions === null
? undefined
: ({ type: 'output', value: endInstructions } as const),
};
if (onToolCompleted) {
await onToolCompleted(completedEvent);
}

return endInstructions ?? undefined;
},
}),
],
});
}
Loading