fix(voice): commit interrupted replies when playback start arrives after forwarding completes (avatar outputs)

[snowmaker]

Description

Interrupted agent replies are silently dropped from the chat context (and conversation_item_added never fires) when using a DataStream avatar output — the same "phantom utterance" symptom that #1916 fixed, reintroduced through a different mechanism. The agent also never enters the speaking state in avatar sessions.

Root cause. With avatar outputs configured with waitPlaybackStart: true (e.g. the LemonSlice plugin since #1406), AudioOutput.EVENT_PLAYBACK_STARTED is only emitted when the remote avatar worker sends the lk.playback_started RPC — which arrives once the worker actually starts playing, typically ~1s after the first audio push (we measured LemonSlice time_to_first_push ≈ 1.0–1.1s). But forwardAudio() pushes frames into the DataStream faster than real time, so for essentially every utterance it completes before that RPC arrives. Its finally block then:

1. detached the playback-started listener, and
2. rejected firstFrameFut (audio forwarding cancelled before playback started) — even when forwarding completed normally.

The deferred lk.playback_started notification is dropped on the floor. Consequences, for every utterance of every avatar session:

• onFirstFrame never runs → the agent never transitions to the speaking state → audioRecognition.onStartOfAgentSpeech() is never called (degraded interruption gating / adaptive endpointing), and startedSpeakingAt / e2eLatency metrics are missing.
• On interruption, the commit gate (firstFrameFut.done && !rejected) fails → the segment is classified played: 'skipped' → forwardedTextFor() returns '' → the audibly-heard reply is never inserted into the chat context. The LLM doesn't know what it already said, and downstream conversation_item_added consumers (transcripts) lose the turn.

The python implementation does not have this bug: perform_audio_forwarding attaches the playback-started listener outside the forwarding task, removes it only when the future settles, and lets the caller settle the future after the playout window (generation.py, "out.first_frame_fut should be cancelled in the caller after the playout is finished or interrupted"). This PR mirrors that.

Real-world reproduction (agents-js 1.5.0 + real LemonSlice avatar session, AgentSession + FakeLLM + a tone ttsNode, interrupt at t+2.5s mid-audible-playback):

[+6.03s] generateReply
[+6.87s] DataStreamAudioOutput emitted playbackStarted   <- RPC arrived, fut already rejected
[+8.53s] session.interrupt({force: true})
[+8.74s] playbackFinished {"playbackPosition":1.875,"interrupted":true}  <- 1.9s audibly played
RESULT: chat history contains only the user turn — assistant reply dropped
         (agent state went thinking -> listening; never 'speaking')

With this fix, the same scenario commits the reply with interrupted: true and the agent state correctly transitions thinking → speaking → listening.

We hit this in production: in one 3-minute session, 4 of 15 spoken utterances were interrupted and all 4 were missing from the conversation history, visibly confusing the LLM (it could not see its own prior answers).

Changes Made

• generation.ts forwardAudio: reject firstFrameFut only when forwarding was aborted before playback started; otherwise keep the playback-started listener attached until the future settles (python parity). The stray-event guard (hasCapturedOwnFrame, fix(voice): scope forwardAudio playback-started listener to its own segment #1786) is preserved.
• agent_activity.ts (forwardSegment in both the pipeline and realtime paths): settle a still-pending firstFrameFut once the segment's playout window ends, bounding the listener's lifetime (leak guard, and preserves the "never started" reject semantics for non-avatar outputs).
• agent_activity.ts interrupted branches: also honor a non-zero playbackPosition reported by waitForPlayout() as evidence of partial playback — covers an interruption racing ahead of the lk.playback_started RPC.
• Tests: two new regression tests in agent_activity_interrupted_commit.test.ts (deferred playback-started notification; position-only evidence) — both fail on main and pass with this fix. Updated the teardown assertion in generation_tts_timeout.test.ts to the new lifecycle (future stays pending after zero-frame forwarding; stray events still ignored).

Pre-Review Checklist

• Build passes: All builds (lint, typecheck, tests) pass locally
• AI-generated code reviewed: Removed unnecessary comments and ensured code quality
• Changes explained: All changes are properly documented and justified above
• Scope appropriate: All changes relate to the PR title

Testing

• Automated tests added/updated
• All tests pass (agents: 88 files, 1180 passed; lint + prettier + tsc clean)
• Verified end-to-end against a live LemonSlice avatar session (dev LiveKit Cloud project): pre-fix the interrupted reply is dropped and the agent never reaches speaking; post-fix it is committed with interrupted: true and full state transitions.

[changeset-bot]

🦋 Changeset detected

Latest commit: 32074f2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hedra	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Jared Friedman seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🟡 Event listener on the shared audio output leaks when using the "say" path with deferred playback notifications

The playback-started listener is never detached from the shared audio output (performAudioForwarding at agents/src/voice/agent_activity.ts:2519) because the say() code path has no finally-block cleanup to settle the pending firstFrameFut, so each call accumulates a stale listener.

Impact: Repeated say() calls with avatar outputs that defer playback-started notifications leak event listeners, causing gradual memory growth.

Mechanism: the say() path was not updated with the new cleanup pattern

The PR changed forwardAudio (agents/src/voice/generation.ts:952) to only reject firstFrameFut when signal?.aborted, keeping the listener attached until the future settles. It then added matching finally-block cleanup in forwardSegment (agents/src/voice/agent_activity.ts:2931-2937) and processOneMessage (agents/src/voice/agent_activity.ts:3503-3509) to settle any still-pending firstFrameFut.

However, the say() path at agents/src/voice/agent_activity.ts:2519-2557 also calls performAudioForwarding but was not updated with the same cleanup. When forwarding completes normally (stream closes, signal not aborted), firstFrameFut stays pending and the onPlaybackStarted listener at agents/src/voice/generation.ts:894 remains attached to audioOutput indefinitely. Before this PR, forwardAudio always rejected firstFrameFut on exit, so the listener was always cleaned up.

(Refers to lines 2550-2557)

Prompt for agents

The say() code path at agents/src/voice/agent_activity.ts:2519-2557 calls performAudioForwarding but does not settle the returned audioOut.firstFrameFut when the playout window ends. The forwardSegment (line 2931-2937) and processOneMessage (line 3503-3509) paths both have a finally block that rejects firstFrameFut if still pending, which triggers the .finally() cleanup in forwardAudio to detach the onPlaybackStarted listener. The say() path needs the same pattern: after the interrupted-handling block (around line 2557), add cleanup that rejects audioOut.firstFrameFut if it is not yet done, similar to the pattern used in forwardSegment's finally block. This could be done by wrapping the relevant section in a try/finally or adding the check after the interrupted block.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[toubatbrian]

The core mechanism looks right and matches the Python semantics (generation.py attaches the playback-started listener, detaches on future settlement, and relies on the caller to settle after playout) — and the stale-position guard on startedForwardingAt correctly lines up with waitForPlayout()'s segment counting, so a zero-frame segment can't be committed off a previous segment's event. The regression tests encode the failure mode well.

One gap though: ttsTask (the say() path) is the third caller of performAudioForwarding and wasn't given the same cleanup. Python's say task does cancel a pending future at the end:

https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/agent_activity.py — _tts_task:

if audio_out is not None and not audio_out.first_frame_fut.done():
    audio_out.first_frame_fut.cancel()

Without the equivalent in ttsTask, the impact goes beyond the listener leak already flagged: pre-PR, forwardAudio's finally always settled the future so the listener always detached. Post-PR, any say() whose playback-started never arrives (deferred avatar with a lost RPC, or a zero-frame TTS timeout) leaves a listener attached whose hasCapturedOwnFrame is true and whose future is still pending. The next utterance's playbackStarted on the shared output then resolves that stale future and fires the old say's onFirstFrame — a spurious speaking transition carrying the dead speech handle's OTel context, corrupted startedSpeakingAt/playbackLatency metrics, and a stray onStartOfAgentSpeech into audio recognition. Ten leaked listeners will also trip Node's max-listeners warning.

Suggested fix: the same three-line settle used in forwardSegment/processOneMessage, in a finally around the post-forwarding section of ttsTask (audioOut needs hoisting out of the else block to be visible there). A regression test would be cheap: two consecutive say() calls against a deferred-start output, asserting listenerCount(AudioOutput.EVENT_PLAYBACK_STARTED) returns to baseline and no speaking transition fires after the first say completes.

Minor, non-blocking: the position-based evidence branch (forwardedAudio && playbackPosition > 0) is a deliberate superset of Python — Python's gate is only first_frame_fut.done() && !cancelled, so it still has the "interruption races ahead of the RPC" hole this closes. Worth calling out in the description, and probably worth upstreaming to Python as a follow-up.

[toubatbrian]

Hi @snowmaker — following up on the review: community PR #1910 fixes the same firstFrameFut rejection from the false-interruption side (#1909) and also covers the say() path, but it conflicts line-for-line with this PR in generation.ts / agent_activity.ts.

To simplify coordination we've opened #1966 combining both fixes: your playback-position evidence gate and the avatar-shaped regression tests (DeferredStartOutput, stale-position guard) are carried over as-is, and you're credited as a co-author on the commit. Would appreciate a look to confirm the avatar scenario is still fully covered — we'll close this PR in favor of the combined one once you've had a chance to review.