Skip to content

feat(elevenlabs): add STT support#1367

Open
u9g wants to merge 1 commit intomainfrom
port-elevenlabs-stt-node
Open

feat(elevenlabs): add STT support#1367
u9g wants to merge 1 commit intomainfrom
port-elevenlabs-stt-node

Conversation

@u9g
Copy link
Copy Markdown
Contributor

@u9g u9g commented May 1, 2026

Summary

  • Port ElevenLabs Scribe STT from Python agents, including batch recognize() and realtime WebSocket stream() support.
  • Export STT types/sample rates and add a Node example mirroring the Python Scribe v2 realtime example.
  • Add REST, realtime event, timestamp, and updateOptions({ serverVad }) regression coverage.

Tests

  • pnpm test plugins/elevenlabs/src/stt.test.ts
  • pnpm test plugins/elevenlabs/src/tts.test.ts
  • pnpm --filter @livekit/agents-plugin-elevenlabs build
  • pnpm --filter @livekit/agents-plugin-elevenlabs lint
  • pnpm format:check
  • git diff --check

Notes

  • pnpm --filter @livekit/agents-plugin-elevenlabs api:check fails because this package has no checked-in API report folder/file yet; API Extractor also reports existing missing release-tag warnings.
  • pnpm --filter livekit-agents-examples build fails on missing declaration files for many unbuilt plugin packages; after fixing this example's AgentSession options, no ElevenLabs-specific example error remained.

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented May 1, 2026

🦋 Changeset detected

Latest commit: 7fa80a8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 28 packages
Name Type
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch
@livekit/agents-plugins-test Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 5 additional findings in Devin Review.

Open in Devin Review

Comment on lines +610 to +614
try {
this.#processStreamEvent(parseStreamEvent(JSON.parse(msg.toString())));
} catch (error) {
this.#logger.error({ error }, 'failed to process ElevenLabs STT message');
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 API errors from #processStreamEvent are silently swallowed by the onMessage catch block

When the ElevenLabs server sends an error message (e.g., auth_error, quota_exceeded, transcriber_error), #processStreamEvent deliberately throws an APIConnectionError at plugins/elevenlabs/src/stt.ts:840. However, this throw occurs inside the onMessage callback, which is wrapped in a generic try-catch at lines 610-614 that only logs the error and does not call reject(). As a result, the receiveMessages promise never rejects for API errors, and the stream continues running silently. Critical conditions like authentication failures, quota exceeded, and transcriber errors are permanently lost — the caller never learns the stream has failed, and no transcriptions will be produced.

Suggested change
try {
this.#processStreamEvent(parseStreamEvent(JSON.parse(msg.toString())));
} catch (error) {
this.#logger.error({ error }, 'failed to process ElevenLabs STT message');
}
try {
this.#processStreamEvent(parseStreamEvent(JSON.parse(msg.toString())));
} catch (error) {
if (error instanceof APIConnectionError) {
reject(error);
} else {
this.#logger.error({ error }, 'failed to process ElevenLabs STT message');
}
}
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same in plugins/deepgram/src/stt_v2.ts: we throw on server Error messages but the on('message') handler catches and only logs.

py has the same shape (caught here) — carried over from the port.

};
}

// Ref: python livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/stt.py - 277-295 lines
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should keep those ref?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants