feature: elevenlabs tts provider#2009
Conversation
Adds ElevenLabs text-to-speech alongside the existing OpenAI and local
(sherpa) speech providers. ElevenLabs is TTS-only — requests for it on
STT or turn-detection slots log a warning and skip those capabilities.
Provider resolution accepts ELEVENLABS_API_KEY (with ELEVENLABS_VOICE_ID
or ELEVENLABS_TTS_VOICE_ID) and ELEVENLABS_BASE_URL/ELEVENLABS_TTS_MODEL
env vars, or matching fields under providers.elevenlabs in the persisted
config. The synthesized MP3 stream flows through the existing audio_output
pipeline unchanged because the client already maps the "mp3" format to
audio/mpeg and decodes it on both web and native audio engines.
Tests cover provider wiring, URL building, error mapping, and empty
input. Server typecheck stays green; existing speech tests stay green.
Note: the project's pre-commit typecheck also runs the CLI workspace,
which surfaces 4 pre-existing CLI-vs-server signature mismatches in
commands/agent/{delete,stop,update}.ts. They reproduce on main without
my changes, so they are out of scope here.
Adds docs/elevenlabs-tts.md covering configuration (env vars and persisted config), defaults, request flow, capability caveats, and gotchas for the new ElevenLabs TTS provider. Registers the doc in CLAUDE.md's docs index.
|
| Filename | Overview |
|---|---|
| packages/server/src/server/speech/providers/elevenlabs/tts.ts | Core ElevenLabsTTS class; correctly injects fetch, uses Readable.fromWeb for streaming, handles error parsing. Minor: DEFAULT_ELEVENLABS_VOICE_ID exported but never used as a default; fetchImpl test seam exposed in public config type. |
| packages/server/src/server/speech/providers/elevenlabs/runtime.ts | Availability check, credential validation, and service initialization for ElevenLabs. Minor: redundant && elevenlabsConfig guard when hasTts already proves it non-null. |
| packages/server/src/server/speech/providers/elevenlabs/config.ts | Config resolver wires persisted-config and env vars into ElevenLabsTtsConfig; priority order (persisted > env) is correct and consistent with other providers. |
| packages/server/src/server/speech/speech-runtime.ts | ElevenLabs slot wired into the initialization pipeline (local → openai → elevenlabs); resolveVoiceTtsLabel simplified to use ttsService.id; localVoiceTtsProvider variable removed cleanly. |
| packages/server/src/server/speech/speech-provider.ts | TextToSpeechProvider gains required readonly id: string; all three TTS implementations (OpenAI, local, ElevenLabs) now set it, collapsing the label function to a single property access. |
| packages/server/src/server/speech/providers/elevenlabs/tts.test.ts | Tests use fetchImpl injection (no vi.hoisted / vi.stubGlobal); covers URL building, path encoding, empty-input rejection, and error response mapping. |
| packages/server/src/server/speech/providers/elevenlabs/runtime.test.ts | Tests cover availability predicate, service construction when provider matches, passthrough when it doesn't, and no-throw for valid configuration. |
| packages/server/src/server/persisted-config.ts | Adds ElevenLabsProviderSchema, extends SpeechProviderIdSchema with "elevenlabs", and widens voice field from OpenAI enum to string — documented as back-compat in the PR. |
Sequence Diagram
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Config as resolveElevenLabsSpeechConfig
participant Runtime as createSpeechService
participant ELRuntime as initializeElevenLabsSpeechServices
participant TTS as ElevenLabsTTS
participant API as ElevenLabs API
Runtime->>Config: "{ env, persisted }"
Config-->>Runtime: "ElevenLabsTtsConfig | undefined"
Runtime->>ELRuntime: "{ providers, elevenlabsConfig, existing }"
ELRuntime->>TTS: new ElevenLabsTTS(config, logger)
ELRuntime-->>Runtime: ExistingSpeechServices (ttsService set)
Note over Runtime: resolveVoiceTtsLabel → ttsService.id = "elevenlabs"
Runtime->>TTS: synthesizeSpeech(text)
TTS->>API: "POST /v1/text-to-speech/{voiceId}?output_format=mp3_44100_128"
API-->>TTS: "ReadableStream<Uint8Array> (MP3)"
TTS-->>Runtime: "{ stream: Readable, format: "mp3" }"
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Config as resolveElevenLabsSpeechConfig
participant Runtime as createSpeechService
participant ELRuntime as initializeElevenLabsSpeechServices
participant TTS as ElevenLabsTTS
participant API as ElevenLabs API
Runtime->>Config: "{ env, persisted }"
Config-->>Runtime: "ElevenLabsTtsConfig | undefined"
Runtime->>ELRuntime: "{ providers, elevenlabsConfig, existing }"
ELRuntime->>TTS: new ElevenLabsTTS(config, logger)
ELRuntime-->>Runtime: ExistingSpeechServices (ttsService set)
Note over Runtime: resolveVoiceTtsLabel → ttsService.id = "elevenlabs"
Runtime->>TTS: synthesizeSpeech(text)
TTS->>API: "POST /v1/text-to-speech/{voiceId}?output_format=mp3_44100_128"
API-->>TTS: "ReadableStream<Uint8Array> (MP3)"
TTS-->>Runtime: "{ stream: Readable, format: "mp3" }"
Reviews (5): Last reviewed commit: "refactor(speech): drop redundant ElevenL..." | Re-trigger Greptile
|
Want your agent to iterate on Greptile's feedback? Try greploops. |
- Drop per-provider identity parameter from resolveVoiceTtsLabel.
Add `id` to the TextToSpeechProvider contract and to all three TTS
classes (openai, local, elevenlabs); the runtime now reads
ttsService.id directly. Future TTS providers only declare their id
and the runtime picks them up without parameter-list growth.
- Inject `fetchImpl` into ElevenLabsTTS via the constructor so tests
can pass a plain stub instead of patching the global. Removes the
vi.hoisted/vi.stubGlobal antipattern from tts.test.ts.
- Drop unused getConfig() from ElevenLabsTTS (was leaking the API key
and had no callers).
- Remove the dead null guard on elevenlabsConfig.tts in
createElevenLabsTts; the type requires tts and the caller already
validates it.
- Drop the `as { ... }` type assertion in resolveElevenLabsSpeechConfig;
PersistedConfigSchema already infers the right type for the
elevenlabs provider block.
All 41 speech tests stay green.
Replace the hand-rolled reader-loop helper with Node's native Readable.fromWeb. Drops ~16 lines and one extra async generator. The fetch Web stream and Readable.fromWeb nominally type as distinct ReadableStream generics (DOM vs stream/web), so the boundary gets a single `as unknown as NodeReadableStream<Uint8Array>` cast — same shape the OpenAI provider already uses to bridge the same gap.
- Make TextToSpeechProvider.id required. The previous optional
declaration forced resolveVoiceTtsLabel to keep a `?? "openai"`
fallback that silently mislabeled any provider missing the field.
All four TTS implementations (openai, local via Sherpa, local via
worker client, elevenlabs) already set id, so making it required
surfaces any future omission at compile time.
- Flatten ElevenLabsSpeechProviderConfig to ElevenLabsTtsConfig. The
outer wrapper carried apiKey/baseUrl as copies of values that lived
on the inner tts object and were the only ones actually consumed;
ElevenLabs is TTS-only, so the OpenAI-shaped { stt, tts } wrapper
is dead weight. Resolver, runtime initializer, validator, and tests
updated to read apiKey/voiceId directly.
The provider config and the TTS config are now the same type, so the internal alias added no information. Consumers (runtime, runtime tests, bootstrap) import ElevenLabsTtsConfig directly from tts.ts; PaseoElevenLabsConfig remains in bootstrap.ts as the public-API alias matching the existing PaseoOpenAIConfig pattern.
Linked issue
Closes #2008
Type of change
What does this PR do
Supports ElevenLabs as a tts provider
How did you verify it
Checklist
npm run typecheckpassesnpm run lintpassesnpm run formatran (Biome)