Skip to content

feature: elevenlabs tts provider#2009

Open
ilteoood wants to merge 8 commits into
getpaseo:mainfrom
ilteoood:feature/elevenlabs-tts-provider
Open

feature: elevenlabs tts provider#2009
ilteoood wants to merge 8 commits into
getpaseo:mainfrom
ilteoood:feature/elevenlabs-tts-provider

Conversation

@ilteoood

Copy link
Copy Markdown
Contributor

Linked issue

Closes #2008

Type of change

  • Bug fix
  • New feature (with prior issue + design alignment)
  • Refactor / code improvement
  • Docs

What does this PR do

Supports ElevenLabs as a tts provider

How did you verify it

Checklist

  • One focused change. Unrelated cleanups split out.
  • npm run typecheck passes
  • npm run lint passes
  • npm run format ran (Biome)
  • UI changes include screenshots or video for every affected platform
  • Tests added or updated where it made sense

ilteoood added 2 commits July 10, 2026 16:53
Adds ElevenLabs text-to-speech alongside the existing OpenAI and local
(sherpa) speech providers. ElevenLabs is TTS-only — requests for it on
STT or turn-detection slots log a warning and skip those capabilities.

Provider resolution accepts ELEVENLABS_API_KEY (with ELEVENLABS_VOICE_ID
or ELEVENLABS_TTS_VOICE_ID) and ELEVENLABS_BASE_URL/ELEVENLABS_TTS_MODEL
env vars, or matching fields under providers.elevenlabs in the persisted
config. The synthesized MP3 stream flows through the existing audio_output
pipeline unchanged because the client already maps the "mp3" format to
audio/mpeg and decodes it on both web and native audio engines.

Tests cover provider wiring, URL building, error mapping, and empty
input. Server typecheck stays green; existing speech tests stay green.

Note: the project's pre-commit typecheck also runs the CLI workspace,
which surfaces 4 pre-existing CLI-vs-server signature mismatches in
commands/agent/{delete,stop,update}.ts. They reproduce on main without
my changes, so they are out of scope here.
Adds docs/elevenlabs-tts.md covering configuration (env vars and
persisted config), defaults, request flow, capability caveats, and
gotchas for the new ElevenLabs TTS provider. Registers the doc in
CLAUDE.md's docs index.
@greptile-apps

greptile-apps Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds ElevenLabs as a third TTS provider alongside OpenAI and local (Sherpa), routed through the existing speech-service pipeline. The feature is well-scoped: it lives in a dedicated elevenlabs/ directory, integrates through the same local→openai→elevenlabs initialization chain used by other providers, and previous review feedback on reader-lock cleanup, the identity-comparison label function, the vi.hoisted/vi.stubGlobal test pattern, and the dead null guard have all been addressed.

  • TextToSpeechProvider gains a required readonly id: string field, collapsing resolveVoiceTtsLabel to a single ttsService.id property access and removing the per-provider localVoiceTtsProvider variable.
  • ElevenLabsTTS uses constructor-injected fetch (via ElevenLabsTtsConfig.fetchImpl) for testability; the test suite covers URL building, path encoding, empty-input rejection, and error response mapping without vi.hoisted or global patching.
  • persisted-config.ts widens the voice field from the OpenAI-specific enum to a plain string and extends SpeechProviderIdSchema with "elevenlabs", both documented as back-compat in the PR.

Confidence Score: 5/5

Safe to merge; the three new files form a contained module and the integration touches only the expected wiring points.

The feature core is cleanly isolated, the initialization pipeline is unchanged in structure, all previous review concerns have been addressed, and the remaining findings are minor naming and defensive-code nits that do not affect correctness.

tts.ts has two minor issues worth a second look: the misleading DEFAULT_ELEVENLABS_VOICE_ID export and the fetchImpl test seam surfacing in the public PaseoElevenLabsConfig type.

Important Files Changed

Filename Overview
packages/server/src/server/speech/providers/elevenlabs/tts.ts Core ElevenLabsTTS class; correctly injects fetch, uses Readable.fromWeb for streaming, handles error parsing. Minor: DEFAULT_ELEVENLABS_VOICE_ID exported but never used as a default; fetchImpl test seam exposed in public config type.
packages/server/src/server/speech/providers/elevenlabs/runtime.ts Availability check, credential validation, and service initialization for ElevenLabs. Minor: redundant && elevenlabsConfig guard when hasTts already proves it non-null.
packages/server/src/server/speech/providers/elevenlabs/config.ts Config resolver wires persisted-config and env vars into ElevenLabsTtsConfig; priority order (persisted > env) is correct and consistent with other providers.
packages/server/src/server/speech/speech-runtime.ts ElevenLabs slot wired into the initialization pipeline (local → openai → elevenlabs); resolveVoiceTtsLabel simplified to use ttsService.id; localVoiceTtsProvider variable removed cleanly.
packages/server/src/server/speech/speech-provider.ts TextToSpeechProvider gains required readonly id: string; all three TTS implementations (OpenAI, local, ElevenLabs) now set it, collapsing the label function to a single property access.
packages/server/src/server/speech/providers/elevenlabs/tts.test.ts Tests use fetchImpl injection (no vi.hoisted / vi.stubGlobal); covers URL building, path encoding, empty-input rejection, and error response mapping.
packages/server/src/server/speech/providers/elevenlabs/runtime.test.ts Tests cover availability predicate, service construction when provider matches, passthrough when it doesn't, and no-throw for valid configuration.
packages/server/src/server/persisted-config.ts Adds ElevenLabsProviderSchema, extends SpeechProviderIdSchema with "elevenlabs", and widens voice field from OpenAI enum to string — documented as back-compat in the PR.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Config as resolveElevenLabsSpeechConfig
    participant Runtime as createSpeechService
    participant ELRuntime as initializeElevenLabsSpeechServices
    participant TTS as ElevenLabsTTS
    participant API as ElevenLabs API

    Runtime->>Config: "{ env, persisted }"
    Config-->>Runtime: "ElevenLabsTtsConfig | undefined"

    Runtime->>ELRuntime: "{ providers, elevenlabsConfig, existing }"
    ELRuntime->>TTS: new ElevenLabsTTS(config, logger)
    ELRuntime-->>Runtime: ExistingSpeechServices (ttsService set)

    Note over Runtime: resolveVoiceTtsLabel → ttsService.id = "elevenlabs"

    Runtime->>TTS: synthesizeSpeech(text)
    TTS->>API: "POST /v1/text-to-speech/{voiceId}?output_format=mp3_44100_128"
    API-->>TTS: "ReadableStream<Uint8Array> (MP3)"
    TTS-->>Runtime: "{ stream: Readable, format: "mp3" }"
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Config as resolveElevenLabsSpeechConfig
    participant Runtime as createSpeechService
    participant ELRuntime as initializeElevenLabsSpeechServices
    participant TTS as ElevenLabsTTS
    participant API as ElevenLabs API

    Runtime->>Config: "{ env, persisted }"
    Config-->>Runtime: "ElevenLabsTtsConfig | undefined"

    Runtime->>ELRuntime: "{ providers, elevenlabsConfig, existing }"
    ELRuntime->>TTS: new ElevenLabsTTS(config, logger)
    ELRuntime-->>Runtime: ExistingSpeechServices (ttsService set)

    Note over Runtime: resolveVoiceTtsLabel → ttsService.id = "elevenlabs"

    Runtime->>TTS: synthesizeSpeech(text)
    TTS->>API: "POST /v1/text-to-speech/{voiceId}?output_format=mp3_44100_128"
    API-->>TTS: "ReadableStream<Uint8Array> (MP3)"
    TTS-->>Runtime: "{ stream: Readable, format: "mp3" }"
Loading

Reviews (5): Last reviewed commit: "refactor(speech): drop redundant ElevenL..." | Re-trigger Greptile

Comment thread packages/server/src/server/speech/providers/elevenlabs/runtime.ts Outdated
Comment thread packages/server/src/server/speech/speech-runtime.ts Outdated
Comment thread packages/server/src/server/speech/providers/elevenlabs/config.ts Outdated
@greptile-apps

greptile-apps Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Want your agent to iterate on Greptile's feedback? Try greploops.

Comment thread packages/server/src/server/speech/providers/elevenlabs/tts.ts Outdated
ilteoood added 2 commits July 11, 2026 21:50
- Drop per-provider identity parameter from resolveVoiceTtsLabel.
  Add `id` to the TextToSpeechProvider contract and to all three TTS
  classes (openai, local, elevenlabs); the runtime now reads
  ttsService.id directly. Future TTS providers only declare their id
  and the runtime picks them up without parameter-list growth.

- Inject `fetchImpl` into ElevenLabsTTS via the constructor so tests
  can pass a plain stub instead of patching the global. Removes the
  vi.hoisted/vi.stubGlobal antipattern from tts.test.ts.

- Drop unused getConfig() from ElevenLabsTTS (was leaking the API key
  and had no callers).

- Remove the dead null guard on elevenlabsConfig.tts in
  createElevenLabsTts; the type requires tts and the caller already
  validates it.

- Drop the `as { ... }` type assertion in resolveElevenLabsSpeechConfig;
  PersistedConfigSchema already infers the right type for the
  elevenlabs provider block.

All 41 speech tests stay green.
Replace the hand-rolled reader-loop helper with Node's native
Readable.fromWeb. Drops ~16 lines and one extra async generator. The
fetch Web stream and Readable.fromWeb nominally type as distinct
ReadableStream generics (DOM vs stream/web), so the boundary gets a
single `as unknown as NodeReadableStream<Uint8Array>` cast — same
shape the OpenAI provider already uses to bridge the same gap.
Comment thread packages/server/src/server/speech/speech-runtime.ts Outdated
ilteoood added 2 commits July 11, 2026 22:23
- Make TextToSpeechProvider.id required. The previous optional
  declaration forced resolveVoiceTtsLabel to keep a `?? "openai"`
  fallback that silently mislabeled any provider missing the field.
  All four TTS implementations (openai, local via Sherpa, local via
  worker client, elevenlabs) already set id, so making it required
  surfaces any future omission at compile time.

- Flatten ElevenLabsSpeechProviderConfig to ElevenLabsTtsConfig. The
  outer wrapper carried apiKey/baseUrl as copies of values that lived
  on the inner tts object and were the only ones actually consumed;
  ElevenLabs is TTS-only, so the OpenAI-shaped { stt, tts } wrapper
  is dead weight. Resolver, runtime initializer, validator, and tests
  updated to read apiKey/voiceId directly.
The provider config and the TTS config are now the same type, so the
internal alias added no information. Consumers (runtime, runtime
tests, bootstrap) import ElevenLabsTtsConfig directly from tts.ts;
PaseoElevenLabsConfig remains in bootstrap.ts as the public-API alias
matching the existing PaseoOpenAIConfig pattern.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support elevenlabs tts

2 participants