Skip to content

Commit 061e765

Browse files
IgorSwatIgorSwat
committed
Update docs & add supertonic info
1 parent 35ef417 commit 061e765

3 files changed

Lines changed: 175 additions & 110 deletions

File tree

‎docs/docs/03-hooks/01-natural-language-processing/useTextToSpeech.md‎

Lines changed: 77 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ keywords: [
66
voice synthesizer,
77
transcription,
88
kokoro,
9+
supertonic,
910
react native,
1011
executorch,
1112
ai,
@@ -19,7 +20,10 @@ description: "Learn how to use text-to-speech models in your React Native applic
1920
Text to speech is a task that allows to transform written text into spoken language. It is commonly used to implement features such as voice assistants, accessibility tools, or audiobooks.
2021

2122
:::info
22-
It is recommended to use models provided by us, which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-kokoro). You can also use [constants](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts) shipped with our library.
23+
It is recommended to use models provided by us, which are available at our Hugging Face repositories:
24+
[Kokoro](https://huggingface.co/software-mansion/react-native-executorch-kokoro) and
25+
[Supertonic 3](https://huggingface.co/software-mansion/react-native-executorch-supertonic).
26+
You can also use [constants](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts) shipped with our library.
2327
:::
2428

2529
## API Reference
@@ -32,6 +36,35 @@ It is recommended to use models provided by us, which are available at our [Hugg
3236

3337
You can play the generated waveform in any way most suitable to you; however, in the snippet below we utilize the react-native-audio-api library to play synthesized speech.
3438

39+
### Supertonic 3
40+
41+
```typescript
42+
import { models, useTextToSpeech } from 'react-native-executorch';
43+
import { AudioContext } from 'react-native-audio-api';
44+
45+
const model = useTextToSpeech(models.text_to_speech.supertonic.m1());
46+
47+
const audioContext = new AudioContext({ sampleRate: 44100 });
48+
49+
const handleSpeech = async (text: string) => {
50+
const waveform = await model.forward({
51+
text,
52+
totalSteps: 8,
53+
lang: 'en',
54+
});
55+
56+
const audioBuffer = audioContext.createBuffer(1, waveform.length, 44100);
57+
audioBuffer.getChannelData(0).set(waveform);
58+
59+
const source = audioContext.createBufferSource();
60+
source.buffer = audioBuffer;
61+
source.connect(audioContext.destination);
62+
source.start();
63+
};
64+
```
65+
66+
### Kokoro
67+
3568
```typescript
3669
import { models, useTextToSpeech } from 'react-native-executorch';
3770
import { AudioContext } from 'react-native-audio-api';
@@ -58,9 +91,10 @@ const handleSpeech = async (text: string) => {
5891

5992
`useTextToSpeech` takes [`TextToSpeechModelConfig`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md) that consists of:
6093

61-
- `model` of type [`TextToSpeechModelSources`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md) containing the [`durationPredictorSource`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#durationpredictorsource), [`synthesizerSource`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#synthesizersource), and [`modelName`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#modelname).
62-
- [`voiceSource`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#voicesource) of type [`ResourceSource`](../../06-api-reference/type-aliases/ResourceSource.md) - configuration of specific voice used in TTS.
63-
- [`phonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#phonemizerconfig) of type [`TextToSpeechPhonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechPhonemizerConfig.md) - configuration of the phonemizer.
94+
- `model` of type [`TextToSpeechModelSources`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md) — model configuration.
95+
- [`voiceSource`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#voicesource) of type [`ResourceSource`](../../06-api-reference/type-aliases/ResourceSource.md) — the voice tensor used for synthesis.
96+
- [`phonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#phonemizerconfig) of type [`TextToSpeechPhonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechPhonemizerConfig.md) — Kokoro only: phonemizer configuration. Unused by Supertonic.
97+
- [`lang`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#lang) of type [`TextToSpeechSupertonicLanguage`](../../06-api-reference/type-aliases/TextToSpeechSupertonicLanguage.md) — Supertonic only: default language token (e.g. `'en'`, `'na'`). Unused by Kokoro.
6498

6599
`useTextToSpeech`'s second optional argument is an object with:
66100

@@ -79,21 +113,28 @@ You need more details? Check the following resources:
79113

80114
## Running the model
81115

82-
The module provides two ways to generate speech using either raw text or pre-generated phonemes:
116+
The module provides two ways to generate speech. The available parameters differ by model family:
117+
118+
| Parameter | Kokoro | Supertonic |
119+
| ------------ | ------ | ---------- |
120+
| `speed` | ✓ | |
121+
| `phonemize` | ✓ | |
122+
| `totalSteps` | | ✓ |
123+
| `lang` | | ✓ |
83124

84125
### Using Text
85126

86-
1. [**`forward({ text, speed, phonemize })`**](../../06-api-reference/interfaces/TextToSpeechType.md#forward): Generates the complete audio waveform at once. Returns a promise resolving to a `Float32Array`.
87-
2. [**`stream({ speed, phonemize, stopAutomatically, onNext, ... })`**](../../06-api-reference/interfaces/TextToSpeechType.md#stream): An async generator-like functionality (managed via callbacks like `onNext`) that yields chunks of audio as they are computed.
127+
1. [**`forward({ text, speed, phonemize, totalSteps, lang })`**](../../06-api-reference/interfaces/TextToSpeechType.md#forward): Generates the complete audio waveform at once. Returns a promise resolving to a `Float32Array`.
128+
2. [**`stream({ speed, phonemize, totalSteps, lang, stopAutomatically, onNext, ... })`**](../../06-api-reference/interfaces/TextToSpeechType.md#stream): An async generator-like functionality (managed via callbacks like `onNext`) that yields chunks of audio as they are computed.
88129
This is ideal for reducing the "time to first audio" for long sentences. You can also dynamically insert text during the generation process using `streamInsert(text)`, force-partition trailing content without an end-of-sentence character via `streamFlush()`, and stop the stream with `streamStop(instant)`.
89130

90131
:::tip Recommendation
91132
In most cases, the **`stream()`** method is recommended over `forward()`. It significantly reduces latency by allowing audio playback to begin as soon as the first chunk is synthesized, rather than waiting for the entire text to be processed.
92133
:::
93134

94-
Both methods accept a `phonemize` parameter (defaults to `true`). When set to `true`, the input `text` is treated as raw text and converted to phonemes internally. When set to `false`, the input is expected to be a string of IPA phonemes.
135+
Both methods accept a `phonemize` parameter (defaults to `true`). This applies to **Kokoro only** — Supertonic maps text directly through a unicode indexer and does not use phonemization. When set to `true`, the input `text` is treated as raw text and converted to phonemes internally. When set to `false`, the input is expected to be a string of IPA phonemes.
95136

96-
### Using Phonemes
137+
### Using Phonemes (Kokoro only)
97138

98139
If you have pre-computed phonemes (e.g., from an external dictionary or a custom G2P model), you can skip the internal phoneme generation step:
99140

@@ -106,7 +147,7 @@ Since `forward` and `stream` process the input, they might take a significant am
106147

107148
## Example
108149

109-
### Speech Synthesis
150+
### Raw Synthesis (forward)
110151

111152
```tsx
112153
import React from 'react';
@@ -115,16 +156,24 @@ import { models, useTextToSpeech } from 'react-native-executorch';
115156
import { AudioContext } from 'react-native-audio-api';
116157

117158
export default function App() {
118-
const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
159+
// Supertonic 3 — multilingual, any voice works for any language
160+
const tts = useTextToSpeech(models.text_to_speech.supertonic.m1());
161+
// Kokoro — language-specific voice bundle:
162+
// const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
119163

120164
const generateAudio = async () => {
121165
const audioData = await tts.forward({
122166
text: 'Hello world! This is a sample text.',
167+
totalSteps: 8,
168+
lang: 'en',
169+
// Kokoro: text only (no totalSteps/lang):
170+
// text: 'Hello world! This is a sample text.',
123171
});
124172

125-
// Playback example
126-
const ctx = new AudioContext({ sampleRate: 24000 });
127-
const buffer = ctx.createBuffer(1, audioData.length, 24000);
173+
// Playback — sample rate depends on the model
174+
const ctx = new AudioContext({ sampleRate: 44100 });
175+
// Kokoro: sampleRate: 24000
176+
const buffer = ctx.createBuffer(1, audioData.length, ctx.sampleRate);
128177
buffer.getChannelData(0).set(audioData);
129178

130179
const source = ctx.createBufferSource();
@@ -150,18 +199,25 @@ import { models, useTextToSpeech } from 'react-native-executorch';
150199
import { AudioContext } from 'react-native-audio-api';
151200

152201
export default function App() {
153-
const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
202+
// Supertonic 3
203+
const tts = useTextToSpeech(models.text_to_speech.supertonic.m1());
204+
// Kokoro:
205+
// const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
154206

155-
const contextRef = useRef(new AudioContext({ sampleRate: 24000 }));
207+
const contextRef = useRef(new AudioContext({ sampleRate: 44100 }));
208+
// Kokoro: sampleRate: 24000
156209

157210
const generateStream = async () => {
158211
const ctx = contextRef.current;
159212

160213
await tts.stream({
161214
text: "This is a longer text, which is being streamed chunk by chunk. Let's see how it works!",
215+
totalSteps: 8,
216+
lang: 'en',
217+
// Kokoro: use speed: 1.0 instead of totalSteps/lang
162218
onNext: async (chunk) => {
163219
return new Promise((resolve) => {
164-
const buffer = ctx.createBuffer(1, chunk.length, 24000);
220+
const buffer = ctx.createBuffer(1, chunk.length, ctx.sampleRate);
165221
buffer.getChannelData(0).set(chunk);
166222

167223
const source = ctx.createBufferSource();
@@ -182,42 +238,9 @@ export default function App() {
182238
}
183239
```
184240

185-
### Synthesis from Phonemes
186-
187-
If you already have a phoneme string obtained from an external source (e.g. the Python `phonemizer` library,
188-
`espeak-ng`, or any custom phonemizer), you can use `forward` or `stream` with the `phonemize: false` flag to synthesize audio directly, skipping the phoneme generation stage.
189-
190-
```tsx
191-
import React from 'react';
192-
import { Button, View } from 'react-native';
193-
import { models, useTextToSpeech } from 'react-native-executorch';
194-
export default function App() {
195-
const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
196-
197-
const synthesizePhonemes = async () => {
198-
// Example phonemes for "Hello"
199-
const audioData = await tts.forward({
200-
text: 'ɐ mˈæn hˌu dˈʌzᵊnt tɹˈʌst hɪmsˈɛlf, kæn nˈɛvəɹ ɹˈiᵊli tɹˈʌst ˈɛniwˌʌn ˈɛls.',
201-
phonemize: false,
202-
});
203-
204-
// ... process or play audioData ...
205-
};
206-
207-
return (
208-
<View style={{ flex: 1, justifyContent: 'center', alignItems: 'center' }}>
209-
<Button
210-
title="Synthesize Phonemes"
211-
onPress={synthesizePhonemes}
212-
disabled={!tts.isReady}
213-
/>
214-
</View>
215-
);
216-
}
217-
```
218-
219241
## Supported models
220242

221-
| Model | Language |
222-
| -------------------------------------------------------------------------------- | :------------------------------------------------------------------: |
223-
| [Kokoro](https://huggingface.co/software-mansion/react-native-executorch-kokoro) | English, French, German, Spanish, Portuguese, Italian, Polish, Hindi |
243+
| Model | Language |
244+
| ------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
245+
| [Supertonic 3](https://huggingface.co/software-mansion/react-native-executorch-supertonic) | 31 languages + `na` (unknown) — Arabic, Bulgarian, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swahili, Swedish, Tagalog, Tamil, Thai, Turkish, Vietnamese |
246+
| [Kokoro](https://huggingface.co/software-mansion/react-native-executorch-kokoro) | English, French, German, Spanish, Portuguese, Italian, Polish, Hindi |

0 commit comments

Comments
 (0)