Skip to content

bug: Allow passing options to useSpeechToText model.stream() #648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mkopcins merged 3 commits into from
Oct 15, 2025
Merged

bug: Allow passing options to useSpeechToText model.stream() #648

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
019be08
Allow passing options to useSpeechToText model.stream()
michaldudek Oct 14, 2025
6dcdbdb
Update model.stream() docs.
michaldudek Oct 14, 2025
017d914
Fix lint.
michaldudek Oct 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/02-hooks/01-natural-language-processing/useSpeechToText.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ For more information on loading resources, take a look at [loading models](../..
| Field | Type | Description |
| --------------------------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `transcribe` | `(waveform: Float32Array \| number[], options?: DecodingOptions \| undefined) => Promise<string>` | Starts a transcription process for a given input array, which should be a waveform at 16kHz. The second argument is an options object, e.g. `{ language: 'es' }` for multilingual models. Resolves a promise with the output transcription when the model is finished. Passing `number[]` is deprecated. |
| `stream` | `() => Promise<string>` | Starts a streaming transcription process. Use in combination with `streamInsert` to feed audio chunks and `streamStop` to end the stream. Updates `committedTranscription` and `nonCommittedTranscription` as transcription progresses. |
| `stream` | `(options?: DecodingOptions \| undefined) => Promise<string>` | Starts a streaming transcription process. Use in combination with `streamInsert` to feed audio chunks and `streamStop` to end the stream. The argument is an options object, e.g. `{ language: 'es' }` for multilingual models. Updates `committedTranscription` and `nonCommittedTranscription` as transcription progresses. |
| `streamInsert` | `(waveform: Float32Array \| number[]) => void` | Inserts a chunk of audio data (sampled at 16kHz) into the ongoing streaming transcription. Call this repeatedly as new audio data becomes available. Passing `number[]` is deprecated. |
| `streamStop` | `() => void` | Stops the ongoing streaming transcription process. |
| `encode` | `(waveform: Float32Array \| number[]) => Promise<Float32Array>` | Runs the encoding part of the model on the provided waveform. Passing `number[]` is deprecated. |
Expand Down
41 changes: 23 additions & 18 deletions packages/react-native-executorch/src/hooks/natural_language_processing/useSpeechToText.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { useEffect, useCallback, useState } from 'react';
import { ETError, getError } from '../../Error';
import { SpeechToTextModule } from '../../modules/natural_language_processing/SpeechToTextModule';
import { SpeechToTextModelConfig } from '../../types/stt';
import { DecodingOptions, SpeechToTextModelConfig } from '../../types/stt';

export const useSpeechToText = ({
model,
Expand Down Expand Up @@ -65,24 +65,29 @@ export const useSpeechToText = ({
[isReady, isGenerating, modelInstance]
);

const stream = useCallback(async () => {
if (!isReady) throw new Error(getError(ETError.ModuleNotLoaded));
if (isGenerating) throw new Error(getError(ETError.ModelGenerating));
setIsGenerating(true);
setCommittedTranscription('');
setNonCommittedTranscription('');
let transcription = '';
try {
for await (const { committed, nonCommitted } of modelInstance.stream()) {
setCommittedTranscription((prev) => prev + committed);
setNonCommittedTranscription(nonCommitted);
transcription += committed;
const stream = useCallback(
async (options?: DecodingOptions) => {
if (!isReady) throw new Error(getError(ETError.ModuleNotLoaded));
if (isGenerating) throw new Error(getError(ETError.ModelGenerating));
setIsGenerating(true);
setCommittedTranscription('');
setNonCommittedTranscription('');
let transcription = '';
try {
for await (const { committed, nonCommitted } of modelInstance.stream(
options
)) {
setCommittedTranscription((prev) => prev + committed);
setNonCommittedTranscription(nonCommitted);
transcription += committed;
}
} finally {
setIsGenerating(false);
}
} finally {
setIsGenerating(false);
}
return transcription;
}, [isReady, isGenerating, modelInstance]);
return transcription;
},
[isReady, isGenerating, modelInstance]
);

const wrapper = useCallback(
<T extends (...args: any[]) => any>(fn: T) => {
Expand Down
Loading