Commit d3182ce
feat!: major performance & accuracy improvements in speech-to-text module (#1132)
## Description
This PR introduces several changes to the speech-to-text module based on
Whisper models:
- CoreML integration - models re-exported to CoreML backend, bringing
significant performance upgrade for iOS devices.
- New streaming algorithm - eliminates duplicates in streaming output,
resulting in a major quality improvement of the live streaming mode.
- Changes in demo apps: removed faulty 'voice mode' screen in LLM demo
app, refactored speech to text screen in 'speech' app by adding new
CoreML models to selection bar and changing the default model for iOS
devices.
- Minor code improvements in speech-to-text module
### Introduces a breaking change?
- [x] Yes
- [ ] No
**Change**: removes predefined constants for quantized models.
**Justification**: the quantized models differ very slightly from the
original ones, introducing unnecessary complexity in this case.
### Type of change
- [x] Bug fix (change which fixes an issue)
- [x] New feature (change which adds functionality)
- [x] Documentation update (improves or adds clarity to existing
documentation)
- [x] Other (chores, tests, code style improvements etc.)
### Tested on
- [x] iOS
- [x] Android
### Testing instructions
Run demo app to test the live streaming mode.
### Screenshots
<!-- Add screenshots here, if applicable -->
### Related issues
#1124
### Checklist
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have updated the documentation accordingly
- [x] My changes generate no new warnings
### Additional notes
---------
Co-authored-by: Mateusz Słuszniak <msluszniak1@gmail.com>1 parent d04a36e commit d3182ce
25 files changed
Lines changed: 670 additions & 1358 deletions
File tree
- apps
- llm/app
- voice_chat
- speech/screens
- docs/docs
- 03-hooks/01-natural-language-processing
- 04-typescript-api/01-natural-language-processing
- packages/react-native-executorch
- common/rnexecutorch
- host_objects
- models/speech_to_text
- common
- schema
- types
- whisper
- tests
- src
- constants
- hooks/natural_language_processing
- modules/natural_language_processing
- types
- skills
- canary/react-native-executorch/references
- react-native-executorch/references
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | 32 | | |
39 | 33 | | |
40 | 34 | | |
| |||
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
| 21 | + | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
| |||
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
32 | 36 | | |
33 | 37 | | |
34 | 38 | | |
| |||
45 | 49 | | |
46 | 50 | | |
47 | 51 | | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
48 | 55 | | |
49 | 56 | | |
50 | | - | |
| 57 | + | |
51 | 58 | | |
52 | 59 | | |
53 | 60 | | |
| |||
148 | 155 | | |
149 | 156 | | |
150 | 157 | | |
151 | | - | |
| 158 | + | |
152 | 159 | | |
153 | 160 | | |
154 | 161 | | |
| |||
178 | 185 | | |
179 | 186 | | |
180 | 187 | | |
| 188 | + | |
181 | 189 | | |
182 | 190 | | |
183 | 191 | | |
| |||
0 commit comments