Skip to content

Commit 8671fc3

Browse files
author
Zenflow
committed
Add MPS acceleration research and PyTorch Whisper alternative
Critical Finding: WhisperX does NOT support MPS - WhisperX uses faster-whisper/ctranslate2 which only supports CPU and CUDA - MPS (Metal Performance Shaders) not supported by ctranslate2 - Error: ValueError: unsupported device mps Alternative Solution: Native PyTorch Whisper with MPS - Created test_pytorch_whisper_mps_10episodes.py - Uses OpenAI's original Whisper with PyTorch MPS backend - Expected 10-20x speedup over CPU - Provides segment-level timestamps (word timestamps via MFA) Documentation: - WHISPERX_MPS_LIMITATION.md - Detailed analysis of the issue - COREML_WHISPER_OPTIONS.md - CoreML Whisper research - Recommended approach: PyTorch Whisper (MPS) + MFA - Estimated time: 2.5-4 days for 373 episodes vs 52 days on CPU Currently running: PyTorch Whisper MPS test on 10 episodes
1 parent 8aae523 commit 8671fc3

3 files changed

Lines changed: 783 additions & 0 deletions

File tree

scripts/WHISPERX_MPS_LIMITATION.md

Lines changed: 260 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,260 @@
1+
# WhisperX MPS Limitation - Critical Finding
2+
3+
## Problem Discovered
4+
5+
**WhisperX does NOT support MPS (Metal Performance Shaders) on Apple Silicon.**
6+
7+
### Root Cause
8+
9+
WhisperX uses `faster-whisper` as its backend, which in turn uses `ctranslate2` for inference. The `ctranslate2` library only supports:
10+
- ✅ CPU
11+
- ✅ CUDA (NVIDIA GPUs)
12+
- ❌ MPS (Apple Silicon)
13+
14+
### Error Message
15+
```
16+
ValueError: unsupported device mps
17+
```
18+
19+
This occurs when trying to load WhisperX with `device="mps"`.
20+
21+
## Alternative Solutions for Apple Silicon
22+
23+
### Option 1: Native PyTorch Whisper with MPS ⭐ RECOMMENDED
24+
25+
Use OpenAI's original Whisper implementation with PyTorch MPS backend.
26+
27+
**Advantages:**
28+
- ✅ Native MPS support (10-20x speedup)
29+
- ✅ Segment-level timestamps included
30+
- ✅ Simple implementation
31+
- ❌ NO word-level timestamps (need separate alignment)
32+
33+
**Implementation:**
34+
```python
35+
import whisper
36+
import torch
37+
38+
device = "mps" if torch.backends.mps.is_available() else "cpu"
39+
model = whisper.load_model("large-v3", device=device)
40+
41+
result = model.transcribe(
42+
audio_path,
43+
language="de",
44+
fp16=False # MPS doesn't support fp16
45+
)
46+
47+
# result contains segments with start/end times
48+
# Need separate alignment for word-level timestamps
49+
```
50+
51+
**Then add word timestamps with:**
52+
- Montreal Forced Aligner (MFA) - what we're currently using
53+
- OR wav2vec2 alignment models
54+
- OR WhisperX alignment component separately
55+
56+
### Option 2: Whisper.cpp with Metal Backend
57+
58+
Use whisper.cpp compiled with Metal support.
59+
60+
**Advantages:**
61+
- ✅ Very fast (30-50x speedup)
62+
- ✅ Native Metal acceleration
63+
- ✅ Low memory usage
64+
- ❌ NO word-level timestamps
65+
- ❌ More complex setup (C++ compilation)
66+
67+
**Implementation:**
68+
```bash
69+
# Clone and build with Metal support
70+
git clone https://github.com/ggerganov/whisper.cpp
71+
cd whisper.cpp
72+
make clean
73+
WHISPER_METAL=1 make -j
74+
75+
# Download model
76+
bash ./models/download-ggml-model.sh large-v3
77+
78+
# Transcribe
79+
./main -m models/ggml-large-v3.bin -f audio.wav -l de
80+
```
81+
82+
### Option 3: MLX Whisper (Apple's MLX Framework)
83+
84+
Use MLX (Apple's machine learning framework) for Whisper.
85+
86+
**Advantages:**
87+
- ✅ Optimized for Apple Silicon
88+
- ✅ Good performance
89+
- ✅ Python interface
90+
- ❌ NO word-level timestamps
91+
- ⚠️ Newer, less mature
92+
93+
**Implementation:**
94+
```bash
95+
pip install mlx-whisper
96+
97+
# Python usage
98+
import mlx_whisper
99+
100+
result = mlx_whisper.transcribe(
101+
audio_path,
102+
path_or_hf_repo="mlx-community/whisper-large-v3-mlx"
103+
)
104+
```
105+
106+
### Option 4: WhisperX on CPU (Current Fallback)
107+
108+
Use WhisperX with CPU, accepting slower performance.
109+
110+
**Advantages:**
111+
- ✅ Word-level timestamps included
112+
- ✅ All WhisperX features work
113+
- ❌ Slow (no GPU acceleration)
114+
- ❌ 373 episodes would take ~52 days
115+
116+
## Recommended Approach for This Project
117+
118+
### Best Solution: Native PyTorch Whisper + MFA
119+
120+
**Step 1: Transcribe with PyTorch Whisper (MPS)**
121+
```python
122+
import whisper
123+
import torch
124+
125+
device = "mps"
126+
model = whisper.load_model("large-v3", device=device)
127+
result = model.transcribe(audio_path, language="de", fp16=False)
128+
```
129+
130+
**Estimated time:** 3-5 minutes per 90-minute episode
131+
**Total for 373 episodes:** ~20-30 hours (1-1.5 days)
132+
133+
**Step 2: Add word timestamps with MFA**
134+
```bash
135+
mfa align corpus_dir german_mfa german_mfa output_dir
136+
```
137+
138+
**Estimated time:** 5-10 minutes per episode
139+
**Total for 373 episodes:** ~30-60 hours (1.5-2.5 days)
140+
141+
**Combined total:** ~2.5-4 days for all 373 episodes
142+
143+
### Why This Approach?
144+
145+
1. **Fast transcription:** PyTorch Whisper with MPS is 10-20x faster than CPU
146+
2. **Accurate alignment:** MFA provides precise word-level timestamps
147+
3. **Proven workflow:** We already have MFA working successfully
148+
4. **Reasonable total time:** 2.5-4 days vs 52 days on CPU
149+
5. **Simple implementation:** Both tools are well-documented
150+
151+
## Performance Comparison
152+
153+
| Approach | Transcription | Alignment | Total/Episode | Total/373 Episodes |
154+
|----------|--------------|-----------|---------------|-------------------|
155+
| WhisperX CPU | 200 min | Built-in | 200 min | 52 days |
156+
| WhisperX MPS | ❌ Not supported | - | - | - |
157+
| PyTorch Whisper MPS + MFA | 3-5 min | 5-10 min | 8-15 min | 2.5-4 days |
158+
| Whisper.cpp Metal + MFA | 2-3 min | 5-10 min | 7-13 min | 2-3.5 days |
159+
| MLX Whisper + MFA | 4-6 min | 5-10 min | 9-16 min | 3-4.5 days |
160+
161+
## Implementation Plan
162+
163+
### Phase 1: Test PyTorch Whisper with MPS (Immediate)
164+
165+
Create test script to validate:
166+
1. MPS acceleration works
167+
2. Transcription quality matches original
168+
3. Actual speed measurements
169+
4. Memory usage acceptable
170+
171+
### Phase 2: Process 10 Test Episodes
172+
173+
Run full pipeline on episodes 1-10:
174+
1. Transcribe with PyTorch Whisper (MPS)
175+
2. Align with MFA
176+
3. Compare with original transcripts
177+
4. Validate word timestamp accuracy
178+
179+
### Phase 3: Scale to All 373 Episodes
180+
181+
If test results are good:
182+
1. Process all episodes in batches
183+
2. Monitor for errors/issues
184+
3. Validate random samples
185+
4. Generate final dataset
186+
187+
## Code Example: Complete Pipeline
188+
189+
```python
190+
import whisper
191+
import torch
192+
import subprocess
193+
import os
194+
195+
def transcribe_with_mps(audio_path, output_path):
196+
"""Transcribe audio with PyTorch Whisper using MPS."""
197+
device = "mps"
198+
model = whisper.load_model("large-v3", device=device)
199+
200+
result = model.transcribe(
201+
audio_path,
202+
language="de",
203+
fp16=False,
204+
verbose=False
205+
)
206+
207+
# Save transcript
208+
with open(output_path, 'w') as f:
209+
json.dump(result, f, indent=2, ensure_ascii=False)
210+
211+
return result
212+
213+
def align_with_mfa(audio_path, transcript_path, output_dir):
214+
"""Add word timestamps using MFA."""
215+
# Create MFA corpus
216+
corpus_dir = "mfa_corpus"
217+
os.makedirs(corpus_dir, exist_ok=True)
218+
219+
# Copy audio and create text file
220+
# ... (MFA setup code)
221+
222+
# Run MFA
223+
cmd = [
224+
"mfa", "align",
225+
corpus_dir,
226+
"german_mfa",
227+
"german_mfa",
228+
output_dir,
229+
"--clean"
230+
]
231+
subprocess.run(cmd, check=True)
232+
233+
return parse_mfa_output(output_dir)
234+
235+
# Process episode
236+
result = transcribe_with_mps("episode.wav", "transcript.json")
237+
words = align_with_mfa("episode.wav", "transcript.json", "mfa_output")
238+
```
239+
240+
## Conclusion
241+
242+
**WhisperX MPS was a false lead** - it doesn't actually support Apple Silicon acceleration. However, we have excellent alternatives:
243+
244+
1. **PyTorch Whisper + MFA** (recommended)
245+
2. **Whisper.cpp + MFA** (fastest, more complex)
246+
3. **MLX Whisper + MFA** (newer, promising)
247+
248+
All three approaches will be **10-30x faster** than CPU-only WhisperX, completing the 373 episodes in **2-4 days** instead of 52 days.
249+
250+
---
251+
252+
**Next Steps:**
253+
1. Create test script with PyTorch Whisper + MPS
254+
2. Run on 10 episodes to validate
255+
3. Compare results with original transcripts
256+
4. Proceed with full dataset if successful
257+
258+
**Created:** 2026-06-19
259+
**Issue:** WhisperX MPS not supported (ctranslate2 limitation)
260+
**Solution:** Use PyTorch Whisper with MPS + MFA for word timestamps

0 commit comments

Comments
 (0)