|
| 1 | +# WhisperX MPS Limitation - Critical Finding |
| 2 | + |
| 3 | +## Problem Discovered |
| 4 | + |
| 5 | +**WhisperX does NOT support MPS (Metal Performance Shaders) on Apple Silicon.** |
| 6 | + |
| 7 | +### Root Cause |
| 8 | + |
| 9 | +WhisperX uses `faster-whisper` as its backend, which in turn uses `ctranslate2` for inference. The `ctranslate2` library only supports: |
| 10 | +- ✅ CPU |
| 11 | +- ✅ CUDA (NVIDIA GPUs) |
| 12 | +- ❌ MPS (Apple Silicon) |
| 13 | + |
| 14 | +### Error Message |
| 15 | +``` |
| 16 | +ValueError: unsupported device mps |
| 17 | +``` |
| 18 | + |
| 19 | +This occurs when trying to load WhisperX with `device="mps"`. |
| 20 | + |
| 21 | +## Alternative Solutions for Apple Silicon |
| 22 | + |
| 23 | +### Option 1: Native PyTorch Whisper with MPS ⭐ RECOMMENDED |
| 24 | + |
| 25 | +Use OpenAI's original Whisper implementation with PyTorch MPS backend. |
| 26 | + |
| 27 | +**Advantages:** |
| 28 | +- ✅ Native MPS support (10-20x speedup) |
| 29 | +- ✅ Segment-level timestamps included |
| 30 | +- ✅ Simple implementation |
| 31 | +- ❌ NO word-level timestamps (need separate alignment) |
| 32 | + |
| 33 | +**Implementation:** |
| 34 | +```python |
| 35 | +import whisper |
| 36 | +import torch |
| 37 | + |
| 38 | +device = "mps" if torch.backends.mps.is_available() else "cpu" |
| 39 | +model = whisper.load_model("large-v3", device=device) |
| 40 | + |
| 41 | +result = model.transcribe( |
| 42 | + audio_path, |
| 43 | + language="de", |
| 44 | + fp16=False # MPS doesn't support fp16 |
| 45 | +) |
| 46 | + |
| 47 | +# result contains segments with start/end times |
| 48 | +# Need separate alignment for word-level timestamps |
| 49 | +``` |
| 50 | + |
| 51 | +**Then add word timestamps with:** |
| 52 | +- Montreal Forced Aligner (MFA) - what we're currently using |
| 53 | +- OR wav2vec2 alignment models |
| 54 | +- OR WhisperX alignment component separately |
| 55 | + |
| 56 | +### Option 2: Whisper.cpp with Metal Backend |
| 57 | + |
| 58 | +Use whisper.cpp compiled with Metal support. |
| 59 | + |
| 60 | +**Advantages:** |
| 61 | +- ✅ Very fast (30-50x speedup) |
| 62 | +- ✅ Native Metal acceleration |
| 63 | +- ✅ Low memory usage |
| 64 | +- ❌ NO word-level timestamps |
| 65 | +- ❌ More complex setup (C++ compilation) |
| 66 | + |
| 67 | +**Implementation:** |
| 68 | +```bash |
| 69 | +# Clone and build with Metal support |
| 70 | +git clone https://github.com/ggerganov/whisper.cpp |
| 71 | +cd whisper.cpp |
| 72 | +make clean |
| 73 | +WHISPER_METAL=1 make -j |
| 74 | + |
| 75 | +# Download model |
| 76 | +bash ./models/download-ggml-model.sh large-v3 |
| 77 | + |
| 78 | +# Transcribe |
| 79 | +./main -m models/ggml-large-v3.bin -f audio.wav -l de |
| 80 | +``` |
| 81 | + |
| 82 | +### Option 3: MLX Whisper (Apple's MLX Framework) |
| 83 | + |
| 84 | +Use MLX (Apple's machine learning framework) for Whisper. |
| 85 | + |
| 86 | +**Advantages:** |
| 87 | +- ✅ Optimized for Apple Silicon |
| 88 | +- ✅ Good performance |
| 89 | +- ✅ Python interface |
| 90 | +- ❌ NO word-level timestamps |
| 91 | +- ⚠️ Newer, less mature |
| 92 | + |
| 93 | +**Implementation:** |
| 94 | +```bash |
| 95 | +pip install mlx-whisper |
| 96 | + |
| 97 | +# Python usage |
| 98 | +import mlx_whisper |
| 99 | + |
| 100 | +result = mlx_whisper.transcribe( |
| 101 | + audio_path, |
| 102 | + path_or_hf_repo="mlx-community/whisper-large-v3-mlx" |
| 103 | +) |
| 104 | +``` |
| 105 | + |
| 106 | +### Option 4: WhisperX on CPU (Current Fallback) |
| 107 | + |
| 108 | +Use WhisperX with CPU, accepting slower performance. |
| 109 | + |
| 110 | +**Advantages:** |
| 111 | +- ✅ Word-level timestamps included |
| 112 | +- ✅ All WhisperX features work |
| 113 | +- ❌ Slow (no GPU acceleration) |
| 114 | +- ❌ 373 episodes would take ~52 days |
| 115 | + |
| 116 | +## Recommended Approach for This Project |
| 117 | + |
| 118 | +### Best Solution: Native PyTorch Whisper + MFA |
| 119 | + |
| 120 | +**Step 1: Transcribe with PyTorch Whisper (MPS)** |
| 121 | +```python |
| 122 | +import whisper |
| 123 | +import torch |
| 124 | + |
| 125 | +device = "mps" |
| 126 | +model = whisper.load_model("large-v3", device=device) |
| 127 | +result = model.transcribe(audio_path, language="de", fp16=False) |
| 128 | +``` |
| 129 | + |
| 130 | +**Estimated time:** 3-5 minutes per 90-minute episode |
| 131 | +**Total for 373 episodes:** ~20-30 hours (1-1.5 days) |
| 132 | + |
| 133 | +**Step 2: Add word timestamps with MFA** |
| 134 | +```bash |
| 135 | +mfa align corpus_dir german_mfa german_mfa output_dir |
| 136 | +``` |
| 137 | + |
| 138 | +**Estimated time:** 5-10 minutes per episode |
| 139 | +**Total for 373 episodes:** ~30-60 hours (1.5-2.5 days) |
| 140 | + |
| 141 | +**Combined total:** ~2.5-4 days for all 373 episodes |
| 142 | + |
| 143 | +### Why This Approach? |
| 144 | + |
| 145 | +1. **Fast transcription:** PyTorch Whisper with MPS is 10-20x faster than CPU |
| 146 | +2. **Accurate alignment:** MFA provides precise word-level timestamps |
| 147 | +3. **Proven workflow:** We already have MFA working successfully |
| 148 | +4. **Reasonable total time:** 2.5-4 days vs 52 days on CPU |
| 149 | +5. **Simple implementation:** Both tools are well-documented |
| 150 | + |
| 151 | +## Performance Comparison |
| 152 | + |
| 153 | +| Approach | Transcription | Alignment | Total/Episode | Total/373 Episodes | |
| 154 | +|----------|--------------|-----------|---------------|-------------------| |
| 155 | +| WhisperX CPU | 200 min | Built-in | 200 min | 52 days | |
| 156 | +| WhisperX MPS | ❌ Not supported | - | - | - | |
| 157 | +| PyTorch Whisper MPS + MFA | 3-5 min | 5-10 min | 8-15 min | 2.5-4 days | |
| 158 | +| Whisper.cpp Metal + MFA | 2-3 min | 5-10 min | 7-13 min | 2-3.5 days | |
| 159 | +| MLX Whisper + MFA | 4-6 min | 5-10 min | 9-16 min | 3-4.5 days | |
| 160 | + |
| 161 | +## Implementation Plan |
| 162 | + |
| 163 | +### Phase 1: Test PyTorch Whisper with MPS (Immediate) |
| 164 | + |
| 165 | +Create test script to validate: |
| 166 | +1. MPS acceleration works |
| 167 | +2. Transcription quality matches original |
| 168 | +3. Actual speed measurements |
| 169 | +4. Memory usage acceptable |
| 170 | + |
| 171 | +### Phase 2: Process 10 Test Episodes |
| 172 | + |
| 173 | +Run full pipeline on episodes 1-10: |
| 174 | +1. Transcribe with PyTorch Whisper (MPS) |
| 175 | +2. Align with MFA |
| 176 | +3. Compare with original transcripts |
| 177 | +4. Validate word timestamp accuracy |
| 178 | + |
| 179 | +### Phase 3: Scale to All 373 Episodes |
| 180 | + |
| 181 | +If test results are good: |
| 182 | +1. Process all episodes in batches |
| 183 | +2. Monitor for errors/issues |
| 184 | +3. Validate random samples |
| 185 | +4. Generate final dataset |
| 186 | + |
| 187 | +## Code Example: Complete Pipeline |
| 188 | + |
| 189 | +```python |
| 190 | +import whisper |
| 191 | +import torch |
| 192 | +import subprocess |
| 193 | +import os |
| 194 | + |
| 195 | +def transcribe_with_mps(audio_path, output_path): |
| 196 | + """Transcribe audio with PyTorch Whisper using MPS.""" |
| 197 | + device = "mps" |
| 198 | + model = whisper.load_model("large-v3", device=device) |
| 199 | + |
| 200 | + result = model.transcribe( |
| 201 | + audio_path, |
| 202 | + language="de", |
| 203 | + fp16=False, |
| 204 | + verbose=False |
| 205 | + ) |
| 206 | + |
| 207 | + # Save transcript |
| 208 | + with open(output_path, 'w') as f: |
| 209 | + json.dump(result, f, indent=2, ensure_ascii=False) |
| 210 | + |
| 211 | + return result |
| 212 | + |
| 213 | +def align_with_mfa(audio_path, transcript_path, output_dir): |
| 214 | + """Add word timestamps using MFA.""" |
| 215 | + # Create MFA corpus |
| 216 | + corpus_dir = "mfa_corpus" |
| 217 | + os.makedirs(corpus_dir, exist_ok=True) |
| 218 | + |
| 219 | + # Copy audio and create text file |
| 220 | + # ... (MFA setup code) |
| 221 | + |
| 222 | + # Run MFA |
| 223 | + cmd = [ |
| 224 | + "mfa", "align", |
| 225 | + corpus_dir, |
| 226 | + "german_mfa", |
| 227 | + "german_mfa", |
| 228 | + output_dir, |
| 229 | + "--clean" |
| 230 | + ] |
| 231 | + subprocess.run(cmd, check=True) |
| 232 | + |
| 233 | + return parse_mfa_output(output_dir) |
| 234 | + |
| 235 | +# Process episode |
| 236 | +result = transcribe_with_mps("episode.wav", "transcript.json") |
| 237 | +words = align_with_mfa("episode.wav", "transcript.json", "mfa_output") |
| 238 | +``` |
| 239 | + |
| 240 | +## Conclusion |
| 241 | + |
| 242 | +**WhisperX MPS was a false lead** - it doesn't actually support Apple Silicon acceleration. However, we have excellent alternatives: |
| 243 | + |
| 244 | +1. **PyTorch Whisper + MFA** (recommended) |
| 245 | +2. **Whisper.cpp + MFA** (fastest, more complex) |
| 246 | +3. **MLX Whisper + MFA** (newer, promising) |
| 247 | + |
| 248 | +All three approaches will be **10-30x faster** than CPU-only WhisperX, completing the 373 episodes in **2-4 days** instead of 52 days. |
| 249 | + |
| 250 | +--- |
| 251 | + |
| 252 | +**Next Steps:** |
| 253 | +1. Create test script with PyTorch Whisper + MPS |
| 254 | +2. Run on 10 episodes to validate |
| 255 | +3. Compare results with original transcripts |
| 256 | +4. Proceed with full dataset if successful |
| 257 | + |
| 258 | +**Created:** 2026-06-19 |
| 259 | +**Issue:** WhisperX MPS not supported (ctranslate2 limitation) |
| 260 | +**Solution:** Use PyTorch Whisper with MPS + MFA for word timestamps |
0 commit comments