Skip to content

Add AMD GPU support: faster_whisper CPU fallback + whisper.cpp backend#522

Closed
Kaihui-AMD wants to merge 2 commits into
collabora:mainfrom
Kaihui-AMD:amd-gpu-support
Closed

Add AMD GPU support: faster_whisper CPU fallback + whisper.cpp backend#522
Kaihui-AMD wants to merge 2 commits into
collabora:mainfrom
Kaihui-AMD:amd-gpu-support

Conversation

@Kaihui-AMD

Copy link
Copy Markdown
Contributor

Summary

Makes WhisperLive work on AMD GPUs (#520):

  1. Fix faster_whisper crashing on AMD/ROCm — fall back to CPU instead of crashing.
  2. Add a whisper_cpp backend — real GPU transcription on AMD (Vulkan/ROCm), Intel, Apple.

Verified on a Radeon AI PRO R9700 (RDNA4, gfx1201) and a Ryzen AI Max+ 395 / Radeon 8060S
("Strix Halo", gfx1151 — the #520 chip). CUDA / TensorRT / OpenVINO paths unaffected.

1. faster_whisper: don't crash on AMD

CTranslate2's PyPI wheels are CUDA-only, but on ROCm torch.cuda.is_available() is True, so
the server picked device="cuda" and crashed (RuntimeError: CUDA failed ...). Guard on the
CTranslate2 device count instead:

use_cuda = torch.cuda.is_available() and ctranslate2.get_cuda_device_count() > 0
device = "cuda" if use_cuda else "cpu"

Applied to all three WhisperModel sites (the websocket backend + both REST endpoints in
server.py). CUDA users are unaffected; diarization/translation use torch and already work on
ROCm.

2. New whisper_cpp backend (GPU on AMD)

python run_server.py -b whisper_cpp

ServeClientWhisperCpp runs whisper.cpp via
pywhispercpp — ggml's Vulkan/ROCm/Metal backends
give real GPU transcription where CTranslate2 can't. Vulkan is the portable path (one build →
AMD/Intel/NVIDIA). It plugs into the existing server (same protocol/clients) and falls back to
faster_whisper if pywhispercpp is missing. Adds docker/Dockerfile.vulkan (turnkey) + README.

Build pywhispercpp with GPU (from git; setup.py forwards GGML_* to CMake):

GGML_VULKAN=1 pip install "git+https://github.com/absadiki/pywhispercpp.git"                    # portable
GGML_HIP=1 AMDGPU_TARGETS=gfx1151 pip install "git+https://github.com/absadiki/pywhispercpp.git" # ROCm

Verification

samples/jfk.wav (11 s), exact-text match; drove the real backend classes plus a full
run_server.py -b whisper_cpp end-to-end over WebSocket.

  • faster_whisper fix: device=cuda crash → now CPU, transcribes correctly.
  • whisper_cpp GPU backend:
box / chip backend device correct? time (cold/warm)¹
R9700 · gfx1201 Vulkan Vulkan0 (RADV GFX1201) 0.055 / 0.049 s
gfx1201 — full server E2E (WebSocket) Vulkan Vulkan0 streamed OK
Ryzen AI Max+ 395 / 8060S · gfx1151 (#520 chip) Vulkan Vulkan0 (RADV GFX1151) 0.336 / 0.071 s
gfx1151 ROCm/HIP ROCm0 0.135 / 0.077 s

¹ ggml-base.en, single runs — verification timings, not a formal benchmark.

Notes

  • pywhispercpp must be source-built for GPU (PyPI is CPU-only); Dockerfile.vulkan handles it.
    RDNA4 needs a recent Mesa/RADV.
  • First version: no_speech_prob is 0.0, word timestamps + whisper.cpp VAD not wired, and
    the OpenAI REST endpoint still uses CTranslate2 (CPU on AMD) — only the websocket backend is GPU.
  • CTranslate2's ROCm forks are intentionally avoided (they produce wrong output on RDNA4).

@makaveli10

Copy link
Copy Markdown
Collaborator

Closing for #523

@makaveli10 makaveli10 closed this Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants