Skip to content

feat(whisper): Vulkan GPU acceleration for AMD/Intel GPUs#1185

Merged
gabrielste1n merged 1 commit into
mainfrom
feat/whisper-vulkan-gpu
Jul 15, 2026
Merged

feat(whisper): Vulkan GPU acceleration for AMD/Intel GPUs#1185
gabrielste1n merged 1 commit into
mainfrom
feat/whisper-vulkan-gpu

Conversation

@gabrielste1n

Copy link
Copy Markdown
Collaborator

Summary

Local Whisper is GPU-accelerated on macOS (Metal) and NVIDIA (CUDA), but AMD Radeon and Intel Arc/iGPU users on Windows/Linux run CPU-only — the largest unaccelerated population. Since binaries release 0.0.7, OpenWhispr/whisper.cpp ships whisper-server-{win32,linux}-x64-vulkan.zip (statically linked; the only runtime dependency is the Vulkan loader every modern GPU driver provides), but nothing consumed them. This wires them in, mirroring the existing CUDA (whisper) and Vulkan (llama) flows: a one-click "Enable GPU" card, 2–5× faster transcription on larger models, automatic CPU fallback if Vulkan misbehaves.

Implements whisper-vulkan-gpu-acceleration.md.

What changed

Shared refactor (net-negative LOC)

  • New src/helpers/gpuBinaryManager.js extracts the download→verify→extract→install pipeline that whisperCudaManager and llamaVulkanManager had duplicated; both are now thin config subclasses. A third copy for whisper-Vulkan would have hit the 3-occurrence DRY threshold.
  • The pipeline adds streaming SHA256 verification, failing closed: against in-app pinned digests when configured, otherwise against the digest the GitHub releases API reports. (The downloaded exes are unsigned and release assets are mutable.)
  • Divergent wrapper semantics preserved: CUDA download() throws "Download cancelled by user" on abort, llama returns { success: false, cancelled: true }; delete result shapes unchanged. Covered by new characterization tests (test/helpers/gpuBinaryManager.test.js).
  • Deliberate deviations: CUDA disk pre-check tightens 2×→2.5× (llama's convention), the CUDA release fetch now sends GITHUB_TOKEN when set (rate limits), and archives stage in the OS temp dir instead of userData/bin.

Whisper Vulkan backend

  • whisperVulkanManager: pinned tag WHISPER_CPP_VERSION || "0.0.8" with pinned per-asset digests, exact asset names, no companion libs (statically linked — also avoids clobbering DLLs in the shared userData/bin).
  • whisperServer: resolves whisper-server-{platform}-{arch}-vulkan when preferVulkan, sticky useVulkan beside useCuda (CUDA wins on conflict), and a dedicated VULKAN_STARTUP_TIMEOUT_MS = 120000 — llama-Vulkan cold starts run 30–40s on Windows and whisper-large loads ~3 GiB before the port binds (see fix(llama): raise Vulkan startup timeout, stop server before re-download #698 for the timeout precedent).
  • Fallback: ANY Vulkan startup rejection ⇒ CPU — early exit, late death (VRAM OOM mid-model-load), or hang — mirroring llama's catch-any-rejection. CUDA keeps its shipped early-exit-only rule. Also: an intentional stop() during a pending startup now rejects the startup instead of resurrecting the server behind the caller (previously possible on the CUDA path too).
  • useVulkan threaded through startup pre-warm, per-dictation starts, and the Feature: Option to leave local transcription model loaded in VRAM while idle #766 wake re-warm, which now replays lastStartOptions instead of hardcoding useCuda: true.

IPC / UI

  • Four channels mirroring CUDA: get-vulkan-whisper-status (bundles Vulkan availability + hasNvidiaGpu so the renderer gates precedence in one call), download-/cancel-/delete-vulkan-whisper-binary, plus vulkan-whisper-download-progress. Download/delete stop the server first (Windows file-lock lesson); CUDA delete now does the same.
  • WHISPER_VULKAN_ENABLED joins PERSISTED_KEYS; gpu-fallback relayed beside the existing cuda-fallback broadcast.
  • The CUDA card in TranscriptionModelPicker is now a GPU card: NVIDIA → CUDA flow (unchanged), otherwise Vulkan-capable GPU → same card on the vulkan API. The Vulkan offer never shows on NVIDIA machines (CUDA is faster) or macOS. ControlPanel's unified GPU banner detects whisper-Vulkan too.
  • Zero new i18n strings — the existing gpu.* keys are backend-agnostic; the one hardcoded "Cancel" now uses t("gpu.cancel").

Verification

  • npm test (432 tests), npm run lint, npm run typecheck, npm run i18n:check all pass.
  • New characterization tests pin the shared pipeline behavior: asset resolution (exact vs regex, pinned tag vs latest), digest fail-closed (pinned + API fallback), raw progress passthrough, divergent cancel semantics, cleanup-on-failure, delete scoping (whisper-Vulkan leaves shared DLLs alone).
  • Verified the 0.0.8 release zips contain exactly whisper-server-{win32-x64,linux-x64}-vulkan(.exe) and that the pinned digests match the release assets.
  • sidecarReaper's whisper-server fragment already matches the vulkan binary name — no sidecar changes needed.
  • macOS untouched: the manager is never constructed and all handlers null-guard.

Before merge (from the plan)

  • Hardware smoke test (merge gate): run whisper-server-win32-x64-vulkan.exe from release 0.0.8 on at least one discrete AMD/Intel GPU and one AMD/Intel iGPU laptop; confirm stderr contains whisper_backend_init_gpu: using Vulkan0 backend (MSVC static builds can silently register no Vulkan backend and still transcribe correctly on CPU), record cold/warm startup times against the 120s timeout, and verify output.
  • End-to-end on AMD/Intel Windows + Linux: offer card → download → restart on vulkan binary → transcribe → fallback paths (corrupt binary, kill >10s into load, digest mismatch fails closed) → remove → sleep/wake re-warm.
  • Repo task: enable immutable releases on OpenWhispr/whisper.cpp (0.0.8 is currently mutable) so the pinned tag + digests can't be swapped.

Wire in the whisper-server Vulkan binaries shipped since binaries release
0.0.7 so AMD Radeon and Intel Arc/iGPU users on Windows/Linux get the same
one-click "Enable GPU" flow NVIDIA users get with CUDA.

- Extract src/helpers/gpuBinaryManager.js: the shared download/extract/
  install pipeline behind whisperCudaManager and llamaVulkanManager (both
  are now thin config subclasses), with streaming SHA256 verification
  against in-app pinned digests (fail closed) or the GitHub API asset
  digest.
- Add whisperVulkanManager: pinned tag 0.0.8 + pinned digests, statically
  linked binaries, no companion libs.
- whisperServer: preferVulkan binary resolution, sticky useVulkan flag,
  120s Vulkan startup timeout, and CPU fallback on ANY Vulkan startup
  rejection (early exit, late VRAM-OOM death, or hang). CUDA keeps its
  existing early-exit rule. An intentional stop() during a pending
  startup now rejects instead of resurrecting the server.
- Thread useVulkan through pre-warm, per-dictation starts, and wake
  re-warm (which now replays lastStartOptions instead of hardcoding CUDA).
- IPC: get/download/cancel/delete vulkan-whisper channels (download and
  delete stop the server first to avoid Windows file locks; CUDA delete
  now does the same), WHISPER_VULKAN_ENABLED persisted, gpu-fallback
  relay beside the existing cuda-fallback one.
- UI: the CUDA card in TranscriptionModelPicker is now a GPU card that
  offers CUDA on NVIDIA machines and Vulkan otherwise; ControlPanel's
  unified GPU banner detects whisper-vulkan too. Reuses the existing
  gpu.* i18n keys.
- Characterization tests for the shared pipeline: asset resolution,
  digest fail-closed, divergent cancel semantics, cleanup on failure.

Deliberate behavior changes kept from the refactor: the CUDA disk
pre-check tightens 2x -> 2.5x, its release fetch now sends GITHUB_TOKEN
when set (rate limits), and download archives stage in the OS temp dir
instead of userData/bin.
@gabrielste1n
gabrielste1n force-pushed the feat/whisper-vulkan-gpu branch from f1d8c78 to 505516b Compare July 15, 2026 17:37
@gabrielste1n
gabrielste1n merged commit 36df3a8 into main Jul 15, 2026
8 checks passed
@gabrielste1n
gabrielste1n deleted the feat/whisper-vulkan-gpu branch July 15, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant