Skip to content

whisper.md

Latest commit

 

History

History
105 lines (86 loc) · 5.34 KB

whisper.md

File metadata and controls

105 lines (86 loc) · 5.34 KB
Raw

Whisper

OpenAI's Whisper family ported to transcribe.cpp. All twelve OpenAI checkpoints — tiny, base, small, medium, large, large-v2, large-v3, large-v3-turbo, plus the four English-only *.en siblings — share the same encoder-decoder transformer architecture and 30-second windowing, so most of the porting and runtime contract is the same across variants. Pick a size based on the accuracy/cost tradeoff you want; pick a .en variant only if you know the audio is English.

For the architecture deep-dive, validation contract, and porting notes, see the family doc at docs/porting/families/whisper.md.

Choosing a variant

  • English-only audio? Prefer the .en checkpoint at your chosen size. They are typically a touch more accurate than the multilingual sibling and slightly cheaper to decode (no language-detection step). They cannot transcribe other languages and cannot translate.
  • Other languages, or auto-detect across languages? Use the multilingual checkpoints (no .en). They cover 99 languages (100 in the v3 family, which adds Cantonese), do automatic language identification, and can produce English translations of non-English audio when invoked with task=translate.
  • Throughput vs accuracy. WER drops as you go up the size ladder, but decode latency scales with parameter count. tiny and base run in near-realtime on CPU; large typically wants Metal or a recent CUDA GPU. large-v3-turbo is large-v3 quality with a much smaller decoder — the best accuracy/speed tradeoff for most multilingual workloads.
  • v3 family quirks. large-v3 and large-v3-turbo use a 128-bin mel input (the rest use 80) and add a Cantonese (yue) language token. The reference dtype shipped is F16, not F32 — they were released in F16 upstream, so transcribe.cpp follows suit.

All variants

WER is on LibriSpeech test-clean for the Q8_0 preset (the default recommended quant), measured by transcribe.cpp's WER pipeline with segment timestamps enabled. See each per-variant doc for the full quant matrix (F32/F16/Q8_0/Q6_K/Q5_K_M/Q4_K_M) and a discussion of how our numbers compare to OpenAI's self-reported figures. Numbers come from single Metal-backed runs; Metal's non-deterministic parallel reductions add ~0.1pp run-to-run variance on the noise floor.

Variant Languages Q8_0 size WER (Q8_0) Doc
whisper-tiny 99 + auto-detect 44 MB 7.53% whisper-tiny.md
whisper-tiny.en English only 44 MB 5.72% whisper-tiny.en.md
whisper-base 99 + auto-detect 81 MB 5.12% whisper-base.md
whisper-base.en English only 81 MB 4.16% whisper-base.en.md
whisper-small 99 + auto-detect 257 MB 3.33% whisper-small.md
whisper-small.en English only 257 MB 3.09% whisper-small.en.md
whisper-medium 99 + auto-detect 793 MB 2.64% whisper-medium.md
whisper-medium.en English only 793 MB 2.72% whisper-medium.en.md
whisper-large 99 + auto-detect 1.55 GB 2.74% whisper-large.md
whisper-large-v2 99 + auto-detect 1.55 GB 2.65% whisper-large-v2.md
whisper-large-v3 100 + auto-detect 1.55 GB 1.82% whisper-large-v3.md
whisper-large-v3-turbo 100 + auto-detect 845 MB 2.01% whisper-large-v3-turbo.md

Pre-built GGUFs for every variant and quant are hosted under handy-computer on Hugging Face; each per-variant doc has direct download links.

Input limits

No practical per-call length limit (transcribe_capabilities.max_audio_ms == 0): Whisper slices long audio into 30-second windows internally and stitches the results, so you can pass arbitrarily long recordings. See the input-length contract.

Quick start

Pick a variant and run:

cmake -B build
cmake --build build

build/bin/transcribe-cli \
  -m models/whisper-base.en/whisper-base.en-Q8_0.gguf \
  samples/jfk.wav

The repo doesn't ship the GGUFs — pull them from the corresponding handy-computer/<variant>-gguf repo on Hugging Face, or convert from the upstream OpenAI checkpoint via the per-variant doc's reproduction section.

Capabilities

All Whisper variants support:

  • Transcription of 16 kHz mono WAV input.
  • Long-form audio via 30-second chunked decoding with the prev-context window assembly described in the family doc.
  • Segment timestamps — the finest granularity the library emits for Whisper (max_timestamp_kind = segment). Word-level timestamps are not currently exposed.
  • Translation (any supported language → English) on multilingual checkpoints — .en variants are transcribe-only.

What's not supported (consistent across the family): real-time streaming (whisper is not streaming-first; chunked 30-second windows only), VAD, speaker diarization. See the family doc for the full runtime contract.