Skip to content

sensevoice-small.md

Latest commit

 

History

History
222 lines (174 loc) · 9.53 KB

sensevoice-small.md

File metadata and controls

222 lines (174 loc) · 9.53 KB
Raw

SenseVoice Small

Alibaba / FunAudioLLM's FunAudioLLM/SenseVoiceSmall ported to transcribe.cpp. A 234M-parameter SAN-M encoder with a single CTC head over a 25,055-token SentencePiece vocabulary covering Chinese, Cantonese, English, Japanese, and Korean.

What it's for

Offline multilingual speech-to-text in zh / yue / en / ja / ko. The model takes a 16 kHz mono WAV (capped at 30 seconds per call) and produces a transcript. It is not a streaming model and does not translate. Long-form audio is the caller's responsibility.

The same CTC head also emits language ID, simple emotion labels (<|HAPPY|>, <|NEUTRAL|>, <|SAD|>, <|ANGRY|>, <|EMO_UNKNOWN|>), audio-event tags (<|Speech|>, <|BGM|>, <|Applause|>, …), and an inverse-text-normalization flag (<|withitn|> / <|woitn|>). These are stripped from the transcript by default; pass --raw-tokens to keep them, and --itn to enable ITN.

See FunAudioLLM's model card for training data, intended use, and upstream evaluation methodology.

Licensed under the FunASR Model Open Source License Agreement — the legacy "model-license" form (MODEL_LICENSE). Ported from upstream commit 3eb3b4e, pinned 2026-05-06.

Input limits

SenseVoice runs on short segments — up to about 30 seconds per call (the window its upstream pipeline feeds via VAD). Longer audio is accepted, but the library logs a WARN and accuracy may degrade; it is not rejected. Segment long recordings (e.g. with VAD) for best results. See the input-length contract.

Download

Quantization Download Size WER (LibriSpeech test-clean)
F32 SenseVoiceSmall-F32.gguf 893 MB 3.13%
F16 SenseVoiceSmall-F16.gguf 449 MB 3.13%
Q8_0 SenseVoiceSmall-Q8_0.gguf 241 MB 3.13%
Q6_K SenseVoiceSmall-Q6_K.gguf 187 MB 3.14%
Q5_K_M SenseVoiceSmall-Q5_K_M.gguf 164 MB 3.18%
Q4_K_M SenseVoiceSmall-Q4_K_M.gguf 139 MB 3.45%

WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy CTC decoding. The publisher does not report a numerical LibriSpeech WER, so the gate baseline is our own FunASR 1.3.1 reference run on the same manifest: 3.13% (95% CI [2.93%, 3.34%]). transcribe.cpp's F32 port matches that baseline within +0.002 percentage-points. Q4_K_M is the only quant with a visible regression (+0.32 pp); F16 / Q8_0 / Q6_K / Q5_K_M are within bootstrap noise of F32.

LibriSpeech is an English benchmark; SenseVoice's strongest case is Mandarin. FLEURS-zh (945 utterances) CER: 10.20% on our FunASR 1.3.1 reference run, 10.11% on the Q8_0 port (95% CI [9.18%, 11.02%]); within bootstrap noise. Reproduce with uv run scripts/wer/run.py --model … --dataset fleurs:zh; reference run via uv run --project scripts/envs/sensevoice scripts/wer/run_reference_sensevoice.py.

Quick Start

cmake -B build
cmake --build build

build/bin/transcribe-cli \
  -m models/SenseVoiceSmall/SenseVoiceSmall-Q8_0.gguf \
  --language en \
  samples/jfk.wav

Pass --language zh / yue / ja / ko (or omit for auto-detection) for the other supported languages. Raw control tokens and ITN are opt-in:

# Keep <|en|><|HAPPY|><|Speech|><|woitn|>… in the output text:
build/bin/transcribe-cli --raw-tokens -m … samples/jfk.wav

# Render numbers/punctuation in formal form:
build/bin/transcribe-cli --itn -m … samples/jfk.wav

If your audio is not already 16 kHz mono WAV, convert it first:

ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav

Performance

Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with speedup over realtime in parentheses. Units: ms below 1 s, s above (2 decimal places).

Apple M4 Max

Backend Sample Q8_0 Q4_K_M
Metal jfk (11.0s) 42 ms (260×) 44 ms (250×)
Metal dots (35.3s) 111 ms (319×) 137 ms (258×)
CPU jfk (11.0s) 208 ms (53×) 213 ms (52×)
CPU dots (35.3s) 700 ms (50×) 727 ms (49×)

macOS 26.4.1, transcribe.cpp 811fe2a.

AMD Ryzen 7 PRO 4750U

Backend Sample Q8_0 Q4_K_M
Vulkan jfk (11.0s) 329 ms (33×) 332 ms (33×)
Vulkan dots (35.3s) 1.11 s (32×) 1.12 s (31×)
CPU jfk (11.0s) 687 ms (16×) 590 ms (19×)
CPU dots (35.3s) 2.31 s (15×) 2.03 s (17×)

Fedora 43, transcribe.cpp 8635bd1. Vulkan device: AMD Radeon Graphics (RADV RENOIR).

Benchmark reproduction:

uv run scripts/bench/run.py \
  --models SenseVoiceSmall \
  --quants q8_0,q4_k_m \
  --samples jfk,dots \
  --backends metal,cpu,vulkan \
  --iters 3 --warmup 1 \
  --name sensevoice-small-publication

Numerical Validation

transcribe.cpp is validated tensor-by-tensor against FunASR 1.3.1 on samples/jfk.wav. All 16 checkpointed tensors fall within family tolerance, and the final transcript matches the FunASR reference verbatim (both spelled … laled out … on token 1089-134686-0000 — a quirk of SenseVoice, not a port defect). Last validated at commit f094d28.

Field Value
Reference FunASR 1.3.1, FunAudioLLM/SenseVoiceSmall (rev 3eb3b4e)
Dump script scripts/dump_reference_sensevoice_funasr.py
Manifest tests/golden/sensevoice/sensevoice-small.manifest.json
Command uv run scripts/validate.py compare --family sensevoice --variant sensevoice-small

Selected tensors:

Tensor Max abs diff Mean abs diff Notes
frontend.fbank.lfr.cmvn.out 3.13e-03 6.34e-04 fp32 FFT vs C++ fp64 STFT round-off
enc.input.with_prefix 3.13e-03 6.20e-04 frontend drift carried by concat (no compute)
enc.embed.out 7.08e-02 1.40e-02 frontend drift × √d_model after sinusoidal PE
enc.encoders0.0.out 1.53e+02 2.52e+00 first SAN-M block (560→512 projection)
enc.encoders.0.out 9.63e+01 2.78e+00 main-tier block 0
enc.encoders.24.out 4.60e+02 1.41e+01 mid-tier (block 24); reference values ~4.5k
enc.encoders.48.out 4.52e+04 7.74e+01 last main block; reference values ~46k
enc.after_norm.out 6.66e+00 4.71e-01 tier-boundary LayerNorm renormalises
enc.tp_encoders.{0,10,19}.out ≤ 1.10e+04 ≤ 9.07e+00 tp-tier 20-block stack
enc.tp_norm.out 1.53e+01 5.86e-01 final encoder output, post-LN
ctc.logits.raw 3.23e+01 1.87e+00 CTC logits — argmax positions identical
ctc.log_probs 3.07e+01 2.90e+00 log-softmax CTC distribution

The expected divergence is fp32 reduction-order drift accumulated through 70 SAN-M blocks. SenseVoice's encoder has no inter-layer normalization (only after_norm between the two tiers and tp_norm at the end), so absolute magnitudes grow ~170× through the main-tier stack — and so does the absolute drift. After tp_norm re-renormalises, the final output is within ~0.6 mean / 15 max. Both reference and C++ produce argmax-equivalent CTC outputs; the transcript is a verbatim match.

Reproduction

Convert

Loads directly from FunASR's model.pt pickle via funasr.AutoModel.

uv run --project scripts/envs/sensevoice \
  scripts/convert-sensevoice.py FunAudioLLM/SenseVoiceSmall

Quantize

Run transcribe-quantize once per target quant.

for Q in F16 Q8_0 Q6_K Q5_K_M Q4_K_M; do
  build/bin/transcribe-quantize \
    models/SenseVoiceSmall/SenseVoiceSmall-F32.gguf \
    models/SenseVoiceSmall/SenseVoiceSmall-${Q}.gguf \
    --quant ${Q}
done

Validate

uv run scripts/validate.py all --family sensevoice --variant sensevoice-small

Score WER

# Reference baseline (FunASR; ~25 min on a single CPU thread for 2620 utts).
uv run --project scripts/envs/sensevoice \
  scripts/wer/run_reference_sensevoice.py \
    --manifest samples/wer/test-clean.manifest.jsonl \
    --out      reports/wer/sensevoice-small-REF.test-clean.jsonl
uv run scripts/wer/score.py reports/wer/sensevoice-small-REF.test-clean.jsonl

# transcribe.cpp ports (one preset shown; loop in the family doc).
uv run scripts/wer/run.py \
  --model models/SenseVoiceSmall/SenseVoiceSmall-F32.gguf \
  --manifest samples/wer/test-clean.manifest.jsonl \
  --out      reports/wer/sensevoice-small-F32.test-clean.jsonl
uv run scripts/wer/score.py reports/wer/sensevoice-small-F32.test-clean.jsonl