Skip to content

voxtral-mini-3b-2507.md

Latest commit

 

History

History
158 lines (125 loc) · 7.37 KB

voxtral-mini-3b-2507.md

File metadata and controls

158 lines (125 loc) · 7.37 KB
Raw

Voxtral Mini 3B (2507)

Mistral's mistralai/Voxtral-Mini-3B-2507 ported to transcribe.cpp. An offline audio-LLM: a Whisper-large-v3 bidirectional audio encoder (32 layers, d_model=1280, 20 heads) feeds a 4-frame-group projector (375 audio tokens per 30 s chunk) into a Ministral-3B causal LM (30 layers, hidden_size=3072, intermediate_size=8192, GQA 32 q / 8 kv heads, NEOX RoPE, SwiGLU) via audio-token injection at the audio_token_id=24 positions in the prompt.

What it's for

Offline speech-to-text and speech-to-text translation. Takes a 16 kHz mono WAV and produces a transcript via greedy decoding.

  • Transcription — auto language detection, or an explicit --language hint. Voxtral advertises English, French, German, Spanish, Italian, Portuguese, Dutch, and Hindi.
  • Translation — --translate --target-language <code> runs the mistral-common instruct template ("Translate this to {Language}.") to translate non-English speech into the target language's text.

See Mistral's model card for training data, intended use, and upstream evaluation.

Licensed Apache-2.0. Ported from upstream commit 3060fe3, pinned 2026-06-06.

Download

Quantization Download Size WER (LibriSpeech test-clean)
BF16 Voxtral-Mini-3B-2507-BF16.gguf 9.37 GB 1.88%
F16 Voxtral-Mini-3B-2507-F16.gguf 9.38 GB 1.89%
Q8_0 Voxtral-Mini-3B-2507-Q8_0.gguf 5.00 GB 1.87%
Q6_K Voxtral-Mini-3B-2507-Q6_K.gguf 3.87 GB 1.87%
Q5_K_M Voxtral-Mini-3B-2507-Q5_K_M.gguf 3.46 GB 1.91%
Q4_K_M Voxtral-Mini-3B-2507-Q4_K_M.gguf 2.98 GB 1.94%

WER measured on the full LibriSpeech test-clean split (2620 utterances) with the Whisper-style English text normalizer, batch size 8 on an NVIDIA L40S. The same-machine HuggingFace transformers reference run (VoxtralForConditionalGeneration, BF16, attn_implementation=eager, greedy) lands at 1.87%, and the BF16 GGUF matches it (1.87% at batch 1).

Quick Start

cmake -B build
cmake --build build

# transcription (auto language)
build/bin/transcribe-cli \
  -m models/Voxtral-Mini-3B-2507/Voxtral-Mini-3B-2507-Q8_0.gguf \
  samples/jfk.wav

# transcription with an explicit language hint
build/bin/transcribe-cli \
  -m models/Voxtral-Mini-3B-2507/Voxtral-Mini-3B-2507-Q8_0.gguf \
  --language de samples/german.wav

# speech translation (non-English audio -> English text)
build/bin/transcribe-cli \
  -m models/Voxtral-Mini-3B-2507/Voxtral-Mini-3B-2507-Q8_0.gguf \
  --translate --target-language en samples/german.wav

If your audio is not already 16 kHz mono WAV, convert it first:

ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav

CLI flags:

  • --language <code> — BCP-47 hint. Omit for auto-detection.
  • --translate --target-language <code> — speech translation via the instruct template.

Performance

Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with speedup over realtime in parentheses. Units: ms below 1 s, s above (2 decimal places).

Apple M4 Max

Backend Sample Q8_0 Q4_K_M
Metal jfk (11.0s) 727.3 ms (15.1×) 656.8 ms (16.7×)
Metal dots (35.3s) 2.40 s (14.7×) 1.90 s (18.6×)
CPU jfk (11.0s) 6.06 s (1.8×) 6.76 s (1.6×)
CPU dots (35.3s) 16.60 s (2.1×) 15.31 s (2.3×)

AMD Ryzen 7 PRO 4750U

Backend Sample Q8_0 Q4_K_M
Vulkan jfk (11.0s) 9.57 s (1.2×) 9.37 s (1.2×)
Vulkan dots (35.3s) 26.49 s (1.3×) 23.82 s (1.5×)
CPU jfk (11.0s) 26.15 s (0.4×) 19.39 s (0.6×)
CPU dots (35.3s) 63.96 s (0.6×) 45.92 s (0.8×)

Fedora Linux 43, transcribe.cpp 91af262. Vulkan device: AMD Radeon Graphics (RADV RENOIR).

Benchmark reproduction:

uv run scripts/bench/run.py \
  --models Voxtral-Mini-3B-2507 \
  --quants q8_0,q4_k_m \
  --samples jfk,dots \
  --backends metal,cpu,vulkan \
  --iters 3 --warmup 1 \
  --name voxtral-mini-3b-2507-publication

Numerical Validation

transcribe.cpp is validated tensor-by-tensor against the HuggingFace transformers reference (VoxtralForConditionalGeneration, BF16, attn_implementation=eager) on samples/jfk.wav with the strict CPU backend. All 43 checkpointed tensors fall within family tolerance, and the BF16 transcript matches the reference verbatim (And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.).

The encoder mel frontend is computed in-process (not injected from the reference), so enc.mel.in is the real frontend-parity gate and matches the reference WhisperFeatureExtractor to 2.2e-5 max / 4.1e-8 mean. The dominant remaining drift is not a bug: the reference casts the mel to BF16 before conv1 and keeps BF16 activations through the whole stack, while the C++ ggml graph runs F32 activations with BF16 weights — so the C++ is the more accurate path, and the cpp-vs-BF16-reference drift is the reference's own BF16 activation rounding (which compounds with depth). A three-way decomposition confirmed this: cpp-vs-F32-reference is 3–4× tighter than cpp-vs-BF16-reference on the encoder, and the transcript is byte-exact against both the BF16 and F32 references. Tolerances and the full mechanism are pinned in tests/tolerances/voxtral.json.

Field Value
Reference HuggingFace transformers v4.57.6 (mistralai/Voxtral-Mini-3B-2507)
Dump script scripts/dump_reference_voxtral_transformers.py
Manifest tests/golden/voxtral/voxtral-mini-3b-2507.manifest.json
Tolerances tests/tolerances/voxtral.json
Command uv run scripts/validate.py all --family voxtral --variant voxtral-mini-3b-2507

Selected tensors (observed on CPU, strict backend; see tolerance file for budgets):

Tensor Shape Max abs diff Mean abs diff Notes
enc.mel.in [128,3000] 2.229e-05 4.060e-08 In-process log-mel vs reference WhisperFeatureExtractor — the frontend-parity gate
enc.out [1500,1280] 1.414e+01 1.121e-02 Final encoder LayerNorm; the drift here is the reference's BF16 activation rounding (cpp-vs-F32-ref is ~4× tighter)
proj.out [375,3072] 3.493e-01 3.158e-03 Projector output (the audio embeddings injected into the LM)