Skip to content

voxtral-realtime.md

Latest commit

 

History

History
156 lines (122 loc) · 7.35 KB

voxtral-realtime.md

File metadata and controls

156 lines (122 loc) · 7.35 KB
Raw

Voxtral Realtime (2602)

Mistral's mistralai/Voxtral-Mini-4B-Realtime-2602 ported to transcribe.cpp. A streaming audio-LLM (~970 M causal audio encoder + ~3.4 B Ministral decoder): a left-pad causal conv stem + 32-layer causal, sliding-window (750), RoPE transformer encoder feeds a 4-frame-group projector whose audio embeddings are added onto the decoder's input embeddings; a 26-layer Ministral decoder (GQA 32/8, sliding-window 8192) with delay-token latency conditioning emits one text token per 80 ms audio slot (12.5 Hz).

Architecturally distinct from the offline Voxtral 2507 family (own arch, streaming frontend with a fixed global log-mel max, causal encoder, additive fusion, ada-norm FFN scaling) — it shares only the projector shape and the tekken tokenizer. Licensed Apache-2.0. Ported from upstream commit 2769294, pinned 2026-06-06.

What it's for

Real-time and offline speech-to-text from a 16 kHz mono WAV.

  • Streaming transcription — incremental emission with a configurable latency/quality trade-off. --stream-chunk-ms <N> feeds audio in N-ms chunks; the final transcript is byte-equal to the offline path.
  • Configurable delay — --stream-voxtral-delay <N> (default 6 = 480 ms; range 80 ms–2.4 s) sets the transcription delay.
  • Auto language detection (the streaming processor is auto-detect only).

Input limits

No practical per-call length limit (transcribe_capabilities.max_audio_ms == 0): the streaming encoder runs in constant memory, so you can feed arbitrarily long audio. The only ceiling is an absolute decoder-position wall at about 2.9 hours of continuous audio — a one-shot or batch clip past it returns TRANSCRIBE_ERR_INPUT_TOO_LONG, and a stream that reaches it keeps its committed text and sets transcribe_was_truncated() (finalize still returns OK). See the input-length contract.

Download

handy-computer/Voxtral-Mini-4B-Realtime-2602-gguf:

Quantization Download Size WER (LibriSpeech test-clean)
BF16 GGUF 8.87 GB 2.08%
F16 GGUF 8.88 GB 2.09%
Q8_0 GGUF 4.73 GB 2.07%
Q6_K GGUF 3.66 GB 2.08%
Q5_K_M GGUF 3.28 GB 2.08%
Q4_K_M GGUF 2.83 GB 2.08%

WER measured on the full LibriSpeech test-clean split (2620 utterances) with the Whisper-style English text normalizer, offline path, batch size 8 on an NVIDIA L40S. The same-machine HuggingFace transformers reference (VoxtralForConditionalGeneration, BF16, greedy) lands at 2.08%, and the BF16 GGUF matches it (2.08%). Every shipped quant stays within bootstrap noise of the reference (2.07–2.09%; 95% CI ≈ ±0.18), so the quantization ladder is WER-neutral down to Q4_K_M.

Quick Start

cmake -B build && cmake --build build

# offline transcription
build/bin/transcribe-cli \
  -m models/Voxtral-Mini-4B-Realtime-2602/Voxtral-Mini-4B-Realtime-2602-Q8_0.gguf \
  samples/jfk.wav

# streaming transcription (500 ms chunks)
build/bin/transcribe-cli \
  -m models/Voxtral-Mini-4B-Realtime-2602/Voxtral-Mini-4B-Realtime-2602-Q8_0.gguf \
  --stream-chunk-ms 500 samples/jfk.wav

CLI flags:

  • --stream-chunk-ms <N> — incremental streaming at N-ms chunk granularity.
  • --stream-voxtral-delay <N> — transcription delay in audio slots (default 6 = 480 ms).
  • --spec-k-drafts <N> — offline-path 1-gram-lookup speculative decoding draft length. -1 (default) uses the family default (2). 0 disables spec (plain autoregression). 1..8 selects an explicit K. Speculation applies to transcribe_run / transcribe-cli only — the streaming path is unaffected.

Performance

Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with speedup over realtime in parentheses. Units: ms below 1 s, s above (2 decimal places). Measured on the offline path at the family-default K=1 speculative decoding.

Apple M4 Max

Backend Sample Q8_0 Q4_K_M
Metal jfk (11.0s) 1.22 s (9.0×) 1.14 s (9.7×)
Metal dots (35.3s) 4.34 s (8.1×) 3.91 s (9.0×)
CPU jfk (11.0s) 4.43 s (2.5×) 4.69 s (2.3×)
CPU dots (35.3s) 13.65 s (2.6×) 13.12 s (2.7×)

macOS 15, transcribe.cpp 483c122. Metal device: Apple M4 Max.

AMD Ryzen 7 4750U Pro

Backend Sample Q8_0 Q4_K_M
Vulkan jfk (11.0s) 12.62 s (0.87×) 10.97 s (1.00×)
Vulkan dots (35.3s) 39.29 s (0.90×) 33.51 s (1.05×)
CPU jfk (11.0s) 19.54 s (0.56×) 13.80 s (0.80×)
CPU dots (35.3s) 58.00 s (0.61×) 41.54 s (0.85×)

Fedora 43, transcribe.cpp 483c122. Vulkan device: AMD Radeon Graphics (RADV RENOIR).

Benchmark reproduction:

uv run scripts/bench/run.py \
  --models Voxtral-Mini-4B-Realtime-2602 \
  --quants q8_0,q4_k_m \
  --samples jfk,dots \
  --backends metal,cpu,vulkan \
  --iters 3 --warmup 1 \
  --name voxtral-mini-4b-realtime-2602-publication

Speculative decoding

The offline decoder runs 1-gram-lookup speculative decoding by default. Each verify pass processes K+1 positions in parallel: position 0 is the model's true next-token decision; positions 1..K verify K draft tokens read from the 1-gram suffix lookup over the already-decoded prefix. Drafts are accepted as long as the model's argmax matches the drafted token; the first mismatch ends the accepted prefix. Because ~60–70% of audio slots emit STREAMING_PAD (id 32), the 1-gram lookup hits high acceptance during silence and during repeated phrases.

The transcript is byte-identical to the K=0 (no-spec) path; only wall-clock time changes.

Capability gate: transcribe_capabilities::supports_spec_decode = true. Families that do not advertise this bit silently ignore the field.

Numerical Validation

The streaming scheduler is validated against the upstream transformers reference for true incremental equivalence: the offline and streaming paths produce byte-equal transcripts, and the encoder StaticCache / decoder sliding-window rings are gated bit-exact (or within matmul reduction-order noise) against the reference. See the family port notes in docs/porting/families/voxtral_realtime.md for the full streaming contract.

Upstream: mistralai/Voxtral-Mini-4B-Realtime-2602.