Skip to content

granite-speech-4.1-2b-nar.md

Latest commit

 

History

History
184 lines (137 loc) · 7.41 KB

granite-speech-4.1-2b-nar.md

File metadata and controls

184 lines (137 loc) · 7.41 KB
Raw

Granite Speech 4.1-2b NAR

IBM's ibm-granite/granite-speech-4.1-2b-nar ported to transcribe.cpp. The non-autoregressive editor variant of Granite-Speech. Shares the Conformer audio encoder with the AR Granite- Speech family but pairs it with a custom MLP-with-attention projector and the Granite-4.0-1b LLM used as a bidirectional editor (causal mask disabled). One forward pass produces logits over the full transcript; CTC decode yields the final text — no token-by-token loop.

What it's for

Offline multilingual speech-to-text in a single non-autoregressive editor pass. Covers English plus French, German, Spanish, and Portuguese. ASR only — no translation, no timestamps, no diarization.

See IBM's model card for training data, intended use, and upstream evaluation methodology.

Licensed Apache-2.0. Ported from upstream commit 99a4df9, pinned 2026-05-24 (single-file modeling_granite_speech_nar.py snapshot — the README's canonical inference target).

Download

Quantization Download Size WER (LibriSpeech test-clean)
BF16 granite-speech-4.1-2b-nar-BF16.gguf 4.20 GB 1.29%
F16 granite-speech-4.1-2b-nar-F16.gguf 4.21 GB 1.29%
Q8_0 granite-speech-4.1-2b-nar-Q8_0.gguf 2.33 GB 1.29%
Q6_K granite-speech-4.1-2b-nar-Q6_K.gguf 1.84 GB 1.29%
Q5_K_M granite-speech-4.1-2b-nar-Q5_K_M.gguf 1.66 GB 1.25%
Q4_K_M granite-speech-4.1-2b-nar-Q4_K_M.gguf 1.45 GB 1.35%

WER measured on the full LibriSpeech test-clean split (2620 utterances). BF16 reference baseline (transformers model.transcribe, MPS, re-run locally): 1.28% — matches the upstream model card's 1.29% to within sampling noise. Text normalizer: Whisper EnglishTextNormalizer. F16, Q8_0, and Q6_K all score the same 1.29% as BF16 — the editor is very robust to weight quantization down through Q5_K_M, where the WER actually dips slightly (1.25%, within overlapping 95% CI of REF). Reference reproduction follows the model card path verbatim (AutoProcessor + AutoModel.transcribe + processor.batch_decode) at HF revision 99a4df9; the older snapshot's bidirectional-mask patch is obsolete in this snapshot.

Quick Start

cmake -B build
cmake --build build

build/bin/transcribe-cli \
  -m models/granite-speech-4.1-2b-nar/granite-speech-4.1-2b-nar-Q8_0.gguf \
  samples/jfk.wav

If your audio is not already 16 kHz mono WAV, convert it first:

ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav

NAR is single-task ASR; there is no --translate flag or --timestamps mode for this variant. The --language flag is accepted but ignored — the editor handles language detection implicitly.

Performance

Cells are wall-clock latency, with speedup over realtime in parentheses. NAR is faster than the AR variants on GPU backends because there is no autoregressive step loop — a single bidirectional forward through 40 LLM layers replaces the per-token decode graph.

Apple M4 Max

Mean over 3 iterations after 1 warmup.

Metal

Sample Q4_K_M Q8_0
jfk (11.0s) 209 ms (53×) 196 ms (56×)
dots (35.3s) 664 ms (53×) 635 ms (56×)

CPU

Sample Q4_K_M Q8_0
jfk (11.0s) 1.87 s (5.9×) 1.99 s (5.5×)
dots (35.3s) 6.50 s (5.4×) 7.71 s (4.6×)

macOS 26.4, transcribe.cpp de05c43.

Apple M4

Mean over 5 iterations after 2 warmups. Q8_0.

Backend Sample Q8_0
Metal jfk (11.0s) 614 ms (18×)
CPU jfk (11.0s) 2.55 s (4×)

macOS 26.1, transcribe.cpp 275332d.

AMD Ryzen 7 PRO 4750U (Vega 8 iGPU)

Mean over 3 iterations after 1 warmup.

Vulkan (RADV)

Sample Q4_K_M Q8_0
jfk (11.0s) 3.16 s (3.5×) 3.06 s (3.6×)
dots (35.3s) 9.85 s (3.6×) 9.57 s (3.7×)

CPU

Sample Q4_K_M Q8_0
jfk (11.0s) 5.71 s (1.9×) 7.05 s (1.6×)
dots (35.3s) 20.39 s (1.7×) 24.81 s (1.4×)

Linux 6.18 (Fedora 43), transcribe.cpp dbe5814. NAR's Vulkan RTF stays flat across short and long samples (jfk and dots both ~3.6×) because the single bidirectional LLM pass dominates over the encoder; on CPU the encoder dominates so RTF tapers slightly with sequence length.

Capabilities

Capability Status
Transcribe (English) Yes
Transcribe (fr/de/es/pt) Yes
Translation No (not supported by the NAR family; use the AR variants)
Word/segment timestamps No (the NAR encoder pools to per-window output; per-token timing is lost)

Numerical Validation

Tensor-level parity with the transformers reference on samples/jfk.wav. Per-tensor max_abs / mean_abs budgets in tests/tolerances/granite_nar.json. Drift on dec.text_logits is dominated by BF16 reduction-order noise over 40 bidirectional LLM layers; absolute magnitudes run O(100) at confident positions, observed drift ~3% relative — the editor is argmax-stable on this drift band, hence the BF16/F16/Q8_0/Q6_K all match WER.

Reproduction

Convert

uv run --project scripts/envs/granite_nar \
  scripts/convert-granite_nar.py ibm-granite/granite-speech-4.1-2b-nar \
  --repo-id ibm-granite/granite-speech-4.1-2b-nar

Quantize

for PRESET in F16 Q8_0 Q6_K Q5_K_M Q4_K_M; do
  build/bin/transcribe-quantize \
    models/granite-speech-4.1-2b-nar/granite-speech-4.1-2b-nar-BF16.gguf \
    models/granite-speech-4.1-2b-nar/granite-speech-4.1-2b-nar-${PRESET}.gguf \
    --quant ${PRESET}
done

Validate

uv run scripts/validate.py all --family granite_nar --variant granite-speech-4.1-2b-nar

Reproduce WER

uv run scripts/wer/run.py \
  --model models/granite-speech-4.1-2b-nar/granite-speech-4.1-2b-nar-BF16.gguf \
  --manifest samples/wer/test-clean.manifest.jsonl \
  --out reports/wer/granite-speech-4.1-2b-nar-BF16.test-clean.jsonl
uv run scripts/wer/score.py reports/wer/granite-speech-4.1-2b-nar-BF16.test-clean.jsonl