Skip to content

Commit c09bbda

Browse files
cjpaiscjpais
committed
Merge branch 'main' into moss-transcribe-diarize
2 parents de370d9 + fd8b8f8 commit c09bbda

22 files changed

Lines changed: 1289 additions & 85 deletions

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ C/C++ speech-to-text inference library. Runs diverse STT model families via [GGU
2121
| FunASR Nano | `fun-asr-nano-2512`, `fun-asr-mlt-nano-2512` | [docs/models/fun-asr-nano.md](docs/models/fun-asr-nano.md) |
2222
| Nemotron Speech Streaming | `nemotron-speech-streaming-en-0.6b` | [docs/models/nemotron-speech-streaming-en-0.6b.md](docs/models/nemotron-speech-streaming-en-0.6b.md) |
2323
| Nemotron 3.5 ASR Streaming | `nemotron-3.5-asr-streaming-0.6b` (multilingual, 40 locales) | [docs/models/nemotron-3.5-asr-streaming-0.6b.md](docs/models/nemotron-3.5-asr-streaming-0.6b.md) |
24+
| Multitalker Parakeet Streaming | `multitalker-parakeet-streaming-0.6b-v1` (single-speaker ASR path only) | [docs/models/multitalker-parakeet-streaming-0.6b-v1.md](docs/models/multitalker-parakeet-streaming-0.6b-v1.md) |
2425
| Granite Speech 4 / 4.1 | `granite-4.0-1b-speech`, `granite-speech-4.1-2b{,-plus,-nar}` | [docs/models/granite-speech.md](docs/models/granite-speech.md) |
2526
| Voxtral | `voxtral-mini-3b-2507`, `voxtral-small-24b-2507` (audio-LLM; transcription + translation) | [docs/models/voxtral.md](docs/models/voxtral.md) |
2627
| Voxtral Realtime | `voxtral-mini-4b-realtime-2602` (streaming audio-LLM) | [docs/models/voxtral-realtime.md](docs/models/voxtral-realtime.md) |

‎docs/models/multitalker-parakeet-streaming-0.6b-v1.md‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
# Multitalker Parakeet Streaming 0.6B v1
2+
3+
NVIDIA's [`nvidia/multitalker-parakeet-streaming-0.6b-v1`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1)
4+
ported to transcribe.cpp. A 0.6B-parameter cache-aware streaming
5+
FastConformer encoder with an RNN-T transducer decoder, fine-tuned from
6+
[`nvidia/nemotron-speech-streaming-en-0.6b`](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b).
7+
8+
## What it's for
9+
10+
Offline and cache-aware **streaming** English speech-to-text with greedy
11+
RNN-T decoding. Outputs cased, punctuated transcripts. Token- and
12+
word-level timestamps are available.
13+
14+
Upstream this is a **multitalker (speaker-attributed)** checkpoint: it can
15+
transcribe several overlapping speakers into per-speaker channels. That
16+
path depends on machinery this port does **not** ship (see
17+
[Known limitations](#known-limitations)). transcribe.cpp exposes only the
18+
model's `single_speaker_mode` ASR path, which disables the multitalker
19+
machinery and runs the checkpoint as a cache-aware streaming RNN-T with
20+
the base model's frontend, encoder backbone, decoder, and tokenizer plus
21+
the checkpoint's always-on layer-0 speaker-kernel injection.
22+
23+
This port runs the model in both **offline** and **cache-aware streaming**
24+
modes.
25+
26+
See NVIDIA's [model card](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1)
27+
for training data, intended use, the multitalker methodology, and the full
28+
latency-vs-accuracy table.
29+
30+
Licensed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).
31+
Ported from upstream commit
32+
[`8749fc7`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1/commit/8749fc71fd6e2d88ef230159bbf2aea69b524ee1),
33+
pinned 2026-07-12.
34+
35+
## Download
36+
37+
| Quantization | Download | Size | WER (LibriSpeech test-clean, offline) |
38+
| --- | --- | ---: | ---: |
39+
| F32 | [multitalker-parakeet-streaming-0.6b-v1-F32.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-F32.gguf) | 2.49 GB | 2.19% |
40+
| F16 | [multitalker-parakeet-streaming-0.6b-v1-F16.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-F16.gguf) | 1.25 GB | 2.19% |
41+
| Q8_0 | [multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf) | 734 MB | 2.18% |
42+
| Q6_K | [multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf) | 604 MB | 2.20% |
43+
| Q5_K_M | [multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf) | 542 MB | 2.18% |
44+
| Q4_K_M | [multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf) | 478 MB | 2.18% |
45+
46+
WER is measured on the full LibriSpeech test-clean split (2620 utterances)
47+
in `single_speaker_mode` with greedy RNN-T decoding, whisper-normalizer
48+
scoring (PnC-stripped), and no external LM. F32 reference baseline: 2.19%.
49+
The measured NeMo `single_speaker_mode` reference on the same split is
50+
2.19%, and NVIDIA's self-reported number is 2.19% (from the
51+
[HF model card](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1)).
52+
53+
## Streaming parity
54+
55+
Cache-aware streaming was validated tensor-by-tensor against NeMo's
56+
`conformer_stream_step` reference via `scripts/validate_streaming.py`. On
57+
`samples/jfk.wav` at `--backend cpu --threads 1`, the always-on layer-0
58+
speaker-kernel injection is applied per chunk on the post-drop block-0
59+
input (matching the offline path), and the harness reports **150/150
60+
streaming-tensor pairs within tolerance and the transcript byte-exact vs
61+
NeMo at R=13** (`att_context_size=[70, 13]`, 1.12s chunk). Streaming
62+
`enc_out` drift (observed 5.5e-6) is tighter than the offline C++
63+
`enc.final` drift (1.9e-3) on the same audio, because clean per-chunk
64+
caches accumulate less error than a single 24-block offline pass.
65+
66+
Reproduce:
67+
68+
```bash
69+
uv run --project scripts/envs/parakeet scripts/validate_streaming.py \
70+
--hf-model nvidia/multitalker-parakeet-streaming-0.6b-v1 \
71+
--gguf models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-F32.gguf \
72+
--audio samples/jfk.wav \
73+
--out build/validate_streaming/multitalker/jfk \
74+
--right 13 6 1 0 \
75+
--backend cpu --threads 1 \
76+
--tolerances tests/tolerances/multitalker-parakeet-streaming-0.6b-v1.streaming.json
77+
```
78+
79+
## Quick Start
80+
81+
```bash
82+
cmake -B build
83+
cmake --build build
84+
85+
build/bin/transcribe-cli \
86+
-m models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf \
87+
samples/jfk.wav
88+
```
89+
90+
If your audio is not already 16 kHz mono WAV, convert it first:
91+
92+
```bash
93+
ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav
94+
```
95+
96+
## Performance
97+
98+
Cells are wall-clock latency (mean over 3 iterations after 1 warmup),
99+
with speedup over realtime in parentheses. Units: `ms` below 1 s, `s`
100+
above (2 decimal places).
101+
102+
### Apple M4 Max
103+
104+
| Backend | Sample | Q8_0 | Q4_K_M |
105+
| ------- | ------------ | ------------: | ------------: |
106+
| Metal | jfk (11.0s) | 67 ms (164×) | 69 ms (159×) |
107+
| Metal | dots (35.3s) | 184 ms (192×) | 185 ms (191×) |
108+
| CPU | jfk (11.0s) | 310 ms (36×) | 307 ms (36×) |
109+
| CPU | dots (35.3s) | 1.05 s (34×) | 1.03 s (34×) |
110+
111+
macOS 26.5.1, transcribe.cpp `c55a09d`.
112+
113+
### AMD Ryzen 7 4750U Pro
114+
115+
| Backend | Sample | Q8_0 | Q4_K_M |
116+
| ------- | ------------ | ------------: | ------------: |
117+
| Vulkan | jfk (11.0s) | 466 ms (24×) | 475 ms (23×) |
118+
| Vulkan | dots (35.3s) | 1.36 s (26×) | 1.39 s (26×) |
119+
| CPU | jfk (11.0s) | 751 ms (15×) | 816 ms (13×) |
120+
| CPU | dots (35.3s) | 2.99 s (12×) | 3.12 s (11×) |
121+
122+
Fedora 43, transcribe.cpp `c55a09d`. Vulkan device: `AMD Radeon
123+
Graphics (RADV RENOIR)`.
124+
125+
Benchmark reproduction:
126+
127+
```bash
128+
uv run scripts/bench/run.py \
129+
--models multitalker-parakeet-streaming-0.6b-v1 \
130+
--quants q8_0,q4_k_m \
131+
--samples jfk,dots \
132+
--backends metal,cpu,vulkan \
133+
--iters 3 --warmup 1 \
134+
--name multitalker-parakeet-streaming-0.6b-v1-publication
135+
```
136+
137+
## Numerical Validation
138+
139+
transcribe.cpp is validated tensor-by-tensor against NeMo on
140+
`samples/jfk.wav` via `scripts/validate.py`. Per-tensor tolerances live
141+
in a per-variant file
142+
([`tests/tolerances/multitalker-parakeet-streaming-0.6b-v1.json`](../../tests/tolerances/multitalker-parakeet-streaming-0.6b-v1.json))
143+
rather than the family-shared one because the unnormalised log-mel
144+
(NeMo's `normalize="NA"` no-op) lands on a different magnitude scale than
145+
the per-feature-normalised siblings, and because the layer-0
146+
speaker-kernel injection is unique to this variant. The family-level
147+
forward map at
148+
[`reports/porting/parakeet/forward-map.md`](../../reports/porting/parakeet/forward-map.md)
149+
documents the per-stage divergence sources.
150+
151+
| Field | Value |
152+
| --- | --- |
153+
| Reference | NeMo, `nvidia/multitalker-parakeet-streaming-0.6b-v1` (`single_speaker_mode`) |
154+
| Dump script | `scripts/dump_reference_parakeet_nemo.py` |
155+
| Manifest | `tests/golden/parakeet/multitalker-parakeet-streaming-0.6b-v1.manifest.json` |
156+
| Command | `uv run scripts/validate.py all --family parakeet --variant multitalker-parakeet-streaming-0.6b-v1` |
157+
158+
## Batch
159+
160+
The model ships an explicit `run_batch()` parallel fast path. Batch output
161+
is WER-neutral: byte-equal to the serial single-stream path at batch sizes
162+
2 / 4 / 8 (golden frozen at
163+
`tests/golden/batch/multitalker-parakeet-streaming-0.6b-v1.cpu.json`),
164+
CPU same-length tensor parity is bit-exact (max_abs 0.0) at batch=4 on
165+
`jfk.wav`, diverse-length flash tensor parity is bit-exact across arbitrary
166+
length mixes, and full test-clean batch-8 WER equals batch-1 (2.19%).
167+
168+
## Known limitations
169+
170+
- **Single-speaker only. The multitalker / speaker-attributed ASR path is
171+
not ported.** Upstream, this checkpoint transcribes several overlapping
172+
speakers into per-speaker channels. That path requires (1) an external
173+
streaming speaker-diarization model
174+
([`nvidia/diar_streaming_sortformer_4spk-v2.1`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1),
175+
a separate Sortformer checkpoint that this repo does not ship), (2)
176+
per-frame speaker-kernel injection driven by that diarizer's supervision,
177+
(3) one encoder+decoder instance per speaker (up to 4), and (4) a
178+
speaker-tagged SegLST output contract. None of these exist in
179+
transcribe.cpp today, so the port runs `single_speaker_mode` and produces
180+
a single flat transcript. It does **not** separate speakers, diarize, or
181+
emit speaker turns.
182+
- **English only.** The model is English-only by training.
183+
- **No translation and no language detection.**
184+
185+
## Reproduction
186+
187+
### Convert
188+
189+
```bash
190+
uv run --project scripts/envs/parakeet \
191+
scripts/convert-parakeet.py nvidia/multitalker-parakeet-streaming-0.6b-v1
192+
```
193+
194+
### Quantize
195+
196+
Run `transcribe-quantize` once per target quant. Example for Q8_0; repeat
197+
with `F16`, `Q6_K`, `Q5_K_M`, `Q4_K_M`:
198+
199+
```bash
200+
build/bin/transcribe-quantize \
201+
models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-F32.gguf \
202+
models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf \
203+
--quant Q8_0
204+
```
205+
206+
### Validate
207+
208+
```bash
209+
uv run scripts/validate.py all --family parakeet --variant multitalker-parakeet-streaming-0.6b-v1
210+
```

‎docs/porting/families/parakeet.md‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,12 @@ CI-overlap grounds; see the family WER summary for rationale.
3131
- **Cache-aware streaming RNN-T (encoder-transducer)**:
3232
`nemotron-speech-streaming-en-0.6b` (English-only),
3333
`nemotron-3.5-asr-streaming-0.6b` (multilingual, 40 locales,
34-
language one-hot conditioning) — intake only, Stages 2-8 TODO
34+
language one-hot conditioning) — intake only, Stages 2-8 TODO,
35+
`multitalker-parakeet-streaming-0.6b-v1` (English-only, fine-tuned
36+
from `nemotron-speech-streaming-en-0.6b`; ships the
37+
`single_speaker_mode` ASR path only — the multitalker /
38+
speaker-attribution machinery is out of scope, see Capability
39+
Validation rows)
3540

3641
Per-variant intake JSON: `reports/porting/parakeet/<variant>/intake.json`.
3742

@@ -146,6 +151,14 @@ Allowed statuses: `PASS` | `SKIP — not exposed by runtime` |
146151
| nemotron-3.5-asr-streaming-0.6b | Batch (offline) | run_batch fast path | `uv run scripts/batch_parity.py --model <gguf> --list <list.txt> --batch-sizes 2,4,8 --backend cpu --language en-US` + `uv run scripts/batch_tensor_parity.py --model <gguf> --wav samples/jfk.wav --batch 4 --backend cpu` | text byte-equal vs serial at sizes 2/4/8 (golden frozen at `tests/golden/batch/nemotron-3.5-asr-streaming-0.6b.cpu.json`); CPU tensor parity bit-exact (max_abs=0.0) at batch=4 on jfk.wav | PASS |
147152
| ctc-1.1b | Transcribe (CTC head) | explicit en | `build/bin/transcribe-cli -m models/parakeet-ctc-1.1b/parakeet-ctc-1.1b-F32.gguf --language en samples/jfk.wav` | English transcript | PASS |
148153
| ctc-0.6b | Transcribe (CTC head) | explicit en | `build/bin/transcribe-cli -m models/parakeet-ctc-0.6b/parakeet-ctc-0.6b-F32.gguf --language en samples/jfk.wav` | English transcript | PASS |
154+
| multitalker-parakeet-streaming-0.6b-v1 | Transcribe (single_speaker_mode, offline / cache-aware att_context_size=[70,13], 1.12s chunk) | explicit en | `build/bin/transcribe-cli -m models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-F32.gguf --language en samples/jfk.wav` | English transcript with PnC | PASS |
155+
| multitalker-parakeet-streaming-0.6b-v1 | Transcribe (single_speaker_mode) — no-hint | auto/default | `build/bin/transcribe-cli -m <gguf> samples/jfk.wav` (no `--language`; English-only checkpoint) | English transcript with PnC | PASS |
156+
| multitalker-parakeet-streaming-0.6b-v1 | Punctuation/casing | output | same as the explicit-en row | output contains capital letters and `,.?!` | PASS |
157+
| multitalker-parakeet-streaming-0.6b-v1 | Streaming (single_speaker_mode, cache reuse across chunks) | streaming | `build/bin/transcribe-cli -m <gguf> --language en --backend cpu --threads 1 --stream-chunk-ms 1120 --stream-att-right 13 samples/jfk.wav` | byte-equal transcript vs single-speaker one-shot at the default `att_context_size=[70,13]` (1.12s chunk) | PASS |
158+
| multitalker-parakeet-streaming-0.6b-v1 | Other latency settings ([70,0]/[70,1]/[70,6]/[70,13]) | runtime-selectable att_context_size | `… --stream-chunk-ms 1120 --stream-att-right {0,1,6,13} …` | all four R settings stream a valid single-speaker transcript; R=6/13 byte-equal to one-shot, R=0/1 differ only in trailing punctuation (lower lookahead) | PASS |
159+
| multitalker-parakeet-streaming-0.6b-v1 | Batch (offline, single_speaker_mode) | run_batch fast path | `uv run scripts/batch_parity.py --model <gguf> --list <list.txt> --batch-sizes 2,4,8 --backend cpu --language en` + `uv run scripts/batch_tensor_parity.py --model <gguf> --wav samples/jfk.wav --batch 4 --backend cpu` | text byte-equal vs serial at sizes 2/4/8 (golden frozen at `tests/golden/batch/multitalker-parakeet-streaming-0.6b-v1.cpu.json`); same-length CPU tensor parity bit-exact (max_abs=0.0) at batch=4 on jfk.wav; **diverse-length** flash tensor parity bit-exact (max_abs=0.0) across arbitrary length mixes; full test-clean batch-8 WER == batch-1 (2.19%) after the causal var-len pre-encode masking fix (see forward-map Deviations) | PASS |
160+
| multitalker-parakeet-streaming-0.6b-v1 | Multitalker / speaker-attributed ASR (speaker-kernel injection + external Sortformer diarization + multi-instance + SegLST) | multitalker | (no runtime surface today) | per-speaker cpWER vs NeMo `speech_to_text_multitalker_streaming_infer.py` on AMI/CH109/Mixer6 | SKIP — not exposed by runtime (OUT OF SCOPE) — the single-speaker kernel path is implemented, but full speaker attribution still requires (1) a ported streaming Sortformer diarizer, (2) target-driven per-frame kernel masks, (3) N-instance-per-speaker orchestration, and (4) a SegLST speaker-tagged output contract. |
161+
| multitalker-parakeet-streaming-0.6b-v1 | Speaker diarization (produce speaker turns) | diarization | (not a capability of this checkpoint) | n/a | SKIP — not exposed by runtime (OUT OF SCOPE) — this checkpoint CONSUMES external diarization (Sortformer); it does not produce speaker turns. Brought back in scope only if a Sortformer diarizer is ported as a separate family. |
149162
| all variants | Word timestamps | only if exposed | `transcribe-cli --timestamps word -m <gguf> <wav>` (any variant) | per-word `t0_ms`/`t1_ms` in JSON output | PASS — derived host-side from emit-frame indices (TDT/RNNT) or per-frame argmax (CTC); same code path as the existing v2/v3 word-timestamp gate, no per-variant differences |
150163

151164
## Open decisions before Stage 3 (convert)
@@ -182,6 +195,39 @@ be resolved before the corresponding port enters porting-3-convert:
182195
convert time from `model.cfg.preprocessor.features` inside the
183196
.nemo archive. Wrong default silently degrades WER without
184197
changing tensor shapes elsewhere.
198+
5. **Layer-0 speaker-kernel injection** (`multitalker-parakeet-streaming-0.6b-v1`):
199+
RESOLVED at Stage 2. The single-speaker path is NOT numerically identical
200+
to `nemotron-speech-streaming-en-0.6b`. `EncDecMultiTalkerRNNTBPEModel`
201+
registers a `forward_pre_hook` on `encoder.layers[0]` (from
202+
`SpeakerKernelMixin`, `spk_kernel_layers=[0]`) that fires
203+
**unconditionally** — even with no diarization / speaker targets. In
204+
single-speaker mode the hook computes, at the INPUT of conformer layer 0:
205+
`x += spk_kernels.0(x)` (speaker mask defaults to all-ones) then
206+
`x += bg_spk_kernels.0(0)` (background mask defaults to zeros → a constant
207+
bias vector). Each kernel is an FF block:
208+
`Linear(1024,1024) → ReLU → Dropout(id at eval) → Linear(1024,1024)` with
209+
bias. Stage 3 MUST export `spk_kernels.0.{0,3}.{weight,bias}` and
210+
`bg_spk_kernels.0.{0,3}.{weight,bias}` (2 FF modules) and emit a GGUF KV
211+
marking layer-0 injection; Stage 4 MUST apply the injection at the layer-0
212+
input. Skipping it silently degrades single-speaker WER (a structural-cfg
213+
distinction, not a shape change). The full multitalker path (per-frame
214+
diarization mask instead of all-ones, N-instance orchestration, SegLST) is
215+
OUT OF SCOPE per Stage 1 — see the multitalker integration brief.
216+
217+
**Stage 3 resolution (DONE).** `convert-parakeet.py` gates emission on a
218+
`spk_kernel_layers` profile key. The 8 source tensors are emitted verbatim
219+
(fp32 passthrough) under the loader's `enc.` prefix, keeping the source
220+
layer index and the parameter-free `.1`/`.2` Sequential slots implicit:
221+
- `spk_kernels.<L>.{0,3}.{weight,bias}` → `enc.spk_kernel.<L>.{0,3}.{weight,bias}`
222+
- `bg_spk_kernels.<L>.{0,3}.{weight,bias}` → `enc.bg_spk_kernel.<L>.{0,3}.{weight,bias}`
223+
224+
Both Linear slots are `[1024,1024]` + `[1024]` bias. The layer-0 injection
225+
is marked by three KVs Stage 4 reads:
226+
`stt.parakeet.encoder.spk_kernel_type` (`"ff"`),
227+
`stt.parakeet.encoder.spk_kernel_layers` (int array, `[0]`), and
228+
`stt.parakeet.encoder.add_bg_spk_kernel` (bool, `true`). The loader binds
229+
these tensors and applies the injection in both offline and streaming
230+
encoder graphs.
185231

186232
## Tooling: NeMo-aware preflight
187233

0 commit comments

Comments
 (0)